638 倍嗰個數唔係 page latency
兩個數,兩個唔同 workload,兩個都唔係量 page latency。嗰個 638 倍,係喺一個啱啱 create 嘅 file 上面做一次 stat(即係 file metadata lookup),EFS 對比 block device。嗰個 4 倍,係一個 synthetic、request 形狀嘅 sweep 打去 local volume,mean aggregate stat time;但 median 嘅差距其實得 1.2 到 1.3 倍左右。Mean 係真嘅,不過佢始終話唔到你邊個 request 慢。
文入面每個 finding 都會用返佢個名去引;全部都喺 study repository 度,連背後啲 run 都喺度。
把尺:需求係固定嘅
擺 WordPress 喺度,係因為佢代表咗大部分 estate 都係由呢一類嘢砌成:喺 object storage 變成常態之前寫落嘅 software,佢假設有個 POSIX filesystem,而佢自己嘅 core code、plugin、theme、template hierarchy 就不斷去問個 filesystem。呢個 study 嘅前提係:當個 application 係黑盒。我會改 task definition、mount topology、container image、php.ini 同 opcache、仲有 proxy;但我不會改個 application。呢個 architecture 嘅 naive 版本我六年前砌過,喺度佢只係 provenance,唔係研究對象。
下面所有數字都係 per warm request。一個 warm request 大約出 4,300 個 filesystem syscall,當中大約 3,900 個係 stat;我試過嘅 php.ini profile 冇一個推得低呢個底(warm-request syscall census,E0-n10)。Cold request 係另一個數,15,400 到 16,400 個 operation,唔會加落 warm 個數度。
| 量度項目 | 數目 |
|---|---|
| Home / product / wp-admin,總 syscall | 5,157 / 5,265 / 5,140 |
| 有真實 item 嘅 cart 同 checkout,總 syscall | 5,941 / 5,884 |
wp-includes(core),filesystem op | 大約 103 |
wpforms-lite,filesystem op | 大約 1,155 |
| Elementor / WooCommerce / theme / ACF | 590 / 528 / 214 / 137 |
ENOENT lookup | 大約 1,600 |
endpoint 嗰兩行係總 syscall,唔係 filesystem call,所以先會高過 census 個數;佢哋喺 content 同 WooCommerce page 之間差 14 到 15%,但重複十次都完全唔變(cart 同 checkout 個數,E0-cart-checkout)。filesystem call 入面大約 1,600 個係回 ENOENT,即係個 application 喺度問「有冇呢個 file」。喺 block device 度呢條問題幾乎唔使錢;喺 network filesystem 度,就係用 latency 去換一個「乜都學唔到」嘅答案。呢場風暴係嚟自 plugin code 而唔係 core,係逐個 component 觀察到嘅。至於每個 probe 點解存在,係推斷返嚟,因為冇一個 call site 追溯到 PHP source。
大家喺呢度成日 recommend 嘅兩個設定,改嘅係 application 幾密去問,唔係問幾多。熄咗 opcache.validate_timestamps,喺 revalidation window 入面大約慳返 700 個 open call,同大約 1.8% 嘅 stat;而開住 validating 嘅 profile,每個 revalidate_freq window 就要畀一次大約 +4,400 個 stat 嘅稅,naive 係 2 秒,tune 完係 60 秒(同一個 census,E0-n10)。我跑過嘅三個 profile,包括一個 realpath_cache 32 MB、TTL 3600 秒嘅,個底全部都係留喺大約 3,900(PHP profile 對比,E0-first-run)。因為呢筆稅係每個 window 畀一次,所以喺低 traffic 嘅 site,佢 per request 嘅價值最高。
兩個數,兩個 workload
每次呼叫 · 一個 stat,新檔案
638.2×
每個 request · 中位數
1.21× / 1.29× / 1.25×
每個 request · 平均值
3.91× / 4.43× / 5.19×
每個 request · p99
88.7× / 125.9× / 87.6×
每次呼叫 · 一個 stat,新檔案
638.2×
每個 request · 中位數
1.21× / 1.29× / 1.25×
每個 request · 平均值
3.91× / 4.43× / 5.19×
每個 request · p99
88.7× / 125.9× / 87.6×
Log₁₀ scale · ratio · Supplied replications
同一個比較,兩個分母。每次呼叫:638.2×,量度喺剛建立嘅檔案上面。每個 request:同一個檔案系統喺貼近真實嘅存取模式下,中位數 1.21× / 1.29× / 1.25×、平均值 3.91× / 4.43× / 5.19×、p99 88.7× / 125.9× / 87.6×(每個 replication 一個值)。對數刻度;run results/E2/20260910T163206Z-66dcff8, results/E5-formal/20261028T160106Z-dd5fce0。
Source: mikeng-io/aws-wordpress@b52192b
results/E2/20260910T163206Z-66dcff8 · results/E5-formal/20261028T160106Z-dd5fce0
Measurement definitions / Command
- 每次呼叫 · 一個 stat,新檔案
- EC2 EFS direct TLS stat p50 / EC2 instance-store stat p50, pooled across three replications. Formula: raw numerator / raw denominator in ns, before display rounding; no ratio field exists in the pinned summary. Source JSON pointers (numerator, denominator): /tiers/ec2~1efs/stat/p50_ns; /tiers/ec2~1instance_store/stat/p50_ns.
- 每個 request · 中位數
- EFS plain per-request summed stat time median / matched-EBS median, delta=1 s. Leaves are numerator, denominator pairs per replication in chronological order of the three eligible formal-v3 replications. Formula: raw numerator / raw denominator in ns, before display rounding; no ratio field exists in the pinned summary. Source JSON pointers (numerator, denominator): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/median.
- 每個 request · 平均值
- Reported amortised per-request stat-time EFS plain / EBS ratios, delta=1 s; the array is the three eligible formal-v3 replications in chronological order. Source JSON pointers (output order): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_over_ebs; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_over_ebs; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_over_ebs.
- 每個 request · p99
- EFS plain per-request summed stat time p99 / matched-EBS p99, delta=1 s. Leaves are numerator, denominator pairs per replication in chronological order of the three eligible formal-v3 replications. Formula: raw numerator / raw denominator in ns, before display rounding; no ratio field exists in the pinned summary. Source JSON pointers (numerator, denominator): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/p99.
python3 analysis/export_figures.py 喺 EFS 上面做一次 stat,時間係喺 block device 度做同一個 call 嘅 638 倍。但係喺一個 request 形狀嘅 sweep 入面,啲 file 至少兩分鐘前已經存在,一個 client 每秒逐個 stat 一次,aggregate stat time 對比打去 local EBS 嘅同一個 sweep:median 大約 1.2 到 1.3 倍,mean 大約 3.9 到 5.2 倍。將呢兩個 ratio 乘埋一齊,就會得出一個冇人量過嘅數——我自己之前一稿 findings 就係咁做;而且 per-request 個結果都冇推翻 per-call 嗰個,只係將佢嘅適用範圍收窄。你見到有人單獨引用嗰個 638 倍,就係我舊嗰稿。兩個 sweep 用嘅係 synthetic tree,唔係 WordPress 嘅 request path,而且呢度冇一樣嘢係 scale 到上面講嘅 3,900 個 call census。
| Tier | stat p50 | open+read p50 | create p50 | unlink p50 |
|---|---|---|---|---|
| Instance store | 1.3 µs | 5.2 µs | 19.7 µs | 7.5 µs |
| EBS gp3 | 1.3 µs | 5.3 µs | 19.9 µs | 7.6 µs |
| Fargate ephemeral | 1.8 µs | 5.7 µs | 21.9 µs | 11.4 µs |
| EC2 + EFS,Elastic throughput / One Zone / General Purpose / NFS 4.1 | 814 µs | 904 µs | 7.34 ms | 3.00 ms |
| Fargate + EFS,同一個 configuration | 778 µs | 679 µs | 6.99 ms | 2.93 ms |
pooled 自三個 replication,每組三個獨立 deployment(per-call tier matrix,E2-block-backed)。全文嘅 ratio 都係用未四捨五入嘅 pooled p50 計,所以用呢個四捨五入咗嘅表再計返,會得出 626 倍而唔係 638 倍。兩個 block substrate 喺度分唔開:instance store 對 EBS,stat 1.00 倍、open+read 1.03 倍、create 1.01 倍、unlink 1.02 倍,p99 都係咁樣行。一過咗條界就完全唔埋身:stat 638 倍、unlink 399 倍、create 373 倍、open+read 174 倍。metadata 類 operation 受嘅苦,遠比細 file 嗰個 open+read pass 多,而呢個就係問題嘅形狀:server-backed 嘅 tier 做呢種 read,比佢答「個 file 點樣」嘅問題好得多。呢個唔係 throughput test,我唔會喺度讀出任何關於搬 byte 嘅結論。
係 protocol 而唔係 hardware——呢個係我嘅推斷,唔係一個獨立跑出嚟嘅結果:冇一個 run 係只變 protocol,所以佢係靠三個 block substrate 講同一件事、再加六個 EFS configuration 講同一件事撐住。四個候選 lever 量出嚟都係 noise,而 replication 之間嘅底已經係 1.49 到 1.82 倍:經 local efs-proxy 行 transit encryption(視乎 mount path,1.00 到 1.10 倍)、mount topology(1.00 到 1.09 倍,而 ECS-managed 嗰條 path 其實叫嘅係同一個 mount helper,同手寫嗰個一樣)、EFS path 上 EC2 對 Fargate(stat 1.06 倍、create 1.01 倍)、仲有六個 EFS configuration,stat p50 由 683 到 786 µs,1.15 倍嘅差距,而逐個 configuration 對 block group 嘅差距係 357 到 630 倍。Pooled 嘅 638 倍係 headline ratio;357 到 630 倍係每個獨立 configuration 自己做到嘅數。呢啲全部係同一把尺上面嘅 per-call finding,冇一個係證實咗嘅 tail lever。
有兩個 collection 要分開講:上面 Fargate ephemeral 嗰行係嚟自 tier matrix,而另一個獨立 collection 量咗三個 Fargate task、4,020 個 sample(Fargate ephemeral finding,E3-fargate-ephemeral)。再獨立一樣嘢:EFS 係每個 task mount 一次,唔係每個 host 一次。同一個 t4g.small 上面兩個 task,同一個 task definition、同一個 filesystem,出咗兩個 NFS4 client mount、兩個 efs-proxy process,同埋各自嘅 attribute cache(mount isolation finding,E1-mount-per-task)。同機 co-locate 嘅 task 默認係唔會幫對方 warm cache 嘅,所以一個 task 可以肯定自己本地揸住嘅 code,就只有佢自己寫落本地嗰啲。刻意 share host mount 嗰個情況我冇量過,所以兩邊我都不作判斷。
File 幾舊、cache 歷史、同個 timer
EFS
7.173 s / 6.502 s / 7.122 s
FSx OpenZFS
3.385 s / 3.408 s / 3.416 s
FSx ONTAP
0.9938 s / 0.9682 s / 0.9875 s
EFS
7.173 s / 6.502 s / 7.122 s
FSx OpenZFS
3.385 s / 3.408 s / 3.416 s
FSx ONTAP
0.9938 s / 0.9682 s / 0.9875 s
Linear scale · s · Supplied replications
喺每個 tier 建立測試檔案所需嘅秒數,每個 replication 一點:EFS 係 7.173 s / 6.502 s / 7.122 s,對比 ONTAP 嘅 0.9938 s / 0.9682 s / 0.9875 s。下一張圖會顯示呢件事對 benchmark 造成咩影響。Run results/E2/20261015T000959Z-85382cc。
Source: mikeng-io/aws-wordpress@b52192b
results/E2/20261015T000959Z-85382cc
Measurement definitions / Command
- EFS
- duration of the 1,000-file create pass, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/efs/replications/rep-1/create_pass_s; /tiers/efs/replications/rep-2/create_pass_s; /tiers/efs/replications/rep-3/create_pass_s.
- FSx OpenZFS
- duration of the 1,000-file create pass, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/fsx_openzfs/replications/rep-1/create_pass_s; /tiers/fsx_openzfs/replications/rep-2/create_pass_s; /tiers/fsx_openzfs/replications/rep-3/create_pass_s.
- FSx ONTAP
- duration of the 1,000-file create pass, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/fsx_ontap/replications/rep-1/create_pass_s; /tiers/fsx_ontap/replications/rep-2/create_pass_s; /tiers/fsx_ontap/replications/rep-3/create_pass_s.
python3 analysis/export_figures.py per-call benchmark 會 create 一批 file,然後幾秒之後去 stat 佢哋;呢把尺行一 pass,就係一個 create sweep 跟住一個 stat sweep。FSx 嗰組就揭示咗呢樣嘢其實量緊乜:喺每一個 NFS tier 上面,快嘅 stat 都係集中喺最近 create 嗰批 file,形成一條連續嘅尾,去到一個固定嘅 age 就斷(server-backed tier matrix,E2-server-backed)。
| Tier,pinned configuration | create 呢批 file 用幾耐 | 有幾多 stat 係快答 | 邊界 age |
|---|---|---|---|
| EFS,Elastic / One Zone / General Purpose / NFS 4.1 | 6.5 到 7.2 秒 | 31 到 38% | 2.30 到 2.48 秒 |
| FSx for OpenZFS,Single-AZ 64 GiB,128 MBps | 3.4 秒 | 68 到 70% | 2.91 到 2.92 秒 |
| FSx for ONTAP,Single-AZ gen-1 1024 GiB,128 MBps | 1.0 秒 | 100% | 成個 sweep 都喺 window 入面 |
create sweep 越快,stat sweep 開始嗰陣就越多 file 仲喺有效 window 入面,所以越多 stat 可以本地答。ONTAP 1,000 個 stat call 只係出咗 8 個 GETATTR RPC,而佢喺同一組度嘅 1.7 µs p50 係 cache hit,唔係一個 round trip。呢個數係屬於佢 Single-AZ gen-1、1024 GiB、128 MBps 嗰個 provisioning point,唔係 ONTAP 成個 service 嘅特性。
條界係 Linux client 嘅 acregmin,默認 3 秒(nfs(5)),而個 backoff 係喺 kernel v6.1 嘅 fs/nfs/inode.c 度讀出嚟,唔係從呢啲 run 推返嚟。每次 revalidation 見到冇變,timeout 就 double:3 → 6 → 12 → 24 → 48 → 60 秒,封頂喺 acregmax,一有變就 reset。所以,唔係「舊過 60 秒嘅 file」就會 miss:個 timer 係跟住呢個 client 上面嗰個 inode 嘅 revalidation 歷史行,所以一個一年前寫落嘅 PHP file,喺 cold mount 上面第一次 lookup,都係同一個全新 file 一樣由 3 秒開始。
咁就得出兩種唔同嘅 miss workload。喺 cold client 上面嘅 first touch——即係 task 開頭、scale-out、或者 deploy 之後——要畀一個 LOOKUP compound,量出嚟係 EFS 嗰幾條 arm 上面 revalidation 成本嘅 2.8 到 4.4 倍;OpenZFS 上面係 1.04 到 1.08 倍,其中一個 replication 係 0.72 倍;ONTAP 上面係 1.08 到 1.44 倍(first-touch 同 revalidation 對比,E5-attribute-cache)。嗰個大 premium 係 EFS 觀察到嘅嘢,唔係 NFS 嘅本性。Steady-state revalidation 每個 file 大約每 acregmax 畀一次 GETATTR:EFS 263 到 357 µs,OpenZFS 69 到 98 µs,ONTAP 109 到 111 µs。呢度要修正我自己之前一個數:per-call benchmark 嗰個 827 µs 嘅 EFS miss 成本,喺呢度重現唔到;呢啲數係佢嘅 0.32 到 0.43 倍,因為嗰個係另一種 miss。
個 timer 到底停唔停得,係 protocol 嘅能力問題。EFS 唔可以發 NFSv4 delegation,所以 OPEN 永遠回 OPEN_DELEGATE_NONE(EFS limits);FSx for OpenZFS 可以發 read delegation(OpenZFS performance guide),啲 run 喺佢身上數到 1,000 個 DELEGRETURN,而 EFS 同 ONTAP 係零。Client 揸住一個 delegation,就可以本地答 stat,連 timer 都唔使起。呢個只係「揸到 delegation 嘅能力」,唔係話 OpenZFS 每次 access 都揸住一個。
呢個機制係有 doc 同 kernel 撐住嘅推斷,而且啱啱好對得上量到嘅邊界;要真正 settle,就要改 actimeo 再睇條界會唔會跟住郁。用 20 µs 做 cutoff、當一個 stat 係「client memory 答」嗰條,係事後(post-hoc)加嘅描述性規則。而個缺陷係出喺個 metric 嘅範圍,唔係出喺 benchmark 作者身上:喺啱啱 create 嘅 file 上面做 per-call 量度,等於喺 cache lifecycle 唔同嘅位去比較唔同 tier——EFS 得 31 到 38% 嘅 stat、OpenZFS 得 68 到 70% 落喺 window 入面,ONTAP 就全部落喺入面。佢量到嘅係一個受 create 歷史影響嘅 hit/miss 混合物,唔係一個均勻 warm 嘅 cache。
心跳:到期之前幾乎打和
- EFS · 每個 request 嘅 stat 時間 ×3
- EBS 對照組 · 每個 request 嘅 stat 時間 ×3
Log₁₀ scale · ms · Ordered positions · straight segments join supplied observations
每個 request 加總嘅 stat 時間,由中位數到最大值排成階梯,EFS 同 EBS 對照組各三個 replication,同一個對數刻度。對照組保持平直;EFS 喺尾部離開佢。間距係次序性,唔係時間軸。Run results/E5-formal/20261028T160106Z-dd5fce0。
Source: mikeng-io/aws-wordpress@b52192b
results/E5-formal/20261028T160106Z-dd5fce0
Measurement definitions / Command
- EFS · 每個 request 嘅 stat 時間
- Per-request summed stat time on EFS plain, delta=1 s, eligible formal-v3 replication 1 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/amortised; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/max.
- EFS · 每個 request 嘅 stat 時間
- Per-request summed stat time on EFS plain, delta=1 s, eligible formal-v3 replication 2 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/amortised; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/max.
- EFS · 每個 request 嘅 stat 時間
- Per-request summed stat time on EFS plain, delta=1 s, eligible formal-v3 replication 3 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/amortised; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/max.
- EBS 對照組 · 每個 request 嘅 stat 時間
- Per-request summed stat time on the matched-EBS control, delta=1 s, eligible formal-v3 replication 1 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/amortised; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/max.
- EBS 對照組 · 每個 request 嘅 stat 時間
- Per-request summed stat time on the matched-EBS control, delta=1 s, eligible formal-v3 replication 2 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/amortised; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/max.
- EBS 對照組 · 每個 request 嘅 stat 時間
- Per-request summed stat time on the matched-EBS control, delta=1 s, eligible formal-v3 replication 3 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/amortised; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/max.
python3 analysis/export_figures.py 第二把尺係喺量度前至少 120 秒,由另一個 mount 度 seed 好啲 file,所以每個量度用嘅 mount 見到佢哋嗰陣都係 cold 嘅。一個 client 每 0.1 秒、每 1 秒或者每 10 秒 stat 每個 file 一次,warm-up 300 秒,量 600 秒。Miss 係用 mountstats 嘅 RPC counter 去分,唔係用 latency threshold,因為 counter 話你知個 call 有冇去到 wire,而 timer 只話你知佢用咗幾耐。不過呢啲 counter 數嘅係 RPC operation,同 file-miss event 唔係同一個量,所以我引用個 rate,唔會換算成 hit percentage。
喺一秒一個 cadence 度,EFS 每個 positive stat 量大約 0.017 個 RPC operation;0.1 秒嘅話係 0.0017,10 秒嘅話係 0.16。所以喺一個現實嘅 cadence 度,wire 上面嘅活動係好少,client 幾乎全部喺 memory 度答。咁樣換到嘅,係 median request 大約 2 ms aggregate stat time,三次 replication 分別 2.14 / 2.09 / 1.88 ms,而 EBS 係 1.77 / 1.63 / 1.50 ms。Mean 係 3.91 倍、4.43 倍、5.19 倍。p99 係 151 到 217 ms,maximum 去到 233 到 262 ms(cadence finding,E5-attribute-cache)。呢啲全部係 EC2 普通 host-mount 嗰條 arm 上面嘅 aggregate stat timing,唔係 end-to-end page latency,亦都唔係 WordPress 嗰條 path。
個形狀之所以睇得清,係因為集中。最慢嗰 5% 嘅 request 揹住咗全部 NFS RPC operation 嘅 82.9%、89.4% 同 82.5%。第一個 replication 嘅 raw per-request timing(我係逐個手睇,唔係經一個 commit 咗嘅 derivation)入面,慢過 20 ms 嘅 request 喺 EFS 上面係一叢一叢咁嚟,每叢相隔 56 到 59 秒,而 EBS control 完全冇。不過呢個檢查只覆蓋一個 replication。佢同 60 秒 window 對得上,亦都同「一齊 compile 嘅 file 一齊到期」對得上,但兩個都仍然係推斷;數據入面冇嘢分辨得出「同步到期」同其他任何一個會令 request 一齊爆嘅原因。
Lustre 係用嚟做對照,睇「完全冇 attribute timer」係咩環境:佢慢 call 嘅佔比喺每個 interval 都停喺 0.157 到 0.335%,而個實驗設嘅上限係 1%;同時 EBS control 係 0.0000 到 0.0243%;Lustre 用嘅係佢 Scratch 2、1200 GiB、version 2.15 嘅 configuration。呢個係 latency 觀察,唔係 lock state 嘅證明。
讀數出錯嘅位就係個 mean。佢量到嘅 aggregate stat time 係真做嘅工,如果你個 model 講嘅係總和,佢係一個企得住嘅 input;但佢描述唔到典型嘅 request(大約 2 ms),亦都描述唔到嗰啲集中嘅停頓(151 到 217 ms)。呢啲 aggregate 等待會唔會變成 fleet 成本,要睇 concurrency、worker occupancy 同你個 latency target——呢個 study 一樣都冇量。
Launch type 決定你個餐牌有咩揀
EFS
33.5% / 37.8% / 30.9%
FSx OpenZFS
69.6% / 68.9% / 67.9%
FSx ONTAP
100% / 100% / 100%
EFS
33.5% / 37.8% / 30.9%
FSx OpenZFS
69.6% / 68.9% / 67.9%
FSx ONTAP
100% / 100% / 100%
Linear scale · percent · Supplied replications
由 client memory 直接回答嘅 stat 佔比,用文中講明、並標明係事後回看定嘅描述性規則:EFS 33.5% / 37.8% / 30.9%,ONTAP 100% / 100% / 100%。建立檔案最快嗰個 tier 睇落最快,因為佢啲檔案仲喺 attribute-cache 窗口入面。Run results/E2/20261015T000959Z-85382cc。
Source: mikeng-io/aws-wordpress@b52192b
results/E2/20261015T000959Z-85382cc
Measurement definitions / Command
- EFS
- share of the 1,000 stat calls answered in client memory under the descriptive, post-hoc 20 us rule (stat latency < 20 us = answered in client memory), declared after inspecting the run's samples and not pre-registered; every absolute threshold in the 15-30 us sensitivity range yields identical shares, so the value is not tuned. It disagrees with the repo's registered E3-gated k-means clustering rule, which leaves ONTAP and Lustre undefined and is kept as per-rep diagnostics (registered_rule) in the derived file, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/efs/replications/rep-1/answered_local_share; /tiers/efs/replications/rep-2/answered_local_share; /tiers/efs/replications/rep-3/answered_local_share.
- FSx OpenZFS
- share of the 1,000 stat calls answered in client memory under the descriptive, post-hoc 20 us rule (stat latency < 20 us = answered in client memory), declared after inspecting the run's samples and not pre-registered; every absolute threshold in the 15-30 us sensitivity range yields identical shares, so the value is not tuned. It disagrees with the repo's registered E3-gated k-means clustering rule, which leaves ONTAP and Lustre undefined and is kept as per-rep diagnostics (registered_rule) in the derived file, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/fsx_openzfs/replications/rep-1/answered_local_share; /tiers/fsx_openzfs/replications/rep-2/answered_local_share; /tiers/fsx_openzfs/replications/rep-3/answered_local_share.
- FSx ONTAP
- share of the 1,000 stat calls answered in client memory under the descriptive, post-hoc 20 us rule (stat latency < 20 us = answered in client memory), declared after inspecting the run's samples and not pre-registered; every absolute threshold in the 15-30 us sensitivity range yields identical shares, so the value is not tuned. It disagrees with the repo's registered E3-gated k-means clustering rule, which leaves ONTAP and Lustre undefined and is kept as per-rep diagnostics (registered_rule) in the derived file, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/fsx_ontap/replications/rep-1/answered_local_share; /tiers/fsx_ontap/replications/rep-2/answered_local_share; /tiers/fsx_ontap/replications/rep-3/answered_local_share.
python3 analysis/export_figures.py 下面個表講緊嘅規則係:你有邊啲 tier 可以用,喺你問任何 performance 問題之前,就已經由你個 launch type 決定咗。
| Tier | 喺 EC2 | 喺 Fargate | 邊個決定 |
|---|---|---|---|
| EFS | ECS-managed volume,或者 host mount 再 bind | ECS-managed volume,platform 1.4.0+ | mount helper、port 2049 |
| EBS gp3 | ECS volume 或者 root volume | AWS 文件寫住係一個 volume option | 冇量過嘅 Fargate arm |
| FSx for Lustre | host mount 再 bind | 唔 support | 唔係 ECS volume type |
| FSx for OpenZFS | host mount 再 bind | 唔 support | 唔係 ECS volume type |
| FSx for ONTAP | task-definition volume,native NFS | 淨係 S3 Access Points,S3 API 唔係 POSIX | 要 native NFS 就用 EC2 |
| S3 Access Points | S3 API | 連得到,但唔係 mount | object semantics,唔係 POSIX |
上面講嘅 eligibility 係睇文件,唔係量出嚟(ECS data volumes)。FUSE 類 tier 冇咗,係因為平台原因而唔係 test 結果:佢哋要 CAP_SYS_ADMIN,而 Fargate 只准你加一個 Linux capability,就係 CAP_SYS_PTRACE(Fargate task considerations)。呢個 study 入面所有 FUSE 量度都係喺 EC2 做,Fargate 上面一個都冇試過。
| Tier | Pinned configuration |
|---|---|
| EFS | Elastic throughput、One Zone、General Purpose、transit encryption 開、NFS 4.1 |
| FSx for OpenZFS | Single-AZ、64 GiB、128 MBps provisioned(唔係 64 MBps 個底)、compression NONE |
| FSx for ONTAP | Single-AZ gen-1、1024 GiB、128 MBps、冇 snapshot、storage efficiency 熄 |
| FSx for Lustre | Scratch 2、1200 GiB、version 2.15、mount 用 -o flock |
每一行都係一個 provisioning point,唔係成個 service 嘅 latency 特性。Provisioned throughput 會決定 FSx file server 嘅 in-memory metadata cache 有幾大,所以 OpenZFS 同 ONTAP 啲數有一部分係個 cap 造成;Scratch 2 唔係 Persistent,呢度亦都冇一樣嘢係講 Persistent 嗰種 deployment type。encrypted EFS 嗰行係屬於 per-call campaign;上面嘅 cadence 結果係嚟自普通 host-mount 嗰條 arm。
量咗三個 filesystem,而老實講,個總結比「淘汰賽」窄好多。JuiceFS 做到 104.6 µs stat p50,create 29.2 ms,主要係花咗喺將 file body 寫上 S3;佢本來係登記嚟答「object-backed 嘅 userspace filesystem 喺 metadata 上面可唔可以贏 EFS 一個數量級」,而 7.5 倍快過 EFS cache-miss p50,結果係 inconclusive,唔算贏唔算輸(FUSE results,E2-fuse)。SeaweedFS 做到 20.2 µs stat、2.28 ms create,11 個 conformance check 過晒 11 個,即係喺 EC2 上面,佢係一個真 option,只係呢個 study 冇攞佢對住個 application 試。Mountpoint for S3 喺每個 replication 都係 11 個 single-node check 錯 8 個,包括 rename 唔會覆蓋原本嗰個、mtime 唔會前進,所以佢冇 latency table:semantic 上錯,唔可以就咁撈入 performance ranking 度平均。三個共通點係都唔 eligible 喺 Fargate 用,但呢個係平台排除,唔係對嗰個 software 嘅判決。
Hydration 抽走咗「到期」呢個事件
- EFS · stat
- 已 hydrate 嘅本地目錄樹 · stat
Log₁₀ scale · us · Ordered positions · straight segments join supplied observations
EFS 同一個喺開工時 hydrate 去 task-local storage 嘅程式碼樹,兩者嘅 stat 延遲,p50 到 p99,同一個對數刻度。hydration 喺開工時要付幾多代價,文中已經講明。間距係次序性。Run results/E4/20261029T091909Z-842d596。
Source: mikeng-io/aws-wordpress@b52192b
results/E4/20261029T091909Z-842d596
Measurement definitions / Command
- EFS · stat
- EFS stat percentile ladder, pooled across three deployments; element order p50, p95, p99; every element ns/1000 at four significant figures, exact ns in raw. Source JSON pointers (output order): /pooled/efs/stat/p50_ns; /pooled/efs/stat/p95_ns; /pooled/efs/stat/p99_ns.
- 已 hydrate 嘅本地目錄樹 · stat
- Hydrated code-tree stat percentile ladder, pooled across three deployments; element order p50, p95, p99; every element ns/1000 at four significant figures, exact ns in raw. Source JSON pointers (output order): /pooled/hydrated/stat/p50_ns; /pooled/hydrated/stat/p95_ns; /pooled/hydrated/stat/p99_ns.
python3 analysis/export_figures.py 喺 startup 嗰陣,將一個 read-only code tree 由 EFS hydrate 落 task 自己嘅 ephemeral storage,並冇消除 filesystem access。佢消除嘅係嗰條 read path 上面嘅 remote attribute-cache 到期。三個獨立 deployment,Fargate 1.4.0、ARM64、0.5 vCPU 同 1 GiB、20 GiB ephemeral(hydration finding,E4-hydration):
| 量度項目 | 喺 EFS | hydrate 之後 | 比例 |
|---|---|---|---|
stat p50 | 1.3884 ms | 3.061 µs | 0.00220 |
open_read p50 | 2.7109 ms | 11.667 µs | 0.00430 |
| Tar 解壓,27,412,480 bytes | n/a | 0.218 / 0.444 / 0.429 秒 | 同一批 byte,差距 2.04 倍 |
| 由第一行 log 到 benchmark 開始,兩條 arm 之間 | n/a | 0.051 / 0.192 / 0.265 秒 | 有 confound,見下 |
上面嘅 3.061 µs 同 0.00220,係呢個 experiment 報告嘅 summary 值,係喺 task-local storage 上面逐個 operation 量。每個 replication 都企得住嘅,係佢哋背後嗰條 acceptance threshold:hydrate 之後唔可以差過 EFS stat 成本嘅十分一,同 open_read 成本嘅五分一;逐個 run 嘅結果係圍住個 summary 上上落落,唔係逐個 digit 複製得出。本地啲 distribution 係單峰嘅,stat p99 係 6.023 µs、ENOENT p99 係 22.6 µs,而呢個就係重點:local filesystem 冇 revalidation timer,所以根本冇嘢會到期。
我事前預測 hydration 會令 benchmark 至少遲 0.5 秒先開得工,結果錯,而且係錯喺「平」嗰邊。解壓本身用咗 0.218 到 0.444 秒,而兩條 arm 之間「第一行 log 到 benchmark」嘅 delta 係 0.051 到 0.265 秒,當中仲包含 remote conformance write 同 scheduling noise,所以呢個 delta 我唔會當係 hydration 嘅因果 overhead,亦都唔會當係成個 startup time。
呢個唔係一個量到嘅 request-latency 結果。hydrate 咗嘅 tree 從來冇喺 cadence 嗰把尺嘅負載下面跑過,所以 151 到 217 ms 嗰個停頓有冇消失,仲未有人證明到;兩者之間嘅關連,係由兩個獨立結果推返嚟嘅推斷。而且佢好 specific:一個 read-only code tree,得 27 MB 而唔係一個 production install,冇 write-back adapter、冇 fleet refresh 或者 invalidation 行為、亦都冇 end-to-end latency 量度。佢處理嘅係一條 path 上面嘅到期事件。佢唔會令 mutable 嘅 application state 變成本地,而且佢只係一個有範圍嘅 option,唔係剩低唯一嗰個。
Audit notes:呢個 study 講得出啲乜
| Study 問過嘅問題 | 結果 |
|---|---|
每個 server-backed tier 喺 stat 上面係咪都會慢過 block group 10 倍以上? | 機械式咁被推翻:表面嘅反例其實係 client-cache hit 而唔係 server lookup,所以呢個推翻本身冇乜資訊量 |
| first touch 係咪貴過 revalidation? | 喺 EFS 係,2.8 到 4.4 倍;事前登記嘅 1.0 到 3.0 區間兩邊都被推翻——EFS 嗰幾條 arm 喺上面超咗,有一個 OpenZFS replication 喺下面 0.72 倍 |
| 一個 request 係咪要畀 per-call 個 ratio? | 唔係:mean 落喺佢 support bound 之內,而且遠低過推翻門檻(cadence finding,E5-attribute-cache) |
| hydration 嘅成本係咪低過我預測、benchmark 開工前嗰 0.5 秒下限? | 唔係;解壓用咗 0.218 到 0.444 秒,所以個預測被推翻,而兩條 arm 之間嘅 delta 太 confounded,唔可以攞嚟代替 |
| hydration 係咪消除咗 request 停頓? | 未證明到;從來冇喺 cadence 嗰把尺下面跑過 |
啲預測喺跑之前已經登記好,設計容許嘅地方做三次 replication,每個 service 都只喺一個 provisioning point 量。Wire 上面有冇活動係靠 RPC counter 講,wall-clock latency 永遠唔會用嚟撐「個 call 有冇離開 client」呢類講法。20 µs 嗰條 cache-hit cutoff、得一個 replication 嘅 clustering 檢查、同「同步到期」嘅解釋,喺每一處出現都標明係 post-hoc。冇量過 end-to-end request latency、冇一個 cadence 對得上嘅 Fargate run、冇試過刻意 share host mount,cart 同 checkout traffic 有數但冇做 end-to-end instrumentation。
星期一開工:decision checklist
- 喺比較任何 storage tier 之前,先喺一個 warm request 度數清楚你嘅 filesystem 需求。你冇量過嘅需求數字,計唔到 capacity。
- 睇清楚你個 launch type 到底 attach 得到啲乜,因為一個對你嚟講唔存在嘅 option,唔係一個慢啲嘅 option。
- 先試你個 application 需要嘅 POSIX semantics。
rename同mtime就係 object-store mount 死嘅位,而 correctness 出事係排除,唔係排名。 - 用你自己嘅 cadence 去量,用 RPC counter 而唔係秒錶,因為「啱啱 create 嘅 file 矩陣」同「request 形狀嘅 sweep」答緊唔同嘅問題。
- 你重複引用任何一個數,都要連佢個 provisioning point 一齊講。Single-AZ OpenZFS 128 MBps 同 Single-AZ gen-1 ONTAP 1024 GiB 係 configuration,唔係成個 service。
- 如果你需求入面「code」嗰部分係 read-only,可以考慮 hydrate 佢落 task-local storage,並且量埋解壓成本同逐個 operation 嘅結果。如果唔係 read-only,呢個 option 唔關你事。
對呢個 workload 嚟講,storage tier 決定條尾個形狀,同佢決定個平均一樣咁多;而真正抽走「到期」呢個事件嘅干預,係改個 code 由邊度讀。Mean 嘅 penalty 仲喺度,任何計總 stat 等待嘅數都要計埋佢;但佢從來都冇描述過一個獨立 request,而用戶記得嘅 request,永遠都係 p99 嗰啲。