Skip to content

The 638× Stat Is Not Page Latency

The 638× Stat Is Not Page Latency

Two figures, two workloads, and neither is measured page latency. The 638× is one stat, a file-metadata lookup, on a freshly created file, EFS against a block device. The 4× is the mean aggregate stat time in a synthetic request-shaped sweep against a local volume, where the median difference is closer to 1.2 to 1.3×. The mean is real, and it still does not tell you which request is slow.

Findings are cited by name throughout; all of them, with the runs behind them, live in the study repository.

The instrument: fixed demand

WordPress is here as a specimen of a class most estates are made of: software written before object storage was normal, which assumes a POSIX filesystem and consults it constantly from its own core code, plugins, themes and template hierarchy. Treating the application as a black box is this study’s constraint. I change task definitions, mount topology, container images, php.ini and opcache, and proxies; I do not change the application. I built the naive version of this architecture six years ago, and it is provenance here, not a subject.

Every count below is per warm request. A warm request issues about 4,300 filesystem syscalls, of which about 3,900 are stat, and no php.ini profile I tested moved that floor (the warm-request syscall census, E0-n10). Cold requests are a separate figure, 15,400 to 16,400 operations, and they do not add to the warm count.

MeasurementCount
Home / product / wp-admin, total syscalls5,157 / 5,265 / 5,140
Cart and checkout with a real item, total syscalls5,941 / 5,884
wp-includes (core), filesystem opsabout 103
wpforms-lite, filesystem opsabout 1,155
Elementor / WooCommerce / theme / ACF590 / 528 / 214 / 137
ENOENT lookupsabout 1,600

The endpoint rows are total syscalls, not filesystem calls, which is why they sit above the census; they vary 14 to 15% between content and WooCommerce pages and not at all across ten repetitions (the cart and checkout counts, E0-cart-checkout). Roughly 1,600 of the filesystem calls return ENOENT, the application asking whether a file exists. On a block device that question costs almost nothing; on a network filesystem it is latency spent to learn nothing. That the storm comes from plugin code rather than core is observed per component. Why each probe exists is inferred, because no call site was traced back to PHP source.

The two settings people recommend here change how often the application asks, not how much it asks. Turning off opcache.validate_timestamps removes about 700 open calls and about 1.8% of stats inside the revalidation window, and a validating profile pays a roughly +4,400-stat tax once per revalidate_freq window, 2 s naive against 60 s tuned (same census, E0-n10). The three profiles I ran, including one at realpath_cache 32 MB with a 3600 s TTL, all left the floor at about 3,900 (the PHP profile comparison, E0-first-run). Because that tax is paid once per window, it is worth most per request on low-traffic sites.

Two numbers, two workloads

Per call · one stat, fresh files

1101001000Per call · one stat, fresh files — replication 1 of 1: 638.2×638.2×

638.2×

Per request · median

1101001000Per request · median — replication 1 of 3: 1.21×1.21×Per request · median — replication 2 of 3: 1.29×1.29×Per request · median — replication 3 of 3: 1.25×1.25×

1.21× / 1.29× / 1.25×

Per request · mean

1101001000Per request · mean — replication 1 of 3: 3.91×3.91×Per request · mean — replication 2 of 3: 4.43×4.43×Per request · mean — replication 3 of 3: 5.19×5.19×

3.91× / 4.43× / 5.19×

Per request · p99

1101001000Per request · p99 — replication 1 of 3: 88.7×88.7×Per request · p99 — replication 2 of 3: 125.9×125.9×Per request · p99 — replication 3 of 3: 87.6×87.6×

88.7× / 125.9× / 87.6×

Per call · one stat, fresh files

1101001000Per call · one stat, fresh files — replication 1 of 1: 638.2×638.2×

638.2×

Per request · median

1101001000Per request · median — replication 1 of 3: 1.21×1.21×Per request · median — replication 2 of 3: 1.29×1.29×Per request · median — replication 3 of 3: 1.25×1.25×

1.21× / 1.29× / 1.25×

Per request · mean

1101001000Per request · mean — replication 1 of 3: 3.91×3.91×Per request · mean — replication 2 of 3: 4.43×4.43×Per request · mean — replication 3 of 3: 5.19×5.19×

3.91× / 4.43× / 5.19×

Per request · p99

1101001000Per request · p99 — replication 1 of 3: 88.7×88.7×Per request · p99 — replication 2 of 3: 125.9×125.9×Per request · p99 — replication 3 of 3: 87.6×87.6×

88.7× / 125.9× / 87.6×

Log₁₀ scale · ratio · Supplied replications

One comparison, two denominators. Per call: 638.2×, measured on freshly created files. Per request, the same filesystem under a realistic access pattern: median 1.21× / 1.29× / 1.25×, mean 3.91× / 4.43× / 5.19×, p99 88.7× / 125.9× / 87.6× (one value per replication). Log scale; run results/E2/20260910T163206Z-66dcff8, results/E5-formal/20261028T160106Z-dd5fce0.

Source: mikeng-io/aws-wordpress@b52192b
results/E2/20260910T163206Z-66dcff8 · results/E5-formal/20261028T160106Z-dd5fce0

Measurement definitions / Command
Per call · one stat, fresh files
EC2 EFS direct TLS stat p50 / EC2 instance-store stat p50, pooled across three replications. Formula: raw numerator / raw denominator in ns, before display rounding; no ratio field exists in the pinned summary. Source JSON pointers (numerator, denominator): /tiers/ec2~1efs/stat/p50_ns; /tiers/ec2~1instance_store/stat/p50_ns.
Per request · median
EFS plain per-request summed stat time median / matched-EBS median, delta=1 s. Leaves are numerator, denominator pairs per replication in chronological order of the three eligible formal-v3 replications. Formula: raw numerator / raw denominator in ns, before display rounding; no ratio field exists in the pinned summary. Source JSON pointers (numerator, denominator): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/median.
Per request · mean
Reported amortised per-request stat-time EFS plain / EBS ratios, delta=1 s; the array is the three eligible formal-v3 replications in chronological order. Source JSON pointers (output order): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_over_ebs; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_over_ebs; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_over_ebs.
Per request · p99
EFS plain per-request summed stat time p99 / matched-EBS p99, delta=1 s. Leaves are numerator, denominator pairs per replication in chronological order of the three eligible formal-v3 replications. Formula: raw numerator / raw denominator in ns, before display rounding; no ratio field exists in the pinned summary. Source JSON pointers (numerator, denominator): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/p99.
python3 analysis/export_figures.py

A stat on EFS takes 638 times as long as the same call on a block device. In a request-shaped sweep over files at least two minutes old, with one client stating each of them about once a second, aggregate stat time costs about 1.2 to 1.3 times the median and 3.9 to 5.2 times the mean of the identical sweep against local EBS. Multiply those ratios and you get a figure nobody measured, which an earlier draft of my own findings did; and the per-request result does not refute the per-call one either, it re-scopes it. That earlier draft is the 638× figure you may have seen quoted on its own. The sweeps are a synthetic tree rather than a WordPress request path, and nothing here is scaled to the 3,900-call census above.

Tierstat p50open+read p50create p50unlink p50
Instance store1.3 µs5.2 µs19.7 µs7.5 µs
EBS gp31.3 µs5.3 µs19.9 µs7.6 µs
Fargate ephemeral1.8 µs5.7 µs21.9 µs11.4 µs
EC2 + EFS, Elastic throughput / One Zone / General Purpose / NFS 4.1814 µs904 µs7.34 ms3.00 ms
Fargate + EFS, same configuration778 µs679 µs6.99 ms2.93 ms

Pooled from three replications with three independent deployments per group (the per-call tier matrix, E2-block-backed). Ratios throughout are computed on unrounded pooled p50s, so recomputing them from this rounded table gives 626× rather than 638×. The block substrates are indistinguishable here: instance store against EBS is 1.00× on stat, 1.03× on open+read, 1.01× on create, 1.02× on unlink, with p99s tracking the same way. Across the boundary nothing ties: stat 638×, unlink 399×, create 373×, open+read 174×. The metadata operations suffer far more than the small-file open+read pass, and that is the shape of the problem: a server-backed tier is much less bad at this read than at answering questions about files. This was not a throughput test, and I would not read a byte-moving conclusion out of it.

Protocol rather than hardware is my inference, not an isolated result: no run varied only the protocol, so it rests on three block substrates agreeing and six EFS configurations agreeing. Four candidate levers measure as noise against a between-replication floor of 1.49 to 1.82×: transit encryption through the local efs-proxy (1.00× to 1.10× depending on mount path), mount topology (1.00 to 1.09×, and the ECS-managed path invokes the same mount helper as a hand-written one), EC2 against Fargate on the EFS path (1.06× on stat, 1.01× on create), and six EFS configurations spanning 683 to 786 µs stat p50, a 1.15× spread whose per-configuration gaps to the block group run 357 to 630×. The pooled 638× is the headline ratio; the 357 to 630× range is what each individual configuration manages. Those are per-call findings on one instrument, and none is a demonstrated tail lever.

Two collections stay apart: the Fargate ephemeral row above comes from the tier matrix, while a separate collection measured three Fargate tasks and 4,020 samples (the Fargate ephemeral finding, E3-fargate-ephemeral). Separately again, EFS mounts once per task rather than once per host. Two tasks on one t4g.small, same task definition and filesystem, produced two NFS4 client mounts, two efs-proxy processes and separate attribute caches (the mount isolation finding, E1-mount-per-task). Co-located tasks warming each other’s cache does not happen by default, which is also why the only code a task can be sure of holding locally is code it wrote locally. The deliberately shared host mount was never measured, and I make no claim either way.

File age, cache history, and the timer

EFS

07.173EFS — replication 1 of 3: 7.173 s7.173 sEFS — replication 2 of 3: 6.502 s6.502 sEFS — replication 3 of 3: 7.122 s7.122 s

7.173 s / 6.502 s / 7.122 s

FSx OpenZFS

07.173FSx OpenZFS — replication 1 of 3: 3.385 s3.385 sFSx OpenZFS — replication 2 of 3: 3.408 s3.408 sFSx OpenZFS — replication 3 of 3: 3.416 s3.416 s

3.385 s / 3.408 s / 3.416 s

FSx ONTAP

07.173FSx ONTAP — replication 1 of 3: 0.9938 s0.9938 sFSx ONTAP — replication 2 of 3: 0.9682 s0.9682 sFSx ONTAP — replication 3 of 3: 0.9875 s0.9875 s

0.9938 s / 0.9682 s / 0.9875 s

EFS

07.173EFS — replication 1 of 3: 7.173 s7.173 sEFS — replication 2 of 3: 6.502 s6.502 sEFS — replication 3 of 3: 7.122 s7.122 s

7.173 s / 6.502 s / 7.122 s

FSx OpenZFS

07.173FSx OpenZFS — replication 1 of 3: 3.385 s3.385 sFSx OpenZFS — replication 2 of 3: 3.408 s3.408 sFSx OpenZFS — replication 3 of 3: 3.416 s3.416 s

3.385 s / 3.408 s / 3.416 s

FSx ONTAP

07.173FSx ONTAP — replication 1 of 3: 0.9938 s0.9938 sFSx ONTAP — replication 2 of 3: 0.9682 s0.9682 sFSx ONTAP — replication 3 of 3: 0.9875 s0.9875 s

0.9938 s / 0.9682 s / 0.9875 s

Linear scale · s · Supplied replications

Seconds to create the test files on each tier, one dot per replication: 7.173 s / 6.502 s / 7.122 s on EFS against 0.9938 s / 0.9682 s / 0.9875 s on ONTAP. The next figure shows what that does to the benchmark. Run results/E2/20261015T000959Z-85382cc.

Source: mikeng-io/aws-wordpress@b52192b
results/E2/20261015T000959Z-85382cc

Measurement definitions / Command
EFS
duration of the 1,000-file create pass, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/efs/replications/rep-1/create_pass_s; /tiers/efs/replications/rep-2/create_pass_s; /tiers/efs/replications/rep-3/create_pass_s.
FSx OpenZFS
duration of the 1,000-file create pass, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/fsx_openzfs/replications/rep-1/create_pass_s; /tiers/fsx_openzfs/replications/rep-2/create_pass_s; /tiers/fsx_openzfs/replications/rep-3/create_pass_s.
FSx ONTAP
duration of the 1,000-file create pass, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/fsx_ontap/replications/rep-1/create_pass_s; /tiers/fsx_ontap/replications/rep-2/create_pass_s; /tiers/fsx_ontap/replications/rep-3/create_pass_s.
python3 analysis/export_figures.py

The per-call benchmark creates a file set and then stats it a few seconds later; one pass of the instrument is one create sweep followed by one stat sweep. The FSx group exposed what that measures: on every NFS tier the fast stats form a contiguous tail of the most recently created files, ending at a fixed age (the server-backed tier matrix, E2-server-backed).

Tier, pinned configurationTime to create the file setStats answered fastBoundary age
EFS, Elastic / One Zone / General Purpose / NFS 4.16.5 to 7.2 s31 to 38%2.30 to 2.48 s
FSx for OpenZFS, Single-AZ 64 GiB, 128 MBps3.4 s68 to 70%2.91 to 2.92 s
FSx for ONTAP, Single-AZ gen-1 1024 GiB, 128 MBps1.0 s100%whole sweep inside the window

A faster create sweep leaves more files inside the validity window when the stat sweep begins, so more of its stats get answered locally. ONTAP sent 8 GETATTR RPCs for 1,000 stat calls, and its 1.7 µs p50 in that same group is a cache hit rather than a round trip. That belongs to its Single-AZ gen-1, 1024 GiB, 128 MBps provisioning point, not to ONTAP as a service.

The boundary is the Linux client’s acregmin, 3 s by default (nfs(5)), with the backoff read from fs/nfs/inode.c at kernel v6.1 rather than inferred from these runs. Each unchanged revalidation doubles the timeout, 3 to 6 to 12 to 24 to 48 to 60 s, capped at acregmax and reset by any change. Files older than 60 seconds are therefore not what produces a miss: the timer follows the inode’s revalidation history on this client, which is why a PHP file written a year ago and looked up for the first time on a cold mount starts at 3 s like a brand-new one.

ServerLinux NFS clientApplicationServerLinux NFS clientApplicationfirst touch on a cold clientattributes valid, up to 60 svalidity elapsesopt[read delegation held, OpenZFScapability only]statLOOKUP compoundattributes, timer starts at acregminansweredstat againanswered from memory, no RPCstat after expiryGETATTRunchanged, timeout doubles toward acregmaxansweredstatanswered locally, timer never runs

Two different miss workloads come out of that. First touch on a cold client, at task start or scale-out or after a deploy, pays a LOOKUP compound, measured at 2.8 to 4.4× the revalidation cost on the EFS arms, 1.04 to 1.08× on OpenZFS with one replication at 0.72×, and 1.08 to 1.44× on ONTAP (the first-touch and revalidation comparison, E5-attribute-cache). The large premium is an EFS observation, not a property of NFS. Steady-state revalidation pays a GETATTR about once per acregmax per file: 263 to 357 µs on EFS, 69 to 98 µs on OpenZFS, 109 to 111 µs on ONTAP. That corrects an earlier figure of my own, the 827 µs EFS miss cost from the per-call benchmark, which does not reproduce here; these numbers are 0.32 to 0.43× it, because it was a different miss.

Whether the timer can be stopped at all is a protocol capability. EFS cannot grant NFSv4 delegations, so OPEN always returns OPEN_DELEGATE_NONE (EFS limits); FSx for OpenZFS can grant read delegations (OpenZFS performance guide), and the runs counted 1,000 DELEGRETURNs on it against zero on EFS and ONTAP. A client holding one answers stat locally without starting the timer. That is the ability to hold a delegation, not a claim that every OpenZFS access holds one.

The mechanism is documentation- and kernel-supported inference that fits the measured boundaries; changing actimeo and watching the boundary move is what would settle it. The 20 µs cut used to call a stat “answered from client memory” is a post-hoc descriptive rule. And the defect is the metric’s scope rather than the benchmark’s authors: a per-call measurement on freshly created files compares tiers at different points in the cache lifecycle, with only 31 to 38% of EFS stats and 68 to 70% of OpenZFS stats landing inside the window, against all of ONTAP’s. It measures a creation-history-dependent mixture of hits and misses, not a uniformly warm cache.

The heartbeat: nearly tied until expiry

  • EFS · stat time per request ×3
  • EBS control · stat time per request ×3
1101001000 EFS · stat time per request — median: 2.14 ms; mean: 6.942 ms; p99: 164.1 ms; max: 233.3 ms EFS · stat time per request — median: 2.14 ms 2.14 ms EFS · stat time per request — mean: 6.942 ms 6.942 ms EFS · stat time per request — p99: 164.1 ms 164.1 ms EFS · stat time per request — max: 233.3 ms 233.3 ms EFS · stat time per request — median: 2.091 ms; mean: 7.221 ms; p99: 216.6 ms; max: 244.1 ms EFS · stat time per request — median: 2.091 ms 2.091 ms EFS · stat time per request — mean: 7.221 ms 7.221 ms EFS · stat time per request — p99: 216.6 ms 216.6 ms EFS · stat time per request — max: 244.1 ms 244.1 ms EFS · stat time per request — median: 1.875 ms; mean: 7.947 ms; p99: 151.3 ms; max: 262.4 ms EFS · stat time per request — median: 1.875 ms 1.875 ms EFS · stat time per request — mean: 7.947 ms 7.947 ms EFS · stat time per request — p99: 151.3 ms 151.3 ms EFS · stat time per request — max: 262.4 ms 262.4 ms EBS control · stat time per request — median: 1.773 ms; mean: 1.775 ms; p99: 1.849 ms; max: 3.454 ms EBS control · stat time per request — median: 1.773 ms 1.773 ms EBS control · stat time per request — mean: 1.775 ms 1.775 ms EBS control · stat time per request — p99: 1.849 ms 1.849 ms EBS control · stat time per request — max: 3.454 ms 3.454 ms EBS control · stat time per request — median: 1.627 ms; mean: 1.63 ms; p99: 1.72 ms; max: 2.932 ms EBS control · stat time per request — median: 1.627 ms 1.627 ms EBS control · stat time per request — mean: 1.63 ms 1.63 ms EBS control · stat time per request — p99: 1.72 ms 1.72 ms EBS control · stat time per request — max: 2.932 ms 2.932 ms EBS control · stat time per request — median: 1.501 ms; mean: 1.533 ms; p99: 1.727 ms; max: 2.763 ms EBS control · stat time per request — median: 1.501 ms 1.501 ms EBS control · stat time per request — mean: 1.533 ms 1.533 ms EBS control · stat time per request — p99: 1.727 ms 1.727 ms EBS control · stat time per request — max: 2.763 ms 2.763 ms medianmeanp99max

EFS · stat time per request — median: 2.14 ms / mean: 6.942 ms / p99: 164.1 ms / max: 233.3 ms

EFS · stat time per request — median: 2.091 ms / mean: 7.221 ms / p99: 216.6 ms / max: 244.1 ms

EFS · stat time per request — median: 1.875 ms / mean: 7.947 ms / p99: 151.3 ms / max: 262.4 ms

EBS control · stat time per request — median: 1.773 ms / mean: 1.775 ms / p99: 1.849 ms / max: 3.454 ms

EBS control · stat time per request — median: 1.627 ms / mean: 1.63 ms / p99: 1.72 ms / max: 2.932 ms

EBS control · stat time per request — median: 1.501 ms / mean: 1.533 ms / p99: 1.727 ms / max: 2.763 ms

1101001000 EFS · stat time per request — median: 2.14 ms; mean: 6.942 ms; p99: 164.1 ms; max: 233.3 ms EFS · stat time per request — median: 2.14 ms 2.14 ms EFS · stat time per request — mean: 6.942 ms 6.942 ms EFS · stat time per request — p99: 164.1 ms 164.1 ms EFS · stat time per request — max: 233.3 ms 233.3 ms EFS · stat time per request — median: 2.091 ms; mean: 7.221 ms; p99: 216.6 ms; max: 244.1 ms EFS · stat time per request — median: 2.091 ms 2.091 ms EFS · stat time per request — mean: 7.221 ms 7.221 ms EFS · stat time per request — p99: 216.6 ms 216.6 ms EFS · stat time per request — max: 244.1 ms 244.1 ms EFS · stat time per request — median: 1.875 ms; mean: 7.947 ms; p99: 151.3 ms; max: 262.4 ms EFS · stat time per request — median: 1.875 ms 1.875 ms EFS · stat time per request — mean: 7.947 ms 7.947 ms EFS · stat time per request — p99: 151.3 ms 151.3 ms EFS · stat time per request — max: 262.4 ms 262.4 ms EBS control · stat time per request — median: 1.773 ms; mean: 1.775 ms; p99: 1.849 ms; max: 3.454 ms EBS control · stat time per request — median: 1.773 ms 1.773 ms EBS control · stat time per request — mean: 1.775 ms 1.775 ms EBS control · stat time per request — p99: 1.849 ms 1.849 ms EBS control · stat time per request — max: 3.454 ms 3.454 ms EBS control · stat time per request — median: 1.627 ms; mean: 1.63 ms; p99: 1.72 ms; max: 2.932 ms EBS control · stat time per request — median: 1.627 ms 1.627 ms EBS control · stat time per request — mean: 1.63 ms 1.63 ms EBS control · stat time per request — p99: 1.72 ms 1.72 ms EBS control · stat time per request — max: 2.932 ms 2.932 ms EBS control · stat time per request — median: 1.501 ms; mean: 1.533 ms; p99: 1.727 ms; max: 2.763 ms EBS control · stat time per request — median: 1.501 ms 1.501 ms EBS control · stat time per request — mean: 1.533 ms 1.533 ms EBS control · stat time per request — p99: 1.727 ms 1.727 ms EBS control · stat time per request — max: 2.763 ms 2.763 ms medianmeanp99max

EFS · stat time per request — median: 2.14 ms / mean: 6.942 ms / p99: 164.1 ms / max: 233.3 ms

EFS · stat time per request — median: 2.091 ms / mean: 7.221 ms / p99: 216.6 ms / max: 244.1 ms

EFS · stat time per request — median: 1.875 ms / mean: 7.947 ms / p99: 151.3 ms / max: 262.4 ms

EBS control · stat time per request — median: 1.773 ms / mean: 1.775 ms / p99: 1.849 ms / max: 3.454 ms

EBS control · stat time per request — median: 1.627 ms / mean: 1.63 ms / p99: 1.72 ms / max: 2.932 ms

EBS control · stat time per request — median: 1.501 ms / mean: 1.533 ms / p99: 1.727 ms / max: 2.763 ms

Log₁₀ scale · ms · Ordered positions · straight segments join supplied observations

Stat time summed per request, as a ladder from median to max, three replications of EFS and of the EBS control on one log scale. The control stays flat; EFS leaves it at the tail. Spacing is ordinal. Run results/E5-formal/20261028T160106Z-dd5fce0.

Source: mikeng-io/aws-wordpress@b52192b
results/E5-formal/20261028T160106Z-dd5fce0

Measurement definitions / Command
EFS · stat time per request
Per-request summed stat time on EFS plain, delta=1 s, eligible formal-v3 replication 1 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/amortised; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/efs_per_request_ns/max.
EFS · stat time per request
Per-request summed stat time on EFS plain, delta=1 s, eligible formal-v3 replication 2 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/amortised; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/efs_per_request_ns/max.
EFS · stat time per request
Per-request summed stat time on EFS plain, delta=1 s, eligible formal-v3 replication 3 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/median; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/amortised; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/p99; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/efs_per_request_ns/max.
EBS control · stat time per request
Per-request summed stat time on the matched-EBS control, delta=1 s, eligible formal-v3 replication 1 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/amortised; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T103017Z-342d6b4~1rep-1/ebs_per_request_ns/max.
EBS control · stat time per request
Per-request summed stat time on the matched-EBS control, delta=1 s, eligible formal-v3 replication 2 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/amortised; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T125412Z-f6659e0~1rep-1/ebs_per_request_ns/max.
EBS control · stat time per request
Per-request summed stat time on the matched-EBS control, delta=1 s, eligible formal-v3 replication 3 in chronological order. Percentile ladder, element order median, mean (amortised), p99, max; every element ns/1e6 (milliseconds) at four significant figures, exact ns in raw. Milliseconds because a request's summed stat time runs from ~1.5 ms to ~260 ms. Source JSON pointers (output order): /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/median; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/amortised; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/p99; /predictions/9/per_replication/20261028T160106Z-dd5fce0~1rep-1/ebs_per_request_ns/max.
python3 analysis/export_figures.py

The second instrument seeds files at least 120 s before measurement from a separate mount, so every measurement mount meets them cold. One client stats each file every 0.1 s, every 1 s or every 10 s, with 300 s of warm-up and 600 s measured. Misses are graded from mountstats RPC counters rather than a latency threshold, because a counter says whether a call went to the wire and a timer only says how long it took. What those counters count is RPC operations, which is not the same quantity as file-miss events, so I quote the rate rather than converting it into a hit percentage.

At the one-second cadence EFS measures about 0.017 RPC operations per positive stat: at 0.1 s it is 0.0017, at 10 s it is 0.16. Wire activity at a realistic cadence is therefore low, and the client is answering nearly all of this from memory. What that buys is a median request of about 2 ms of aggregate stat time, 2.14 / 2.09 / 1.88 ms across three replications, against 1.77 / 1.63 / 1.50 ms on EBS. The mean is 3.91×, 4.43× and 5.19×. The p99 is 151 to 217 ms, with maxima at 233 to 262 ms (the cadence finding, E5-attribute-cache). All of these are aggregate stat timings on the EC2 plain host-mounted arm, not end-to-end page latency, and not the WordPress path.

Concentration is what makes the shape legible. The top 5% of requests carried 82.9%, 89.4% and 82.5% of all NFS RPC operations. In the raw per-request timings of the first replication, which I inspected by hand rather than through a committed derivation, requests slower than 20 ms arrive in clusters 56 to 59 s apart on EFS and never on the EBS control. That inspection covers one replication. It is consistent with the 60 s window and with files compiled together expiring together, and both remain inference; nothing in the data separates synchronised expiry from any other reason a burst would appear.

Lustre is the control for what no attribute timer buys: its share of slow calls stayed at 0.157 to 0.335% at every interval, against a 1% ceiling the experiment set as its bound, with the same-time EBS control at 0.0000 to 0.0243%, at its Scratch 2, 1200 GiB, version 2.15 configuration. That is a latency observation, not proof of lock state.

The mean is where a reading goes wrong. The aggregate stat time it measures is real work, and it is a defensible input to a model of total waiting, provided the model is about totals; what it cannot do is describe the typical request, which sits near 2 ms, or the concentrated stalls, which sit at 151 to 217 ms. Whether that aggregate waiting turns into fleet cost depends on concurrency, worker occupancy and your latency target, none of which this study measured.

The launch type chooses the menu

EFS

0100EFS — replication 1 of 3: 33.5%33.5%EFS — replication 2 of 3: 37.8%37.8%EFS — replication 3 of 3: 30.9%30.9%

33.5% / 37.8% / 30.9%

FSx OpenZFS

0100FSx OpenZFS — replication 1 of 3: 69.6%69.6%FSx OpenZFS — replication 2 of 3: 68.9%68.9%FSx OpenZFS — replication 3 of 3: 67.9%67.9%

69.6% / 68.9% / 67.9%

FSx ONTAP

0100FSx ONTAP — replication 1 of 3: 100%100%FSx ONTAP — replication 2 of 3: 100%100%FSx ONTAP — replication 3 of 3: 100%100%

100% / 100% / 100%

EFS

0100EFS — replication 1 of 3: 33.5%33.5%EFS — replication 2 of 3: 37.8%37.8%EFS — replication 3 of 3: 30.9%30.9%

33.5% / 37.8% / 30.9%

FSx OpenZFS

0100FSx OpenZFS — replication 1 of 3: 69.6%69.6%FSx OpenZFS — replication 2 of 3: 68.9%68.9%FSx OpenZFS — replication 3 of 3: 67.9%67.9%

69.6% / 68.9% / 67.9%

FSx ONTAP

0100FSx ONTAP — replication 1 of 3: 100%100%FSx ONTAP — replication 2 of 3: 100%100%FSx ONTAP — replication 3 of 3: 100%100%

100% / 100% / 100%

Linear scale · percent · Supplied replications

Share of stat calls answered from client memory, by a descriptive rule the text states and labels as chosen after looking: EFS 33.5% / 37.8% / 30.9%, ONTAP 100% / 100% / 100%. The tier that creates files fastest looks fastest, because its files are still inside the attribute-cache window. Run results/E2/20261015T000959Z-85382cc.

Source: mikeng-io/aws-wordpress@b52192b
results/E2/20261015T000959Z-85382cc

Measurement definitions / Command
EFS
share of the 1,000 stat calls answered in client memory under the descriptive, post-hoc 20 us rule (stat latency < 20 us = answered in client memory), declared after inspecting the run's samples and not pre-registered; every absolute threshold in the 15-30 us sensitivity range yields identical shares, so the value is not tuned. It disagrees with the repo's registered E3-gated k-means clustering rule, which leaves ONTAP and Lustre undefined and is kept as per-rep diagnostics (registered_rule) in the derived file, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/efs/replications/rep-1/answered_local_share; /tiers/efs/replications/rep-2/answered_local_share; /tiers/efs/replications/rep-3/answered_local_share.
FSx OpenZFS
share of the 1,000 stat calls answered in client memory under the descriptive, post-hoc 20 us rule (stat latency < 20 us = answered in client memory), declared after inspecting the run's samples and not pre-registered; every absolute threshold in the 15-30 us sensitivity range yields identical shares, so the value is not tuned. It disagrees with the repo's registered E3-gated k-means clustering rule, which leaves ONTAP and Lustre undefined and is kept as per-rep diagnostics (registered_rule) in the derived file, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/fsx_openzfs/replications/rep-1/answered_local_share; /tiers/fsx_openzfs/replications/rep-2/answered_local_share; /tiers/fsx_openzfs/replications/rep-3/answered_local_share.
FSx ONTAP
share of the 1,000 stat calls answered in client memory under the descriptive, post-hoc 20 us rule (stat latency < 20 us = answered in client memory), declared after inspecting the run's samples and not pre-registered; every absolute threshold in the 15-30 us sensitivity range yields identical shares, so the value is not tuned. It disagrees with the repo's registered E3-gated k-means clustering rule, which leaves ONTAP and Lustre undefined and is kept as per-rep diagnostics (registered_rule) in the derived file, per replication; derived by analysis/e2_cache_window.py from the pinned FSx run's raw CSVs; the array is the three replications. Source JSON pointers (output order): /tiers/fsx_ontap/replications/rep-1/answered_local_share; /tiers/fsx_ontap/replications/rep-2/answered_local_share; /tiers/fsx_ontap/replications/rep-3/answered_local_share.
python3 analysis/export_figures.py

The rule the table below encodes: which tiers exist for you is settled by your launch type, before any performance question is asked.

TierOn EC2On FargateWhat decides it
EFSECS-managed volume, or host mount and bindECS-managed volume, platform 1.4.0+mount helper, port 2049
EBS gp3ECS volume or root volumedocumented by AWS as a volume optionno measured Fargate arm
FSx for Lustrehost mount and bindnot availablenot an ECS volume type
FSx for OpenZFShost mount and bindnot availablenot an ECS volume type
FSx for ONTAPtask-definition volume, native NFSS3 Access Points only, S3 API not POSIXEC2 for native NFS
S3 Access PointsS3 APIreachable, not a mountobject semantics, not POSIX

Eligibility above is documented rather than measured (ECS data volumes). FUSE tiers are absent for a platform reason rather than a test result: they need CAP_SYS_ADMIN, and Fargate permits adding exactly one Linux capability, CAP_SYS_PTRACE (Fargate task considerations). Every FUSE measurement in the study was made on EC2, and none was attempted on Fargate.

TierPinned configuration
EFSElastic throughput, One Zone, General Purpose, transit encryption on, NFS 4.1
FSx for OpenZFSSingle-AZ, 64 GiB, 128 MBps provisioned (not the 64 MBps floor), compression NONE
FSx for ONTAPSingle-AZ gen-1, 1024 GiB, 128 MBps, no snapshots, storage efficiency off
FSx for LustreScratch 2, 1200 GiB, version 2.15, mounted -o flock

Each row is a provisioning point, not a service-wide latency property. Provisioned throughput sizes an FSx file server’s in-memory metadata cache, so the OpenZFS and ONTAP figures are partly the cap; Scratch 2 is not Persistent, and nothing here describes the Persistent deployment type. The encrypted EFS row belongs to the per-call campaign; the cadence results above came from the plain host-mounted arm.

Three filesystems measured, and the honest summary is narrower than an elimination. JuiceFS reached 104.6 µs stat p50 and a create of 29.2 ms dominated by writing the file body to S3; it was registered to answer whether an object-backed userspace filesystem could beat EFS by an order of magnitude on metadata, and at 7.5× faster than the EFS cache-miss p50 it landed inconclusive rather than a win or a loss (the FUSE results, E2-fuse). SeaweedFS reached 20.2 µs stat, 2.28 ms create, and passed 11 of 11 conformance checks, which makes it, on EC2, a real option this study did not test against the application. Mountpoint for S3 failed 8 of 11 single-node checks in every replication, including rename that does not replace and mtime that does not advance, so it gets no latency table: a semantic failure must not be averaged into a performance ranking. What all three share is Fargate ineligibility, which is a platform exclusion and not a verdict about the software.

Hydration removes the expiry event

  • EFS · stat
  • Hydrated local tree · stat
110100100010000 EFS · stat — p50: 1388 µs; p95: 1745 µs; p99: 5106 µs EFS · stat — p50: 1388 µs 1388 µs EFS · stat — p95: 1745 µs 1745 µs EFS · stat — p99: 5106 µs 5106 µs Hydrated local tree · stat — p50: 3.061 µs; p95: 3.938 µs; p99: 6.023 µs Hydrated local tree · stat — p50: 3.061 µs 3.061 µs Hydrated local tree · stat — p95: 3.938 µs 3.938 µs Hydrated local tree · stat — p99: 6.023 µs 6.023 µs p50p95p99

EFS · stat — p50: 1388 µs / p95: 1745 µs / p99: 5106 µs

Hydrated local tree · stat — p50: 3.061 µs / p95: 3.938 µs / p99: 6.023 µs

110100100010000 EFS · stat — p50: 1388 µs; p95: 1745 µs; p99: 5106 µs EFS · stat — p50: 1388 µs 1388 µs EFS · stat — p95: 1745 µs 1745 µs EFS · stat — p99: 5106 µs 5106 µs Hydrated local tree · stat — p50: 3.061 µs; p95: 3.938 µs; p99: 6.023 µs Hydrated local tree · stat — p50: 3.061 µs 3.061 µs Hydrated local tree · stat — p95: 3.938 µs 3.938 µs Hydrated local tree · stat — p99: 6.023 µs 6.023 µs p50p95p99

EFS · stat — p50: 1388 µs / p95: 1745 µs / p99: 5106 µs

Hydrated local tree · stat — p50: 3.061 µs / p95: 3.938 µs / p99: 6.023 µs

Log₁₀ scale · us · Ordered positions · straight segments join supplied observations

Stat latency on EFS and on a code tree hydrated to task-local storage at start, p50 to p99, one log scale. What hydration costs at task start is stated in the text. Spacing is ordinal. Run results/E4/20261029T091909Z-842d596.

Source: mikeng-io/aws-wordpress@b52192b
results/E4/20261029T091909Z-842d596

Measurement definitions / Command
EFS · stat
EFS stat percentile ladder, pooled across three deployments; element order p50, p95, p99; every element ns/1000 at four significant figures, exact ns in raw. Source JSON pointers (output order): /pooled/efs/stat/p50_ns; /pooled/efs/stat/p95_ns; /pooled/efs/stat/p99_ns.
Hydrated local tree · stat
Hydrated code-tree stat percentile ladder, pooled across three deployments; element order p50, p95, p99; every element ns/1000 at four significant figures, exact ns in raw. Source JSON pointers (output order): /pooled/hydrated/stat/p50_ns; /pooled/hydrated/stat/p95_ns; /pooled/hydrated/stat/p99_ns.
python3 analysis/export_figures.py

Hydrating a read-only code tree from EFS onto the task’s own ephemeral storage at startup does not remove filesystem access. It removes remote attribute-cache expiry from that read path. Three independent deployments, Fargate 1.4.0, ARM64, 0.5 vCPU and 1 GiB, 20 GiB ephemeral (the hydration finding, E4-hydration):

MeasurementOn EFSAfter hydrationRatio
stat p501.3884 ms3.061 µs0.00220
open_read p502.7109 ms11.667 µs0.00430
Tar extraction, 27,412,480 bytesn/a0.218 / 0.444 / 0.429 s2.04× spread on identical bytes
First log to benchmark, between armsn/a0.051 / 0.192 / 0.265 sconfounded, see below

The 3.061 µs and 0.00220 above are the reported summary values for the experiment, measured per operation on task-local storage. What held in every replication was the acceptance threshold behind them, that hydration be no worse than a tenth of the EFS stat cost and a fifth of the open_read cost; the per-run results vary around the summary rather than reproducing it digit for digit. The local distributions are unimodal, with stat p99 at 6.023 µs and the ENOENT p99 at 22.6 µs, which is the point: a local filesystem has no revalidation timer, so there is no event to expire.

I predicted hydration would add at least 0.5 s before the benchmark could start, and that was wrong in the cheap direction. Extraction itself took 0.218 to 0.444 s, and the between-arm first-log-to-benchmark delta came in at 0.051 to 0.265 s, which includes remote conformance writes and scheduling noise, so I would not book that delta as hydration’s causal overhead or as full startup time.

What this is not is a measured request-latency result. The hydrated tree never ran under the cadence instrument’s load, so the 151 to 217 ms stall has not been shown to disappear, and the connection between the two is an inference from two separate results. It is also specific: a read-only code tree, 27 MB rather than a production install, with no write-back adapter, no fleet refresh or invalidation behaviour, and no end-to-end latency measurement. It addresses the expiry event on one path. It does not make mutable application state local, and it is one bounded option rather than the only one left standing.

Audit notes: what the study can establish

Question the study askedOutcome
Would every server-backed tier stay more than 10× slower than the block group on stat?Refuted, mechanically: the apparent counterexamples were client-cache hits rather than server lookups, which is why the refutation carries no information
Does a first touch cost more than a revalidation?Yes on EFS, 2.8 to 4.4×; the registered band of 1.0 to 3.0 was refuted from both sides, by the EFS arms above it and by one OpenZFS replication at 0.72× below it
Does a request pay the per-call ratio?No: the mean landed inside its support bound and far under its refutation threshold (the cadence finding, E5-attribute-cache)
Does hydration cost less than the 0.5 s floor I predicted before the benchmark could start?No; extraction took 0.218 to 0.444 s, so the prediction is refuted and the between-arm delta is too confounded to substitute for it
Does hydration remove the request stall?Not shown; never run under the cadence instrument

Predictions were registered before the runs, at three replications where the design allowed, with every service measured at a single provisioning point. Wire activity is grounded in RPC counters, and wall-clock latency never grounds a claim about whether a call left the client. The 20 µs cache-hit cut, the one-replication clustering inspection and the synchronised-expiry explanation are labelled post-hoc wherever they appear. No end-to-end request latency was measured, no cadence-matched Fargate run exists, no deliberately shared host mount was tested, and cart and checkout traffic was counted but not instrumented end to end.

Monday: decision checklist

  1. Count your filesystem demand on a warm request before you compare any storage tiers. You cannot size capacity from a demand number you did not measure.
  2. Check what your launch type can attach at all, because an option that does not exist for you is not a slower option.
  3. Test the POSIX semantics your application needs first. rename and mtime are where the object-store mounts fail, and a correctness failure is an exclusion rather than a ranking.
  4. Measure at your own cadence, with RPC counters rather than a stopwatch, because a fresh-file matrix and a request-shaped sweep answer different questions.
  5. Quote a provisioning point with every number you reuse. Single-AZ OpenZFS at 128 MBps and Single-AZ gen-1 ONTAP at 1024 GiB are configurations, not the services.
  6. If the code side of your demand is read-only, consider hydrating it to task-local storage and measure both the extraction cost and the per-operation result. If it is not, that option does not apply to you.

For this workload the storage tier sets the shape of the tail as much as it sets the average, and the intervention that removes the expiry event is a change to where the code is read from. The mean penalty is still there, and it belongs to any accounting of total stat waiting; what it never described was an individual request, and the requests a user remembers are the ones at the p99.