---
title: "The 638× Stat Is Not Page Latency"
description: "A 638× EFS stat and a 4× sweep mean less than they look. Here's what each benchmark measures, and how to size filesystem demand before choosing a tier."
pubDate: 2026-11-13
tags: ["filesystem-benchmarking", "efs-latency", "nfs-attribute-cache", "container-storage-attachment", "posix-legacy-workloads"]
locale: en
canonical: https://mikeng.io/writing/638-stat-not-page-latency/
author: "Mike Ng Sin Ching"
authorUrl: https://mikeng.io/
---

Two figures, two workloads, and neither is measured page latency. The 638× is one `stat`, a file-metadata lookup, on a freshly created file, EFS against a block device. The 4× is the mean aggregate `stat` time in a synthetic request-shaped sweep against a local volume, where the median difference is closer to 1.2 to 1.3×. The mean is real, and it still does not tell you which request is slow.

Findings are cited by name throughout; all of them, with the runs behind them, live in the [study repository](https://github.com/mikeng-io/aws-wordpress).

## The instrument: fixed demand

WordPress is here as a specimen of a class most estates are made of: software written before object storage was normal, which assumes a POSIX filesystem and consults it constantly from its own core code, plugins, themes and template hierarchy. Treating the application as a black box is this study's constraint. I change task definitions, mount topology, container images, `php.ini` and opcache, and proxies; I do not change the application. I built the naive version of this architecture six years ago, and it is provenance here, not a subject.

Every count below is per warm request. A warm request issues about 4,300 filesystem syscalls, of which about 3,900 are `stat`, and no `php.ini` profile I tested moved that floor (the warm-request syscall census, `E0-n10`). Cold requests are a separate figure, 15,400 to 16,400 operations, and they do not add to the warm count.

| Measurement | Count |
| --- | --- |
| Home / product / wp-admin, total syscalls | 5,157 / 5,265 / 5,140 |
| Cart and checkout with a real item, total syscalls | 5,941 / 5,884 |
| `wp-includes` (core), filesystem ops | about 103 |
| `wpforms-lite`, filesystem ops | about 1,155 |
| Elementor / WooCommerce / theme / ACF | 590 / 528 / 214 / 137 |
| `ENOENT` lookups | about 1,600 |

The endpoint rows are total syscalls, not filesystem calls, which is why they sit above the census; they vary 14 to 15% between content and WooCommerce pages and not at all across ten repetitions (the cart and checkout counts, `E0-cart-checkout`). Roughly 1,600 of the filesystem calls return `ENOENT`, the application asking whether a file exists. On a block device that question costs almost nothing; on a network filesystem it is latency spent to learn nothing. That the storm comes from plugin code rather than core is observed per component. Why each probe exists is inferred, because no call site was traced back to PHP source.

The two settings people recommend here change how often the application asks, not how much it asks. Turning off `opcache.validate_timestamps` removes about 700 `open` calls and about 1.8% of stats inside the revalidation window, and a validating profile pays a roughly +4,400-stat tax once per `revalidate_freq` window, 2 s naive against 60 s tuned (same census, `E0-n10`). The three profiles I ran, including one at `realpath_cache` 32 MB with a 3600 s TTL, all left the floor at about 3,900 (the PHP profile comparison, `E0-first-run`). Because that tax is paid once per window, it is worth most per request on low-traffic sites.

## Two numbers, two workloads

![One comparison, two denominators. Per call: 638.2×, measured on freshly created files. Per request, the same filesystem under a realistic access pattern: median 1.21× / 1.29× / 1.25×, mean 3.91× / 4.43× / 5.19×, p99 88.7× / 125.9× / 87.6× (one value per replication). Log scale; run results/E2/20260910T163206Z-66dcff8, results/E5-formal/20261028T160106Z-dd5fce0.](https://mikeng.io/writing/638-stat-not-page-latency/figures/per-call-vs-per-request.svg)

A `stat` on EFS takes 638 times as long as the same call on a block device. In a request-shaped sweep over files at least two minutes old, with one client `stat`ing each of them about once a second, aggregate `stat` time costs about 1.2 to 1.3 times the median and 3.9 to 5.2 times the mean of the identical sweep against local EBS. Multiply those ratios and you get a figure nobody measured, which an earlier draft of my own findings did; and the per-request result does not refute the per-call one either, it re-scopes it. That earlier draft is the 638× figure you may have seen quoted on its own. The sweeps are a synthetic tree rather than a WordPress request path, and nothing here is scaled to the 3,900-call census above.

| Tier | stat p50 | open+read p50 | create p50 | unlink p50 |
| --- | --- | --- | --- | --- |
| Instance store | 1.3 µs | 5.2 µs | 19.7 µs | 7.5 µs |
| EBS gp3 | 1.3 µs | 5.3 µs | 19.9 µs | 7.6 µs |
| Fargate ephemeral | 1.8 µs | 5.7 µs | 21.9 µs | 11.4 µs |
| EC2 + EFS, Elastic throughput / One Zone / General Purpose / NFS 4.1 | 814 µs | 904 µs | 7.34 ms | 3.00 ms |
| Fargate + EFS, same configuration | 778 µs | 679 µs | 6.99 ms | 2.93 ms |

Pooled from three replications with three independent deployments per group (the per-call tier matrix, `E2-block-backed`). Ratios throughout are computed on unrounded pooled p50s, so recomputing them from this rounded table gives 626× rather than 638×. The block substrates are indistinguishable here: instance store against EBS is 1.00× on `stat`, 1.03× on open+read, 1.01× on create, 1.02× on unlink, with p99s tracking the same way. Across the boundary nothing ties: `stat` 638×, `unlink` 399×, `create` 373×, open+read 174×. The metadata operations suffer far more than the small-file `open+read` pass, and that is the shape of the problem: a server-backed tier is much less bad at this read than at answering questions about files. This was not a throughput test, and I would not read a byte-moving conclusion out of it.

Protocol rather than hardware is my inference, not an isolated result: no run varied only the protocol, so it rests on three block substrates agreeing and six EFS configurations agreeing. Four candidate levers measure as noise against a between-replication floor of 1.49 to 1.82×: transit encryption through the local `efs-proxy` (1.00× to 1.10× depending on mount path), mount topology (1.00 to 1.09×, and the ECS-managed path invokes the [same mount helper](https://docs.aws.amazon.com/efs/latest/ug/mount-helper-setting.html) as a hand-written one), EC2 against Fargate on the EFS path (1.06× on `stat`, 1.01× on `create`), and six EFS configurations spanning 683 to 786 µs `stat` p50, a 1.15× spread whose per-configuration gaps to the block group run 357 to 630×. The pooled 638× is the headline ratio; the 357 to 630× range is what each individual configuration manages. Those are per-call findings on one instrument, and none is a demonstrated tail lever.

Two collections stay apart: the Fargate ephemeral row above comes from the tier matrix, while a separate collection measured three Fargate tasks and 4,020 samples (the Fargate ephemeral finding, `E3-fargate-ephemeral`). Separately again, EFS mounts once per task rather than once per host. Two tasks on one `t4g.small`, same task definition and filesystem, produced two NFS4 client mounts, two `efs-proxy` processes and separate attribute caches (the mount isolation finding, `E1-mount-per-task`). Co-located tasks warming each other's cache does not happen by default, which is also why the only code a task can be sure of holding locally is code it wrote locally. The deliberately shared host mount was never measured, and I make no claim either way.

## File age, cache history, and the timer

![Seconds to create the test files on each tier, one dot per replication: 7.173 s / 6.502 s / 7.122 s on EFS against 0.9938 s / 0.9682 s / 0.9875 s on ONTAP. The next figure shows what that does to the benchmark. Run results/E2/20261015T000959Z-85382cc.](https://mikeng.io/writing/638-stat-not-page-latency/figures/cache-window-create-pass.svg)

The per-call benchmark creates a file set and then `stat`s it a few seconds later; one pass of the instrument is one create sweep followed by one stat sweep. The FSx group exposed what that measures: on every NFS tier the fast `stat`s form a contiguous tail of the most recently created files, ending at a fixed age (the server-backed tier matrix, `E2-server-backed`).

| Tier, pinned configuration | Time to create the file set | Stats answered fast | Boundary age |
| --- | --- | --- | --- |
| EFS, Elastic / One Zone / General Purpose / NFS 4.1 | 6.5 to 7.2 s | 31 to 38% | 2.30 to 2.48 s |
| FSx for OpenZFS, Single-AZ 64 GiB, 128 MBps | 3.4 s | 68 to 70% | 2.91 to 2.92 s |
| FSx for ONTAP, Single-AZ gen-1 1024 GiB, 128 MBps | 1.0 s | 100% | whole sweep inside the window |

A faster create sweep leaves more files inside the validity window when the stat sweep begins, so more of its `stat`s get answered locally. ONTAP sent 8 `GETATTR` RPCs for 1,000 `stat` calls, and its 1.7 µs p50 in that same group is a cache hit rather than a round trip. That belongs to its Single-AZ gen-1, 1024 GiB, 128 MBps provisioning point, not to ONTAP as a service.

The boundary is the Linux client's `acregmin`, 3 s by default ([nfs(5)](https://man7.org/linux/man-pages/man5/nfs.5.html)), with the backoff read from `fs/nfs/inode.c` at kernel v6.1 rather than inferred from these runs. Each unchanged revalidation doubles the timeout, 3 to 6 to 12 to 24 to 48 to 60 s, capped at `acregmax` and reset by any change. Files older than 60 seconds are therefore not what produces a miss: the timer follows the inode's revalidation history on this client, which is why a PHP file written a year ago and looked up for the first time on a cold mount starts at 3 s like a brand-new one.

```mermaid
sequenceDiagram
    participant App as Application
    participant C as Linux NFS client
    participant S as Server

    Note over App,S: first touch on a cold client
    App->>C: stat
    C->>S: LOOKUP compound
    S-->>C: attributes, timer starts at acregmin
    C-->>App: answered

    Note over C: attributes valid, up to 60 s
    App->>C: stat again
    C-->>App: answered from memory, no RPC

    Note over C: validity elapses
    App->>C: stat after expiry
    C->>S: GETATTR
    S-->>C: unchanged, timeout doubles toward acregmax
    C-->>App: answered

    opt read delegation held, OpenZFS capability only
        App->>C: stat
        C-->>App: answered locally, timer never runs
    end
```

Two different miss workloads come out of that. First touch on a cold client, at task start or scale-out or after a deploy, pays a `LOOKUP` compound, measured at 2.8 to 4.4× the revalidation cost on the EFS arms, 1.04 to 1.08× on OpenZFS with one replication at 0.72×, and 1.08 to 1.44× on ONTAP (the first-touch and revalidation comparison, `E5-attribute-cache`). The large premium is an EFS observation, not a property of NFS. Steady-state revalidation pays a `GETATTR` about once per `acregmax` per file: 263 to 357 µs on EFS, 69 to 98 µs on OpenZFS, 109 to 111 µs on ONTAP. That corrects an earlier figure of my own, the 827 µs EFS miss cost from the per-call benchmark, which does not reproduce here; these numbers are 0.32 to 0.43× it, because it was a different miss.

Whether the timer can be stopped at all is a protocol capability. EFS cannot grant NFSv4 delegations, so `OPEN` always returns `OPEN_DELEGATE_NONE` ([EFS limits](https://docs.aws.amazon.com/efs/latest/ug/limits.html)); FSx for OpenZFS can grant read delegations ([OpenZFS performance guide](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/performance.html)), and the runs counted 1,000 `DELEGRETURN`s on it against zero on EFS and ONTAP. A client holding one answers `stat` locally without starting the timer. That is the ability to hold a delegation, not a claim that every OpenZFS access holds one.

The mechanism is documentation- and kernel-supported inference that fits the measured boundaries; changing `actimeo` and watching the boundary move is what would settle it. The 20 µs cut used to call a `stat` "answered from client memory" is a post-hoc descriptive rule. And the defect is the metric's scope rather than the benchmark's authors: a per-call measurement on freshly created files compares tiers at different points in the cache lifecycle, with only 31 to 38% of EFS stats and 68 to 70% of OpenZFS stats landing inside the window, against all of ONTAP's. It measures a creation-history-dependent mixture of hits and misses, not a uniformly warm cache.

## The heartbeat: nearly tied until expiry

![Stat time summed per request, as a ladder from median to max, three replications of EFS and of the EBS control on one log scale. The control stays flat; EFS leaves it at the tail. Spacing is ordinal. Run results/E5-formal/20261028T160106Z-dd5fce0.](https://mikeng.io/writing/638-stat-not-page-latency/figures/heartbeat.svg)

The second instrument seeds files at least 120 s before measurement from a separate mount, so every measurement mount meets them cold. One client `stat`s each file every 0.1 s, every 1 s or every 10 s, with 300 s of warm-up and 600 s measured. Misses are graded from `mountstats` RPC counters rather than a latency threshold, because a counter says whether a call went to the wire and a timer only says how long it took. What those counters count is RPC operations, which is not the same quantity as file-miss events, so I quote the rate rather than converting it into a hit percentage.

At the one-second cadence EFS measures about 0.017 RPC operations per positive `stat`: at 0.1 s it is 0.0017, at 10 s it is 0.16. Wire activity at a realistic cadence is therefore low, and the client is answering nearly all of this from memory. What that buys is a median request of about 2 ms of aggregate stat time, 2.14 / 2.09 / 1.88 ms across three replications, against 1.77 / 1.63 / 1.50 ms on EBS. The mean is 3.91×, 4.43× and 5.19×. The p99 is 151 to 217 ms, with maxima at 233 to 262 ms (the cadence finding, `E5-attribute-cache`). All of these are aggregate `stat` timings on the EC2 plain host-mounted arm, not end-to-end page latency, and not the WordPress path.

Concentration is what makes the shape legible. The top 5% of requests carried 82.9%, 89.4% and 82.5% of all NFS RPC operations. In the raw per-request timings of the first replication, which I inspected by hand rather than through a committed derivation, requests slower than 20 ms arrive in clusters 56 to 59 s apart on EFS and never on the EBS control. That inspection covers one replication. It is consistent with the 60 s window and with files compiled together expiring together, and both remain inference; nothing in the data separates synchronised expiry from any other reason a burst would appear.

Lustre is the control for what no attribute timer buys: its share of slow calls stayed at 0.157 to 0.335% at every interval, against a 1% ceiling the experiment set as its bound, with the same-time EBS control at 0.0000 to 0.0243%, at its Scratch 2, 1200 GiB, version 2.15 configuration. That is a latency observation, not proof of lock state.

The mean is where a reading goes wrong. The aggregate stat time it measures is real work, and it is a defensible input to a model of total waiting, provided the model is about totals; what it cannot do is describe the typical request, which sits near 2 ms, or the concentrated stalls, which sit at 151 to 217 ms. Whether that aggregate waiting turns into fleet cost depends on concurrency, worker occupancy and your latency target, none of which this study measured.

## The launch type chooses the menu

![Share of stat calls answered from client memory, by a descriptive rule the text states and labels as chosen after looking: EFS 33.5% / 37.8% / 30.9%, ONTAP 100% / 100% / 100%. The tier that creates files fastest looks fastest, because its files are still inside the attribute-cache window. Run results/E2/20261015T000959Z-85382cc.](https://mikeng.io/writing/638-stat-not-page-latency/figures/cache-window-fast-share.svg)

The rule the table below encodes: which tiers exist for you is settled by your launch type, before any performance question is asked.

| Tier | On EC2 | On Fargate | What decides it |
| --- | --- | --- | --- |
| EFS | ECS-managed volume, or host mount and bind | ECS-managed volume, platform 1.4.0+ | mount helper, port 2049 |
| EBS gp3 | ECS volume or root volume | documented by AWS as a volume option | no measured Fargate arm |
| FSx for Lustre | host mount and bind | not available | not an ECS volume type |
| FSx for OpenZFS | host mount and bind | not available | not an ECS volume type |
| FSx for ONTAP | task-definition volume, native NFS | S3 Access Points only, S3 API not POSIX | EC2 for native NFS |
| S3 Access Points | S3 API | reachable, not a mount | object semantics, not POSIX |

Eligibility above is documented rather than measured ([ECS data volumes](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html)). FUSE tiers are absent for a platform reason rather than a test result: they need `CAP_SYS_ADMIN`, and Fargate permits adding exactly one Linux capability, `CAP_SYS_PTRACE` ([Fargate task considerations](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-tasks-services.html)). Every FUSE measurement in the study was made on EC2, and none was attempted on Fargate.

| Tier | Pinned configuration |
| --- | --- |
| EFS | Elastic throughput, One Zone, General Purpose, transit encryption on, NFS 4.1 |
| FSx for OpenZFS | Single-AZ, 64 GiB, 128 MBps provisioned (not the 64 MBps floor), compression NONE |
| FSx for ONTAP | Single-AZ gen-1, 1024 GiB, 128 MBps, no snapshots, storage efficiency off |
| FSx for Lustre | Scratch 2, 1200 GiB, version 2.15, mounted `-o flock` |

Each row is a provisioning point, not a service-wide latency property. Provisioned throughput sizes an FSx file server's in-memory metadata cache, so the OpenZFS and ONTAP figures are partly the cap; Scratch 2 is not Persistent, and nothing here describes the Persistent deployment type. The encrypted EFS row belongs to the per-call campaign; the cadence results above came from the plain host-mounted arm.

Three filesystems measured, and the honest summary is narrower than an elimination. JuiceFS reached 104.6 µs `stat` p50 and a `create` of 29.2 ms dominated by writing the file body to S3; it was registered to answer whether an object-backed userspace filesystem could beat EFS by an order of magnitude on metadata, and at 7.5× faster than the EFS cache-miss p50 it landed inconclusive rather than a win or a loss (the FUSE results, `E2-fuse`). SeaweedFS reached 20.2 µs `stat`, 2.28 ms `create`, and passed 11 of 11 conformance checks, which makes it, on EC2, a real option this study did not test against the application. Mountpoint for S3 failed 8 of 11 single-node checks in every replication, including `rename` that does not replace and `mtime` that does not advance, so it gets no latency table: a semantic failure must not be averaged into a performance ranking. What all three share is Fargate ineligibility, which is a platform exclusion and not a verdict about the software.

## Hydration removes the expiry event

![Stat latency on EFS and on a code tree hydrated to task-local storage at start, p50 to p99, one log scale. What hydration costs at task start is stated in the text. Spacing is ordinal. Run results/E4/20261029T091909Z-842d596.](https://mikeng.io/writing/638-stat-not-page-latency/figures/hydration.svg)

Hydrating a read-only code tree from EFS onto the task's own ephemeral storage at startup does not remove filesystem access. It removes remote attribute-cache expiry from that read path. Three independent deployments, Fargate 1.4.0, ARM64, 0.5 vCPU and 1 GiB, 20 GiB ephemeral (the hydration finding, `E4-hydration`):

| Measurement | On EFS | After hydration | Ratio |
| --- | --- | --- | --- |
| `stat` p50 | 1.3884 ms | 3.061 µs | 0.00220 |
| `open_read` p50 | 2.7109 ms | 11.667 µs | 0.00430 |
| Tar extraction, 27,412,480 bytes | n/a | 0.218 / 0.444 / 0.429 s | 2.04× spread on identical bytes |
| First log to benchmark, between arms | n/a | 0.051 / 0.192 / 0.265 s | confounded, see below |

The 3.061 µs and 0.00220 above are the reported summary values for the experiment, measured per operation on task-local storage. What held in every replication was the acceptance threshold behind them, that hydration be no worse than a tenth of the EFS `stat` cost and a fifth of the `open_read` cost; the per-run results vary around the summary rather than reproducing it digit for digit. The local distributions are unimodal, with `stat` p99 at 6.023 µs and the `ENOENT` p99 at 22.6 µs, which is the point: a local filesystem has no revalidation timer, so there is no event to expire.

I predicted hydration would add at least 0.5 s before the benchmark could start, and that was wrong in the cheap direction. Extraction itself took 0.218 to 0.444 s, and the between-arm first-log-to-benchmark delta came in at 0.051 to 0.265 s, which includes remote conformance writes and scheduling noise, so I would not book that delta as hydration's causal overhead or as full startup time.

What this is not is a measured request-latency result. The hydrated tree never ran under the cadence instrument's load, so the 151 to 217 ms stall has not been shown to disappear, and the connection between the two is an inference from two separate results. It is also specific: a read-only code tree, 27 MB rather than a production install, with no write-back adapter, no fleet refresh or invalidation behaviour, and no end-to-end latency measurement. It addresses the expiry event on one path. It does not make mutable application state local, and it is one bounded option rather than the only one left standing.

## Audit notes: what the study can establish

| Question the study asked | Outcome |
| --- | --- |
| Would every server-backed tier stay more than 10× slower than the block group on `stat`? | Refuted, mechanically: the apparent counterexamples were client-cache hits rather than server lookups, which is why the refutation carries no information |
| Does a first touch cost more than a revalidation? | Yes on EFS, 2.8 to 4.4×; the registered band of 1.0 to 3.0 was refuted from both sides, by the EFS arms above it and by one OpenZFS replication at 0.72× below it |
| Does a request pay the per-call ratio? | No: the mean landed inside its support bound and far under its refutation threshold (the cadence finding, `E5-attribute-cache`) |
| Does hydration cost less than the 0.5 s floor I predicted before the benchmark could start? | No; extraction took 0.218 to 0.444 s, so the prediction is refuted and the between-arm delta is too confounded to substitute for it |
| Does hydration remove the request stall? | Not shown; never run under the cadence instrument |

Predictions were registered before the runs, at three replications where the design allowed, with every service measured at a single provisioning point. Wire activity is grounded in RPC counters, and wall-clock latency never grounds a claim about whether a call left the client. The 20 µs cache-hit cut, the one-replication clustering inspection and the synchronised-expiry explanation are labelled post-hoc wherever they appear. No end-to-end request latency was measured, no cadence-matched Fargate run exists, no deliberately shared host mount was tested, and cart and checkout traffic was counted but not instrumented end to end.

## Monday: decision checklist

1. Count your filesystem demand on a warm request before you compare any storage tiers. You cannot size capacity from a demand number you did not measure.
2. Check what your launch type can attach at all, because an option that does not exist for you is not a slower option.
3. Test the POSIX semantics your application needs first. `rename` and `mtime` are where the object-store mounts fail, and a correctness failure is an exclusion rather than a ranking.
4. Measure at your own cadence, with RPC counters rather than a stopwatch, because a fresh-file matrix and a request-shaped sweep answer different questions.
5. Quote a provisioning point with every number you reuse. Single-AZ OpenZFS at 128 MBps and Single-AZ gen-1 ONTAP at 1024 GiB are configurations, not the services.
6. If the code side of your demand is read-only, consider hydrating it to task-local storage and measure both the extraction cost and the per-operation result. If it is not, that option does not apply to you.

For this workload the storage tier sets the shape of the tail as much as it sets the average, and the intervention that removes the expiry event is a change to where the code is read from. The mean penalty is still there, and it belongs to any accounting of total stat waiting; what it never described was an individual request, and the requests a user remembers are the ones at the p99.
