Capacity-write transaction admission

Issue #3185 closes two pre-staging requirements: durable-mode disagreements must produce a typed HTTP response rather than a 500, and capacity-sensitive source writes must read the durable mode in their own transaction. The existing allocation eligibility, atomic capacity filters, reservation pipeline and optimistic version checks remain the source-write contract.

When materialized writes are enabled, standalone assignment and screening capacity saves own a snapshot transaction. The annotation capacity save never owns one: its only caller (SubmitAnnotationSessionService) opens the transaction that also carries the presence rows and the annotation delta, and the durable-mode read joins it. Calling it without a session while the flag is on fails closed rather than reading the fleet-wide mode outside the write's snapshot. The statistics-enabled screening overload checks the durable mode in its existing source-and-statistics transaction before preparing or replacing the Study. A disagreement aborts before source mutation. A rejected statistics commit aborts the preceding source replacement. Each request captures MaterializedProjectStatisticsWrites once, at entry, and passes that value to both the transaction decision and the durable-mode agreement check. RuntimeFeatureFlagProvider.ApplyBackendFlags mutates the shared FeatureFlags instance in place every thirty seconds, so re-reading the property at the second decision point could admit a write whose mode read never joined a snapshot, or refuse one for a flag change that arrived after the transaction was already chosen. One request sees one value.

A capacity guard that matched nothing aborts its transaction and, for the screening save, runs its diagnosis outside it, matching the statistics overload: nothing was written, so nothing is committed. An assignment filter miss is the ordinary "already reserved, already completed or at capacity" answer, so it also leaves the repository's Study cache untouched; only a committed assignment evicts. The screening save evicts however its attempt ends, because PrepareScreeningSave runs OnSaving on the cached instance before Mongo is called. Every statistics-writing transaction — the owned capacity transactions and the three overloads that open their own (SaveScreeningWithCapacityGuardAsync with a commit, SaveWithAnnotationStatisticsAsync and TrySaveScreeningWithStatisticsAsync) — starts at ReadConcern.Snapshot from one shared TransactionOptions, so a path's durable-mode read, its capacity guard and its source replace cannot land on different committed snapshots. They all commit through the shared bounded unknown-commit-result retry, so a partition that returns UnknownTransactionCommitResult retries the idempotent commit instead of surfacing as a 500.

MongoDB's TransientTransactionError label means the transaction aborted, whichever half raised it — an unknown commit result carries UnknownTransactionCommitResult instead and has already been retried by the shared helper — so no answer given for one can hide a durable write. What decides the response is therefore whether the callback may be re-run, not which half failed. Assignment builds its filter and pipeline from immutable arguments, so it is retried in a fresh snapshot for at most three whole-transaction attempts; when that budget is spent it answers its ordinary null rather than raising a driver exception at the client, because a caller cannot tell a lost race from a full stage. A mutable Study callback has already applied OnSaving and a classified transition, so replaying it would apply them to newer source data: it is never re-run, and answers the caller's existing optimistic-concurrency result (VersionStale or false) for the caller to reload and reapply. The screening statistics paths carry a second net: TrySaveScreeningAsync mints its operation envelope once outside its retry loop, so a replay is rejected as DuplicateOperation and treated as saved.

With materialized writes disabled, these standalone paths create no statistics transaction and do not read durable controls. This preserves the default deployment's write behavior and hot-path cost. Which overload a caller picks is the one capacity decision no in-transaction check can repair: choosing the plain optimistic save from a stale cached flag commits a screening with no capacity guard, and the chosen overload cannot add the guard back. ReviewController.TrySaveScreeningAsync and SubmitAnnotationSessionService therefore branch on ResolveVerifiedActiveReviewerTrackingModeAsync, which runs the same fail-closed agreement check. That resolution is gated on materializedProjectStatisticsWrites alone, not on whether a statistics commit was prepared: the family flag and the project allowlist decide whether a delta is written, not whether this replica's capacity semantics are the fleet's, so a project outside the allowlist would otherwise still save unguarded on a stale replica.

That resolution happens once per attempt, not once per request: the untracked overloads refuse when the mode has moved under them, and re-entering the same branch on every retry would spend the whole budget on a conflict no reload can resolve. The same rule governs which mutation a deletion applies, since restoring a reviewer's slot reservation is a capacity decision too.

Both untracked statistics saves — screening and annotation — re-check inside their transaction and refuse when the mode they read there says tracking is available: between the caller's resolution and the transaction, the durable mode and the process flag can both move to "tracked", which the agreement check accepts because they now agree, and the version-only replacement would commit unguarded. Nothing has been written, so they answer the optimistic-conflict result and the caller reloads, re-resolves and reapplies. The comparison is against the mode the caller assumed, not merely against "tracked": a reconciliation submission and a session deletion need no capacity guard, but both were still shaped by that mode — a deletion chooses whether to restore the reviewer's slot reservation from it — so a moved mode makes their decision stale too. RemoveSession therefore carries its own bounded reload loop that re-resolves the mode and re-chooses the mutation.

A save with no delta to write — the family off, or the project outside the allowlist — still goes through the transactional overload while writes are on, with a null statistics commit, and that includes the session deletion. assumedTrackingMode has no default, so every caller states the mode it shaped its write from rather than inheriting one. What the family flag and the allowlist decide is whether a delta is written, not whether the mode the caller branched on is still the fleet's when the replace happens, and the plain SaveAsync / TrySaveExistingAsync calls re-read nothing. With writes off those calls are unchanged and open no transaction.

Every caller-side early return that hands a reviewer a study with no reservation follows the same rule, because the atomic assignment's own in-transaction gate is never reached when the caller returns before it: the direct and random assignment paths in StageReviewService and the JoinStudyReview grant in NotificationHub all resolve the verified mode when writes are on, and read nothing when they are off. Direct navigation goes further, because it is the one grant that both decides "no reservation is needed" and reads the study it hands over: TryLoadStudyWhenUntrackedAsync takes both from one snapshot, so no window remains between them.

The random-assignment entry points also resolve the mode before their pre-loop existing-reservation lookup, and re-run that lookup inside the loop when a skipped lookup is followed by a verified tracked answer — otherwise the reviewer would be given a second slot. The direct-request controller path decides whether to run the guarded direct-navigation method from the same verified mode.

JoinStudyReview takes the same snapshot-bound treatment, because its untracked grant creates no reservation, connection or presence and so has nothing later that would revalidate the mode. The direct-request controller path runs the guarded method whenever writes are enabled rather than deciding from a standalone read of its own, and an indeterminate annotation commit evicts the Study cache before propagating, since that commit may already be durable.

What remains a recorded residual for #3308 is narrower: the three random-assignment loops hand back a candidate loaded by a preceding query, so re-reading it inside the mode's transaction would be a different document. The mode they decide from is verified and a late existing-reservation lookup stops a transition from granting a second slot, but a transition after the final decision is not observed by any later operation on those paths.

An indeterminate commit — the bounded unknown-result retry spent — may already be durable, so the capacity-write wrapper evicts its cache before rethrowing; an abort or a filter miss keeps the cache, because neither can have changed anything. Callers that own their transaction commit through the same bounded retry, and a caller that replaces SaveAsync with a repository statistics overload dispatches the aggregate's domain events itself after a confirmed commit, because the overload cannot.

Every review action that can reach one of these writes declares the typed 503, and RemoveSession declares its optimistic 409, so the generated client models both instead of treating them as unexpected errors. The API maps only ProjectStatisticsDurableModeDisagreementException to application/problem+json, HTTP 503, with code statistics_durable_mode_disagreement; unrelated failures retain their normal handling. 503 is the contract the technical plan and CLAUDE.md already bind for this condition, and it is the honest one: a disagreement means this replica's process flag does not match the fleet-wide durable mode, which is deployment skew the caller cannot fix by resubmitting. A 409 would invite a reload-and-resubmit loop that keeps failing for as long as the skew lasts.

Acceptance evidence

• Replica-set tests exercise assignment, screening assignment, annotation capacity save and screening capacity save with statistics enabled, and verify the durable read occurs inside a transaction.
• A real durable singleton disagreement rejects a statistics-enabled screening save before changing the Study, operation receipt or committed projection revision.
• A duplicate statistics operation rejected after the source replacement leaves the persisted Study, source revision and projection revision at their prior committed values.
• A screening save whose capacity/version guard does not match reaches no commit and leaves the stored document byte-identical, and an assignment that matched nothing reaches no commit either.
• A writes-flag refresh landing between the transaction decision and the agreement check cannot split them: the request that owned a transaction still validated the mode inside it.
• An assignment that matched nothing leaves a cached Study instance in place, while a committed one evicts it.
• Every statistics-writing transaction start is pinned to snapshot read concern by test, including the one SubmitAnnotationSessionService opens and lends to the annotation capacity save.
• A branch that consumes the durable mode follows the value its transaction verified, not a runtime flag refresh that arrives afterwards.
• Assignment recovers from a transient abort inside its budget and answers null when the budget is spent; a mutating callback runs exactly once.
• The annotation statistics save answers false on a stale version and on a transient abort, and fails closed on a durable-mode disagreement before writing anything.
• With the durable mode available and this replica's flag stale, screening and annotation saves are capacity-guarded rather than committed unguarded — including when no statistics commit is prepared — and an unverifiable mode refuses before the study is mutated.
• Both untracked statistics saves refuse when the mode moves under them, and allow a save whose caller assumed the mode it actually finds.
• A deletion re-chooses its mutation when the mode moves under it, rather than committing a plain delete for a reviewer whose slot should have been restored.
• A stale replica cannot hand out review work without a reservation through the service or the hub, and neither reads a control row when materialized writes are off.
• A save with no delta still opens its transaction and compares the assumed mode while writes are on, and still uses the plain save when they are off.
• The direct-navigation grant reads the study in the snapshot that verified the mode, and answers null — sending the caller to the atomic claim — when that mode says tracking is available, and the controller decides whether to run it from that same verified mode.
• A random assignment whose mode turns on after its pre-loop lookup was skipped returns the existing reservation rather than claiming a second slot, in all three loops — annotation, screening and the combined stage.
• A refused untracked save is followed by a re-resolution on the next attempt, so the retry takes the guarded overload instead of exhausting the budget on a conflict.
• An indeterminate assignment commit evicts the cached Study; an indeterminate submission commit is retried rather than failing the submission.
• Domain events raised by a statistics save reach the dispatcher exactly once, and only after commit.
• Existing flags-off tests prove no durable-mode read, while API filter tests prove both a recorded disagreement and an unverifiable mode produce the typed 503 and unrelated failures do not.

Accepted costs

• Under ReadConcern.Snapshot a concurrent version bump now raises a write conflict where the session-less guard previously returned a precise AtCapacity / VersionStale / NotFound diagnosis. The conflict is flattened to VersionStale, and ReviewController.TrySaveScreeningAsync recovers the precise answer on its bounded reload loop, so a genuinely at-capacity study can burn retries before reporting AtCapacity.
• The transaction is opened on materializedProjectStatisticsWrites alone, not on the narrower screening family flag and project allowlist. Enabling the global flag therefore costs a transaction on every assignment and every screening capacity save, including projects with no delta to write. Latency and retry acceptance at ½/5/10 reviewers remain separate measured evidence (#3255).

Reservation-family materialization, transaction domain-event delivery and staging activation retain their separate programme boundaries. This change introduces no schedule, consumer cutover or flag activation.