mirror of
https://github.com/huggingface/xet-core.git
synced 2026-06-04 13:30:29 +08:00
3051478cdd
Currently, simulation global dedup shard queries return full shard bytes
with no configurable shard footer expiration, and simulation control
knobs are split between partially implemented paths. This PR adds global
dedup shard expiration control to simulation clients and servers, and
extends /simulation/set_config to cover shard expiration, max range
splitting, V2 reconstruction disabling, API delay, and URL expiration in
one path. This enables rapid simulation of the GC paths by setting the
global dedup expiration to a sub-epoch value.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches simulation client/server APIs and shard serialization behavior
(including new trait methods and HTTP knobs), so downstream implementors
and tests may break if not updated. Changes are scoped to simulation/GC
tooling paths but affect how global-dedup shard bytes are produced and
validated.
>
> **Overview**
> Adds a new simulation control to set **global-dedup shard
expiration**: `DirectAccessClient::set_global_dedup_shard_expiration`
now makes `query_for_global_dedup_shard` optionally return *minimal*
shard bytes (file section stripped) with `shard_key_expiry = now +
expiration` (sub-second durations round up).
>
> Extends `MDBMinimalShard` serialization with
`serialize_xorb_subset_with_expiry` to write an optional
`shard_key_expiry` footer, and updates `LocalClient`/`MemoryClient` to
use it when expiration is enabled.
>
> Unifies and expands runtime simulation knobs under
`/simulation/set_config` (global dedup expiration, max ranges per fetch,
disable V2 reconstruction, API delay, URL expiration) and updates
`SimulationControlClient` to apply them via a retried async POST. Also
moves integrity/reachability checks to `DeletionControlableClient`, adds
`verify_all_reachable`, and wires new `/simulation/verify_all_reachable`
with 501 behavior when no deletion client is configured.
>
> Separately, introduces **simulation-only xorb cut thresholds**
(`XORB_CUT_THRESHOLD_*`) driven by new `xet_runtime` xorb config
overrides, and updates upload/dedup code paths to use these thresholds.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
42bd9c3f4f. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->