mirror of
https://github.com/huggingface/xet-core.git
synced 2026-06-04 13:30:29 +08:00
7aea9fac07
Currently, GC simulation coverage depends on deletion-control operations
that were only partially wired through the disk-backed HTTP simulation
path, and deletion behavior in LocalClient needed to preserve shard-hash
stability across GC epochs. This PR adds shard dedup-entry cleanup to
the deletion-control surface and updates the file deletion behavior so
shard files are not rewritten.
Note: this introduces a breaking change against LocalClient in that
current LocalClient repositories won't persist across this commit.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Changes affect disk-backed simulation CAS behavior and persistence
(new LMDB tables + deletion semantics) and add a new deletion-control
API used by GC; regressions could break existing local test data or GC
integration paths.
>
> **Overview**
> Enables *correct* GC integration testing against the disk-backed
simulation server by switching `LocalClient::delete_file_entry` to a
**soft-delete** backed by a new LMDB `file_status_table`, and updating
listing/reconstruction/direct-file-access paths to hide and reject
deleted files without rewriting shard files.
>
> Extends the deletion-control surface with `remove_shard_dedup_entries`
(plus a new `DELETE /simulation/shards/{hash}/dedup_entries` route and
client support) and fixes `LocalTestServerBuilder` to actually wire a
`deletion_client` for disk-backed servers so `/simulation/*` deletion
routes stop returning `501`.
>
> Reworks `verify_integrity` to validate XORB references *across shards*
(global dedup aware) and skip soft-deleted files, adds targeted
unit/integration tests for the new behaviors, and tightens log cleanup
to avoid protecting stale logs on PID reuse by comparing process start
time to the log’s embedded timestamp.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
fdca297600. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->