mirror of https://github.com/huggingface/xet-core.git synced 2026-06-04 13:30:29 +08:00

Files

Hoyt Koepke 0d9f78aaf4 Add README.md files and Cargo.toml updates needed for publishing hf-xet (#773 )

This PR adds crates.io-facing metadata (homepage, readme, keywords,
categories) for the publishable crates, along with crate README files
and concise crate-level docs so crates.io and docs.rs pages have better
context.

2026-04-03 12:34:47 -07:00

benches

fix: truncate local file on full-file download to prevent corruption (#764 )

2026-03-30 17:57:23 +02:00

examples

Code reorganization towards release of xet cargo package (#693 )

2026-03-11 12:02:38 -07:00

src

Add README.md files and Cargo.toml updates needed for publishing hf-xet (#773 )

2026-04-03 12:34:47 -07:00

tests

feat: optional chunk cache in download path for cross-file dedup (#731 )

2026-04-01 17:29:28 +02:00

Cargo.toml

Add README.md files and Cargo.toml updates needed for publishing hf-xet (#773 )

2026-04-03 12:34:47 -07:00

README.md

Add README.md files and Cargo.toml updates needed for publishing hf-xet (#773 )

2026-04-03 12:34:47 -07:00

README.md

xet-data

Data processing pipeline for chunking, deduplication, and file reconstruction. Intended to be used through the API in the hf-xet package.

Overview

Content-defined chunking — Gear-hash based chunking for deduplication
Deduplication — Probe and register chunks against metadata shards
File reconstruction — Reassemble files from deduplicated chunk references
Progress tracking — Hooks for upload/download progress reporting

This crate is part of xet-core.

License

Apache-2.0