pub_soft/xet-core

mirror of https://github.com/huggingface/xet-core.git synced 2026-06-04 13:30:29 +08:00

Files

History

Di Xiao 23ec2940bb Expose XetSession APIs to Python (#792 )

Replaces the old `upload_files` / `download_files` / `hash_files` Python
functions with a new object-oriented API that exposes `XetSession` and
its child objects directly as PyO3 classes. This gives Python callers
full control over session lifecycle, connection pooling, and progress
reporting.

The previous module-level functions are kept under `hf_xet/src/legacy/`
and remain importable as `from hf_xet import upload_files` etc., but now
emit `DeprecationWarning`.

2026-05-01 03:05:51 -07:00

..

Move XetRuntime model away from thread-local statics (#801 )

2026-04-21 09:17:19 -07:00

Code reorganization towards release of xet cargo package (#693 )

2026-03-11 12:02:38 -07:00

Expose XetSession APIs to Python (#792 )

2026-05-01 03:05:51 -07:00

feat: optional chunk cache in download path for cross-file dedup (#731 )

2026-04-01 17:29:28 +02:00

Cargo.toml

set version 1.5.2 (#805 )

2026-04-20 15:06:14 -07:00

README.md

Add README.md files and Cargo.toml updates needed for publishing hf-xet (#773 )

2026-04-03 12:34:47 -07:00

README.md

xet-data

Data processing pipeline for chunking, deduplication, and file reconstruction. Intended to be used through the API in the hf-xet package.

Overview

Content-defined chunking — Gear-hash based chunking for deduplication
Deduplication — Probe and register chunks against metadata shards
File reconstruction — Reassemble files from deduplicated chunk references
Progress tracking — Hooks for upload/download progress reporting

This crate is part of xet-core.

License

Apache-2.0