mirror of
https://github.com/huggingface/xet-core.git
synced 2026-06-04 13:30:29 +08:00
b43c0aec0e
This PR moves the XetRuntime model away from using thread-local statics
and decouples the XetConfig and XetCommon structs from a single runtime.
It introduces a struct XetContext that gives the runtime context for
operations:
```
struct XetContext {
pub runtime : Arc<XetRuntime>, // The current tokio runtime wrapper, minus the config and common objects..
pub common : Arc<XetCommon>, // The common cache objects, semaphores, rate trackers, etc.
pub config : Arc<XetConfig> // The config
}
```
Now, instead of using functions like `xet_runtime()` and `xet_config()` that examine the thread-local storage, we now explicitly passing through a XetContext instance from the session creation that gets stored in each major processing struct.
This allows decoupling between the runtime, config, and common caches, especially:
- Running multiple config settings and/or endpoints within the same pre-existing tokio runtime.
- Running multiple runtimes that share the same XetCommon object.
hf-xet
Client library for the Hugging Face Xet data storage system. Provides the high-level session API for uploading and downloading files with chunk-based deduplication.
Overview
- XetSession — Top-level session managing authentication, configuration, and concurrent file transfers
- Upload & download — Stream files to/from Hugging Face Hub with automatic chunking, deduplication, and local caching
Crate Ecosystem
hf-xet ties together the lower-level xet-core crates:
| Crate | Role |
|---|---|
xet-runtime |
Async runtime, config, logging |
xet-core-structures |
Merkle hashes, shards, Xorb objects |
xet-client |
HTTP client for CAS and Hub APIs |
xet-data |
Chunking, dedup, file reconstruction |
This crate is part of xet-core, the Rust backend for huggingface_hub.
License
Apache-2.0