mirror of https://github.com/huggingface/xet-core.git synced 2026-06-04 13:30:29 +08:00

Files

Hoyt Koepke e443ee9260 Upgrade package dependencies (#644 )

This PR updates all the package dependencies that would not cause
significant API breakages to the current version. The package versions
in hf_xet_wasm and hf_xet are also updated to match the versions in the
base package. There should be no functional change.

2026-02-11 12:19:29 -08:00

src

Upgrade package dependencies (#644 )

2026-02-11 12:19:29 -08:00

Cargo.toml

Clean up dependencies (#595 )

2025-12-15 15:26:02 -08:00

README.md

Adding README to few crates for documentation (#492 )

2025-09-15 11:15:05 -07:00

README.md

mdb_shard

MDB -> Merkle Database

The mdb_shard crate exposes multiple interfaces for working with shards.

This includes particularly the shard file format as used as API payloads as well as used internally within xet-core to manage and store state during and between processes to deduplicate and upload data.

Serialization and Deserialization Interfaces

The mdb_shard crate provides multiple interfaces for serializing and deserializing shard data, organized by their purpose and usage patterns. These interfaces allow you to work with shard data at different levels of abstraction, from low-level binary serialization to high-level streaming processing.

Core Shard Format Interfaces

src/shard_format.rs

These interfaces handle the core shard file format and metadata:

MDBShardInfo::load_from_reader() - Loads complete shard metadata (header + footer) from a reader
MDBShardInfo::serialize_from() - Serializes an in-memory shard to binary format

Streaming and Processing Interfaces

src/streaming_shard.rs

MDBMinimalShard::from_reader() - Creates a minimal shard representation for lightweight operations from a reader
MDBMinimalShard::from_reader_async() - Creates a minimal shard representation for lightweight operations, from an async reader

File Handle Interfaces

src/shard_file_handle.rs

MDBShardFile::load_from_file() - Loads shard from a file path with caching