Commit Graph

571 Commits

Author SHA1 Message Date
Assaf Vayner
b83054b3d5 revert offset comment (#22) 2024-09-24 08:48:56 -07:00
Rajat Arya
dcd20e2958 CasObject validation with updated tests (#21)
* CasObject validation with updated tests

- also added Copy trait to HexMerkleHash struct

* Pulled fence-post out of loop
2024-09-23 17:39:53 -07:00
Brian Ronan
6950115971 Plumb endpoint and access token (#19)
* Basic plumbing of the endpoint and access token through to calls to xetcas
2024-09-23 14:24:06 -07:00
Rajat Arya
148b7205ed CasObject::serialize validates hash (#18)
* Added tests for CAS hash generation

* Added hash validation to CasObject::serialize

- Removed from LocalClient::put
- Updated unit-tests across LocalClient, StagingClient, CacheClient

Now all paths to creating Xorbs will validate the hash provided matches
the contents being offered.

* Fix clippy
2024-09-23 11:43:58 -07:00
Assaf Vayner
9ba46cbf8c fix lints for stable (#14)
* fix lints

* more lints
2024-09-20 15:14:04 -07:00
Rajat Arya
9392e592e4 Add LZ4 compression usage to CasObject (#17)
* Consolidate CompressionScheme enum

- removed from utils crate
- moved from cas_types crate to cas_object

* Adds LZ4 compression, with unit-tests
2024-09-20 15:13:22 -07:00
Rajat Arya
037d267f07 CasObject v2 (#16)
CasObject v2 includes:
* Major changes to metadata / info kept with each Xorb
* Header -> Footer (now called CasObjectInfo)
* Chunks include efficient 8-byte header, can be compressed
* More unit-tests
* Lots of documentation
---------

Co-authored-by: Hoyt Koepke <hoytak@xethub.com>
Co-authored-by: Assaf Vayner <assafvayner@gmail.com>
Co-authored-by: Assaf Vayner <assaf@huggingface.co>
2024-09-20 12:16:53 -07:00
Assaf Vayner
202005145e change range to u32 (#15)
* change range to u32

* remove conversions
2024-09-17 09:45:00 -07:00
Joseph Godlewski
1936d72900 Fix endpoint propagation (#13)
Co-authored-by: Brian Ronan <brian.ronan@huggingface.co>
2024-09-13 20:08:49 -07:00
Rajat Arya
d819d0b219 Fixing RemoteClient XORB upload (#12)
* Fixing upload

* fixing merge
2024-09-13 19:17:56 -07:00
Assaf Vayner
539a4fc7c1 smudge pointer with new' (#11)
* smudge pointer with new'

* scheme

* lint

* no grpc mock test
2024-09-13 18:20:49 -07:00
Rajat Arya
ec98cef9b8 CAS HTTP remote client (#6)
* Partial Remote Client - not building yet.

* Everything builds, but unlikely anything works

* comments

* Rust file reconstruction

Co-authored by Assaf

* Fixing clippy & build
2024-09-13 18:02:49 -07:00
Di Xiao
d553005188 http shard client (#7)
* import cas_types definitions

* http based shard client

* use cas_types::Key

* add test

* fix linting
2024-09-13 17:54:11 -07:00
Joseph Godlewski
2f7116afde Adding hf_xet integration (#10) 2024-09-13 17:39:45 -07:00
Assaf Vayner
db47085e8d Update cas_types Cargo.toml (#9) 2024-09-13 15:06:37 -07:00
Assaf Vayner
f077033146 add cas_types (#8) 2024-09-13 15:00:05 -07:00
Rajat Arya
4844be430d CasObject + LocalClient (#5)
* CasObject with tests as cas_object crate

* LocalClient using CasObject

- all unit-tests are running successfully.

* Added back unit-tests from prev local_client.rs

* Removed unused variants for CasObjectError enum

* Removed unused use

* Add github actions (#4)

* Fixed clippy unhappiness

---------

Co-authored-by: Di Xiao <di@huggingface.co>
2024-09-12 15:44:10 -07:00
Di Xiao
1f006e2c12 Add github actions (#4) 2024-09-12 15:12:48 -07:00
Di Xiao
b65f57feaf examples of clean and smudge (#3) 2024-09-11 16:44:25 -07:00
Di Xiao
d4fe96147c move over trimmed xet-core (#1) 2024-09-11 13:30:20 -07:00
seanses
b812e40a89 first commit 2024-09-10 14:03:20 -07:00