mirror of
https://github.com/huggingface/xet-core.git
synced 2026-06-04 13:30:29 +08:00
15011cb230
This PR makes two significant, breaking API redesign:
1. Auth tokens move from session-level (shared by all operations) to
per-operation level (per `UploadCommit`, `FileDownloadGroup`, and
`DownloadStreamGroup`). This enables uploads and downloads from the same
session to carry different access-level tokens — a sensible design for
HF's write-vs-read token split.
2. Instead of letting users provide a callback to refresh tokens, this
new API now let users provide a token refresh URL and access credential
in an HTTP header map.
### Why
1. CAS JWT have short life, but `XetSession` is intended to be held long
time -- thus it makes more sense to configure CAS auth on the operation
level (`UploadCommit` or `FileDownloadGroup` or `DownloadStreamGroup`)
and it will be discarded once the operation is done.
2. For different access level (write vs. read) and different operation
target (repo and commit), CAS JWT token will be different and the token
refresh URL will be different. `UploadCommit` and `FileDownloadGroup`
and `DownloadStreamGroup` they each also function as a single auth
group.
3. Providing an URL is considered easier than writing a callback, and is
more safe when crossing the GIL Python - Rust boundary.
Examples:
```
// Upload token (write access)
let mut upload_headers = HeaderMap::new();
upload_headers.insert("Authorization", "Bearer hub-write-token".parse().unwrap());
let commit = session
.new_upload_commit()?
.with_token_info("CAS_WRITE_JWT", 900)
.with_token_refresh_url("https://huggingface.co/api/repos/token/write", upload_headers)
.build_blocking()?;
```
```
// File download token (read access)
let mut dl_headers = HeaderMap::new();
dl_headers.insert("Authorization", "Bearer hub-read-token".parse().unwrap());
let group = session
.new_file_download_group()?
.with_token_info("CAS_READ_JWT", 900)
.with_token_refresh_url("https://huggingface.co/api/repos/token/read", dl_headers)
.build_blocking()?;
```
Secondary changes include:
- `DirectRefreshRouteTokenRefresher` consolidated into
`xet_client::cas_client::auth`.
- HTTP client module moved from `cas_client` to `xet_client::common` for
shared use between `xet_client::cas_client` and
`xet_client::hub_client`.
- New `DownloadStreamGroup` type (streaming downloads moved off
`XetSession`).
- Fix Session ID type regression: this was fixed once in
https://github.com/huggingface/xet-core/pull/738 but regressed again,
seems AI agents don't learn.
- HTTP client cache key now incorporates custom headers