b43c0aec0e
This PR moves the XetRuntime model away from using thread-local statics
and decouples the XetConfig and XetCommon structs from a single runtime.
It introduces a struct XetContext that gives the runtime context for
operations:
```
struct XetContext {
pub runtime : Arc<XetRuntime>, // The current tokio runtime wrapper, minus the config and common objects..
pub common : Arc<XetCommon>, // The common cache objects, semaphores, rate trackers, etc.
pub config : Arc<XetConfig> // The config
}
```
Now, instead of using functions like `xet_runtime()` and `xet_config()` that examine the thread-local storage, we now explicitly passing through a XetContext instance from the session creation that gets stored in each major processing struct.
This allows decoupling between the runtime, config, and common caches, especially:
- Running multiple config settings and/or endpoints within the same pre-existing tokio runtime.
- Running multiple runtimes that share the same XetCommon object.
Git-Xet is a Git LFS custom transfer agent that implements upload and download of files using the Xet protocol. Install git-xet, follow your regular workflow to git lfs track ... & git add ... & git commit ... & git push, and your files are uploaded to Hugging Face repos using the Xet protocol. Enjoy the dedupe!
Installation
Prerequisite
Make sure you have git and git-lfs installed and configured correctly.
macOS or Linux (amd64 or aarch64)
To install using Homebrew:
brew install git-xet
git xet install
Or, using an installation script, run the following in your terminal (requires curl and unzip):
curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/huggingface/xet-core/refs/heads/main/git_xet/install.sh | sh
To verify the installation, run:
git xet --version
Windows (amd64)
Using winget:
winget install git-xet
Using an installer:
- Download
git-xet-windows-installer-x86_64.zip(available here) and unzip. - Run the
msiinstaller file and follow the prompts.
Manual installation:
- Download
git-xet-windows-x86_64.zip(available here) and unzip. - Place the extracted
git-xet.exeunder aPATHdirectory. - Run
git-xet installin a terminal.
To verify the installation, run:
git xet --version
Uninstall
macOS or Linux
Using Homebrew:
git xet uninstall
brew uninstall git-xet
If you used the installation script (for MacOS or Linux), run the following in your terminal:
git xet uninstall
sudo rm $(which git-xet)
Windows
If you used winget:
winget uninstall git-xet
If you used the installer:
- Navigate to Settings -> Apps -> Installed apps
- Find "Git-Xet".
- Select the "Uninstall" option available in the context menu.
If you manually installed:
- Run
git xet uninstallin a terminal. - Delete the
git-xet.exefile from the location where it was originally placed.
How It Works
Git-Xet works by registering itself as a custom transfer agent to Git LFS by name "xet". On git push, git fetch or git pull, git-lfs negotiates with the remote server to determine the transfer agent to use. During this process, git-lfs sends to the server all locally registered agent names in the Batch API request, and the server replies with exactly one agent name in the response. Should "xet" be picked, git-lfs delegates the uploading or downloading operation to git-xet through a sequential protocol.
For more details, see the Git LFS Batch API and Custom Transfer Agent documentation.