mirror of
https://github.com/huggingface/xet-core.git
synced 2026-06-04 13:30:29 +08:00
c3c726bed5
Human context: in integrating new hf-hub usage in tokenizers, tokenizers
also generated a napi binary. so we should validate that hf-hub/hf-xet
are napi compat (hf-hub is pretty trivial if given that hf-xet is
compatible).
## Summary
- Adds `examples/xet_pkg_napi/` — a minimal napi-rs binding that links
`hf-xet` (the `xet` crate at `xet_pkg/`) into a Node.js native addon.
- Exposes `initLogging(version)` and `smokeTest()`. The smoke test
builds a `XetSession` synchronously and constructs upload-commit +
file-download-group builders to exercise lazy runtime startup.
- Crate is excluded from the xet-core workspace and carries its own
`[workspace]` table so it stays standalone under git worktrees (where
cargo would otherwise resolve through the canonical repo path).
- Build artifacts (`*.node`, `index.js`, `index.d.ts`, `node_modules/`)
are gitignored; `Cargo.lock` and `package-lock.json` are committed for
reproducibility.
The point of the smoke test is **not** a full JS API — it's to verify
hf-xet compiles, links, and starts inside libuv (no pyo3, no host-owned
tokio). If you can run `npm run smoke` and see `xet session built;
runtime initialized`, the integration is ready for a fuller binding
(async upload/download via `#[napi]` async fns, progress callbacks via
`ThreadsafeFunction`).
## Test plan
- [x] `npm install` (in `examples/xet_pkg_napi/`)
- [x] `npm run build:debug` — compiles `hf-xet`, `xet-runtime`,
`xet-client`, `xet-data`, `xet-core-structures` and the napi shim
against napi 2.16
- [x] `npm run smoke` — outputs:
```
loaded addon, exports: [ 'initLogging', 'smokeTest' ]
smokeTest: xet session built; runtime initialized
```
- [x] Verify on Linux (only tested on darwin-arm64 locally)
- [x] Decide whether to wire into CI, or keep as a manual example
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Adds a standalone example project and build scripts without changing
production crates; primary risk is repo bloat/noise from the committed
lockfiles and an extra exclusion in the workspace.
>
> **Overview**
> Adds a new standalone `examples/xet_pkg_napi` project to smoke-test
that `hf-xet` can compile/link as a `napi-rs` Node native addon and
perform a real file download via the blocking download APIs.
>
> Updates the root `Cargo.toml` to **exclude** this example from the
workspace, and includes the example’s build/run scaffolding
(`package.json`, `smoke.mjs`, `build.rs`) plus committed lockfiles and a
`.gitignore` for generated artifacts.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
cb628956f7. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
23 lines
447 B
JSON
23 lines
447 B
JSON
{
|
|
"name": "xet-pkg-napi",
|
|
"version": "0.0.1",
|
|
"description": "napi-rs smoke test for hf-xet",
|
|
"private": true,
|
|
"main": "index.js",
|
|
"type": "commonjs",
|
|
"napi": {
|
|
"name": "xet-pkg-napi"
|
|
},
|
|
"scripts": {
|
|
"build": "napi build --platform --release",
|
|
"build:debug": "napi build --platform",
|
|
"smoke": "node smoke.mjs"
|
|
},
|
|
"devDependencies": {
|
|
"@napi-rs/cli": "^2.18.4"
|
|
},
|
|
"engines": {
|
|
"node": ">=18"
|
|
}
|
|
}
|