Files
xet-core/.github/workflows/release.yml
Adrien 21aab77be4 ci(release): publish PEP 740 attestations to PyPI for hf-xet (#848)
## Summary

Switches the `release-pypi` job from `PyO3/maturin-action`'s upload
command to `pypa/gh-action-pypi-publish` with `attestations: true`, so
that every `hf-xet` artifact on PyPI ships with **PEP 740 / Sigstore
provenance**.

Today, `hf-xet 1.5.0` on PyPI shows `attestations: None` and
`https://pypi.org/integrity/hf-xet/1.5.0/<wheel>/provenance` returns
404. The existing `actions/attest-build-provenance` step does the work,
but its output never reaches PyPI: `gh attestation verify` works for
wheels people downloaded from GitHub releases, but `pip download
--verify-attestations` (and the equivalent uv / SLSA verifier flows)
have nothing to chain to.

## What changes

In `.github/workflows/release.yml`, `release-pypi` job:

- Add a step that collects every `wheels-*/<file>` artifact into a
single `dist/` directory. Debug-symbol artifacts (`dbg-*`) are
explicitly excluded from this directory and therefore from the upload.
- Point `actions/attest-build-provenance` at `dist/*` so the GitHub
attestation covers exactly the same bytes that get uploaded.
- Replace `PyO3/maturin-action@... command: upload` with
`pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b`
(v1.14.0, SHA-pinned) and `attestations: true`, `skip-existing: true`,
`print-hash: true`.

## Why two attestation steps?

They are complementary, not redundant:

| Step | Verifier | Where it lives |
|---|---|---|
| `actions/attest-build-provenance` | `gh attestation verify`,
sigstore-go | github.com attestations endpoint |
| `pypa/gh-action-pypi-publish` with `attestations: true` | `pip
download --verify-attestations`, uv, slsa-verifier | PyPI
`/integrity/.../provenance` |

The first one verifies wheels distributed anywhere; the second one is
what PyPI clients actually look at.

## Verification after merge

On the first release that runs this workflow:

```bash
$ pip download --no-deps hf-xet==<new-version>
$ python -c "from pypi_attestations import Distribution; print(Distribution.from_file('hf_xet-*.whl').identity)"
# should print the workflow / environment / repo identity
```

And on PyPI's web UI: each wheel should show a "Provenance" link instead
of "no attestations".

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Changes the release publishing path and artifact selection for PyPI
uploads; mistakes could block releases or upload the wrong artifacts
despite being confined to CI.
> 
> **Overview**
> Switches the `release-pypi` job to **collect all wheel/sdist artifacts
into a single `dist/` directory** (excluding `dbg-*`) and generate the
GitHub build provenance attestation over exactly those files.
> 
> Replaces the PyPI upload step from `PyO3/maturin-action` to
`pypa/gh-action-pypi-publish` using **Trusted Publishing + PEP 740
attestations** (`attestations: true`, `skip-existing: true`,
`print-hash: true`), and clarifies required workflow permissions.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
4f0c7f8db9. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
2026-06-02 08:20:34 +02:00

434 lines
16 KiB
YAML

# This file is autogenerated by maturin v1.7.0
# To update, run
#
# maturin generate-ci github
#
name: Release
run-name: hf-xet release, tag ${{ inputs.tag || 'main' }}
on:
push:
branches:
- main
workflow_dispatch:
inputs:
tag:
description: 'Semantic version for PyPI release (tag will share the same name)'
required: true
default: 'v0.1.0'
permissions:
contents: read
jobs:
linux:
runs-on: ${{ matrix.platform.runner }}
strategy:
matrix:
platform:
- runner: ubuntu-22.04
target: x86_64
manylinux: auto
- runner: ubuntu-22.04
target: aarch64
manylinux: manylinux_2_28
python-version:
- 3.14
- 3.13t
- 3.14t
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install wheel tooling
run: |
python -m pip install --upgrade pip
pip install wheel
- name: Update version in toml
env:
TAG: ${{ github.event.inputs.tag }}
run: |
VERSION=${TAG#v}
if [ -n "$VERSION" ]; then
sed -i '/^version /s/=.*$/= "'"$VERSION"'"/' hf_xet/Cargo.toml
fi
- uses: ./.github/actions/set-build-profile
with:
tag: ${{ github.event.inputs.tag }}
- name: Build wheels
uses: PyO3/maturin-action@04ac600d27cdf7a9a280dadf7147097c42b757ad # v1
with:
target: ${{ matrix.platform.target }}
args: --profile ${{ env.BUILD_PROFILE }} -i ${{ matrix.python-version }} --out dist
sccache: 'true'
manylinux: ${{ matrix.platform.manylinux }}
working-directory: hf_xet
before-script-linux: |
if command -v apt-get &> /dev/null; then
apt-get update && apt-get install libatomic-ops-dev -y
elif command -v yum &> /dev/null; then
yum install devtoolset-10-libatomic-devel perl-IPC-Cmd -y
else
echo "Neither apt-get nor yum is installed. Please install a package manager."
exit 1
fi
git config --global --add safe.directory "*"
- name: Strip debug symbols into a separate file
if: env.IS_RELEASE == 'true'
run: |
shopt -s expand_aliases
if [[ ${{ matrix.platform.target }} == "aarch64" ]]
then
sudo apt-get update
sudo apt-get install -y binutils-aarch64-linux-gnu
alias objcopy=aarch64-linux-gnu-objcopy
fi
mkdir hf_xet/dbg
mkdir dist
pushd dist
cp ../hf_xet/dist/* .
LATEST_WHEEL=$(ls -tr1 *.whl | tail -n 1)
WHEEL_NAME=$(basename $LATEST_WHEEL .whl)
WHEEL_VERSION=$(echo $LATEST_WHEEL | cut -d '-' -f 2)
SYMBOL_FILE=${WHEEL_NAME}.so.dbg
wheel unpack $LATEST_WHEEL
rm *.whl
objcopy --only-keep-debug hf_xet-${WHEEL_VERSION}/hf_xet/hf_xet.*.so hf_xet-${WHEEL_VERSION}/hf_xet/${SYMBOL_FILE}
objcopy --strip-debug hf_xet-${WHEEL_VERSION}/hf_xet/hf_xet.*.so
objcopy --add-gnu-debuglink=hf_xet-${WHEEL_VERSION}/hf_xet/${SYMBOL_FILE} hf_xet-${WHEEL_VERSION}/hf_xet/hf_xet.*.so
mv hf_xet-${WHEEL_VERSION}/hf_xet/${SYMBOL_FILE} ../hf_xet/dbg/${SYMBOL_FILE}
wheel pack hf_xet-${WHEEL_VERSION}
rm -rf hf_xet-${WHEEL_VERSION}
- name: Copy unstripped wheels to upload directory
if: env.IS_RELEASE != 'true'
run: |
mkdir dist
cp hf_xet/dist/* dist/
- name: Upload debug symbols
if: env.IS_RELEASE == 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: dbg-linux-${{ matrix.python-version }}-${{ matrix.platform.target }}
path: hf_xet/dbg
- name: Upload wheels
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: wheels-linux-${{ matrix.python-version }}-${{ matrix.platform.target }}
path: dist
musllinux:
runs-on: ${{ matrix.platform.runner }}
strategy:
matrix:
platform:
- runner: ubuntu-22.04
target: x86_64
- runner: ubuntu-22.04
target: aarch64
python-version:
- 3.14
- 3.13t
- 3.14t
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install wheel tooling
run: |
python -m pip install --upgrade pip
pip install wheel
- name: Update version in toml
env:
TAG: ${{ github.event.inputs.tag }}
run: |
VERSION=${TAG#v}
if [ -n "$VERSION" ]; then
sed -i '/^version /s/=.*$/= "'"$VERSION"'"/' hf_xet/Cargo.toml
fi
- uses: ./.github/actions/set-build-profile
with:
tag: ${{ github.event.inputs.tag }}
- name: Build wheels
uses: PyO3/maturin-action@04ac600d27cdf7a9a280dadf7147097c42b757ad # v1
with:
target: ${{ matrix.platform.target }}
args: --profile ${{ env.BUILD_PROFILE }} -i ${{ matrix.python-version }} --out dist
sccache: 'true'
manylinux: musllinux_1_2
working-directory: hf_xet
before-script-linux: |
git config --global --add safe.directory "*"
- name: Strip debug symbols into a separate file
if: env.IS_RELEASE == 'true'
run: |
shopt -s expand_aliases
if [[ ${{ matrix.platform.target }} == "aarch64" ]]
then
sudo apt-get update
sudo apt-get install -y binutils-aarch64-linux-gnu
alias objcopy=aarch64-linux-gnu-objcopy
fi
mkdir hf_xet/dbg
mkdir dist
pushd dist
cp ../hf_xet/dist/* .
LATEST_WHEEL=$(ls -tr1 *.whl | tail -n 1)
WHEEL_NAME=$(basename $LATEST_WHEEL .whl)
WHEEL_VERSION=$(echo $LATEST_WHEEL | cut -d '-' -f 2)
SYMBOL_FILE=${WHEEL_NAME}.so.dbg
wheel unpack $LATEST_WHEEL
rm *.whl
objcopy --only-keep-debug hf_xet-${WHEEL_VERSION}/hf_xet/hf_xet.*.so hf_xet-${WHEEL_VERSION}/hf_xet/${SYMBOL_FILE}
objcopy --strip-debug hf_xet-${WHEEL_VERSION}/hf_xet/hf_xet.*.so
objcopy --add-gnu-debuglink=hf_xet-${WHEEL_VERSION}/hf_xet/${SYMBOL_FILE} hf_xet-${WHEEL_VERSION}/hf_xet/hf_xet.*.so
mv hf_xet-${WHEEL_VERSION}/hf_xet/${SYMBOL_FILE} ../hf_xet/dbg/${SYMBOL_FILE}
wheel pack hf_xet-${WHEEL_VERSION}
rm -rf hf_xet-${WHEEL_VERSION}
- name: Copy unstripped wheels to upload directory
if: env.IS_RELEASE != 'true'
run: |
mkdir dist
cp hf_xet/dist/* dist/
- name: Upload debug symbols
if: env.IS_RELEASE == 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: dbg-musllinux-${{ matrix.python-version }}-${{ matrix.platform.target }}
path: hf_xet/dbg
- name: Upload wheels with separated debug symbols
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: wheels-musllinux-${{ matrix.python-version }}-${{ matrix.platform.target }}
path: dist
windows:
runs-on: ${{ matrix.platform.runner }}
strategy:
matrix:
platform:
- runner: windows-latest
target: x64
rust_target: x86_64-pc-windows-msvc
- runner: windows-11-arm
target: aarch64
rust_target: aarch64-pc-windows-msvc
python-version:
- 3.14
- 3.13t
- 3.14t
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Update version in toml
shell: bash
env:
TAG: ${{ github.event.inputs.tag }}
run: |
VERSION=${TAG#v}
if [ -n "$VERSION" ]; then
sed -i '/^version /s/=.*$/= "'"$VERSION"'"/' hf_xet/Cargo.toml
fi
- name: Build wheels
uses: PyO3/maturin-action@04ac600d27cdf7a9a280dadf7147097c42b757ad # v1
with:
target: ${{ matrix.platform.target }}
args: --release -i python${{ matrix.python-version }} --out dist
sccache: 'true'
working-directory: hf_xet
- name: Copy debug symbols
shell: bash
run: |
pushd hf_xet/dist
LATEST_WHEEL=$(ls -tr1 *.whl | tail -n 1)
WHEEL_NAME=$(basename $LATEST_WHEEL .whl)
SYMBOL_FILE=${WHEEL_NAME}.pdb
popd
mkdir hf_xet/dbg
cp hf_xet/target/${{ matrix.platform.rust_target }}/release/hf_xet.pdb hf_xet/dbg/${SYMBOL_FILE}
- name: Upload debug symbols
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: dbg-windows-${{ matrix.python-version }}-${{ matrix.platform.target }}
path: hf_xet/dbg
- name: Upload wheels
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: wheels-windows-${{ matrix.python-version }}-${{ matrix.platform.target }}
path: hf_xet/dist
macos:
runs-on: ${{ matrix.platform.runner }}
strategy:
matrix:
platform:
- runner: macos-15-intel
target: x86_64
rust_target: x86_64-apple-darwin
- runner: macos-14
target: aarch64
rust_target: aarch64-apple-darwin
python-version:
- 3.14
- 3.13t
- 3.14t
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install wheel tooling
run: |
python -m pip install --upgrade pip
pip install wheel
- name: Update version in toml
env:
TAG: ${{ github.event.inputs.tag }}
run: |
VERSION=${TAG#v}
if [ -n "$VERSION" ]; then
sed -i '' '/^version /s/=.*$/= "'"$VERSION"'"/' hf_xet/Cargo.toml
fi
- uses: ./.github/actions/set-build-profile
with:
tag: ${{ github.event.inputs.tag }}
- name: Build wheels
uses: PyO3/maturin-action@04ac600d27cdf7a9a280dadf7147097c42b757ad # v1
with:
target: ${{ matrix.platform.target }}
args: --profile ${{ env.BUILD_PROFILE }} -i ${{ matrix.python-version }} --strip --out dist
sccache: 'true'
working-directory: hf_xet
- name: Retarget and copy debug symbols
if: env.IS_RELEASE == 'true'
run: |
pushd hf_xet/dist
LATEST_WHEEL=$(ls -tr1 *.whl | tail -n 1)
WHEEL_NAME=$(basename $LATEST_WHEEL .whl)
WHEEL_VERSION=$(echo $LATEST_WHEEL | cut -d '-' -f 2)
SYMBOL_FILE=${WHEEL_NAME}.dylib.dSYM
wheel unpack $LATEST_WHEEL
LIB_NAME=$(basename hf_xet-${WHEEL_VERSION}/hf_xet/*.so)
rm -rf hf_xet-${WHEEL_VERSION}
popd
mkdir hf_xet/dbg
pushd hf_xet/target/${{ matrix.platform.rust_target}}/release-dbgsymbols
sed -i '' "s/binary-path:.*/binary-path: '.\/${LIB_NAME}'/" libhf_xet.dylib.dSYM/Contents/Resources/Relocations/${{ matrix.platform.target }}/libhf_xet.dylib.yml
cp -r libhf_xet.dylib.dSYM ../../../dbg/${SYMBOL_FILE}
- name: Upload debug symbols
if: env.IS_RELEASE == 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: dbg-macos-${{ matrix.python-version }}-${{ matrix.platform.target }}
path: hf_xet/dbg
- name: Upload wheels
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: wheels-macos-${{ matrix.python-version }}-${{ matrix.platform.target }}
path: hf_xet/dist
sdist:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Update version in toml
env:
TAG: ${{ github.event.inputs.tag }}
run: |
VERSION=${TAG#v}
if [ -n "$VERSION" ]; then
sed -i '/^version /s/=.*$/= "'"$VERSION"'"/' hf_xet/Cargo.toml
fi
- name: Build sdist
uses: PyO3/maturin-action@04ac600d27cdf7a9a280dadf7147097c42b757ad # v1
with:
command: sdist
args: --out dist
working-directory: hf_xet
- name: Patch sdist
run: |
pushd hf_xet/dist
DISTFILE=$(ls -tr1 hf_xet-*.tar.gz | tail -n 1)
DISTNAME=${DISTFILE%.tar.gz}
# maturin sdist will overwrite an edited pyproject.toml in the repo root, so we need to
# update the python path in the built pyproject.toml to account for that.
tar -xvzf ${DISTFILE}
sed -i '/^python\-source =/ s/= .*/= "hf_xet\/python"/' ${DISTNAME}/pyproject.toml
tar -cvzf ${DISTFILE} ${DISTNAME}
rm -rf ${DISTNAME}
- name: Upload sdist
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: wheels-sdist
path: hf_xet/dist
release-pypi:
name: Release PyPi
runs-on: ubuntu-latest
if: ${{ github.event_name == 'workflow_dispatch' }}
needs: [github-release]
permissions:
# Used by both attest-build-provenance and the PyPI Trusted Publisher (OIDC).
id-token: write
# Used to upload release artifacts and to generate artifact attestations.
contents: write
attestations: write
environment: release
steps:
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
- name: Collect wheels and sdist into dist/
run: |
set -euo pipefail
mkdir -p dist
# Each `wheels-*` artifact directory holds one or more wheels/sdists.
# Debug-symbol artifacts (`dbg-*`) must NOT be uploaded to PyPI.
cp wheels-*/* dist/
ls -l dist
- name: Generate GitHub build provenance attestation
uses: actions/attest-build-provenance@977bb373ede98d70efdf65b84cb5f73e068dcc2a # v3
with:
subject-path: 'dist/*'
- name: Publish to PyPI (Trusted Publishing + PEP 740 attestations)
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # v1.14.0
with:
attestations: true
skip-existing: true
print-hash: true
github-release:
name: Create GitHub release
runs-on: ubuntu-latest
permissions:
contents: write
if: ${{ github.event_name == 'workflow_dispatch' }}
needs: [ linux, musllinux, windows, macos, sdist ]
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
- name: Create GitHub Release
env:
GH_TOKEN: ${{ github.token }}
RELEASE_TAG: ${{ github.event.inputs.tag }}
run: |
ls -l
echo "wheels"
ls -l wheels-*/*
mkdir dbg-symbols
echo "dbg"
ls -l dbg-*/*
cp -r dbg-*/* dbg-symbols/
echo "zipping debug symbols"
zip -r dbg-symbols.zip dbg-symbols
gh release create "$RELEASE_TAG" wheels-*/* dbg-symbols.zip --generate-notes --prerelease --target ${{github.sha}}