Skip to content

ci: add benchmarking via conbench#415

Open
kentkwu wants to merge 1 commit intoapache:mainfrom
kentkwu:gh-203
Open

ci: add benchmarking via conbench#415
kentkwu wants to merge 1 commit intoapache:mainfrom
kentkwu:gh-203

Conversation

@kentkwu
Copy link
Contributor

@kentkwu kentkwu commented Mar 17, 2026

What's Changed

This PR adds a github action workflow to run benchmarks and upload them to conbench.

Note: Results are only submitted to conbench on push to main. This requires CONBENCH_EMAIL and CONBENCH_PASS to be configured as GitHub Actions secrets

Test Plan

Ran against a locally running conbench container

bash ci/scripts/bench.sh $(pwd) --json

# Transform verification (without submitting):
python3.11 - <<'EOF'
import sys
sys.path.insert(0, '.')
from ci.scripts.bench_adapt import JSAdapter
adapter = JSAdapter(result_fields_override={"info": {}})
results = adapter._transform_results()
print(f"{len(results)} results parsed")
print("First result tags:", results[0].tags)
print("First result stats:", results[0].stats)
EOF

# Local conbench (Docker):
git clone https://github.com/conbench/conbench
cd conbench
# create buildinfo.json (required by the app)
echo '{"commit":"0000000000000000000000000000000000000000","branch_name":"main","build_time_rfc3339":"2026-03-10T00:00:00Z","build_hostname":"localhost","version_string"
:"local"}' > buildinfo.json
docker compose up --build -d
docker compose port app 5000  # get the port
# register at http://127.0.0.1:<port>/register?key=innocent-registration-key

# Full end-to-end submission:
CONBENCH_URL=http://127.0.0.1:<port> \
CONBENCH_EMAIL=<email> \
CONBENCH_PASSWORD=<password> \
CONBENCH_REF=main \
GITHUB_SERVER_URL=https://github.com \
GITHUB_REPOSITORY=apache/arrow-js \
GITHUB_SHA=<commit-sha> \
python3.11 ci/scripts/bench_adapt.py

Viewed runs in conbench UI
Screenshot 2026-03-16 at 8 24 25 PM
Screenshot 2026-03-16 at 8 24 41 PM

Closes #203

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: Add benchmarking CI

1 participant