feat: conformance reports persist

[LesnyRumcajs]

Summary of changes

Changes introduced in this pull request:

• this workflow along with the scripts will generate a markdown document from the existing report and put it into new category in docs. Done monthly.

Reference issue to close (if applicable)

Closes #6283

Other information and links

Change checklist

• I have performed a self-review of my own code,
• I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
• I have added tests that prove my fix is effective or that my feature works (if possible),
• I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Sample

Summary by CodeRabbit

• Chores

  • Added a scheduled (monthly) and manually-triggerable automated RPC parity conformance report that runs and opens an automated PR with results.

• Documentation

  • Introduced a "Reports" section and an "API Conformance Reports" subsection to host generated conformance reports.
  • Generates dated Markdown-formatted API conformance reports for publication.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Adds a scheduled GitHub Actions workflow plus helper scripts to extract RPC conformance JSON from Docker volumes, convert it to Markdown under docs, commit a signed change, and open a PR; also adds documentation category files for persisted reports.

Changes

Cohort / File(s)	Summary
CI/CD workflow /.github/workflows/rpc-parity-report.yml	New monthly/manual GitHub Actions workflow that runs RPC parity tests (calibnet), relocates Docker volumes, builds/overrides image, dumps logs, runs mise to produce a conformance report, signs a commit and opens a PR.
Documentation categories docs/docs/users/reports/_category_.json, docs/docs/users/reports/api_conformance/_category_.json	Add "Reports" and "API Conformance Reports" category metadata files to host generated report pages.
Report orchestration mise-tasks/add_conformance_report.sh	New mise task that invokes extraction and conversion scripts and writes a dated Markdown report to docs.
Report extraction & conversion scripts/tests/api_compare/extract_conformance_report.sh, scripts/tests/api_compare/convert_report_to_markdown.sh	New scripts: one extracts a JSON conformance report from a Docker volume via a temporary container; the other converts the JSON into a dated Markdown table of method conformance.

Sequence Diagram(s)

sequenceDiagram
  participant GH as GitHub Actions
  participant Docker as Docker Engine
  participant TempCtr as Temp Container
  participant Volume as Docker Volume
  participant Mise as mise action
  participant Scripts as Repo Scripts
  participant GHAPI as GitHub API

  GH->>Docker: start job, mount volumes (larger mount)
  GH->>Docker: build/run test image (image ref)
  Docker->>Volume: write API parity results (/data/api-compare-report)
  GH->>TempCtr: run extract_conformance_report.sh (mount volume read-only)
  TempCtr->>Volume: find/copy report.json
  GH->>Scripts: run convert_report_to_markdown.sh -> docs/.../report_<DATE>.md
  GH->>Mise: run mise action to generate artifacts
  GH->>GHAPI: commit signed changes and open PR with report

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• chore: remove unused rpc repeat workflow #6200 — Related GitHub Actions changes for RPC parity testing and Dockerfile/image override usage.

Suggested reviewers

• hanabi1224
• akaladarshi

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: conformance reports persist' accurately reflects the main change: adding functionality to persist conformance reports through a new workflow and supporting scripts.
Linked Issues check	✅ Passed	The PR implements the primary requirement from issue #6283: persisting RPC conformance test results and automatically opening a PR with the report to the Forest docs.
Out of Scope Changes check	✅ Passed	All changes are directly related to persisting conformance reports: workflow setup, documentation categories, and scripts to extract and convert reports to Markdown format.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Fix all issues with AI agents

In @.github/workflows/rpc-parity-report.yml:
- Around line 2-5: Add an inputs block to the workflow_dispatch trigger so
github.event.inputs.image is defined when the workflow is manually triggered;
update the workflow_dispatch section (symbol: workflow_dispatch) to include an
input named "image" (symbol: github.event.inputs.image) with a description and
an appropriate required/default value so downstream steps can reliably read it.
- Around line 41-42: Add the missing mise task registration for the existing
script by adding a [tasks.add_conformance_report] entry to mise.toml; set
description to "Extracts conformance report from the API compare compose volume
and adds it to the docs." and run to "bash
./mise-tasks/add_conformance_report.sh" so the workflow step that calls `mise
add_conformance_report` succeeds (refer to task name add_conformance_report and
script mise-tasks/add_conformance_report.sh).

In `@scripts/tests/api_compare/extract_conformance_report.sh`:
- Around line 21-22: The inline comment is wrong: update the comment above the
REPORT_PATH_IN_VOLUME assignment to accurately describe what the script searches
for (a JSON report under /data/api-compare-report, not an .md file). Locate the
line that sets REPORT_PATH_IN_VOLUME (uses docker exec with TEMP_CONTAINER_NAME
and find '*.json') and change the comment to mention
`/data/api-compare-report/*.json` so the comment matches the code behavior.

🧹 Nitpick comments (3)

scripts/tests/api_compare/extract_conformance_report.sh (1)

8-8: Missing argument validation.

The script accesses $1 without checking if an argument was provided. With set -u, this will produce an unhelpful error message if the script is called without arguments.

Proposed fix

+if [[ $# -lt 1 ]]; then
+    echo "Usage: $0 <output_path>"
+    exit 1
+fi
+
 OUTPUT_PATH="$1"

scripts/tests/api_compare/convert_report_to_markdown.sh (1)

8-9: Missing argument validation.

The script accesses $1 and $2 without checking if both arguments were provided. With set -u, this will fail with an unclear error if arguments are missing.

Proposed fix

+if [[ $# -lt 2 ]]; then
+    echo "Usage: $0 <input_json> <output_markdown>"
+    exit 1
+fi
+
 INPUT_FILE="$1"
 OUTPUT_FILE="$2"

mise-tasks/add_conformance_report.sh (1)

1-7: Add strict mode and cleanup temp directory.

The script lacks set -euo pipefail, which could cause silent failures if any command fails. Additionally, the temp directory created on line 4 is never cleaned up.

Proposed fix

 #!/bin/bash
 `#MISE` description="Extracts conformance report from the API compare compose volume and adds it to the docs."

+set -euo pipefail
+
-TMP_REPORT=$(mktemp -d)/report.json
+TMP_DIR=$(mktemp -d)
+trap 'rm -rf "$TMP_DIR"' EXIT
+TMP_REPORT="$TMP_DIR/report.json"
 DATE=$(date +%Y-%m-%d)
 ./scripts/tests/api_compare/extract_conformance_report.sh "$TMP_REPORT"
 ./scripts/tests/api_compare/convert_report_to_markdown.sh "$TMP_REPORT" ./docs/docs/users/reports/api_conformance/report_"$DATE".md

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.76%. Comparing base (de93062) to head (19aa2a3).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

see 8 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de93062...19aa2a3. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.