refactor: adapt evaluation pipeline to MCP architecture

[ShuxinLin]

Background

The current evaluation pipeline is tightly coupled to the legacy src/ architecture (AgentHive, ReAct+Reflexion, reactxen library). The new mcp/plan_execute/ runner produces OrchestratorResult objects with rich traces but has no path to grading or MLflow assessment. Meanwhile, aobench/scenario-server/ has the right pluggable-grader design but is disconnected from mcp/.

Current state

Layer	Location	Status
Scenario loading	src/assetopsbench/core/scenarios.py	Legacy, no MCP integration
Grading strategies	aobench/scenario-server/grading/graders.py	Good design; depends on reactxen for evaluation_agent strategy
Grading pipeline	aobench/scenario-server/grading/grading.py	Async, MLflow-ready; not wired to mcp/
EvaluationAgent	src/evaluation_agent/agent.py	Uses reactxen WatsonX wrapper (legacy LLM interface)
MLflow tracking	aobench/scenario-server/grading/grading.py	Implemented but unreachable from mcp/
Benchmark runner	benchmark/cods_track{1,2}/run_track_*.py	Hardcoded to AgentHive workflows
MCP runner output	mcp/plan_execute/models.py (OrchestratorResult)	Has plan + history; no grading hookup

What needs to change

1. Evaluation runner for mcp/

Add mcp/evaluation/ (or mcp/benchmark/) that:

• Loads scenarios from HuggingFace (ibm-research/AssetOpsBench) using the existing Scenario model
• Feeds each scenario's text through PlanExecuteRunner.run()
• Collects OrchestratorResult (answer + full step history as trace)
• Passes results to the grading pipeline

2. MLflow tracing in PlanExecuteRunner

Instrument mcp/plan_execute/runner.py (or executor.py) to emit MLflow traces tagged with scenario_id, matching the schema that aobench/scenario-server/grading/grading.py already expects.

3. Decouple EvaluationAgent from reactxen

aobench/scenario-server/grading/graders.py::evaluation_agent() instantiates EvaluationAgent (from reactxen) which hard-codes a WatsonX LLM wrapper. Replace with the LLMBackend abstraction already in mcp/llm/ so it works with any LiteLLM-compatible model.

4. Connect grading strategies to OrchestratorResult

The three graders (exact_string_match, numeric_match, evaluation_agent) expect (actual, expected, ...). Map:

• actual ← OrchestratorResult.answer
• trace ← serialized OrchestratorResult.history
• grading strategy selected by Scenario.deterministic / Scenario.type

5. Update benchmark entry points

Replace (or wrap) benchmark/cods_track{1,2}/run_track_*.py with MCP-based runners that invoke plan-execute and feed results through the new grading pipeline.

Out of scope

• Changes to aobench/ scenario-server REST API
• Changes to the src/ legacy code (keep as-is for reproducibility)
• Deferred grading / PostgreSQL backend (can be added later)

Acceptance criteria

• uv run mcp-evaluate --scenario-type iot runs all IoT scenarios through PlanExecuteRunner and prints per-scenario grades
• Grading results are logged to MLflow (traces tagged with scenario_id, 6-dimensional assessments as feedbacks)
• evaluation_agent grader uses LiteLLMBackend — no reactxen dependency in mcp/
• Unit tests for the new evaluation runner (mocked PlanExecuteRunner + mocked graders)

[DhavalRepo18]

@ShuxinLin

Please ensure "Innovation aspect is well separated from the tooling aspect," as our focus will be on which aspect the community may be interested in, and as long as it's easy. We may be seeking more input on

• How to make the evaluation more accurate?
• How to contribute the let say new way of doing orchestration?
• How to bring an MCP Server into the echo system?
• Also, I do not want to bring any differentiation in our terminology, like "IoT" level, "WO" level.

# this one we do not want
uv run mcp-evaluate --scenario-type iot

# we may plan to have a category at the asset class level or at light/difficult/ etc
uv run mcp-evaluate --bench-mode light or some other

There is potentially an MCP Server for AssetOpsBench to potentially interest with, say - give me all the scenarios for Asset Chiller? This will make the code be maintained to a very minimum.

[DhavalRepo18]

#148 Should also be taken in account a comment from community members

[bradleyjeck]

Suggest bringing together the scenario server and mlfow parts sitting in aobench with the mcp design. Issue 171 is making a demo of how this works.

In the above, suggest 1. Evaluation Runner should get scenarios from scenario server rather than hugging face. That will also provide the id management for using mlflow.

[DhavalRepo18]

@bradleyjeck We are using the HF Dataset as a single source; it helps us track usage. Also, now HFDataset can also be available as an MCP Server, so any natural language interaction will become easy, saying "Give me all the scenarios of X, Y, Z, and they are able to do it.

We have ensured all the IDs on the HF dataset are unique, but moving forward, we will create various modes (asset-wise, task-wise, or complexity-wise to fetch the group of scenarios from HF.

And the code base sitting inside "mlfow parts sitting in aobench with the mcp" will also be pulled over and refactored, mostly simplified (core part, tooling part), as well as a few from ReActXen code.

[ShuxinLin]

@jtrayfield I create the placeholder directory for evaluation code. please start from there.

[bradleyjeck]

The acceptance criteria here are related to PR #186.

Scenario server + client are are very helpful for managing the various IDs mlfow needs, so that runs are easily comparable etc. Scenario server gets scenarios from HF, and uses same IDs. It can be a parallel approach to an MCP server for HFDataset.
The asset-wise, task-wise- complexity-wise modes sound like "scenario sets" already supported, so it's aligning nicely.

Evaluation is the key part -- it would be great if the eval code developed as part of this issue was worked based on the scenario IDs.

[jtrayfield]

I'll look at it when I have time @ShuxinLin