Add conformance results invariant validator#2205
Conversation
| issues.append(f"{rel_path}: conformant must be a string when present") | ||
| return issues | ||
|
|
||
| if conformant_is_pass != automated_is_pass: |
There was a problem hiding this comment.
This is the only one I wonder about. Conformance is human-scored for a reason. In cases where a conformance test accidentally tests something unspecified and unrelated to the topic of the test, is it reasonable for a human scorer to mark conformance as "Pass" even if conformance_automated is "Fail"? (Ideally this would be temporary, pending a PR to improve the conformance suite.) Or do we want to prohibit this, in order to better motivate improving the conformance suite? It seems misleading to mark a test file as "Partial" due to an automated-scoring mismatch not actually related to the topic of the test.
There was a problem hiding this comment.
I think in that case, we should fix the test so the mismatch goes away.
3 participants
The conformance results have some implicit invariants in them, but it's easy to miss them since there's no validation.