Optional EvalPort export for task datasets and pass@k results

[adhabnr-ux]

Hi BigCode team — I maintain EvalPort, an open, framework-agnostic JSON spec for portable LLM eval datasets and results (a TestCase/Suite/ResultSet schema with a validator, so a dataset or a graded run can move between tools without hand-writing a converter each time). Filing this as an issue first per CONTRIBUTING.md before writing any code.

I read the real Task interface rather than guessing, e.g. bigcode_eval/tasks/humaneval.py:

def get_prompt(self, doc):
    return doc["prompt"] if self.strip_prompt else doc["prompt"]

def get_reference(self, doc):
    test_func = doc["test"]
    entry_point = f"check({doc['entry_point']})"
    return "\n" + test_func + "\n" + entry_point

def process_results(self, generations, references):
    results, _ = compute_code_eval(references=references, predictions=generations, k=self.k, ...)
    return results

and evalport-sdk's real dataclasses (installed and inspected, not from docs):

@dataclass
class TestCase:
    id: str
    input: Union[str, List[str]]
    graders: List[Union[str, Grader]]
    expected_output: Optional[str] = None
    metadata: Dict[str, Any] = field(default_factory=dict)
    ...

@dataclass
class EvalSuite:
    version: str
    id: str
    test_cases: List[TestCase]
    ...

The mapping is fairly direct for any Task subclass here: get_prompt(doc) → TestCase.input, get_reference(doc) → TestCase.expected_output, and the task's execute grader (e.g. HumanEval's pass@k via compute_code_eval) → an EvalPort Grader of type code_execution. On the results side, process_results()'s pass@k dict maps onto EvalPort's ResultSet, which already has a defined shape for per-case pass/fail plus aggregate metrics.

I know several of the individual datasets here (HumanEval, MBPP, GSM8K) already exist as standalone EvalPort benchmark conversions in benchmarks/ — this issue isn't about those. What's specific to this repo is the harness layer itself: running a model through many Tasks and getting back real execution-graded results, which is exactly the "graded run" half of EvalPort that a static dataset export doesn't capture.

Two ways I could see this landing, no strong preference:

1. A standalone adapter script/package in the EvalPort repo (bigcode-eval-openeval-adapter) that wraps main.py's output into an EvalPort ResultSet, with a small Task→TestCase exporter for the dataset side. Zero footprint on this repo.
2. A small optional export flag inside this repo's main.py if maintainers would rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock. Let me know if this is worth pursuing, or if it's not a priority right now — no worries either way.