Eval: evaluate prompt builder configuration against real models in CI.

[polina-c]

• Configure stub key on github
• Get key - go/eval-for-open-source
• Create smoke test
• Evaluate prompt builder
• Document important details in go/flutter-genui-internal

[polina-c]

'Normal' secrets are not visible for workflows. We need environment secrets.

I need admin permissions to set them up.

@jacobsimionato or @yjbanov can you please give me permissions?

[yjbanov]

I'm wondering if we should be evaling at the A2UI level rather than at the GenUI SDK level. The prompt builder is a deterministic piece of code that can be tested using normal unit tests. The structure of the system instruction that the builder produces, and which affects the eval results the most, is dictated by the A2UI spec. This means that most of the work done to reach high eval scores would happen in A2UI, and so the eval testing should take place there. Or is my logic flawed?

[polina-c]

Thanks for kicking off discussion.

GenUI SDK participates in prompt building and we want confidence that our prompts produce expected results and our users do not struggle trying to understand if things work as expected and, if not, when they stopped working.

Ideally, we want two things:

1. Evals for GenUI SDK.
2. Testing API for users to make it easy for them to eval their apps.

I expect #2 to appear as side product of #1.

Am i missing something?