Commit ff898bb
Duc Hoang
Fix TruthfulQA generative evaluation bugs
1. Remove KeyError: mc1_targets field only exists in multiple_choice subset,
not generation subset used by truthfulqa:gen task
2. Fix backwards answer processing logic that was replacing correct answers
with periods instead of preserving answer text
These fixes make truthfulqa:gen functional for proper evaluation.
Task format: lighteval|truthfulqa:gen|01 parent 9b2ca83 commit ff898bb
1 file changed
+2
-3
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2478 | 2478 | | |
2479 | 2479 | | |
2480 | 2480 | | |
2481 | | - | |
| 2481 | + | |
2482 | 2482 | | |
2483 | 2483 | | |
2484 | 2484 | | |
2485 | 2485 | | |
2486 | | - | |
| 2486 | + | |
2487 | 2487 | | |
2488 | 2488 | | |
2489 | 2489 | | |
2490 | 2490 | | |
2491 | 2491 | | |
2492 | 2492 | | |
2493 | 2493 | | |
2494 | | - | |
2495 | 2494 | | |
2496 | 2495 | | |
2497 | 2496 | | |
| |||
0 commit comments