We benchmarked 4 @DSPyOSS adapters, 2 modules, and 5 models on a real structured extraction task. The results were not what we expected.
𝗧𝗵𝗲 𝘁𝗮𝘀𝗸: Extract a list of typed @pydantic objects (decisions and learnings with literal fields, optional values, nested lists) from
4
57
14
4.1K
Snapshot History (19)
| Date | ||||
|---|---|---|---|---|
| Mar 16 · 12:00 | 4 | 57 | 14 | +22 4.1K |
| Mar 16 · 11:00 | 4 | 57 | 14 | +24 4.0K |
| Mar 16 · 10:00 | 4 | 57 | 14 | +82 4.0K |
| Mar 16 · 07:00 | 4 | 57 | 14 | +21 3.9K |
| Mar 16 · 06:00 | 4 | 57 | 14 | +105 3.9K |
| Mar 16 · 03:00 | 4 | 57 | 14 | +37 3.8K |
| Mar 16 · 02:00 | 4 | 57 | 14 | +96 3.8K |
| Mar 15 · 23:00 | 4 | 57 | 14 | +43 3.7K |
| Mar 15 · 22:00 | 4 | +2 57 | 14 | +215 3.6K |
| Mar 15 · 18:00 | 4 | 55 | 14 | +54 3.4K |
| Mar 15 · 17:00 | 4 | 55 | 14 | +141 3.4K |
| Mar 15 · 14:00 | 4 | +1 55 | 14 | +46 3.2K |
| Mar 15 · 13:00 | 4 | 54 | 14 | +157 3.2K |
| Mar 15 · 10:00 | 4 | 54 | 14 | +82 3.0K |
| Mar 15 · 09:00 | 4 | +3 54 | +1 14 | +226 2.9K |
| Mar 15 · 05:00 | 4 | 51 | 13 | +73 2.7K |
| Mar 15 · 04:00 | 4 | 51 | 13 | +281 2.6K |
| Mar 15 · 00:00 | 4 | +1 51 | 13 | +85 2.3K |
| Mar 14 · 23:00 | +4 4 | +50 50 | +13 13 | +2.3K 2.3K |
| Mar 14 · 14:09 | 0 | 0 | 0 | 0 |