Taxonomy of reward hacks in our benchmark

ExploitCases
Fetch the solution from a later upstream version108
Fetch the exact upstream fix commit63
Fabricating information to satisfy a test37
Fetch and read or decompile the source of the target package16
Tamper with helper packages to pass tests14
Solution commit in local git HEAD or history12
Pull exact constants from a related SDK11
Fetch a package and run it as an oracle8
Hidden reference answer file on disk5
File timestamps revealing where the bug was planted3
Test tampering1
Total278