Example: air traffic controller
Evaluating Large Language Models Trained on Code

Evaluating Large Language Models Trained on Code

Back to document page

samples from the models, and check if any of them pass the unit tests. With just a single sample, a 12B parameter Codex solves 28.8% of these problems, and a 300M parameter Codex solves 13.2% of these problems. In contrast, the 6B parameter GPT-J (Wang & Komatsuzaki,2021) achieves 11.4% on the same dataset, while all GPT models achieve near 0%.

  Model

Download Evaluating Large Language Models Trained on Code


Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Advertisement

Related search queries