Example: air traffic controller
Evaluating Large Language Models Trained on Code
samples from the models, and check if any of them pass the unit tests. With just a single sample, a 12B parameter Codex solves 28.8% of these problems, and a 300M parameter Codex solves 13.2% of these problems. In contrast, the 6B parameter GPT-J (Wang & Komatsuzaki,2021) achieves 11.4% on the same dataset, while all GPT models achieve near 0%.
Download Evaluating Large Language Models Trained on Code
Information
Domain:
Source:
Link to this page: