Background
OpenAI is an AI research laboratory founded in December 2015 by Sam Altman, Elon Musk, Greg Brockman, Ilya Sutskever, and others. Originally established as a nonprofit, it transitioned to a capped-profit structure in 2019 to attract the capital needed for large-scale model training. The organization is headquartered in San Francisco.
Key Contributions
OpenAI drove the scaling-laws paradigm for large language models through the gpt series. gpt-1 introduced generative pre-training on unlabeled text followed by discriminative fine-tuning. gpt-2 demonstrated that scaling model size and data produced emergent capabilities, raising concerns about misuse. gpt-3 revealed powerful in-context-learning abilities via few-shot prompting at 175B parameters. gpt-4-technical-report described a multimodal model with substantially improved reasoning.
OpenAI also pioneered rlhf for alignment through instructgpt, showing that reinforcement learning from human feedback could make models more helpful and less harmful. Their scaling-laws-neural-lm paper established empirical power-law relationships between compute, data, model size, and loss.
Notable Publications
- gpt-1 (2018), gpt-2 (2019), gpt-3 (2020), gpt-4-technical-report (2023)
- scaling-laws-neural-lm (2020)
- instructgpt (2022)
Influence
OpenAI’s work catalyzed the modern LLM era. The GPT series set successive capability benchmarks, while InstructGPT and rlhf became the dominant paradigm for aligning language models. Their research directly influenced anthropic, founded by former OpenAI researchers seeking alternative alignment approaches.
Sources
- Improving Language Understanding by Generative Pre-Training (File, URL)
- Language Models are Unsupervised Multitask Learners (File, URL)
- Language Models are Few-Shot Learners (File, DOI)
- GPT-4 Technical Report (File, DOI)
- Scaling Laws for Neural Language Models (File, DOI)
- Training language models to follow instructions with human feedback (File, DOI)