Research
1. RESEARCH
1A. RESEARCH PAPERS
https://arxiv.org/abs/2304.05332 Emergent autonomous scientific research capabilities of large language models
https://arxiv.org/abs/2304.03442 Generative Agents: Interactive Simulacra of Human Behavior
https://arxiv.org/abs/2303.08774 * GPT-4 Technical Report
https://arxiv.org/abs/2303.12712 * Sparks of Artificial General Intelligence: Early experiments with GPT-4
https://arxiv.org/abs/2303.17564 BloombergGPT: A Large Language Model for Finance
https://arxiv.org/abs/2303.18223 ??? A Survey of Large Language Models ???
https://arxiv.org/abs/2203.02155 * [InstructGTP?] Training language models to follow instructions with human feedback (https://github.com/openai/following-instructions-human-feedback)
https://arxiv.org/abs/2303.15772 Ecosystem Graphs: The Social Footprint of Foundation Models
https://arxiv.org/abs/2302.04761 Toolformer: Language Models Can Teach Themselves to Use Tools
https://arxiv.org/abs/2212.10560 Self-Instruct: Aligning Language Model with Self Generated Instructions
https://arxiv.org/abs/2211.01910 Large Language Models Are Human-Level Prompt Engineers
https://arxiv.org/abs/2105.14103 GPT-Neo
https://arxiv.org/abs/2005.14165 Language Models are Few-Shot Learners (GPT-3)
https://arxiv.org/abs/2002.12327 ??? A Primer in BERTology: What we know about how BERT works ???
https://arxiv.org/abs/1706.03762 * Attention Is All You Need
what are the most influential large language model papers on arxiv? 1. “Improving Language Understanding by Generative Pre-training” by Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever (arXiv: 1801.06146) 2. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” by Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (arXiv: 1810.04805) 3. “GPT-2: Language Models are Unsupervised Multitask Learners” by Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever (arXiv: 1901.02860) 4. “XLNet: Generalized Autoregressive Pretraining for Language Understanding” by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le (arXiv: 1906.08237) 5. “T5: Text-to-Text Transfer Transformer” by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu (arXiv: 1910.10683) 6. “ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators” by Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning (arXiv: 2003.10555) 7. “DeBERTa: Decoding-enhanced BERT with Disentangled Attention” by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han (arXiv: 2006.03654) 8. “GPT-3: Language Models are Few-Shot Learners” by Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. (arXiv: 2005.14165)
1B. WEBSITES
- https://llm.garden/ (With so many Large Language Models (LLMs) released daily, we put together a list of everything available)
- https://transformer-circuits.pub/
- https://distill.pub/ (Distill was a scientific journal which operated 2016-2021.)
- https://aman.ai/primers/ai/chatGPT/
- https://wandb.ai/site/prompts (Uncover granular insights about your LLMs)
- https://github.com/ray-project/llm-numbers (Numbers every LLM Developer should know)
- https://flowgpt.com/?sort=most-popular
1C. BLOGS
- ! https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
- ! https://amatriain.net/blog/transformer-models-an-introduction-and-catalog-2d1e9039f376/
- https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens
- https://huyenchip.com/2023/05/02/rlhf.html
- https://huyenchip.com/2023/06/07/generative-ai-strategy.html
- https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
- eugeneyan.com
- https://www.philschmid.de/getting-started-trainium (Fine-tune BERT for Text Classification on AWS Trainium)
- https://huggingface.co/blog/falcon (The Falcon has landed in the Hugging Face ecosystem)
1D. AWESOME LLM LISTS
- https://github.com/Hannibal046/Awesome-LLM
- https://github.com/Mooler0410/LLMsPracticalGuide
- https://github.com/eugeneyan/open-llms
- https://github.com/ChandlerBang/awesome-self-supervised-gnn
- https://github.com/promptslab/Awesome-Prompt-Engineering
1E. VIDEOS
- https://www.youtube.com/watch?v=kCc8FmEb1nY - Let’s build GPT: from scratch, in code, spelled out. (Karpathy)
- ! https://www.youtube.com/playlist?list=PLDw5cZwIToCvXLVY2bSqt7F2gu8y-Rqje - A series of videos on the transformer (Lennart Svensson)
- https://www.youtube.com/playlist?list=PLujxSBD-JXgmB1AnewzycdtUtf5YVUyzU - 2 Minute papers: Stable Diffusion, DALL-E, GPT-4, OpenAI and more!
- ??? https://www.youtube.com/playlist?list=PLoyGOS2WIonajhAVqKUgEMNmeq3nEeM51 - transformer-circuits.pub
- https://www.youtube.com/watch?v=VPRSBzXzavo - How ChatGPT is Trained
- https://www.youtube.com/@ai_io
- https://www.deeplearning.ai/short-courses/
- https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/
- https://www.deeplearning.ai/short-courses/langchain-for-llm-application-development/
- https://www.cloudskillsboost.google/journeys/118 (google Generative AI learning path)
- https://www.youtube.com/watch?v=XSSTuhyAmnI - What are Transformer Neural Networks?
2. PROMPTING
3. FINE-TUNING
- https://github.com/hpcaitech/ColossalAI/tree/main/applications/Chat
- ? https://huggingface.co/docs/autotrain/index & https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/llm_finetuning.mdx
- https://github.com/huggingface/peft & https://huggingface.co/docs/peft/index
4. MODEL UNIVERSES / LISTS
- https://crfm.stanford.edu/ecosystem-graphs/
- (via https://arxiv.org/abs/2303.15772 - Ecosystem Graphs: The Social Footprint of Foundation Models)
- https://github.com/stanford-crfm/ecosystem-graphs
5. DATA CONNECTORS
General
gpt4all
gpt4all: an ecosystem of open-source chatbots trained on a massive collections of clean assistant data including code, stories and dialogue
- https://github.com/nomic-ai/gpt4all
- https://gpt4all.io/index.html
- https://static.nomic.ai/gpt4all/2023_GPT4All-J_Technical_Report_2.pdf
- https://s3.amazonaws.com/static.nomic.ai/gpt4all/2023_GPT4All_Technical_Report.pdf
Llama Index
LlamaIndex (GPT Index) is a project that provides a central interface to connect your LLM’s with external data.
LangChain
LangChain is a toolkit for composability and gluing together different LLMs and utility packages
- https://github.com/hwchase17/langchain
- https://docs.langchain.com/docs/
- https://python.langchain.com/en/latest/index.html
AutoGPT
An experimental open-source attempt to make GPT-4 fully autonomous.
BabyAGI
This Python script is an example of an AI-powered task management system. The system uses OpenAI and Chroma to create, prioritize, and execute tasks. The main idea behind this system is that it creates tasks based on the result of previous tasks and a predefined objective.
🧵Thread (above) generated by GPT4 based on paper 📄Paper generated by GPT4 based on code 📊Graphs in paper generated by GPT4 based on code 💻Code generated by GPT4 based on prompt
- https://github.com/yoheinakajima/babyagi
- https://twitter.com/yoheinakajima/status/1640934493489070080?s=20
- https://twitter.com/yoheinakajima/status/1640934503324717056/photo/1
6. EMBEDDINGS
- https://txt.cohere.com/embedding-archives-wikipedia/ & https://huggingface.co/Cohere?ref=txt.cohere.com
7. VECTOR / EMBEDDING DATABASES
Moved
8. INTERESTING CHAT/LLM/GENAI COMPANIES
Moved
- LLM ARCHITECTURES Moved