#Pretraining
Showing 27 of 27 repositories tagged #pretraining, ranked by stars
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
Llama中文社区，实时汇总最新Llama学习资料，构建最好的中文Llama大模型开源生态，完全开源可商用
General technology for enabling AI capabilities w/ LLMs and MLLMs
mPLUG-Owl: The Powerful Multi-modal Large Language Model Family
Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).
[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
EntitySeg Toolbox: Towards Open-World and High-Quality Image Segmentation
X-modaler is a versatile and high-performance codebase for cross-modal analytics(e.g., image captioning, video captioning, vision-language pre-training, visual question answering, visual commonsense reasoning, and cross-modal retrieval).
A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
PITI: Pretraining is All You Need for Image-to-Image Translation
BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training
Paper List for Recommend-system PreTrained Models
A Chinese Open-Domain Dialogue System
LLMInternSkill: LLM internship resume and job-search Codex Skill for resume polish, JD tailoring, evidence guard, interview grilling, and Project Scout. 大模型实习简历与求职工具箱。
[ICLR 2026] GRAPE: Group Representational Position Encoding (https://arxiv.org/abs/2512.07805)
逐行对照 MiniMind 源码精读、并延伸到大模型技术体系的中文学习笔记 —— 预训练 / SFT / DPO / PPO / GRPO、训练机制、MiniMind2→3 版本对照、真实实验证据。
[ACL 2025 Findings] Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts (https://huggingface.co/papers/2402.07625)
[ECCV 2022] Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
[NeurIPS 2023 Main Track] This is the repository for the paper titled "Don’t Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner"
🌹 Rose: Range-Of-Slice Equilibration PyTorch optimizer. Stateless optimization through range-normalized gradient updates.
A toolkit for scaling law research ⚖
[ECCV 2024] Dyadic Interaction Modeling for Social Behavior Generation
Taught by AI genius Andrew NG, this course entails the cutting edge topics such as, How generative AI works including what it can and can't do, Common uses cases such as Reading, Writing, and Chatting, Life Cycle of GenAI projects, Advanced Technology options such as RAG, Fine tunning, and Pre-Training, Implications of GenAI on business & Society.
A curated list of vintage large language models — also called historical or time-capsule LLMs — trained from scratch on text from bounded historical periods, along with the papers, datasets, demos, and discussions surrounding them.