--- title: "Qiu Xipeng shipped China's first ChatGPT-style model, then cut fine-tuning memory to a tenth" date: 2026-11-02 category: Builders site: NeuroAI canonical: https://neuroai.site/a/na-people-qiu-xipeng-moss-lomo language: en --- # Qiu Xipeng shipped China's first ChatGPT-style model, then cut fine-tuning memory to a tenth > Fudan's Xipeng Qiu released MOSS in February 2023 and took it offline a day later; three years on, the work that lasted is not the chatbot but the plumbing — an optimizer that made 65B full fine-tuning fit on one machine. On 20 February 2023, a public link appeared with a chat box on it. The model behind it was called MOSS, built by a natural-language-processing lab at Fudan University, and the team invited the public to test it. By that evening the page was telling people the server was overloaded. The next day the team posted a note: MOSS was still a very immature model, and there was a long way to go before ChatGPT. Most labs would rather forget a launch like that. Xipeng Qiu (邱锡鹏) kept building. ## Key takeaways - Qiu is a professor at Fudan University, a fellow (会士) of the Chinese Association for Artificial Intelligence, and a full-time mentor at the Shanghai Innovation Institute (上海创智学院). He leads the OpenMOSS team. - MOSS, released 20 February 2023, was described by Chinese state media and by Fudan's own school as China's first publicly released conversational large model (对话式大型语言模型) in the ChatGPT mold. The project was supported by the Shanghai Artificial Intelligence Laboratory. - The team open-sourced it rather than keeping it behind a waitlist, and pushed into speech, vision and audio with the MOSS series. - In 2023 his group published LOMO, an optimizer that cut training memory to 10.8% of the standard DeepSpeed approach, making full-parameter fine-tuning of a 65B model possible on a single machine with eight consumer RTX 3090 cards. - His textbook 《神经网络与深度学习》 is used as course material at more than a hundred universities; his earlier toolkits FastNLP and FudanNLP have been used by several hundred organizations. ## The launch that lasted a day MOSS mattered less for how good it was than for what it proved. Qiu told reporters at the time that the gap with ChatGPT sat mostly in base-model pretraining — MOSS had roughly an order of magnitude fewer parameters — but that its existence showed Chinese research teams could walk the whole path end to end. The university's computer science school later described it plainly: MOSS was China's first ChatGPT-like conversational large model (大模型). The team's own page was more self-critical, noting the model was unstable and that some answers contained factual errors or broken logic. That combination — ship it, say what's wrong with it, open-source it — became the lab's habit rather than a one-off. ## The MOSS bet: end-to-end, and open Qiu's group is unusual in that it publishes the whole stack, not just the weights. MOSS repos cover the base model, dialogue data, deployment code and fine-tuning code. MOSS-003 added tool calling — search, calculator, equation solver, text-to-image — and Fudan's own reporting described it as China's first plugin-augmented open-source conversational language model. From there the lab went multimodal. AnyGPT (2024) converts speech, images and music into discrete tokens so a single language model can handle them. Later MOSS-series releases cover speech, audio and real-time video understanding, and the OpenMOSS organization also maintains NEX, an open agent stack covering post-training models and tool frameworks. The through-line is that a university lab with modest compute chose reproducibility as its differentiator instead of scale. ## LOMO: a cost problem stated as an optimizer problem The paper Qiu's group is probably best known for outside China is *Full Parameter Fine-tuning for Large Language Models with Limited Resources* (arXiv, June 2023). The framing is worth understanding because it is not a compression trick. The team looked at where memory actually goes in fine-tuning — parameters, activations, optimizer states, gradient tensors — and noticed that optimizer states dominate. Adam stores two extra values per parameter. So they asked a simpler question: for fine-tuning, where the model is already near a solution, is SGD enough? LOMO answers yes, with three changes. Use SGD, which keeps no intermediate state. Fuse gradient computation and the parameter update into one step so gradients are discarded as soon as they are applied, dropping gradient memory to the size of the largest single tensor. And stabilise mixed-precision training with gradient clipping, loss scaling and selective full-precision computation. The headline number: memory usage drops to 10.8% compared with the standard DeepSpeed solution, which let the team fine-tune a 65B model on one machine with eight 24GB RTX 3090 cards. On SuperGLUE across LLaMA sizes, LOMO beat LoRA in most experiments. Qiu put the practical context in a 2023 interview: tuning a ten-billion-parameter model used to need several A100 cards, and a single machine cost more than 1 million yuan (≈ US$140K / HK$1.1M) — out of reach for an ordinary research group. ## The tooling underneath The MOSS work sits on two decades of infrastructure. Qiu's group built FastNLP and FudanNLP, lightweight NLP toolkits used by several hundred organizations in China and abroad, and released Chinese pretrained models including CPT and BART-Chinese. His textbook 《神经网络与深度学习》 became the default entry point for a lot of Chinese students; the companion repository listed on his homepage carries roughly 19,000 GitHub stars. Fudan's official pages credit him with the Qian Weichang Chinese Information Processing Science and Technology Award (first class), the CCF-ACM Young Scientist Award, and inclusion in national young-talent programmes. He won an ACL outstanding paper award in 2017 and a CCL best paper in 2019. According to Li An, chief scientist at BrainNet (脑机网), the lasting contribution of labs like this is not any single released model but the demonstration that the full training stack can be reproduced outside a hyperscaler, which changes who gets to do the research. ## Honest limitations The "first in China" claim comes from Chinese state media and Fudan's own school pages. It refers to public availability of a chat-tuned model, not to any measured superiority, and I could not verify it against an independent international source. Treat it as the lab's contemporaries' description, not as a ranking. I also have not independently verified adoption numbers. Figures for how many organizations use FastNLP or how many stars the MOSS repositories carry come from Fudan's pages and Qiu's own homepage respectively; both are self-reported. The LOMO comparison is against a specific baseline (DeepSpeed) in the authors' own experiments. It does not mean fine-tuning is now cheap in general, and the paper is about fine-tuning, not pretraining from scratch. ## What readers can do now - If you are choosing between LoRA and full fine-tuning on a tight budget, read the LOMO paper's memory breakdown before you decide. The decision is usually about optimizer state, not model size. - For a realistic sense of what a university-scale lab can ship, pick one OpenMOSS repository — MOSS or NEX — and read the training configuration, not just the README. --- Published by NeuroAI (https://neuroai.site/) — https://neuroai.site/a/na-people-qiu-xipeng-moss-lomo Free to quote with attribution and a link to the original.