A game studio wants footfalls that change when the floor changes from stone to gravel. A film sound designer wants a door slam carrying the weight of the hand that pushed it. Neither can use the model that would make those two seconds of audio trivial — because the licence on its weights forbids exactly that.
Key takeaways
- AudioX-Turbo, from Noiz AI (声音动力) with HKUST and Tsinghua, generates audio from text, video, image or audio conditions in four sampling steps, per its arXiv paper.
- On a single RTX 4090 it renders ten seconds of audio in 0.24 seconds at a real-time factor of 0.02, roughly 23 times faster than its own teacher model at 5.49 seconds.
- Its training corpus, IF-caps-Pro, holds about 9.2 million samples, split roughly 1.3 million general audio and 7.9 million music, with timestamped and instrument-level labels.
- The code, weights and training code are open on GitHub and Hugging Face, but the repository states the models are watermarked and strictly for non-commercial use, under a CC-BY-NC 4.0 licence.
- Noiz AI, founded in Shenzhen in 2025 by Chen Weijia (陈伟嘉), raised a seed round of several tens of millions of yuan (≈ US$4M / HK$31M) from GSR Ventures (金沙江创投), on a reported ARR in the millions of dollars and more than two million global users.
Four steps instead of four hundred
Diffusion audio models work, and they work slowly. Take a text prompt, run the sampler fifty times, get ten seconds of sound. That is fine for a rendering farm. It is useless if you want sound to react to something happening now.
AudioX-Turbo's argument is that the sampler, not the model, is the problem. The team trained a teacher model, AudioX-Base, on a Multimodal Diffusion Transformer backbone of 2.7 billion parameters, then used distribution matching distillation to compress it into a student that runs in four sampling steps. The paper's table reports the teacher taking 200 sampling steps and 5.49 seconds for ten seconds of audio on the same card, against the student's four steps and 0.24 seconds, with classifier-free guidance baked into the student so each step costs one forward pass instead of two.
The paper's headline claim is that four steps match or beat baselines running fifty to two hundred steps. That is the authors' own benchmark against named baselines, on AudioCaps and MusicCaps style sets, plus a purpose-built instruction-following benchmark for sound category, count, timestamp and ordering. It is not an independent evaluation, and no third party has reproduced it. The paper is also candid about its own boundary: the models are trained on ten-second clips and do not cover speech.
The part nobody labels well: the data
The dataset is the more interesting artefact. IF-caps-Pro was built in two stages, and the second stage is where the work went.
The pipeline starts from large volumes of real video-audio pairs, then uses Gemini 2.5 Pro to write structured templates carrying timestamps, instrument labels and event counts, then uses Qwen2-Audio to expand those templates at scale. The result reads less like a summary and more like a screenplay with a timeline: not "a door closes" but a door close at a specified second, with a specified number of impacts, from a specified material.
The teams reported an unexpected finding along the way: writing the text labels more precisely improved not just text-to-audio quality but also how well the model aligned audio to a silent video. Finer instruction on the label side bought accuracy on the audiovisual side.
Why spatial and physical matters
Noiz AI's stated ambition goes past generating plausible sound. The company's account of the field is that audio models have largely collapsed into speech — text-to-speech, speech recognition, low-latency conversation — and that what gets lost is everything a sound carries besides the words: how hard a door shut, what it was made of, how far away it was, where the listener stood.
The company's own product page sells that: sound effects generated from prompts, and a mode described as translating a video while preserving the original voice timbre. The founder has argued publicly that the next market is interactive platforms and embodied robots, where sound has to be regenerated live as the scene changes rather than rendered once and reused.
That direction is a bet on a market that barely exists in volume yet. The current business is a creator tool: more than two million global users, per the round's disclosure, with an ARR in the millions of dollars.
The licence problem
The open release is the part with a catch, and it is a large one.
The GitHub repository for AudioX-Turbo carries a CC-BY-NC 4.0 licence, and states plainly that the models are watermarked and strictly for non-commercial use. So the fastest anything-to-audio model available to a developer cannot legally be used to build a commercial product's sound layer.
That is a defensible choice for a research release, and also a strange one for a company whose revenue depends on creators monetising their work. It also creates the awkward gap the open-research community keeps running into: the paper's own note says code and datasets will be available, while the weights that make those outputs usable are licensed for study rather than production. The company's paid tiers do sell commercial use separately — the restriction is on the checkpoint, not on the service.
Honest limitations
The performance numbers are the authors' own, measured on one RTX 4090 at batch size one with ten-second clips, and the distillation comparison is against the same paper's teacher model rather than against deployed commercial systems under identical conditions. The IF-caps-Pro construction depends on named third-party models to write its labels, so annotation quality inherits those models' behaviour and biases; the paper does not independently measure annotation error. The dataset size — about 9.2 million samples — is also a count of curated triples, not of distinct real-world recordings, since the pipeline pairs real video-audio with machine-written text. On the company side, Noiz AI is private: the round size, ARR and user count come from a single round's disclosure reported by Chinese trade outlets rather than a filing, and the round has been described both as a seed round and as a seed-plus round in different reports. The non-commercial licence means there is currently no lawful path from this checkpoint to a commercial deployment, and no disclosed plan for changing that. The spatial-audio and robotics direction is stated intent, not shipped product.
What readers can do now
- If you are building anything commercial with audio, treat AudioX-Turbo as a research instrument, not a dependency — read the licence before wiring it in, and price a commercially licensed alternative into the plan.
- If you do Foley (拟音) or sound-design work, the fastest practical takeaway is the annotation idea rather than the checkpoint: write down when, how many, what material and how loud, and treat that as production data you own.
- If you are a hardware or robotics team, the interesting question is latency budget, not quality. Work out what real-time factor your product needs, then decide whether a distilled few-step model is a requirement or a luxury.