We fine-tuned a 235B model on 4 GPUs. Then we didn't need it

A team set out to replace expensive API models by fine-tuning a 235-billion-parameter Mixture-of-Experts model (Qwen3-235B) on four H100 GPUs. Using a QLoRA-style approach - 4-bit compressing the base and training small adapter layers - they ran one epoch over ~3,600 examples and improved F1 for every customer tenant, averaging +7.63 points (individual gains ranged roughly from +4.8 to +12). The run took 26 hours, used 960 GB of system RAM and four 80 GB H100s, and produced a model that beat their previous fine-tuned API models both in accuracy and in per-token cost (about one-third the cost).

Getting that far required overcoming a string of loader and memory pitfalls: compression libraries need full weight matrices while splitting libraries want pieces early, so several tooling combos failed. Axolotl and LlamaFactory triggered out-of-memory behavior or duplicated RAM loads; Unsloth hit a GPU/CPU tensor bug; Answer.AI’s FSDP+QLoRA implementation was incompatible with newer Transformers and Qwen3. The successful setup used DeepSpeed ZeRO-3 with CPU offload, streaming 4-bit layers from RAM to GPU and sharding adapter and optimizer state, then saving per-process adapter shards and reconstructing them later. The result was never deployed: serving a 235B “thinking” model cheaply and with acceptable latency proved a larger engineering lift, and new open models soon closed the gap. The takeaway: only invest in fine-tuning when your data or use case truly can’t be solved by public releases.