LegoGPT builds 3D Lego designs from text, and makes sure they don’t fall over

For decades, Lego has been the playground reserved for patient builders, running around with elaborate instruction manuals. Now, an AI model called LegoGPT is flipping the bricks, literally and figuratively.

Developed by researchers at Carnegie Mellon University, this open-source project aims to find out whether artificial intelligence can generate 3D Lego designs that don’t just look cool, but also stand up, physically.

StableText2Lego has experimental data for days

Built on a fine-tuned version of LLaMA-3.2-Instruct, LegoGPT responds to text prompts like “streamline elongated vessel” or “backless bench with armrest” and spits out detailed 3D Lego structures.

These aren’t just visual suggestions; each creation is designed to be buildable in the real world. The secret is a two-part system. First is the language model that generates the structure, and then a mathematical solver named Gurobi runs a physics check to make sure the design won’t topple like a toddler’s first tower.

ALSO READ: Google’s new AI could let us chat with dolphins – No, really

The backbone of LegoGPT is StableText2Lego, a hefty dataset containing over 47,000 Lego structures and over 28,000 unique 3D designs. Each entry includes text captions, building instructions, and 3D models, enough brick-based knowledge to teach a robot how to become a master builder.

And that’s not theoretical. The researchers tested LegoGPT’s outputs with both robots and humans. A dual-arm robotic setup assembled the designs to verify their stability, and human builders followed suit to test for practical durability. According to the team, a whopping 99.8 per cent of the structures passed.

Even better, LegoGPT is now available on GitHub under the MIT license, meaning anyone can use or tinker with it. So yes, your next Lego set could be AI-generated. No manual required. All you need is a prompt, a printer, and maybe a little faith in robot engineering.