sqiddster

[MOC] The Universal LEGO Sorting Machine

Recommended Posts

Over the past two and a half years, I've been working on my most insane project to date: the world's first Universal LEGO Sorting Machine.

 

I call it 'Universal' because, thanks to the use of cutting-edge AI, it's capable of recognizing and sorting any LEGO part that has ever been produced. The machine itself is built from over 10,000 LEGO bricks, and, even though it's small enough to fit on a desk, it is capable of taking a large bucket of completely unsorted parts and categorizing them into 18 different output buckets at near-human speed.

First, the machine uses a series of belts and a vibrating table to separate the parts out. It uses a camera and a Raspberry Pi to take streaming video footage of the parts running along a belt, and sends the footage wirelessly to a more powerful computer that runs a convolutional neural network - that's where the magic happens. 

A convolutional neural network is an AI technology that's designed to work with images. To build a network that is capable of recognizing almost 3000 different LEGO parts, the network needs a lot of training data. Using 3D models of LEGO bricks, I generated over 25 million images which were then used to train the network, allowing it to accurately recognize even the most obscure parts.

I've also released a second video which demonstrates specifically how a neural network is able to recognize LEGO parts, and how I was able to train the system. The video also functions as a helpful beginner's introduction to the real technology behind AI.

stages_small.jpg.280cd0b1a639b326d595d96742d9b212.jpg

49164390963_e5b8e5ae34_k.jpg

Edited by sqiddster

Share this post


Link to post
Share on other sites

Wow, that's brilliant! Can you expand the machine and software to sort the parts into even more, detailed categories?

Share this post


Link to post
Share on other sites
4 minutes ago, Zerobricks said:

Wow, that's brilliant! Can you expand the machine and software to sort the parts into even more, detailed categories?

Yes! Because it recognizes parts individually, it would be easy to sort the parts into any categories that you choose ^_^

Share this post


Link to post
Share on other sites

Awesome stuff! I'm an AI researcher myself and we have quite a few Lego enthusiasts in the lab, so your video is doing the rounds here! :laugh:

Share this post


Link to post
Share on other sites
1 hour ago, sqiddster said:

To build a network that is capable of recognizing almost 3000 different LEGO parts, the network needs a lot of training data. Using 3D models of LEGO bricks, I generated over 25 million images which were then used to train the network

Impressive! How many days of GPU training did that require?

Share this post


Link to post
Share on other sites
1 hour ago, m00se said:

Impressive! How many days of GPU training did that require?

Training itself doesn't take too long, maybe 2 days or so.

The real computational difficulty is in rendering the 25 million images. This took about 2 years of CPU core time, parallelized over 800 AWS cores to be completed in about a day.

Share this post


Link to post
Share on other sites

wow this is amazing! I'm currently sorting through 20KG of Technic parts, this wiould save a lot of time haha 

Share this post


Link to post
Share on other sites

Wow - this has an obvious (dollar) value to me as an AFOL based purely on my time that i'd rather spend on building than sorting, and i'm sure every bricklink type store who takes in piles of used stuff would find it economically useful as well. 

It's also brilliantly geeky to build it out of lego in a serpent eating its own tail kind of way, but perhaps you might angle toward some kind of crowd sourced manufacturing implementation of this. A brick built shaker table would fall apart in short order and isn't necessary for a production version - same with the bins. But the potential is huge! You could even provide a common data set for updating users entities with improved AI  datasets, or choose to license it.

Sounds like with your 18 bins, it would be fairly simple to broad sort element types, then keep refining - one bin becomes 18 recursively until you've got unique part types - or ultimately even part type and colour if you want to go that deep down the hole. :)

Share this post


Link to post
Share on other sites
41 minutes ago, bonox said:

Wow - this has an obvious (dollar) value to me as an AFOL based purely on my time that i'd rather spend on building than sorting, and i'm sure every bricklink type store who takes in piles of used stuff would find it economically useful as well. 

It's also brilliantly geeky to build it out of lego in a serpent eating its own tail kind of way, but perhaps you might angle toward some kind of crowd sourced manufacturing implementation of this. A brick built shaker table would fall apart in short order and isn't necessary for a production version - same with the bins. But the potential is huge! You could even provide a common data set for updating users entities with improved AI  datasets, or choose to license it.

Sounds like with your 18 bins, it would be fairly simple to broad sort element types, then keep refining - one bin becomes 18 recursively until you've got unique part types - or ultimately even part type and colour if you want to go that deep down the hole. :)

I've thought about it, but I have a feeling that the dollar value that an interested AFOL (or even Bricklink store) would be willing to pay is quite a bit less than the cost of designing, manufacturing and selling something of this complexity for real. Perhaps it could be put to the test with croudsourcing but I'd need a realistic prototype and cost estimate first, which in itself is very expensive to obtain!

As for 18 bins, a 'real' manufactured version of this could be created with any arbitrary number of bins. There is no inherent mechanical or software limitation to how many you can have. The machine currently is even designed to be modular; each 'layer' of bins is completely independent and repeatable. 

You're also spot on regarding recursive sorting, with enough runs through you could have as much granularity as you want.

Edited by sqiddster

Share this post


Link to post
Share on other sites

Fair points - but you'd be amortising the development value across (hopefully) a horde of people would use, with perhaps the ability to use lego parts where suitable and cheap - for example, using the track links and motors from lego might be a good cost reduction idea for your widget - many of us already have rpi sets we could use, and so on. 

 

Interesting thought experiment nonetheless :) Congratulations on a great demonstration project anyway. I always found that the hardest part of any development project (particularly software) was showing it to people and having them play with it. A random pile of parts in your case is very different to a carefully selected test sample you built your system around and to see it work is outstanding :)

Share this post


Link to post
Share on other sites

DAMN! you did it!

I understand the difficulty in marketing a device such as this, mostly because several aspects will likely fall under other companies patent.

Having said that, it's potential use goes quite a bit beyond LEGO orientated applications.

whatever you end up doing with it, I'm impressed!

PS: perhaps a poll would be fun to see what people here would pay for a device like this :classic:

Share this post


Link to post
Share on other sites

This is huge! Brilliant work, hats off. I cannot even imagine the amount of work, research and time you put into this. But the end result is amazing, thanks for sharing your creation :sweet:

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.