[MOC] The Universal LEGO Sorting Machine

[sqiddster]

Over the past two and a half years, I've been working on my most insane project to date: the world's first Universal LEGO Sorting Machine.

 

I call it 'Universal' because, thanks to the use of cutting-edge AI, it's capable of recognizing and sorting any LEGO part that has ever been produced. The machine itself is built from over 10,000 LEGO bricks, and, even though it's small enough to fit on a desk, it is capable of taking a large bucket of completely unsorted parts and categorizing them into 18 different output buckets at near-human speed.

First, the machine uses a series of belts and a vibrating table to separate the parts out. It uses a camera and a Raspberry Pi to take streaming video footage of the parts running along a belt, and sends the footage wirelessly to a more powerful computer that runs a convolutional neural network - that's where the magic happens. 

A convolutional neural network is an AI technology that's designed to work with images. To build a network that is capable of recognizing almost 3000 different LEGO parts, the network needs a lot of training data. Using 3D models of LEGO bricks, I generated over 25 million images which were then used to train the network, allowing it to accurately recognize even the most obscure parts.

I've also released a second video which demonstrates specifically how a neural network is able to recognize LEGO parts, and how I was able to train the system. The video also functions as a helpful beginner's introduction to the real technology behind AI.

Edited December 3, 2019 by sqiddster

[Zerobricks]

Wow, that's brilliant! Can you expand the machine and software to sort the parts into even more, detailed categories?

[sqiddster]

4 minutes ago, Zerobricks said:

Wow, that's brilliant! Can you expand the machine and software to sort the parts into even more, detailed categories?

Yes! Because it recognizes parts individually, it would be easy to sort the parts into any categories that you choose ^_^

[astyanax]

Awesome stuff! I'm an AI researcher myself and we have quite a few Lego enthusiasts in the lab, so your video is doing the rounds here! 

[m00se]

1 hour ago, sqiddster said:

To build a network that is capable of recognizing almost 3000 different LEGO parts, the network needs a lot of training data. Using 3D models of LEGO bricks, I generated over 25 million images which were then used to train the network

Impressive! How many days of GPU training did that require?

[sqiddster]

1 hour ago, m00se said:

Impressive! How many days of GPU training did that require?

Training itself doesn't take too long, maybe 2 days or so.

The real computational difficulty is in rendering the 25 million images. This took about 2 years of CPU core time, parallelized over 800 AWS cores to be completed in about a day.

[nahpets]

wow this is amazing! I'm currently sorting through 20KG of Technic parts, this wiould save a lot of time haha 

[Jeroen Ottens]

This is absolutely amazing! I never thought it would be possible to make something like this, but it does an amazing job.

[bonox]

Wow - this has an obvious (dollar) value to me as an AFOL based purely on my time that i'd rather spend on building than sorting, and i'm sure every bricklink type store who takes in piles of used stuff would find it economically useful as well. 

It's also brilliantly geeky to build it out of lego in a serpent eating its own tail kind of way, but perhaps you might angle toward some kind of crowd sourced manufacturing implementation of this. A brick built shaker table would fall apart in short order and isn't necessary for a production version - same with the bins. But the potential is huge! You could even provide a common data set for updating users entities with improved AI  datasets, or choose to license it.

Sounds like with your 18 bins, it would be fairly simple to broad sort element types, then keep refining - one bin becomes 18 recursively until you've got unique part types - or ultimately even part type and colour if you want to go that deep down the hole. :)

[sqiddster]

41 minutes ago, bonox said:

Wow - this has an obvious (dollar) value to me as an AFOL based purely on my time that i'd rather spend on building than sorting, and i'm sure every bricklink type store who takes in piles of used stuff would find it economically useful as well. 

It's also brilliantly geeky to build it out of lego in a serpent eating its own tail kind of way, but perhaps you might angle toward some kind of crowd sourced manufacturing implementation of this. A brick built shaker table would fall apart in short order and isn't necessary for a production version - same with the bins. But the potential is huge! You could even provide a common data set for updating users entities with improved AI  datasets, or choose to license it.

Sounds like with your 18 bins, it would be fairly simple to broad sort element types, then keep refining - one bin becomes 18 recursively until you've got unique part types - or ultimately even part type and colour if you want to go that deep down the hole. :)

I've thought about it, but I have a feeling that the dollar value that an interested AFOL (or even Bricklink store) would be willing to pay is quite a bit less than the cost of designing, manufacturing and selling something of this complexity for real. Perhaps it could be put to the test with croudsourcing but I'd need a realistic prototype and cost estimate first, which in itself is very expensive to obtain!

As for 18 bins, a 'real' manufactured version of this could be created with any arbitrary number of bins. There is no inherent mechanical or software limitation to how many you can have. The machine currently is even designed to be modular; each 'layer' of bins is completely independent and repeatable. 

You're also spot on regarding recursive sorting, with enough runs through you could have as much granularity as you want.

Edited December 5, 2019 by sqiddster

[bonox]

Fair points - but you'd be amortising the development value across (hopefully) a horde of people would use, with perhaps the ability to use lego parts where suitable and cheap - for example, using the track links and motors from lego might be a good cost reduction idea for your widget - many of us already have rpi sets we could use, and so on. 

 

Interesting thought experiment nonetheless :) Congratulations on a great demonstration project anyway. I always found that the hardest part of any development project (particularly software) was showing it to people and having them play with it. A random pile of parts in your case is very different to a carefully selected test sample you built your system around and to see it work is outstanding :)

[shadow_elenter]

DAMN! you did it!

I understand the difficulty in marketing a device such as this, mostly because several aspects will likely fall under other companies patent.

Having said that, it's potential use goes quite a bit beyond LEGO orientated applications.

whatever you end up doing with it, I'm impressed!

PS: perhaps a poll would be fun to see what people here would pay for a device like this 

[braker23]

This is huge! Brilliant work, hats off. I cannot even imagine the amount of work, research and time you put into this. But the end result is amazing, thanks for sharing your creation 

[BusterHaus]

Fantastic work, congratulations on your achievement and thank you for sharing the development process. 

[natesroom]

Yeah if I could get something like this for somewhere between 500 and 1500 I think you'd have a cast amount of people interested. I don't have nearly as large a collection as many others but I'm literally looking at 1-2 years of constant sorting in my own time to get my full collection organized... I'd gladly pay somewhere in that range to speed and mindlessly sort with a little oversight