[MOC] EV3 Catching a Ball in Mid-Air

[oracid]

What do you mean about "Stock controller" ?

Do you mean that "Custom controller" is your own PID ?

[Jim]

I saw this on Facebook and I am really glad you took the time to join EB and post this. Absolutely awesome.

I have frontpaged it and I have given you the Mindstorms tag.

[dsharlet]

What do you mean about "Stock controller" ?

Do you mean that "Custom controller" is your own PID ?

By stock controller, I mean the controller that is built in to the EV3 (ev3dev that is...).

The custom controller is indeed my own PID controller. It is designed to allow the setpoint to vary without reseting the internal state of the PID.

The catching action requires the robot to anticipate where the ball will land and start moving there very quickly. However, even while the arms are still in motion, the robot gets more observations from the cameras, which gives an improved estimate of the trajectory (and where the ball will land). So, we want to change the setpoint for the arms, without resetting the PID controllers so they continue moving smoothly and quickly. My custom controller allows this to work.

You can kind of see why this is necessary in one of the failed catches in the video starting at 29 seconds:

You can almost see how the initial estimate of the trajectory was pretty far off. It then appears to get a very late improved estimate of where the ball will land that *might* be correct, but it's too late, and so the grabber just hits the ball when it tries to move there, instead of getting under it in time. It all happens in a split second, so it takes a few watches to see it.

BTW, if you want to see it, my implementation of the PID controller (and the motor controller that uses it) is here:

https://github.com/d...id_controller.h

https://github.com/d...c/ev3/servo.cpp

I saw this on Facebook and I am really glad you took the time to join EB and post this. Absolutely awesome.

I have frontpaged it and I have given you the Mindstorms tag.

Wow, thank you! I'm honored :)

Edited September 7, 2016 by dsharlet

[J_C]

Anything I would say would be an understatement.

So I am just gonna say: "Thank you very much for sharing this. Seriously, thank you!"

Edited September 7, 2016 by J_C

[Foton]

Catching ball is kind of tasks I never expect to be accomplished by EV3 brick only.

I love the technic details you post there.

This is most mature (standalone) robot from Mindstorm series I ever seen!

Hat tip to You!

Edited September 7, 2016 by Foton

[Victor Imaginator]

@dsharlet: Congratulations with frontpaging and your tags)

[AndersI]

I must say I'm seriously impressed! This is by far the most interesting LEGO Mindstorms project I've seen! If anyone had asked me, I would have said "No way, you can't get a LEGO robot to catch a ball unless perfectly thrown", but you have proved otherwise.

[lcvisser]

Thanks! Any particular piece you want more info on? I'm happy to go into more detail if you have some specific questions/curiosity about a particular part of the project :)

I'm particularly interested in the two nonlinear optimisations that you're doing (for the camera calibration and the trajectory estimation). I've always been interested in nonlinear problems, but never had much chance to do anything cool with nonlinear optimisation. I can make an educated guess on your approach based on what you've told so far, but I'd love to see the math and theory behind it.

I'm out of academia for a while now and in either case I wasn't to much into vision problems, but the two approaches appear very innovative to me. Are they original ideas from you or adaptations of existing theory?

[MajorAlvega]

Thanks for the details about the controller. The video that show the difference between native and yours is very interisting, I wonder if it is possible to do the same in python (or include it somehow).

I got curious about the pneumatics, I have no experience with pneumatics. Can you detail it further?

[Obi1by1]

Stunning! I know little/next to nothing about robotics, but have always been fascinated by the ability of the human brain to make thousands of on the fly calculations in order to catch something (a ball, etc.), and to see a lego-built robot do the same thing is nothing short of a marvel. It's funny that I've happened to see your post/video while I'm in hospital having undergone one spinal cord surgery to correct a neurological hiccup, and waiting on a second. I'm taking this as a good sign, and it's cheered my up no end! Marvelous! If Lego can do this, so can I (although I bet you robot has better clutch power than I do!)

[1980-Something-Space-Guy]

Wow. Great job! I'm a sucker for controls (mechanical engineer here trying to get into the robotics industry), so it always amazes me the kind of things that can be done. I also found it interesting that you had to change the controllers for the servos. I would not have expected that.

I've been out of the Mindstorms world for quite a few years, and I see things have changed a lot. I wasn't even aware that it was possible to program EV3 using C++. There have been some great developments in the last few years.

Just out of curiosity, what do you do for a living?

[dsharlet]

I must say I'm seriously impressed! This is by far the most interesting LEGO Mindstorms project I've seen! If anyone had asked me, I would have said "No way, you can't get a LEGO robot to catch a ball unless perfectly thrown", but you have proved otherwise.

Well, you do have to throw the ball pretty carefully ;)

I'm particularly interested in the two nonlinear optimisations that you're doing (for the camera calibration and the trajectory estimation). I've always been interested in nonlinear problems, but never had much chance to do anything cool with nonlinear optimisation. I can make an educated guess on your approach based on what you've told so far, but I'd love to see the math and theory behind it.

I'm out of academia for a while now and in either case I wasn't to much into vision problems, but the two approaches appear very innovative to me. Are they original ideas from you or adaptations of existing theory?

The calibration method is new to me, I'm not sure if it's been used before. The thing is, it's really not a good method :) It's not very reliable and it requires very good initialization, otherwise it won't converge. It's hard to do too, it requires moving the ball slowly for minutes, being careful not to let the ball go out of view of the cameras. The only advantage to this method is that it doesn't need actual images, it works with the object tracking data that NXTcam gives you. It's hard to imagine any realistic system with this limitation. BTW: as much as I complain about NXTcam, it's still a great tool for EV3 and I'm glad I bought them, they just require creative solutions to problems like this :)

The trajectory estimation method is also new to me, but I'm sure that it's been done before. It's a very straightforward approach: I just assume the trajectory of the ball is a parabola, and given a trajectory estimate, compute the error between the estimated parabola and the observations in (calibrated) 2D camera space. This is the loss function, when optimized it gives a good estimate of the trajectory.

I've been working on a better writeup of this part, I'll try to finish it soon. Doing it properly requires writing down a bit of math, which are hard to do in text.

My code is not the most readable, but if you want to look at it, the calibration code is solved here. The header has a pretty large comment describing the calibration procedure.

https://github.com/d...n/calibration.h

https://github.com/d...bration.cpp#L31

The trajectory estimation and some other related code is here:

https://github.com/d.../trajectory.cpp

Thanks for the details about the controller. The video that show the difference between native and yours is very interisting, I wonder if it is possible to do the same in python (or include it somehow).

I got curious about the pneumatics, I have no experience with pneumatics. Can you detail it further?

It surely could be implemented in python too, which works on ev3dev! I haven't tried using python on ev3dev myself, but I think it's actually a much more popular choice than C++.

The use of pneumatics in this robot is really basic, there's only one valve directly connected to the smaller EV3 servo motor. I used the pneumatics for the grabber because they're very light, which helps the arm move quickly. Here's a better view of the grabber and the control for it:

https://goo.gl/photo...7nEFt4ZtnLX52B6

https://goo.gl/photo...Eir42MArHYc25A6

Wow. Great job! I'm a sucker for controls (mechanical engineer here trying to get into the robotics industry), so it always amazes me the kind of things that can be done. I also found it interesting that you had to change the controllers for the servos. I would not have expected that.

I've been out of the Mindstorms world for quite a few years, and I see things have changed a lot. I wasn't even aware that it was possible to program EV3 using C++. There have been some great developments in the last few years.

Just out of curiosity, what do you do for a living?

I'm really impressed with EV3, thanks to LEGO for giving us such an awesome system :)

I'm a software engineer, I do lots of DSP (on images) for work, no robots though!

Stunning! I know little/next to nothing about robotics, but have always been fascinated by the ability of the human brain to make thousands of on the fly calculations in order to catch something (a ball, etc.), and to see a lego-built robot do the same thing is nothing short of a marvel. It's funny that I've happened to see your post/video while I'm in hospital having undergone one spinal cord surgery to correct a neurological hiccup, and waiting on a second. I'm taking this as a good sign, and it's cheered my up no end! Marvelous! If Lego can do this, so can I (although I bet you robot has better clutch power than I do!)

Best of luck to you, I'm glad I could provide a bit of entertainment :)

Edited September 7, 2016 by dsharlet

[oracid]

Thank you for sharing.

[jono rocky]

Totally amazed, getting Lego(to say) to catch a ball is just beyond impressive. :thumbup:

[Captainowie]

Doing it properly requires writing down a bit of math, which are hard to do in text.

Depending on what you're using to write, you might be able to get a LaTeX plugin to show the math. In a pinch, you could always use an external LaTeX renderer to make little pictures full of mathematics, and then embed those in your text.

Owen.

[ejred]

It's just like coaching t-ball: if you throw the ball exactly where their glove is, they catch it... at least most of the time

But seriously, that is mind-blowing!

[Aethersprite]

That's incredible. I know next to nothing about Mindstorms but this still blew my mind! Never knew Mindstorms sets were capable of this.

[Boxerlego]

Incredible! If anything, this highlights how amazing it is that humans are so accurate at catching, and we don't have to do advanced maths like the robot does.

There is a reason why we don't rely on math to catch a ball. Lego Robots lacks the Two Eyes That helps with catching. One day Lego Robot can Stick its Lego hand out and Throw a ball in the air and Catch the ball with out moving its catching hand. Might sound unimpressive with words but when you see it the reaction says otherwise...

[nerdsforprez]

Sitting at the airport, catching yet another flight, and I finally get around to checking this out.

Thank you so much for sharing; after a day of dealing with difficult people while traveling, I am reminded of awesome people can be and the lack of limitations of the human mind. Great stuff......

[LEGO Train 12 Volts]

Oh my gosh ...this lego is alive! :excited:

Fantastic work!

[dsharlet]

Depending on what you're using to write, you might be able to get a LaTeX plugin to show the math. In a pinch, you could always use an external LaTeX renderer to make little pictures full of mathematics, and then embed those in your text.

Owen.

Ah, I would not have expected a TeX plugin on a LEGO forum! I'll take a stab at explaining the trajectory estimation this weekend, that's the more interesting and fun problem here. The calibration barely works, and requires a lot more background.

There is a reason why we don't rely on math to catch a ball. Lego Robots lacks the Two Eyes That helps with catching. One day Lego Robot can Stick its Lego hand out and Throw a ball in the air and Catch the ball with out moving its catching hand. Might sound unimpressive with words but when you see it the reaction says otherwise...

Well, this robot does have two eyes, and it still has to do lots of math!

It really is pretty interesting to try and understand how humans catch things, while robots have to do incredible amounts of arithmetic to do the same.

I read an interesting paper back while I was working on this project that provided a reasonable explanation of how humans catch things, but I can't find it again. The gist was that humans (and animals) are basically doing a similar triangles analysis (unconsciously, of course) constantly while watching the ball, and by moving to keep the triangle the same relative shape (i.e. similar) leads the person to where the ball will land. That's not enough information to reproduce the thought exactly. I clearly didn't understand it well enough to make it stick... maybe one of you can connect the dots from my lame attempt at reproducing the argument... :)

[Vorkosigan]

Amazing job, its very creative to make something like this work within the limitations of the NXT system and LEGO.

[baard]

Fantastic creation!

You say the model is simple, but the ideas and thought including math and programming involve makes this just jaw dropping

Thank for sharing!

[Boxerlego]

Ah, I would not have expected a TeX plugin on a LEGO forum! I'll take a stab at explaining the trajectory estimation this weekend, that's the more interesting and fun problem here. The calibration barely works, and requires a lot more background.

Well, this robot does have two eyes, and it still has to do lots of math!

It really is pretty interesting to try and understand how humans catch things, while robots have to do incredible amounts of arithmetic to do the same.

I read an interesting paper back while I was working on this project that provided a reasonable explanation of how humans catch things, but I can't find it again. The gist was that humans (and animals) are basically doing a similar triangles analysis (unconsciously, of course) constantly while watching the ball, and by moving to keep the triangle the same relative shape (i.e. similar) leads the person to where the ball will land. That's not enough information to reproduce the thought exactly. I clearly didn't understand it well enough to make it stick... maybe one of you can connect the dots from my lame attempt at reproducing the argument... :)

I think what you did is amazing here with Lego. But when it come about catching objects thrown I look at it from a more logical side rather then a big math equation. Sure we got to start some where and baby steps are needed but sometimes we need to ask how will this "fit in the future design". Sure your Lego Robot can catch a ball but can it throw it as well and you now get the idea how that eventually ends. Its all natural progression. If your Lego Robot can throw and catch a ball then try something along the lines having the Lego Robot bounce a ball off the wall and catch it or make another Identical Lego catching Robot that can pass a ball back and fourth. That would be the next expansion for your build to push the concept further. Its incredible to see a Lego Robot programmed to catch a ball, Lost of possible ideas to be developed around that concept alone.

[dsharlet]

Fantastic creation!

You say the model is simple, but the ideas and thought including math and programming involve makes this just jaw dropping

Thank for sharing!

I just meant the inside of the robot is boring, there's no moving parts. It's just lots and lots of beams to make the robot rigid :)

Depending on what you're using to write, you might be able to get a LaTeX plugin to show the math. In a pinch, you could always use an external LaTeX renderer to make little pictures full of mathematics, and then embed those in your text.

Owen.

It's not clear to me how to use the plugin. Instead, I wrote up a post describing the trajectory estimation in more detail: http://www.dsharlet.com/2016/09/11/estimating-the-3d-trajectory-of-a-flying-object-from-2d-observations/

Please let me know if any of it is unclear! However, I have to warn you, that without some background in some of these topics (non-linear optimization, computer vision/camera geometry) it might be difficult to understand, I didn't attempt to explain the math that isn't specific to this problem.