[MINDSTORMS] LEGO Mindstorms and Fischertechnik

[fotoopa]

Connection Lego and Fischertechnik.
By combining both systems we can enjoy their advantages. The data and command exchange occurs on 2 levels. There is a serial duplex connection via  the hubE and the DE0-Nano_Soc board. On the other hand, the data lines of the other hubs are also read by the board. The latter can thereby process data in real time and without interaction of the running software. This data can mainly be used by the TXT Controller to display the results on the PC screen. Motor speed, position, sensor color information, distance information and pressure can all be read out. From the Lego side there is no need to write a program. The DE0 board simply listens in real time.
The DE0 board responds very quickly ( only a few usec). The connection is via 115200 baud but can be much faster if needed. I now need to work out some protocols to build a large application.

Lego-Fischertechnik by Frans, on Flickr

With the DE0 board, I have up to 120 inputs available. I can now use those to send to the lego hub and process them there. Commands can also be sent from the lego hub to the DE0 board and so on to the FT Controller. The possibilities are immense. I hope to present a larger project a little later. On the Fischertechnik side, I have a lot of outputs. Now I can already connect 16 motors, 16 servos, various digital outputs, etc.

Frans

 

Edited March 12, 2022 by fotoopa
add more info

[JopieK]

Great idea, but I don't think it will be suitable for virtually anyone here including myself even though I have taught VHDL with a DE0 in the past :) Also maybe you should explain it 'a little' more in laymen's terms.

[Toastie]

7 hours ago, JopieK said:

you should explain it 'a little' more in laymen's terms

You mean this?

@fotoopa Cool, very cool project!!!

Best,
Thorsten

 

[Mr Jos]

I'm very interested in the part that shows a Bluetooth connection from the 51515 hub to a windows PC, and able to show some stuff on screen then.

What I'm wanting for a very long time already is to build a realistic HMI (Human-Machine-Interface) for my Logistics center/other Lego machines. Now I need to do that on the small EV3 screen, but I would love to be able to build a great visual on a large screen that shows the status of the installation (and if it could have feedback/work together with the keyboard that would be even more awesome).

But I'm just self taught with programming MicroPython, and I don't know anything technical from the connections etc. Just knowing if it would be possible to do what I want would be great allready. Normally end of this month going to my first event (in Torhout Belgium), would be great to show a HMI on a large screen, and have photo's/video's from real installations next to it to see the similarities, and how technical installations in the industry work.

[fotoopa]

1 hour ago, Mr Jos said:

What I'm wanting for a very long time already is to build a realistic HMI (Human-Machine-Interface) for my Logistics center/other Lego machines.

That's what I'm making with this. A GUI where everything can be seen, where you get an overview. I have no knowledge of bluetooth, Linux or C programming. That is a big handicap for me.
However, I have been working with Fischertechnik for several years. They use their RoboPro program for this. This is pretty much like scratch programming. You have a very nice interface directly on the PC. You can arrange everything nicely on the 32 inch screen, make layout, place buttons, slide controls, LEDs etc. They have an I2C line. I use them between the TXT and the FPGA.

And I'm over 78 years old, now I have to learn Python too. That's going much slower now.I'm more of a hardware guy, hence my FPGA modules. But for others, this might as well be your favorite micro controller. It doesn't matter anyway. In that board you make an exchange of data between your GUI (here the TXT of FT) and your other hardware. For the other hardware, this is where you add the Lego Mindstorms hub. That connection then goes through a serial duplex connection on  the hubE.

My drawing showed a second part where I can intercept the communications of all the Lego hubs. There you can collect useful information about all the sensors and motors and make it available for your GUI. This is done in the FPGA in my case. The TXT can read and write in a 512 byte register block. The FPGA can also read and write data from the Lego hubs. So you have everything on the big screen. There should be no program running on the Lego. At powerup, it looks up all the attached modules by itself and starts communication. As a result, the data is already immediately available. Therefore I listen on the line of each pin5. This costs nothing in terms of load on the hub.

1 hour ago, Mr Jos said:

. Normally end of this month going to my first event (in Torhout Belgium).

Dat zal dan in het Nederlands zijn, ook mijn moedertaal!

I know that much is not clear now. The intention is to provide more and more details as the project moves forward. I also need to make up some of my own protocols to use with the Lego hub. I know that many elements are not yet clear. The intention is to provide more and more details as the project moves forward. I also need to make up some of my own protocols to use with the Lego hub. On the Lego side, I mainly need to learn to work with Python. Also the multikasking aspect is not always easy ( was easy with the TXT though). This takes time.

Frans

 

[JopieK]

11 hours ago, Toastie said:

You mean this?

@fotoopa Cool, very cool project!!!

Best,
Thorsten

 

Haha, well, sort of. Nice video :)

Let me better explain what I meant: I don't think a lot people know FPGA's in the first place, even fewer people know how to use FPGA's, even fewer people understand FPGA's and... fotoopa who is obviously some kind of embedded engineer (I can already tell that from his [for me] insightful diagram, a lot of undergraduate students struggle with making such a clear diagram, by the time of graduation not all reach this level, we might have a vacancy for you at our Applied University @fotoopa, don't worry about Python apart from VHDL we mostly teach C/C++), forgot to even mention DE10 being an FPGA.

Thanks to initiatives like Arduino, Microbit and Raspberry Pi microcontrollers become at least popular in schools, but FPGA's are a whole other deal. They are really cool, but also very intimidating for laypersons. We use VHDL with Quartus for letting our students program DE10 boards. Quartus is not a very easy tool either. There are alternative hardware description languages including Verilog, but still it is a challenge as you really need to know what you are doing. They are really versatile, but you need to really understand basic computer logic very well and on top of that different flip-flops and other digital logic components and also at a conceptual level (parallel vs. serial logic).

But seems a really cool project even though not all might understand how complicated it is.

[fotoopa]

1 hour ago, JopieK said:

Let me better explain what I meant: ......

Nicely explained what an FPGA board is and does.

The biggest difference and problem compared to microcontroller boards is that with an FPGA board you hardly have a library and have to write almost everything yourself. With microcontrollers, the users are also library users, not writers. So they only have to write their application. With my FPGA, I first have to provide my ownlibrary.

Yes the DE-10 board is the bigger brother of my DE0-Nano-Soc that I use. But the software is the same.I am also using Quartus with version 20.1.1 build 720. I write everything in verilog.

I use the TXT Controller primarily as a user interface for the PC. With it I can display all the results real time on the big screen. Without the TXT connected, I still have the 4x20 char LCD display. There I can also display most results in multiple pages. This is especially useful if I want to look at one of the 120 available inputs to see how their status is. I can also read the position up to 12 FT motors, the 16 servo positions and the digital outputs. On my small box there are 4 more potentiometers and 4 additional analog inputs, 12 bit and these can also be read on the LCD display. All this data is in the 512 byte register array and that is accessible by the TXT. Now the Lego hub information is added which also goes to the register array for exchange between all programs. I am currently using: Logic utilization (in ALMs):  5,590 / 15,880 ( 35 % ).

That is the general operation principle.

Frans.

[fotoopa]

Lego hub monitoring for the Mindstorms 51515.

A brief update:

Lego Hub monitoring for the PC. by Frans, on Flickr

With this interface I can now intercept all 6 hub signals. The information obtained is the same as what Lego reads. It can't be done any faster. No program needs to be written. The hub itself looks at powerup to see what is connected. After auto configuration, the connected device transmits its data continuously. This happens upon request of the hub (every 100 msec) or upon internal change of the data values. The data transmission depends on the device type. This varies from 20 msec to every 1 msec. The FPGA can very quickly process all data from the 6 hubs and make it available in registers. Also, the FPGA can execute commands. For this I do need to use one of the hubs lines. Normally this will be hubE. Because the FPGA stores every data change in registers, they are also available to the TXT Controller of Fischertechnik. Results can now be displayed on a large monitor via the PC. If necessary, I can selectively turn the measurement leads on and off via a jumper. On the pcb there are also some resistors for impedance adjustment. I use twisted pair cable for the signals.

The picoscope is connected via the FPGA module. Through the Picoscope, I can view up to 4 hub signals at any given time. There I see all the data appearing from the 115200 baud lines. On the FPGA keypad I can select which hub I want to monitor on the scope. All data up to 10 sec deep I can monitor this way.

The problem of the extension cables I solved myself by cutting the flat cables of the devices in the middle and reconnecting them with 6-pin low-cost connectors. This way I can make extension cables myself. This was really necessary because with the 25cm of existing cables I could not work any further. The only drawback is that you have to check for yourself if the connectors are connected correctly. For this I have of each cable pin1 marked with a black stroke

Now I can continue working on a practical application with Lego and Fischertechnik. You may expect new information from this soon.

Frans.

[fotoopa]

Lego en Fischertechnik connected.

Lego Mindstorms and Fischertechnik by Frans, on Flickr

This is a screen dump of the data from Lego and Fischertechnik together. All values of the 6 hubs can be read out real time by listening on the serial communication lines. These results are stored in a register array in the FPGA. In this way, the results are available to the Fischertechnik TXT Controller via the RoboPro program. The results are displayed realtime on the PC screen. The motors, servos and analog values of the DE0-NanO-Soc module are also available in this way. The recording speed is the same as Lego's internal hub speed.

The FPGA reads the data sent by the device on the hub at the same time. In this way the color that the color sensor reads can be displayed on the PC screen. Also the leds of the distance sensor are transmitted in this way. With these leds you can send commands to the FT-program. You can easily control these leds in a Lego program. The Fischertechnik program is here primarily as a GUI interface of all Lego hub values. But extending it further can make the working together of FT and Lego easy. If a lot of commands have to be used and in both directions then it is better with a serial line on one of the hubs. For this I will use HubE. This leaves only 5 hubs for the Lego devices. I have no Bluetooth available on my current FPGA code. Hence I have to extend via the serial line.

Frans

[Toastie]

Hi Frans,

fantastic work, I really love it! Getting that close to the hardware level is always the coolest thing, the fastest approach, and allows the greatest flexibility as every layer "above" abstracts more and more (naturally and intentionally, of course).

Now, just to get this "obstacle" out of my (personal) way of thinking: I believe you are using "hub" with more than one meaning, is that correct? I may well have misunderstood some of TLGs new name game definitions. For me, a "hub" is the microcontroller thingy (City hub, Technic hub, and whatever they came up with). The peripherals are actuators or sensors, and these are connected to the "ports" of the hub.

So when you write "All values of the 6 hubs" - is that all values of the six sensors? I believe so. The hub is also providing data to some of the sensors/actuators, correct? And you are intercepting this two-way traffic. Is that what it is?

Forgive me my ignorance!

All the best and keep on with this terrific work!

Thorsten

[fotoopa]

Thanks for the reply.

12 hours ago, Toastie said:

Now, just to get this "obstacle" out of my (personal) way of thinking: I believe you are using "hub" with more than one meaning, is that correct? I may well have misunderstood some of TLGs new name game definitions. For me, a "hub" is the microcontroller thingy (City hub, Technic hub, and whatever they came up with). The peripherals are actuators or sensors, and these are connected to the "ports" of the hub.

So when you write "All values of the 6 hubs" - is that all values of the six sensors? I believe so.

Yes I need to be a little clearer. There are quite a few possibilities for naming Lego devices. With the 6 hubs, I mean the 6 ports of a unit. The Mindstorms 51515 software always shows the hubA to hubF numbering on their help page when formatting the configuration. Hence why I always spoke of the 6 hubs as well. The screendump is also a little clearer in that regard, I have drawn the 6 connections there with their hubA..hubF ports.

12 hours ago, Toastie said:

 The hub is also providing data to some of the sensors/actuators, correct? And you are intercepting this two-way traffic. Is that what it is?

Yes it is. Normally you only have 1 of the 2 lines that you have to listen to. That's where all the low level data come in. If you need a serial connection then you have to use the 2 lines and you only have 5 ports left for the Lego devices. The nice thing is that I now read all the data here without having to run a program. Just a powerup of the hub module is enough because Lego will scan all 6 hub ports to see what is connected. Normally after max 6 sec all ports are set to their default configuration and you can listen to and use that data.

If you want to send commands from Lego to FT you can do so by driving the 4 LEDs on the distance sensor. Because you are listening you can see the data immediately and you can execute the commands with the FT RoboPro program.

Now suppose for a moment that you wish to send a command from FT to Lego you can do that using either the distance sensor or the color sensor. An example: with a servo on FT you place a panel at a certain distance from the distance sensor. In the Lego program you can react to this. By placing different panels at different distances you can create more than 1 command. With the color sensor, it would also go by bringing a different color block in front of the sensor as a command.

However, if you use a serial port then the commands are unlimited in both directions. You must then write the necessary software on both sides. With the Mindstorms 51515 word program a serial port is not provided, They have no blocks for this. A solution is to execute it via Python but also there you have to search for the necessary configuration yourself. They didn't provide it by default either!

As you can see, there is quite a bit of work to do. The next 2 weeks I won't really be able to do much testing because the grandkids are coming on vacation. They would like to make one of the original robots version. Then I'll have to put my testing on hold for a while.

Meanwhile, I can do some dreaming about how to make an FT-Lego practical implementation to make a video of it. After all, I need to be able to keep myself busy.

Frans

 

[Toastie]

Good evening Frans,

thank you very much for your excellent explanations!!!

Enjoy your next two weeks as much as you possibly can (time goes by so slowly - for those who wait, as Madonna said). I wish you all the fun!

Best,
Thorsten

 

[fotoopa]

Lego and Fischertechnik linked together.
All 6 ports of the Lego hub are connected to the DE0-Nano-Soc module. This allows all data from Lego to be monitored. Many results can be displayed via the Fischer TXT controller. This is done via a simple program with the RoboPro software. The TXT Controller has a very nice feature where you can make a logging of the signals. This allows you to visualize the data in real time on the PC screen. All lego ports operate at 115200 baud rate. On the Lego side, you do not need to write a program for this because the data logging is done at low level. This is the same data the hub receives from the motors and sensors.

I use the port hubE for serial data exchange between the two systems. Everything is written in Python because in blocks not everything is available. For this serial connection I do need to write a bit of software. A small problem is the configuration at the first startup of the program. Lego uses an auto config system so the port is not ready for serial connections. I need to place an additional delay. Once started up you would no longer need the delay. I now use a delay of 1sec for this. If nothing was found at startup Lego searches every 500 msec for a new configuration. Once you have configured the port as serial everything is solved.

In the Python example, I sent some commands to the FPGA module ($D5). The Pico-scope 2406B is also always connected to the FPGA module. The 4 scope channels can be selected from 4x8 signal sources via the keyboard on the module. This scope thus also has real-time access to all Lego data. I use this mainly for time analysis and for testing the Python programs. In the video you can see how the data is displayed in real time by the TXT. You can see the motor displacement, the motor speed and direction, the sensor distance and the status of the 4 LEDs on the sensor. The leds are turned on and off at certain times. You can see that in the Python program and in the video itself.

Lego and Fischertechnik by Frans, on Flickr

Lego and Fischertechnik linked together. by Frans, on Flickr

 

python-prog-distance-diagram by Frans, on Flickr

 

Update:

RoboPro program added:

FT program for Lego interface by Frans, on Flickr

 

Frans.

Edited April 6, 2022 by fotoopa

[Mr Jos]

Woah that looks very interesting to see how the distance sensor really reacts to a movement. Seems to jump shortly backwards before overshooting. All the rest looks like Chinese for me (even though I worked in China over 6months).

Very curious to see what you will be able to make with this coöperation between 2 nice systems.

[fotoopa]

1 hour ago, Mr Jos said:

Woah that looks very interesting to see how the distance sensor really reacts to a movement. Seems to jump shortly backwards before overshooting.

Yes the value of the distance sensor is not clear to me either. The Lego status only gives cm as unit. Either the higher accuracy is just not usable. If I would display only cm everything is correct. The data measured with the real-time Picoscope is also the same. So the plot curve is not an incorrect representation . I would need to make measurements over a greater distance. For that I need to change the setup. I currently have very few Lego parts.

The movement itself does proceed regularly. You can also see this in the video. If you watch the video frame by frame you can see the digital readout counter display the value. There too the jumps can be noticed.

I will remake this movement with FT parts for displacement though. But first now vacation with the grandchildren.

Frans.

[fotoopa]

Adapters FT-Lego.
To use both motors together, a number of adapters are needed. For example, a FT chain can be placed on a Lego motor and and a Lego chain can be placed on a FT motor. Now we can really start using both systems together. Attaching a Lego part to a Fischertechnik grid is also now possible.

Adapters FT-Lego by Frans, on Flickr

[fotoopa]

Some new images for the FT-Lego assembly:

DSC07899 by Frans, on Flickr

 

DSC07923-ketting by Frans, on Flickr

 

DSC07899 by Frans, on Flick

Parts I don't have are usually made via 3D printing.

Frans

Edited April 13, 2022 by fotoopa
Typo

[fotoopa]

Here is a new test today to link both systems.

DSC07979-Lego-FT-set by Frans, on Flickr

Via 3D printing, a number of additional blocks are being printed to connect the two systems. Now modules can be coupled in both directions. Because FT motors have no absolute encoder I use a proprietary encoder disc with hall detectors. Via quadrature decoding I have 32 pulses/revolution. On the Lego chain, I use small pins that are secured via a screw M2.5x8mm. They are now printed yellow but in the future I will print them black to work better together with the color sensor.

Step by step we go through...

Frans.

[fotoopa]

Some additional 3D printed adaptors for Lego and Fischertechnik.
The next step is a small application. I am in the process of preparing it.

DSC08007 by Frans, on Flickr

DSC07980 by Frans, on Flickr
Both systems have certain advantages. It is now a matter of presenting these in a working solution.

Frans.

[fotoopa]

First small application of the Lego Mindstorms hub. I am also using parts from Fischertechnik for construction here. The base plate is a FT plate. To build FT and Lego parts together I made some adaptors via 3D printing. This way the grid problem between FT (15mm) and Lego (16mm) can be solved. You could also build this application with Lego parts only but that is not my intention. I was able to solve a number of problems during construction. To be able to use more FT modules I first have to cut the flat cables of all the motors and connect them via 6 pin connectors. This way I can increase the distance between hub and motor. Then I can also re-enable my interface which can listen to all Lego Uart signals. From that moment on I can let FT and Lego fully interact with each other.
First the setup:

DSC08056-sorting-balls by Frans, on Flickr

Then the program:

Sorting-colored-balls by Frans, on Flickr

And the video:

Lego Sorting 14mm balls by Frans, on Flickr

The following application will use both hardware modules FT and Lego and the PC screen as the GUI.

Frans.

 

[fotoopa]

Z axis with gripper for colored 14mm balls Version 2.
In this version, the Z axis works much faster. It is now driven directly by the motor.

foto:

DSC08092-gripper-module by Frans, on Flickr

video:

Ball gripper module by Frans, on Flickr

The gripper itself is 3D printed along with a small impact block for opening and closing. The rest are Lego technic blocks and the whole thing is mounted with Fischertechnik parts.

Frans.

[Johnny1360]

Been following this for awhile now, I finally am starting to understand just what you are doing, lol. Very well executed and great seeing it put to use, this is really cool and as you say opens up many possibilities.

[fotoopa]

Finally I can continue working again. My eyes are seeing better again after some operations.
I have now also bought the Lego technic hub ( €25) and the remote control (€17). They are quite cheap to find at the moment.
The technic hub can only work with the battery box. You have to take the batteries out to recharge them. Very annoying especially if you just want to run tests. I now use Pybricks python for this. Now I replaced the battery box with a 3D printed box with an external connection plug for a 9V adaptor. If you want to go back to the battery box you just swap both. The basic cover plate remains the same.
Some pictures give an overview of how I made this.

technic hub powered by Frans, on Flickr

powerbox-3D-view by Frans, on Flickr

hub pcb by Frans, on Flickr

hub pcb en wires by Frans, on Flickr

hub input power side by Frans, on Flickr

hub electodes by Frans, on Flickr

As a power adaptor I use the Fischertechnic adaptor of 9V 2.5A Now I can quietly spend hours learning to work with python without worry. The hub indicates a voltage of 9.39V. To connect the external 9V plug, a small hole must be made in the hub box. If you don't want that, you can bring out both connecting wires on the side and fit a connector there as desired.

 

Update:

link to the .stl file: https://www.thingiverse.com/thing:5557377

Frans

Edited October 10, 2022 by fotoopa

[Mr Jos]

That's an awesome solution! If only I could find the same for the EV3, then I would get like 10+ of them.

Now I'm running all original rechargeable Lego batteries with homemade 12V 5A adapter with 8 recharge plugs on 1 adapter. I can imagine this constant charging isn't doing good for the lifetime, would love to just have dummy boxes directly to the adapter (but then I would need to get a correct voltage [~8/9V?] adapter instead of the 12V I use now I think). DIY'ing the electric part would be fine, but I don't have a 3D printer.

Glad you can continue your research again, have fun!

Edited October 15, 2022 by Mr Jos
Typo

[Toastie]

On 10/15/2022 at 8:53 AM, Mr Jos said:

Now I'm running all original rechargeable Lego batteries with homemade 12V 5A adapter with 8 recharge plugs on 1 adapter. I can imagine this constant charging isn't doing good for the lifetime

Wait: You run the rechargeables: a) with the 12V power supply plugged in almost all the time and just using the rechargeables for power interruptions to keep the EV3 memory alive, or b) you charge them "off-line" and then drain them without power supply attached and then re-charge them with 12V?

Case a) would not do much harm (other than unavoidable, natural material degradation), case b) may stress them much more. I have 8878's here that live "forever" (so far ) being constantly "charged" off from 15V DC rail power on my PF/PUp trains.

Best,
Thorsten

On 10/10/2022 at 10:57 AM, fotoopa said:

Now I replaced the battery box with a 3D printed box with an external connection plug for a 9V adaptor. If you want to go back to the battery box you just swap both. The basic cover plate remains the same.

Very nice(!!!), I believe @dr_spock did something comparable, but much simpler, for the RCX and other intelligent bricks, using 2 AA dummy cells, if I remember correctly. The thing though is: Once power is lost, memory is lost, right? I like to have some buffer power for such situations. In my world, which is trains on 9V track, power is a) dirty and may b) change polarity. This needs insulation and thus power loss. Yes, you can use super caps or the like - but then I also like to bridge larger power outages (i.e. running on all plastic track).

This is why I like the "charging while running" option, which is, as far as I am concerned, brought to perfection in 8878 - and maybe even in the rechargeable packs for the NXT. No idea about EV3 though ...

Best,
Thorsten