Documenting the LEGO PoweredUp! System

[treczoks]

Hi!

A few days ago I joined to find answers on the communication with the elements of the PoweredUp! train control system. Well, I didn't get a reply, and found the solution myself.

Yesterday, I made a reverse engineering session and wrote a document listing all the information I could find - hopefully in a concise way. I also added some questions both about the Handset as well as the Smart Hub. Questions I intend to direct at LEGO, too, via the LEGO Ambassador Network.

I want to thank Jorge Pereira of the BOOSTreveng project for helpful pointers about the command structure - The old "I could only see that far because I'm standing on the shoulders of giants" definitely applies here.

I will continue my work on this document - I'm waiting for the BOOST color sensor to arrive, and want to get my hands on the motor used in the Batmobile - and if someone here has suggestions or additional information, don't hesitate to contact me.

PoweredUp! Documentation

The next step will be the design of a going-in-between app, which allows me to control my trains with the PUP Handset, but relayed over a computer (maybe a Raspberry Pi) that does things like drive both motors simultaneously, control the lighting, or react to the readings of the color sensor.

UPDATE December 16th, 2018: Expanded the documentation with new findings: Battery power readings and conversion factors, blinking and shutdown thresholds. Document is now Version 0.3

Edited December 16, 2018 by treczoks
Document Update

[sed6]

There is a similar thread here you might find of interest. Your paper looks very complete. I bet you'll find some good reading here. 

 

Edited December 11, 2018 by sed6
cnat spele

[treczoks]

5 hours ago, sed6 said:

There is a similar thread here you might find of interest.

That was actually the thread where I started. But that thread is going into too many directions at the moment, so I started a new one to get a new focus (one of the problems I see with a linear forum like this instead of a tree-like one).

This thread here is just about the BTLE communication of the PoweredUp! system, not about hardware tear downs, battery sizes, or prices for the individual parts.

[imurvai]

Hi @treczoks! Thanks for the doc you made. I've already created an Android and iOS app that can utilize the PUP hub but it has a little limitation: I can't operate the two outputs with a single characteristic write. Do you know if there is a way to send commands on both the outputs with one single command?

[JopieK]

We could create (or maybe you already did) a git repo for protocol specifications? A fusion between the four platforms (PoweredUP, Wedo 2.0, Boost and Duplo) would also be a good idea I think, but I lack the time because of my other research project and hoped that some students of mine would be picking it up but for now that is not the case. 

[treczoks]

34 minutes ago, imurvai said:

I can't operate the two outputs with a single characteristic write. Do you know if there is a way to send commands on both the outputs with one single command?

Sorry, no, I don't know any such command. But why don't you just send two commands? The communication is quite fast, fast enough that with all the inertia and the tolerances of a LEGO construction you'll probable never notice a difference.

35 minutes ago, JopieK said:

We could create (or maybe you already did) a git repo for protocol specifications?

Call me old-fashioned, but I really don't like GIT. I prefer a classical linear document or a hypertext system over GIT with its funny document formats at any given time.

A Wiki would be a more agreeable and natural place for such a collection of information. But I don't have one, either.

[imurvai]

1 hour ago, treczoks said:

Sorry, no, I don't know any such command. But why don't you just send two commands? The communication is quite fast, fast enough that with all the inertia and the tolerances of a LEGO construction you'll probable never notice a difference.

Yes, this is what I'm doing but it's not fully sufficient eg. for the batmobile.

[AVCampos]

Yes, from my subjective observations with the Batmobile and the official PUp app, technically it's possible to activate several outputs at the same time.

It would be really useful indeed if TLG published their specs for the PUp protocol. I'm optimistic about that, given their history of openness with PF and MINDSTORMS.

[treczoks]

2 hours ago, AVCampos said:

Yes, from my subjective observations with the Batmobile and the official PUp app, technically it's possible to activate several outputs at the same time.

It would be useful if someone with a BT sniffer could observe the Batmobile communication, then.

And if one is doing that, I would not mind to learn the answer to question 9 on the hub about the actual power levels associated with the +-10 speeds on the remote.

2 hours ago, AVCampos said:

It would be really useful indeed if TLG published their specs for the PUp protocol. I'm optimistic about that, given their history of openness with PF and MINDSTORMS

Well, I posted the document on LAN, too, with the explicit idea that the community team forwards it with all the questions to the developers. I would not even mind if they just sent some half-baked notes - we'll figure it out.

[AVCampos]

1 minute ago, treczoks said:

It would be useful if someone with a BT sniffer could observe the Batmobile communication, then. 

If a BT sniffer doesn't require hardware, I may be able to help: I have a Batmobile, a Windows PC with integrated BT 4.0 (I think it's also BLE; any way to check?), and several Android devices with BLE. What software would be needed?

[Mr Hobbles]

Nice documentation! I like the style, very traditional network protocol style which I find very easy to read. It’s good to have it all written down somewhere. :)

You may be interested in this Node.js library I’ve been working on, which provides most of the documented functionality, and brings compatibility between WeDo 2.0, Boost, Powered UP, and the Duplo Trains - https://github.com/nathankellenicki/node-poweredup

[treczoks]

2 hours ago, Mr Hobbles said:

You may be interested in this Node.js library I’ve been working on

I definitely am.

And I already found some oddities.

You claim in your source that the "power level" delivered from port 3b on the Handset and port 3c on the Smart Hub is (n/4096*100)%

This does not really relate to what I measured:

Handset reading from port 3B: 0X0b50=2896 => According to your formula ~70%. While my voltmeter might not be the best, I'd still say that 5.98V (of 6V) is a bit better than 70%

Smart Hub reading from port 3C: 0x0d50=3408 => According to your formula ~83%. Measured voltage: 8.58V (of 9V). A bit closer, but no cigar. So either the formula or the readings are off. I don't know which, but at the moment I would not bet important body parts on the (n/4096*100)% formula. :-)

And no, I don't have a solution for those readings, either. I suspect that the truth is a bit more complex, but for that I would need some more readings.

Where did you get this information about the supposed voltage level and the current readings on port 3C of the Handset?

Another interesting information I got from your source is the virtual port 57, but that seems only be available with two identical motors attached to the Smart Hub. Looks like I have to pillage another motor to test this - it it even works with two train motors.

Well, at least I now know how typescript looks like. And I'm amazed by the amount of information you must have had about the hardware to write all this.

[Mr Hobbles]

2 hours ago, treczoks said:

I definitely am.

And I already found some oddities.

You claim in your source that the "power level" delivered from port 3b on the Handset and port 3c on the Smart Hub is (n/4096*100)% 

This does not really relate to what I measured: 

Handset reading from port 3B: 0X0b50=2896 => According to your formula ~70%. While my voltmeter might not be the best, I'd still say that 5.98V (of 6V) is a bit better than 70% 

Smart Hub reading from port 3C: 0x0d50=3408 => According to your formula ~83%. Measured voltage: 8.58V (of 9V). A bit closer, but no cigar. So either the formula or the readings are off. I don't know which, but at the moment I would not bet important body parts on the (n/4096*100)% formula. :-) 

And no, I don't have a solution for those readings, either. I suspect that the truth is a bit more complex, but for that I would need some more readings.

Where did you get this information about the supposed voltage level and the current readings on port 3C of the Handset?

Another interesting information I got from your source is the virtual port 57, but that seems only be available with two identical motors attached to the Smart Hub. Looks like I have to pillage another motor to test this - it it even works with two train motors.

Well, at least I now know how typescript looks like. And I'm amazed by the amount of information you must have had about the hardware to write all this.

Most of the voltage and current readings I've estimated based on whats looks reasonable, as I haven't done any measurements with a voltmeter. :) If you have a better formula, I'll take a look!

I have access to all the hardware as I've bought it, but I have no inside links within Lego for information about the hardware. It's all reverse engineering. I also worked with Jorge, as well as a few others as mentioned in the credits - we have a mostly quiet Slack chat we use when new hardware is released!

Yes, virtual port 57 only appears when two identical *motors* are installed. This allows you to send a single motor command and drive both at the same time (Different from command 0x60, which allows you to control both ports independently in a single command). Note there is an oddity: if you plug in two Boost motors, the PUP hub fully crashes (blinky lights, disconnect, reset). I don't think it properly knows how to handle a device that is both a motor and a sensor at the same time.

Edited December 12, 2018 by Mr Hobbles

[treczoks]

5 hours ago, Mr Hobbles said:

I don't think it properly knows how to handle a device that is both a motor and a sensor at the same time.

That may be the key - the Boost motor has its own processor, and communicates via serial with the Smart Hub. I would not be surprised if the Smart Hub only had one UART to spare, and gets into a badly handled scheduling conflict.

[treczoks]

6 hours ago, Mr Hobbles said:

Most of the voltage and current readings I've estimated based on whats looks reasonable, as I haven't done any measurements with a voltmeter. :) If you have a better formula, I'll take a look!

OK, I just asked - I can take the department's good power supply home for the weekend. I can then feed the Handset and the Smart Hub with whatever voltage I like, and get good readings on voltage and current. I'll make a measured series for both devices, especially to get them into low power conditions, and see what happens. So either I'll find a good formula over the weekend, or I will just drop the list of readings on the net for the world to figure it out.

[JopieK]

On 12/11/2018 at 11:31 AM, treczoks said:

Call me old-fashioned, but I really don't like GIT. I prefer a classical linear document or a hypertext system over GIT with its funny document formats at any given time.

But it is the software industries standard nowadays, from Maker movement to Windows / Linux / Webtech / Embedded / Android / iOS programming. So I understand your undertakings are just a hobby but it would be much more beneficial to have everything in one place and not across Eurobricks / 1000steine / whatever.

[treczoks]

1 hour ago, JopieK said:

But it is the software industries standard nowadays

Not necessarily. I still prefer SVN.

[Toastie]

1 hour ago, JopieK said:

But it is the software industries standard nowadays, from Maker movement to Windows / Linux / Webtech / Embedded / Android / iOS programming. So I understand your undertakings are just a hobby but it would be much more beneficial to have everything in one place and not across Eurobricks / 1000steine / whatever.

That may very well be - but "having all in one place" is not necessarily an "all in one cryptic place". I do not pretend that I know how to program, none what so ever. I believe though that I can comprehend a linear "PF 1.X protocol document" for example created by TLG. Or stuff provided by Dave Baum. Or Dick Swan. Trying to figure out what the Github files so many extremely well educated and experienced folks are wanting to tell me, is (for me) another world. I simply don't get it.

When TLG released the PF protocol documents, I began to look into that. With - I believe - some success. It was like the last "linear VB6 Documentation" - 4 rather thick books documenting the things VB6 can do. Don't blame me on still using VB6 - the net is full of discussions regarding that issue. 

Trying to figure out what PuP is all about is - for me - still completely mysterious. Is it "Serial over BT (LE)"? Is it something else? May I access a hub via Microsoft's "mscomctl" calls? TLG provided the USB driver for the "new" IR tower - with all supporting documents. They cancelled any support because - of what??? Bricks live forever, but their software is supported like next to nothing.

So yes, you Gurus (this is meant in a >fully< positive way) know how to handle such things. I am closely following the "PuP Tear down" and these and that related threads - but I do not understand a thing. This is - to me(!) - so weird, as I was able to (partly but mostly) follow RCX, NXT, EV3 stuff.

All the best,
Thorsten 

[TrainDragon]

I have been looking at this from the perspective of the physical wires that connect the Hub and the peripherals.  The two devices communicate over the "ID" pins via standard serial UART, 115200 baud.  The train motors wire the ID pins to Vcc and Ground, as there is no onboard processor to communicate with.  But the motor from the Batman build appears to be identical to the one from the BOOST Creative Kit, which I have disassembled and analyzed.

I have been working on a plugin for the free and open source "sigrok" signal analyzer tool that will decode the signals from the wires and turn them into something human-readable.  I hope to have something published on GitHub soon.  I want to clean up the code and add some comments before I post it, but it will be at https://github.com/dracode/sigrok-lego-boost when it's ready.

I'm sure it will not surprise you to learn that many of the bytes I'm seeing on the wire are identical to the bytes contained in the BTLE messages.    I was able to decipher several message types by reading the reverse engineering documentation that others have written for WeDo/BOOST BTLE.

[treczoks]

4 hours ago, TrainDragon said:

I have been looking at this from the perspective of the physical wires that connect the Hub and the peripherals.

That would be something I wanted to do in the future, but with a different approach:

4 hours ago, TrainDragon said:

The two devices communicate over the "ID" pins via standard serial UART, 115200 baud.

Here I would just have taken a SoC, maybe an arduino, and have him take care of this. If it is "just" a serial, then programming a signal analyzer might be a bit much.

4 hours ago, TrainDragon said:

I hope to have something published on GitHub soon.  I want to clean up the code and add some comments before I post it, but it will be at https://github.com/dracode/sigrok-lego-boost when it's ready.

And yet another github repository...

I think it is time to find something better than github for documenting all this. There now is a bunch of githubs, all different, caring about one product family, with varying degrees of accuracy and being current (or not...)

[JopieK]

15 hours ago, treczoks said:

Not necessarily. I still prefer SVN.

Well I'm not that legalistic about git or something, so SVN is also a super idea I think!

[Bartosz]

16 minutes ago, JopieK said:

Well I'm not that legalistic about git or something, so SVN is also a super idea I think!

Right, I think it's important not to confuse the tool used for collaboration (git/github, svn, wiki) with the format itself. The source code versioning tools are just tools for keeping files there and tracking updates, they also allow for contributions.

The actual formats of the files are entirely up to the user - the text formats are preferred because it's easier to track changes for them. And here I guess the problem lies - to generate PDF like above, one would want to use the tool he prefers, but that's problematic (unless it'd produce some intermediate text file from which PDF could be generated)...

It's also important that workflow that works now, might not work in the future. If @treczoks is commited to do some more reverse engineering and putting info in that document, then there might be no point in forcing him to some other tool. On the other hand, if, after some time he stops, then such document might easily get lost, as no one would be able to contribute there...

Edited December 13, 2018 by Bartosz

[treczoks]

3 hours ago, Bartosz said:

If @treczoks is commited to do some more reverse engineering and putting info in that document, then there might be no point in forcing him to some other tool. On the other hand, if, after some time he stops, then such document might easily get lost, as no one would be able to contribute there...

At the moment, the document is in an intermediate stage: I'm still researching the whole issue, and the whole research results are collected in one .odt file. This is neither a perfect, nor a permanent solution. But it has become clear to me that we need to bring this documentation thing to a whole new level on the long run, in a way that allows for cooperation. One solution would be a wiki, maybe with limited write access for a certain group of people only. I've done some wiki writing, so it would be no problem to turn my findings into a wiki.

But then we need other people to contribute what they found on devices they have (and I don't). And it has to be checked and double checked - basically peer reviewed like proper scientific research.

[Bartosz]

2 hours ago, treczoks said:

odt file

So, to have it git/svn friendly, you can store it as fodt: https://wiki.documentfoundation.org/Libreoffice_and_subversion

[TrainDragon]

16 hours ago, treczoks said:

Here I would just have taken a SoC, maybe an arduino, and have him take care of this. If it is "just" a serial, then programming a signal analyzer might be a bit much.

Well, when I began this, there was no information on the method used for communication.  As far as I can tell, nobody else has ever published any information about the wired signals, only the BTLE.  So hooking up the probes of a logic analyzer was the best choice to quickly figure that out. :)  The protocol decoder grew out of that.  You absolutely could replicate this with an Arduino or similar, sure.

 

16 hours ago, treczoks said:

I think it is time to find something better than github for documenting all this. There now is a bunch of githubs, all different, caring about one product family, with varying degrees of accuracy and being current (or not...)

Absolutely.  I think the Wiki suggestion is the best for documentation.  It's easy for many people to collaboratively edit.  I view my work as source code, not documentation in itself.  There may be messages that I've decoded that aren't replicated in any other project, and I'd be happy to contribute documentation for those on a Wiki (or wherever the consensus ends up.)