[Software] LIFCreator.py - a tool to package folders and files into a lif file

[M2m]

I finished a Python program that (re)creates a lif file from a folder structure. So basically you can first use LIFExtractor.py (by JrMasterModelBuilder) to extract db.lif, then add mods and new bricks and afterwards repack the modified db folder with this tool (LIFCreator.py).

It is tested on macOS 10.14.6 and Windows 10. It worked with LDD 4.3.11, bluerender 0.6 (that is the new added bricks where loaded and rendered) and of course I also tested it with JrMasterModelBuilders LIFExtractor.py (packing with this tool, unpacking with JrMasterModelBuilders LIFExtractor and vice versa). Still I don't guarantee any functionality and suggest you to keep backups (especially of the original db.lif).

WARNING: You accept full responsibility of using it yourself !

https://github.com/sttng/LIF-Creator

How to:

./LIFCreator.py /Users/your_user_name/Library/Application\ Support/LEGO\ Company/LEGO\ Digital\ Designer/db

On Windows type the following into the command-line

LIFCreator.exe "C:\Users\<YOUR USER>\AppData\Roaming\LEGO Company\LEGO Digital Designer\db"

Below a bluerender test (the leaves are a new custom part):

Edited November 29, 2020 by M2m
Add repository

[Corellian Corvette]

As I posed in the other thread THANK YOU.

I used the instructions on this board to drag my .LIF file in an executable to get the db/ folder. I didn't run a python script. 

Assuming I know just enough to get myself into trouble, what would be the suggested process to run this?

I downloaded python for Windows, and get a dos-like Dialog screen but after that I'm lost...

I've tried running LIFCreator.py with the folder location of my db file (I'm using a backup to not screw anything up. 

So I assume with Python installed, I run a command prompt like this

LIFCreator.py \Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor\db\

and nothing happens :(

 

Edited June 21, 2020 by Corellian Corvette

[M2m]

32 minutes ago, Corellian Corvette said:

As I posed in the other thread THANK YOU.

I used the instructions on this board to drag my .LIF file in an executable to get the db/ folder. I didn't run a python script. 

Assuming I know just enough to get myself into trouble, what would be the suggested process to run this?

I downloaded python for Windows, and get a dos-like Dialog screen but after that I'm lost...

I've tried running LIFCreator.py with the folder location of my db file (I'm using a backup to not screw anything up. 

So I assume with Python installed, I run a command prompt like this

LIFCreator.py \Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor\db\

and nothing happens :(

 

I assume that command is more or less correct (which you have to start from the command prompt - cmd.exe)

LIFCreator.py \Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor\db\

Depending on what version and where you installed it you may have to start as ( this assumes you have installed Python2.7 in a directory C:\Python27)

C:\Python27\python.exe LIFCreator.py \Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor\db\

Do you get any error message or any output ?

What happens if you just type

python

into the command line ?

 

Edited June 21, 2020 by M2m

[M2m]

I checked blueprint 0.26 and I had some succes in loading some new parts

 

While on most of my files using new parts I got this error:

Not sure if it is related to particular parts and which ones...

Edited June 21, 2020 by M2m

[Equilibrium]

It doesn't work.

In windows cmd while providing folder path it is required to add " " if the path contains spaces.

"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe" "D:\XXX\Różne\LEGO\Programy\LIF Packer\LIF Packer\LIF_Packer.py" "C:\Users\XXX\AppData\Roaming\LEGO Company\LEGO Digital Designer\db\"

So I did this and started getting error with incorrect file name. Script tried to create file "db".lif" but it obviously failed. I have trimed walk_dir = walk_dir.strip('"') from one quotation mark (print function showed correct path (C:\Users\XXX\AppData\Roaming\LEGO Company\LEGO Digital Designer\db) and script at least started and it immediately fell into some infinite loop using 20% of my cpu and doing nothing else.

[M2m]

28 minutes ago, Equilibrium said:

It doesn't work.

In windows cmd while providing folder path it is required to add " " if the path contains spaces.

"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe" "D:\XXX\Różne\LEGO\Programy\LIF Packer\LIF Packer\LIF_Packer.py" "C:\Users\XXX\AppData\Roaming\LEGO Company\LEGO Digital Designer\db\"

So I did this and started getting error with incorrect file name. Script tried to create file "db".lif" but it obviously failed. I have trimed walk_dir = walk_dir.strip('"') from one quotation mark (print function showed correct path (C:\Users\XXX\AppData\Roaming\LEGO Company\LEGO Digital Designer\db) and script at least started and it immediately fell into some infinite loop using 20% of my cpu and doing nothing else.

Very sorry to hear :(

I'm totally aware that folder parsing in Windows vs Linux vs Unix (macOS) is quite s**t

I made some updates to print out all files it is trying to parse. Also (stupid) macOS comes with python2.7 and you are using 3.7 (I know 2.7 is out of support already - I blame apple for that).

Maybe you can give it a try. Be aware you have to press a key twice for it to start.

Edited June 21, 2020 by M2m

[Equilibrium]

Ok, I trimed walk_dir again, fixed print syntaxes (I forgot to tell you that :P). And looks like it does work. I can see that files are being added, man it's slow... Maybe it worked whole time but I thought it would take only a min :D Anyway output file was db_4.lif but I can't find this file anywhere.

[M2m]

Just now, Equilibrium said:

Ok, I trimed walk_dir again, fixed print syntaxes (I forgot to tell you that :P). And looks like it does work. I can see that files are being added, man it's slow... Maybe it worked whole time but I thought it would take only a min :D Anyway output file was db_4.lif but I can't find this file anywhere.

I'm having a 2017 Macbook with a i5 2.7GHz and 8GB RAM - so nothing too new. It takes about 35 secs on my machine to create the db.lif (which is the original + all the new parts) and the file is around 1.04GB.

It should create the file in the same dir where you started the script from. Now you already ran the script 4 time (db_4.lif). It will check if a file is already there with the same name and create a new one (and not just overwrite the file).

[Equilibrium]

I'm on Ryzen 5 1600AF and 16GB ram so something is wrong. I have found the files... in C:\Windows\System32. All of them are 0 bytes (even the one currently running) so again something is wrong :P

This is how it looks like:

Adding: C:\Users\XXX\AppData\Roaming\LEGO Company\LEGO Digital Designer\db\Primitives\LOD0600880.g1

etc. Shouldn't there be one more slash? After LOD0.

Edited June 21, 2020 by Equilibrium

[M2m]

Hmm... do you get any output ?

For me it looks like
 

m2ms-MBP:LegoToR m2m$ ./LIFCreator.py /Users/m2m/Library/Application\ Support/LEGO\ Company/LEGO\ Digital\ Designer/db/
Choosen directory: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db

Will create: db.lif


Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/MaterialNames/DElocalizedStrings.loc
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/MaterialNames/ENlocalizedStrings.loc
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/Assemblies76537.lxfml
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/Assemblies64786.lxfml
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/Assemblies99415.lxfml
.....

....

Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/MainGroupDividers226.png
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/MainGroupDividers385.png
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/MainGroupDividers218.png
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/MainGroupDividers225.png
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/MainGroupDividers351.png
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/info.xml
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/CurrentMaterials.xml
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/DecorationMapping.xml
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/Materials.xml
Adding: /Users/m2m/Library/Application Support/LEGO Company/LEGO Digital Designer/db/pes.xml

    COMPLETED: 11806 files processed and added to db.lif.

 

[M2m]

22 minutes ago, Equilibrium said:

I'm on Ryzen 5 1600AF and 16GB ram so something is wrong. I have found the files... in C:\Windows\System32. All of them are 0 bytes (even the one currently running) so again something is wrong :P

This is how it looks like:

Adding: C:\Users\XXX\AppData\Roaming\LEGO Company\LEGO Digital Designer\db\Primitives\LOD0600880.g1

etc. Shouldn't there be one more slash? After LOD0.

Yeah - there should be.

I pushed another update into github. I really hoped folder processing is standardized in Python...

[Equilibrium]

File size is still 0 and it goes and goes... Maybe I'm doing something wrong.

File itself is being created depending on where I start the command:

C:\Windows\system32>"C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\python.exe" "D:\XXX\Różne\LEGO\Programy\LIF Packer\LIF Packer\LIF_Packer.py" "C:\Users\XXX\AppData\Roaming\LEGO Company\LEGO Digital Designer\db\"

This will create db in system32, if I cd\ to C: it will be created directly on C etc.

[M2m]

14 minutes ago, Equilibrium said:

This will create db in system32, if I cd\ to C: it will be created directly on C etc.

This is normal. I pushed another update. Do you get any other output with the latest update?

[Equilibrium]

Sizes are being printed in addition to file paths. But now I can't find new db.lif anywhere.

[M2m]

Yeah I pushed the writing to the end of the program (so the filehandle isn't open forever). What Windows are you using ? I might install Win in VM. Seems to be more fruitful...

Anyway I just pushed another change. Maybe you can have a look where it stops.

 

 

[Equilibrium]

Windows 10, it doesn't stop or error if I fix print syntaxes and trim " from directory. It just goes on and on. After 10min I was at LOD0\24XX. I'm just terminating it. We need someone else to test this on windows as well.

[Corellian Corvette]

I'm using the new script you uploaded and I still can't get anything to work.

I installed Python 3.8.3 x64 version. In any command prompt, if I type py I get the python terminal (this was the test they recommended to make sure python was installed.) 

I have all the files, and the script, in the same folder LIFE EXTRACTOR

This is the path to the folder:  C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor

This has the db folders, and the LIFCreator.py script.

I open up a CMD window and move to the directory with the files so my command prompt has this

C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor>

Per above, I'm using quotations since both my user and several folders have spaces

C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor>LIFCreator.py "C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor\db\"

Absolutely nothing happens. Just goes to the next command prompt.

Is there a test of the actual LIFECreator.py script just to make sure it's working?

Thanks! Just trying to help test. 

 

[M2m]

Ok I made a couple of changes and for me its working on macOS Python 2.7 and Win10 Python 3.8 (current).

https://raw.githubusercontent.com/sttng/LDD/master/LIFCreator.py

I first installed Python 3.8.3 on Windows 10 (with the standard options and selected to have python added to my path)

Then I start the Command Line (cmd.exe) and go to the "Downloads" folder (cd Downloads) where I have the DB folder and the LIFCreator.py script as shown below in my Win10 VM (Python installer is also still there)

To confirm python is working i type python and after i press CTRL-Z to exit the python environment:

Next I start the LIFCreator program with

python LIFCreator.py db\

 

I get a couple of outputs and hit <ENTER> to confirm all (Choosen directory: db, Will create: db.lif) and then it will start and list all files and sizes it processes. On my VM this is super sloooooowwwww. On my mac it takes approx 30sec.

The last file to be processed is pes.xml - after this is will write the lif file:

Done :)

 

Comparison on my Mac (not VM) - just 21secs

 

Edited June 22, 2020 by M2m

[Corellian Corvette]

Ok well... Good news and bad news!

The *good* news is that I was able to get the script to run and create a db.lif file using your new instructions. The problem on Windows 10 is that I needed to install the Windows Store version of Python to be able to run the script in a command prompt. Not sure why, but it works exactly on my PC as it does on your Mac and mine takes a LONG time as well. But still, it works!!

I was able to use that file to run LDD and was able to open up the "Custom Parts Test.lxf" file.

So as far as LDD goes, it appears the .LIF file is working!!

The bad news is that the intended reason to do this (to get Blueprint working) still has an error. 

When I launch Blueprint, I get the following error dialog box:

"Header checksum broken. Store was not closed correctly and might be corrupted. Use 'DBMaker.checksumHeaderBypass()' to recover your data. Use clean shutdown or enable transactions to protect the store in the future"

In the command window I see:

Operating system detected: Windows
lengths.txt file not found.
Starting Blueprint v0026.. Good luck!
Current folder: C:\Users\Albert Penello.000\OneDrive\Lego Creator\Blueprint0026
Initializing database..
Database is located in: C:\Users\Albert Penello.000\AppData\Roaming\Blueprint\database\brickcache.db
org.mapdb.DBException$DataCorruption: Header checksum broken. Store was not closed correctly and might be corrupted. Use `DBMaker.checksumHeaderBypass()` to recover your data. Use clean shutdown or enable transactions to protect the store in the future.
        at org.mapdb.StoreDirectAbstract.fileHeaderCheck(Unknown Source)
        at org.mapdb.StoreDirect.<init>(Unknown Source)
        at org.mapdb.StoreDirect$Companion.make(Unknown Source)
        at org.mapdb.StoreDirect$Companion.make$default(Unknown Source)
        at org.mapdb.DBMaker$Maker.make(Unknown Source)
        at blueprint.b.b.a.b.a(Unknown Source)
        at blueprint.b.b.a.b.<init>(Unknown Source)
        at blueprint.b.b.a.<init>(Unknown Source)
        at blueprint.Blueprint.b(Unknown Source)
        at blueprint.Blueprint.initApp(Unknown Source)
        at a.a.start(Unknown Source)
        at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$8(LauncherImpl.java:863)
        at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$7(PlatformImpl.java:326)
        at com.sun.javafx.application.PlatformImpl.lambda$null$5(PlatformImpl.java:295)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.sun.javafx.application.PlatformImpl.lambda$runLater$6(PlatformImpl.java:294)
        at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
        at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
        at com.sun.glass.ui.win.WinApplication.lambda$null$4(WinApplication.java:186)
        at java.lang.Thread.run(Unknown Source)

FYI blueprint doesn't work either...

 

[M2m]

Did blueprint work with the original db.lif file for you ?

As written before I got blueprint working with some new parts but also not all. With quite some of the new parts it simply crashed. So I expect these problems most likely are related to the way blueprint handles parts and the new parts may be just so slightly different that blueprint gets confused (LDD in all definitly is just the more professional software and just is more resilient to data not 100% 'compliant').

[Corellian Corvette]

Yes, it did work. For some reason, with the new db.lif file the program won't start :(

I guess now it's will have to be up to @msx80 to hopefully take a look. I know it's been a long time since he's worked on the program but perhaps he can make a tweak to get this working!

[M2m]

5 hours ago, Corellian Corvette said:

Yes, it did work. For some reason, with the new db.lif file the program won't start :(

I guess now it's will have to be up to @msx80 to hopefully take a look. I know it's been a long time since he's worked on the program but perhaps he can make a tweak to get this working!

I made some more performance related tweaks, so the creation of lif files should be faster. But I don't think it'll help with Blueprint.

I checked now all other applications I know which rely on db.lif (LDD, Bluerenderl, busufl) and all of them accept the packaged db.lif file. Below a busufl render - so for blueprint I think it is more about how it (not) handles the new parts it self and not about the lif file.

Edited June 23, 2020 by M2m

[Equilibrium]

Ok, I managed to successfully run it. Took 20 min :/

Looks like order of data blocks is incorrect. First block of data inside data block in vanilla is assemblies while yours outputs CurrentMaterials.xml maybe that's why blueprint crashes as it expects certain order of data.

[M2m]

33 minutes ago, Equilibrium said:

Ok, I managed to successfully run it. Took 20 min :/

Looks like order of data blocks is incorrect. First block of data inside data block in vanilla is assemblies while yours outputs CurrentMaterials.xml maybe that's why blueprint crashes as it expects certain order of data.

Hmmm... might be. Then on the other hand blueprint is working for me - just not with the ALL the new parts. But some DO work. So I'm more expecting blueprint has problems in loading parts then the lif file. I'm still surprised the lack of speed for the conversion as it honestly only takes 20-30sec for me.

Edited June 23, 2020 by M2m

[Corellian Corvette]

3 hours ago, Equilibrium said:

Ok, I managed to successfully run it. Took 20 min :/

Looks like order of data blocks is incorrect. First block of data inside data block in vanilla is assemblies while yours outputs CurrentMaterials.xml maybe that's why blueprint crashes as it expects certain order of data.

So I wonder if this is the problem I seem to recall on another thread/post somewhere that the order of the folders in the LIF file matters for some programs. It's also possible that the windows version is more restrictive on this than the Mac version as, IIRC, the Mac version is a port. 

Is this an easy change @M2m? If so I'm happy to test it again!