[Software] LIFCreator.py - a tool to package folders and files into a lif file

[Equilibrium]

I've tried packing vanilla files and blueprint worked so I it doesn't have anything to do with blocks order after all. Now I'm trying to pack vanilla db with my custom brick which proved to work a year ago. Unfortunately this is gonna take a while.

Edited June 23, 2020 by Equilibrium

[Corellian Corvette]

OK well I noticed something interesting - when I went to use blueprint again it wouldn't work after I replaced with the backup db.lif file.

I had to delete the file in the App Data/Users/Roaming for blueprint and it "recreated" the file which allowed me to work with it using the original db.lif file. 

So I tried this with the new parts .lif file and i got an completely different error

java.lang.RuntimeException: File not found /info.xml

Operating system detected: Windows
Starting Blueprint v0026.. Good luck!
Current folder: C:\Users\Albert Penello.000\OneDrive\Lego Creator\Blueprint0026
Initializing database..
Database is located in: C:\Users\Albert Penello.000\AppData\Roaming\Blueprint\database\brickcache.db
java.lang.RuntimeException: java.lang.RuntimeException: File not found /info.xml
        at blueprint.b.b.a.b.a(Unknown Source)
        at blueprint.b.b.a.b.<init>(Unknown Source)
        at blueprint.b.b.a.<init>(Unknown Source)
        at blueprint.Blueprint.b(Unknown Source)
        at blueprint.Blueprint.initApp(Unknown Source)
        at a.a.start(Unknown Source)
        at com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$8(LauncherImpl.java:863)
        at com.sun.javafx.application.PlatformImpl.lambda$runAndWait$7(PlatformImpl.java:326)
        at com.sun.javafx.application.PlatformImpl.lambda$null$5(PlatformImpl.java:295)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.sun.javafx.application.PlatformImpl.lambda$runLater$6(PlatformImpl.java:294)
        at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
        at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
        at com.sun.glass.ui.win.WinApplication.lambda$null$4(WinApplication.java:186)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: File not found /info.xml
        at lib.ldd.lif.LIFReader.readInternalFile(Unknown Source)
        ... 15 more

 

So where is this /info.xml file? Is this something that's supposed to be in the db.lif file or somewhere else? And why is this not a problem with regular blueprint and only comes up with the backup?

OK one other strange thing - for giggles I tried to use LIFExtractor on the new parts db file, and I got a blank folder. So for some reason on my computer using LIFextractor on the new parts db.lif file doesn't seem to work. 

NEW QUESTION: I looked at the info.xml file in the db.lif file and I was wondering - when you add new parts to db FOLDER, is there a new info.xml file that should be included as well? 

Edited June 23, 2020 by Corellian Corvette
One more thing

[Equilibrium]

When I packed vanilla + my recent release it crashed, but packing vanilla + random 2 of them do work. When I get some time I'll try to debug it.

[Corellian Corvette]

@Equilibrium you're on a PC as well, right?

I wonder what I'm doing wrong that I cannot even get Blueprint to start...

Were you getting the same error I was?

[Equilibrium]

Yes on PC/W10, I had different errors. In info.xml there's only db version.

[Corellian Corvette]

OK so I just tried only adding 3 parts - the main three parts I really need for a bunch of my models. 38640, 38641, and 32803.

When I try and start blueprint I get the same /info.xml error as above. 

It's worth noting again, that if I try and use LIFExtractor on the same db.lif file that was created - I get an empty db folder. Is there perhaps a setting in python I'm doing wrong when creating the file? @Equilibrium what happens when you try and extract the newly created db.lif file? Does it work?

 

[Equilibrium]

It seems like it doesn't pack all the files I'd say. Are you typing "db\" with slash at the end? And does your db folder contain this file for example?

Edited June 23, 2020 by Equilibrium

[Corellian Corvette]

OK something is definitely up and I'm not sure if it's where I'm choosing to compact the file. 

So @Equilibrium it appears that your version of Blueprint likes the 2672 monorail track so I built a new db.lif file with ONLY that part added, and I still get the same error on blueprint (no /info.xml)

I do use the db/  at the end of Python script, and I see the cmd line go through all the files (like you about 15+ min to compile)

At the end, I get a db.lif file that's 905,561 KB. LDD accepts the file.

@M2m would it matter where or how I'm packing the file that would prevent me from being able to unpack?

Ok in short here's where I'm at: 

• I'm able to create a db.lif file, but I'm unable to then UNPACK that same file.
• I have added only 1 piece (2672) to the db.lif file, and LDD does see that piece and it works fine. 
• Blueprint still crashes with only a single piece added, saying it cannot find /info.xml, and I think that may be related to my inability to extract the file that I've created.

Is there a way to see inside the new DB file and see what's going on?

[M2m]

7 hours ago, Corellian Corvette said:

I do use the db/  at the end of Python script, and I see the cmd line go through all the files (like you about 15+ min to compile)

I think on windows you should use the \ (back slash). On Linux, macOS you should use the /

So for Windows:

python LIFCreator.py db\

I'm still trying to figure out why the script is so slow on Windows :(

 

[Corellian Corvette]

@M2m Sorry I was using \ not the forward slash /

python LIFCreator.py db\

Still same error on blueprint, still same unable to unpack using LIFExtractor. 

So I am doing these in a completely different folder (using backup files) would the folder structure have anything to do with it? LDD must be very resilient because your packer works for LDD. For some reason on my PC I'm having an issue...

[SylvainLS]

9 hours ago, M2m said:

I think on windows you should use the \ (back slash). On Linux, macOS you should use the /

Theoretically, Windows has a POSIX compatibility and accepts / too.  But then, it’s Windows….

[Equilibrium]

I have debuged an error I was getting. I know what was wrong, why and how to fix it in relatively fast way.

[M2m]

On 6/24/2020 at 7:59 AM, Corellian Corvette said:

@M2m would it matter where or how I'm packing the file that would prevent me from being able to unpack?

Ok in short here's where I'm at: 

• I'm able to create a db.lif file, but I'm unable to then UNPACK that same file.
• I have added only 1 piece (2672) to the db.lif file, and LDD does see that piece and it works fine. 
• Blueprint still crashes with only a single piece added, saying it cannot find /info.xml, and I think that may be related to my inability to extract the file that I've created.

Is there a way to see inside the new DB file and see what's going on?

Its all a big mystery to me.

For me I can pack a folder with LIFCreator.py and then unpack the resulting file with LIFExtractor.py in Windows 10:

Edited June 25, 2020 by M2m

[Corellian Corvette]

Here's what happens when I do exactly the same thing

C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor>python LIFExtractor.py db.lif
PROCESSING: db.lif
        EXTRACTING: Please wait.
        COMPLETE: 0 files in 0 folders extracted.

C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor>

[M2m]

2 hours ago, Corellian Corvette said:

Here's what happens when I do exactly the same thing

C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor>python LIFExtractor.py db.lif
PROCESSING: db.lif
        EXTRACTING: Please wait.
        COMPLETE: 0 files in 0 folders extracted.

C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor>

How did you create db.lif ?

[Corellian Corvette]

I extracted the original db.lif file that comes as part of LDD. That gave me the big folder, in which I added 1 part - the 2672 monorail track (.g and .xml) that seemed to work in blueprint for @Equilibrium

I make a copy of the new uncompressed db folder and move it to a new folder

C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor

This folder has the uncompressed db folder, as well as LIFExtractor.exe and LIFExtractor.py, and your LIFCreator.py files. 

I run a command prompt (.cmd) and do a change directory so the command prompt is now where your python script is. Then I run this command 

C:\Users\Albert Penello.000\OneDrive\Lego Creator\LIFE Extractor>python LIFCreator.py db\

I get your prompts to start creating the file, wait 20 min (LOL) and get your compacted db.lif file. The file properties show it to be 883 MB (904,642 KB) and it works fine with LDD.

However as mentioned, when I start Blueprint i get the error, and when I try to re-extract the same db.lif file I get the error above. It's almost like I have a huge file that LDD can see, but appears empty to both blueprint and LIFExtractor. 
 

 

[Equilibrium]

On 6/24/2020 at 9:03 AM, M2m said:

I'm still trying to figure out why the script is so slow on Windows :(

Would any of this help?

https://stackoverflow.com/a/48726957

EDIT:

All my bricks are now working

Edited June 25, 2020 by Equilibrium

[SylvainLS]

Just for information: I tried on Linux (my LDD is in Wine) and it was killed because it took too much memory.  I guess you’re building the whole db.lif in memory first.  Might need a warning in the readme or something….

[Corellian Corvette]

OK @Equilibrium GOOD JOB!

Can you do me two huge favors? Can you try three pieces for me and see if they work? 38640, 38641, and 32803. These pieces will unlock a lot of my MOC's with open holes :)

If you CAN get them to work - is there way to share your db.lif file with me in the meantime while I try and diagnose what I'm doing wrong?

[M2m]

3 hours ago, SylvainLS said:

Just for information: I tried on Linux (my LDD is in Wine) and it was killed because it took too much memory.  I guess you’re building the whole db.lif in memory first.  Might need a warning in the readme or something….

Yes. True. Can add that add the start-up and user have to confirm with <Enter>.

Where you able to run it on Linux after all ?

[SylvainLS]

10 hours ago, M2m said:

Where you able to run it on Linux after all ?

I didn’t try again on db/.  I would need to kill other applications first (I’ve 4GiB on this PC, about 1.5GB are free).  It worked on LDDExtended.lif though (2 files…).

It seems to be running fine: file names going rather fast.  It “exploded” after only 20s or 30s for db/.

[SylvainLS]

New info: even with 3+GiB of free memory, it gets oom-killed for a 1GiB db.

How much memory do you guys have?  And isn’t the sluggishness simply due to swapping?

 

[M2m]

Having 8GiB of RAM. As said it it runs in 25secs for me (with all new parts)

[M2m]

On 6/26/2020 at 4:08 AM, Equilibrium said:

Would any of this help?

https://stackoverflow.com/a/48726957

EDIT:

All my bricks are now working

Thanks for the hints. I get that it seems windows python is slow on long strings (and as I build the whole lif file in memory I run into the same problem), but I didn't see a solution :(

Anyway how did you get your bricks working ?

[Corellian Corvette]

Yes please share! :)