S_Bartfast

Very good @deraven! Yes, I dare say you are correct. I think it must have been the Dragon that I was thinking of. Thanks:

I've been trying to remember a set for days and just can't recall what it was. It's a large classic set consisting many, many of 2x4 bricks of all the same colour (yellow I think) and the model was a large duck or something. It's an old set and builds one of those pixilated sculpture type of models, kind of like this: It's not this model in particular, I just posted it as an example of the style of model I'm thinking of... I'm pretty sure the one I'm trying to recall was an official Lego set and as I say I think it was a large duck or something but I just can't seem to find it. Any ideas?

To be clear, the reason Brickficiency was slow is not because of the page scraping. Admittedly the scraping was slow but the great majority of the time was in the actual searching algorithm itself, which is necessarily very computationally intensive. As it is I find it hard to imagine how an exhaustive search (such as what Brickficiency does) could be made much faster. Mdoupe has most certainly implemented several clever optimisations to reduce the search time as much as he could and, while there are no doubt still other optimisations available that are yet to be implemented, doing an exhaustive search of the entire space (which is what is required to be guaranteed to find the best solution) is unavoidably expensive. There are however several non-exhaustive search methods available that focus on just key parts of the space. On the whole these non-exhaustive searches do a remarkably good job in just a fraction of the time, they are however not guaranteed to find the best solutions - though they can get surprisingly close. It is no doubt one of these non-exhaustive algorithms that BrickLink must be using to get the results they do.

There are a few places you could ask. Probably the best would be 'mkdir's subredit: https://www.reddit.com/r/brickficiency/ Of you could create an issue on the GitHub project page itself: https://github.com/WindmillHelix/Brickficiency2/issues/13 Oh, I see you've already done that - any got a solution :), well I guess that makes the rest of my post kind of redundant. The problem seems to be with your "Settings.xml" file. It will most likely be in: C:\Users\<username>\AppData\Roaming\Brickficiency\Brickficiency-Settings.xml or similar. See if you can't workout the problem my examining or deleting and replacing the file. If you're still having trouble post a copy of your Settings.xml file some where and I'll take a look at it myself. All the best, Slarti. By the way, what is going on with the formatting in this forum? I appreciate there implementing an upgrade or something but what's with all the broken links and images.

Yeah, it's kind of easy to miss that one. Enjoy ;)

And here is a direct link to the package: https://github.com/W...iency_2.0.0.zip

It is, and I've done it for you ;) You should now find a file called "Debug-parsesource.html" in your 'debug' folder which will be filled with the last page that was parsed for pricing data. I also corrected what I believe was a small typo in regard to extraction colour options for pieces. There was a line that read: foreach (Match colourmatch in Regex.Matches(page, "http://www\\.bricklink\\.com/catalogPG\\.asp\\?" + item.type + "=" + item.number + "&colorID=([0-9]+)")) { However the page being parsed actually contains a capital 'C' for "ColorD=..." and seemed to be being missed by this search. Hopefully this will help with your issues. In particular I found the optimisation was getting tripped up by rck004 (Jet - Trans-Neon Green Visor), probably because it's colour is assigned as "(Not Applicable)". With the modification I was able to get the optimisation to run however no 1, 2 or 3 store solutions could be found to satisfy all your pieces. After 'excluding' all pieces that were available from fewer than 100 stores, but adding back all 4 pieces that had their colour set to "(Not Applicable)" Brickficiency was able to find 14 solutions. Anyway, here's a link to the 'Alpha' again to save you hunting back to find it: https://dl.dropboxus...storesAlpha.exe

Curious... It's a little hard to debug without being able to reproduce the problem, but it appears that the issue is that a page is received however the page does not contain the information expected. In particular the source for the page received does not contain the string: <B>Currently Available</B> What do you get when you click the "Show Bricklink Price Guide" option from the context menu: This should open the following page in your browser: http://www.bricklink...2013&colorID=11 Does that page open properly? BTW the "roaming/brickficiency/debug/debug-priceguide.txt" file is empty for me as well.

Hmm, well, I'm afraid all I can say is "Works for me":

Arr, I see. I'd thought it was something I'd done. Care to post us a parts list and I'll investergate.

Hmm, curious. Is the original program working and performing the optimisations for you properly? I did add a little fix to do with receiving blank pages from BrickLink as I was getting "Excpetions" while I was debugging. I have now removed the "little fix" to set it back to be the same as the regular program. Try downloading the 'exe' again and see if it's now behaving the same as the original. If that still fails I'll swap it over to a "Debug" version (rather than "Release") which is what I believe the original executable was compiled as. BTW, I'm using Win10 as well.

Okay, here is a link to the 'alpha' build I mentioned above: https://dl.dropboxus...storesAlpha.exe You should be able to just copy this executable into the same directory as your regular "Brickficiency.exe" file and if you want to give the alpha a look just run the "Brickficiency - AvailstoresAlpha.exe" instead. Note, you should still be able to run the regular Brickficiency program in whatever way you have been previously. Just be aware that looking up the number of available stores for each piece takes a little time. It's not much of a problem when you add pieces individually but if you import a whole list the program can bog down for quite a while as it fetches all the pages. Note that the 'alpha' program fetches the piece information in the same way the regular program does (which it does when you click the "Calculate" button) however it does this when the pieces are added rather than just before the optimisation is carried out.

I was working on that and I got it to a kind of "proof of concept" stage which was when I posted the screenshot. However, not long after that there was this post from mdoupe: So I just kind of gave up on the project. There were a few issues with my 'alpha' to do with prompting for the users password and things so I wasn't intending to let it loose on the public and I just kind of let it slide when I saw the "impending obsolescence" comment. I do however have some spare time on my hands again now so I could take another swipe it it if it's worth doing. Unfortunately the code I attempted to commit doesn't seem to have made it to repository after all so I'll have to start a fresh, but I certainly can do it if theirs interest.

Yeah, it was designed with King of Tokyo in mind. For those that don't know King of Tokyo is a dice rolling game that uses oversized dice and this tower has been designed to accommodate KoT sized dice (these are the dice used in the videos). While 6 KoT dice work almost flawlessly 7 and even 8 dice do fit however there is an increased chanced that one of the dice may spill out of the tray at the bottom or end up stacking ontop of one of the other dice. It's not really a problem when this happens as it's still clear what was rolled and easy to resolve but it does tend to happen more when more than 6 dice are used. Note it's not that common to roll 7 dice in KoT and very rear to need all 8. Rest assured however this tower will handle itit should all 8 be required ;) As random as dice rolls are. I haven't done any statistical sampling or anything but there are enough baffles and edges in there to give the dice good scattering.

Okay, I've finally finished this project. Here is a video showing the completed tower in action: And here is a clip showing just the tray:

Alright, finally got back to taking another look at this. This is what I've been working on: It still has some stability issues but is this going to satisfy what you are after?

I have been working on something related to this. Rather than counting sets and quantity and such it simply queries BrickLink and determines the availability by the number of stores that currently have that part in stock. As it is, when the report is generated, this number is display for each part at the top but I have been working on a way to fetch this information when pieces are added to the list rather than as part of the optimisation and displaying this information to the user in a "Stores Available" column. You can sort by this column in order to see the least available (i.e. the rarest pieces) and optionally exclude them from the optimisation if you wish. I discussed the idea in an earlier post: Rather than lookup the availability of all the pieces in one batch however I have been working on looking it up when pieces are added to the list. I have the general concept working on my machine but it a bit flakey at the moment. Does this approach satisfy what you are after?

Okay, I have a solution. It's not what you would call an overly elegant solution but so far as I can tell it works and I have pushed it to the repository.

Arr, yes. I had come across that issue before. I don't think that one's a result of any changes with BrickLink though as I'm pretty sure I encountered some pretty weird behavior around that before. I never quite worked out exactly what was going on but even after increasing the quantity to a non-zero value weird things still continued to happen and a restart was required. At the moment I'm trying to workout a solution to prevent the table rearranging the rows mid operation. As in if you sort by "condition", select all the elements and change their "condition" value, only half of the rows seem to get modified. I'm pretty sure it's because the rows get rearranged mid operation and some of the rows get set twice while others miss out altogether. I'm not exactly sure what can be done about it but I'll get back to you when I have a solution. Cheers, Slarti.

Yeah, that definitely would be possible. Things inside are a bit of a monolithic blob at the moment and trying to tease out the 'configuration' elements from the mass would be a bit of a challenge but it is certainly doable. On the whole though it is all remarkably well conceived for someone who is not a programmer and I must say I am impressed with mdoupe's effort. :)

That would be possible but it probably won't be all that beneficial. For instance this is the expression that gleans the price/availability information from the web page: .*<TR ALIGN="RIGHT"><TD NOWRAP>.*?<A HREF="/store.asp\?sID=(\d*?)&.*?<IMG SRC="/images/box16(.).png".*?TITLE="Store: (.*?)" ALIGN="ABSMIDDLE">.*?</TD><TD>(\d*)</TD><TD.*? \D*([\d,]*)\.(\d+)$ This expression is used to extract 6 values from a line such as this: (The values it extracts are shown in red) While it wouldn't be too difficult to store this expression in a user editable file the likelihood of a non-programer being able to maintain it should the format change is slim, and, well, if it's going to be maintained by a programmer then they might as well just edit the source.

Woo-Hoo! Orrr, Hang on... it seems to still be working fine for me. Is anyone else having this problem? If so could I get you to post your BSX file? Reading someone else's regular expressions is kind of like reading tea leaves (in that you just make up what you want it to say and hope for the best...) but given a hard night's hacking and a healthy dose of "single-malt" I'm fairly confident I'll be able to knock something out. So anyone able to provide details of how to get Brickficiency to fail consistently? The night is young and I feel the urge for some hacking (not to mention some single-malt )

Yeah, I have spent a lot of time thinking about these issues. Part of the problem with showing 'best effort' results (as in fewest leftovers) is that we then start to have multiple objectives and multi objective optimisation quickly becomes an unwieldy beast! As the program is currently it does several brute force optimisations, one with a single store constraint, one with a two store constraint, one with a three store constraint etc. In a way this program kind of achieves a multi objective optimisation by performing several single objective optimisations then ranking the results. If you start to add 'fitness' metrics that take 'near misses' into consideration this increases the search space immensely. As in, if you had say 100 distinct pieces in your list then there are 100 different ways you could exclude one piece (which you could possibly still bruit force) but there are 9900 different ways to exclude 2 pieces and almost one million ways to exclude three pieces, and each of those different lists would still need to have the entire optimisation run on it, and if you were able to do all that you would still have a problem of how to rank the results. Now admittedly you wouldn't necessarily have to brute force all possible combinations of pieces to leave out but never the less, if you start to consider 'near misses' things can get out of hand very quickly! In terms of thinning out the list to give Brickficiency a better chance to find valid combinations of stores I would be inclined to leave this up to the user. If it were upto me I would have the data gathering step and the optimisation steps as two separate processes. As in, once the user has completed their list I would have the program interrogate the database to determine availability, and possibly pricing information, of each of the pieces and display this to the user before the optimisation is commenced. If any parts are unavailable or of very low availability the user could either exclude these from the list or make appropriate substitutions before starting the optimisation. This availability information should be relatively easy to include in the table and once obtained it should allow the user to sort their list by 'availability' and manually exclude all those parts that are unavailable (or have an availability of less than 25 if you wish ). If this could be done with a group select it would be easy enough for a user to do and I think it would essentially achieve what you're after. I'm also not sure what setting the state of pieces to "Extra" (the 'blue cross') is meant to do but it would be nice if it meant they were not taken into consideration when performing the optimisation however when complete, if any of the stores happen to have any of the "Extra" pieces available they could be added to the order. It would effectively mean they were "Excluded" for purposes of the optimisation but in the final report they are noted as either being available from the given stores or not and could be added to the order accordingly.

Cool, if you do, please do let me know. BTW I had also tried to contact you through your Google+ account but it seems you don't use that account much anymore. This is what I'd said: In addition to the weight details I'd also like to add a few little tweaks, such as adding increment/decrement buttons to the 'Quantity' field and adding an "Either" option for condition (as in to search for both new OR used bricks). Obviously you'd still be in charge of the program but I'd like to be able to submit such changes for you to verify which you could then either include or not.

Yeah, there are a number of little tweaks like that I would like to have. I've even tried to contact the creator a couple of times and have offered to add them myself but haven't heard anything back. Hopefully in time such things will be added because it really is a great little package and a few simple tweaks like that would make things a lot nicer to work with