Nathan

Rebrickable V2 - Status

Recommended Posts

OK maybe a flashy title, but I feel what I've been working on is so big that it deserve the "v2"  :D

 

As I have hinted in various other posts, I am working on completely rewriting the backend database and build engine to make use of relationships between parts - e.g. different mold variations or printed parts can sometimes be substituted for functionally identical builds. This will allow for much more flexible build calculations and other things like automatic replacement part suggestions, highlighting incompatible tyre/wheel combos for example. I've no doubt it will allow plenty of other cool stuff I can barely imagine right now.

 

It requires a database and build engine that is much smarter than what I came up with 2 years ago, which I have now designed and am in the processing of writing. So now there are actually two full copies of the inventory data that is being maintained simultaneously (old v1 data and the new v2 awesomely-structured-kickass-data). I am running continuous checks to be sure both old and new databases are reconciling. So far the results are great, and I will be transitioning all of the page queries to pull data out of the new v2 database over the coming weeks.

 

Some of the other changes that have occurred as part of this migration which isn't really visible to non-admins:

- Created part relationships functionality and pre-populated thousands of relationships. You cannot see this data yet, but as things come together I will make it available on part details pages and other relevant places.

- Decommissioned the old part mappings. These were forcibly replacing an old part with a new part to get the build results working better. The cost of this was inaccurate inventories for the purists who like to see the mold variations over time. The mappings are there now only for imports/exports between different systems and to remove duplicate part numbers where they exist. I will also try to reverse as much of these forced mappings as I can using various logs that were generated.

- Replaced functionality like similar parts (for missing images) and useless parts (e.g. minifigs stickers!) with v2 equivalent features.

 
At a high level:
 
Phase 1 = Create Part Relationships and pre-populate as much as possible (DONE)
Phase 2 = Decommission old functionality making it very difficult to migrate to v2 (DONE)
Phase 3 = Build and auto maintain the new Set Inventory tables (DONE)
Phase 4 = Build and auto maintain the new User Parts tables (DONE)
Phase 5 = Migrate all code to query from the new v2 tables, setting build engine code to use default options equivalent to v1 (WIP)
Phase 6 = Add new options to build features
Phase 7 = Reverse old forced mappings to get original mold variants back into the set inventories (build results won't be affected by this stage)
Phase 8 = Enhance various pages to use part relationship data
 

I am past the halfway mark, but phases 3, 4 and 5 are definitely the hardest  <_<

 

Share this post


Link to post
Share on other sites

Thanks for the update! :wub:  All your hard work is appreciated especially knowing it's your summer down under. :)

Share this post


Link to post
Share on other sites

Yeah... 1st December was first official day of summer so of course the first thing I did was go out and get burnt and mild sunstroke  :huh:

Share this post


Link to post
Share on other sites

I have just completed migrating the core build/matching calculations over to use the new v2 tables. It took me a lot longer than anticipated to get this done as I struggled to make the calculations fast enough. In the end it runs about 5-10% slower on average than the old calculations, but I upgraded the server to use faster CPUs and more memory to make up for it :)

 

I've been testing this stuff to death for weeks so it should be rock solid, but if you find any strange calculation results that look wrong please let me know! For now, there are no changes to the interface so you can't take advantage of any of its new capabilities yet, but that will come soon.

Share this post


Link to post
Share on other sites

I'm now finished with phase 5:

 

Phase 5 = Migrate all code to query from the new v2 tables, setting build engine code to use default options equivalent to v1 (DONE)

 

There are a few remnants of the v1 database still in use, but 90% of everything is using the v2 data and the majority of v1 maintenance code has been removed. The next big step is to expose the options for the new build features and I'll probably revamp the search options a bit too. Meanwhile, I've started showing things like alternate molds of parts in various places and will continue to tweak what gets shown. For example 4085 shows the four alternate mold images and the number of sets they appear in. I think those pages are starting to look a little confusing though so might redesign a bit.

Share this post


Link to post
Share on other sites

No it's not. import is broken, build is broken (ERROR: relation "mysets" does not exist LINE 17: ... WHERE p.set_id not in (select m.set_id from mysets m) ^) You still have work todo.

Share this post


Link to post
Share on other sites

No it's not. import is broken, build is broken (ERROR: relation "mysets" does not exist LINE 17: ... WHERE p.set_id not in (select m.set_id from mysets m) ^) You still have work todo.

Can you please give me details of what options you are using when building/importing as I can't replicate the error and don't have access to the logs atm.

 

EDIT: I see the import error, but can you please still provide details for the build options used.

Share this post


Link to post
Share on other sites

Build with only my parts list, exact (this part works i get a first list, but this one is allways unwanted)

 

then I change the options to No of parts = 0, From year = 0, tic off exclude accessories and MOC, leaving only offical sets ticed. Press filter results. and the error appear.

Share this post


Link to post
Share on other sites

I played around with it and if I change year to something else like 1950 it starts spinning instead of giving me the error: so check Year = 0.

Share this post


Link to post
Share on other sites

OK thanks, fixed the problem. There was a hole in the logic that didn't handle partlist only + min year < default. Sounds strange I know, but there is some crazy complexity in that code for the sake of performance. Also, the imports are fixed.

Share this post


Link to post
Share on other sites
Phase 6 = Add new options to build features (DONE)


The last parts I will do slowly over the next few months:

 

Phase 7 = Reverse old forced mappings to get original mold variants back into the set inventories (WIP)

Phase 8 = Enhance various pages to use part relationship data (WIP)

Share this post


Link to post
Share on other sites

First Bugreport: all the CMF series comes up as 100% match. And I didn't exclude minifigs and accessories. I only own four CMF figurines so I miss most of them.

 

My settings:

 

Exact, My Loose parts only, Exclude None Lego parts everything else included, Any theme, 0-unlimited parts, 1950-2014 in year. Offical sets and accessories.

Share this post


Link to post
Share on other sites

First Bugreport: all the CMF series comes up as 100% match. And I didn't exclude minifigs and accessories. I only own four CMF figurines so I miss most of them.

 

My settings:

 

Exact, My Loose parts only, Exclude None Lego parts everything else included, Any theme, 0-unlimited parts, 1950-2014 in year. Offical sets and accessories.

Sorry I can't reproduce it on my test server. Can you copy/paste the URL you get after searching eg http://rebrickable.com/pick_set?p=2&partlist=1&s1=&s2=&s3=&c=1&ignore_print=1&ignore_mold=1&ignore_alt=1&ignore_nonlego=1&theme=088&numpieces=0&maxpieces=1050&year=1950&yearto=2014&inc_official=1&inc_custom=1&inc_premium=1&sortby=0#setlist

Share this post


Link to post
Share on other sites

It's the exclude none lego parts that give an unexpected result. it works if I include them.

 

http://rebrickable.com/pick_set?p=1&partlist=1&s1=&s2=&s3=&c=1&ignore_nonlego=1&theme=0&numpieces=0&maxpieces=1050&year=1950&yearto=2014&inc_official=1&inc_bmodels=1&inc_accessory=1&sortby=0

 

THat isn't what I would expect for excluding none Lego parts that it starts ignoring whole sets. I thought it was strings,rubberbands,printed paper and cloth. Not whole boxes of parts.

Share this post


Link to post
Share on other sites

Another thing you need to tweak is how the system is handling "undetermined molds". for example 3962, the system currently treat them as a third mold but it's really a question mark. When I build indepedent of setting, logical I should get it for either 3962a or 3962b. because the system don't know which it is. This would open upp for marking up sets with molds like 3962a?b ie the set exist with either version but it's know that they come with either of them it's not unknown which of them it was, like 3962 might suggest. This would allow you to handle the time period when LEGO was moving from one mold to another. Without having two version of the set or in worst case one of each varation combination.

 

it would give us MOC builder the power to tell the system if a specific mold is really needed or not. and set up the inventory as either or fixed.

Share this post


Link to post
Share on other sites

Hi

First V2 works great.

But i have lost about 200 moc's in the build option, i have tryed everything to get them out but nothing worked.

But also in the Lego set option there are many sets not found anymore, mainly i didn't find al the bigger sets anymore.

I'm based on Technic, i have not searched for other themes.

 

The normal search function is working but the build not.

 

To test my account i did empty my ignored sets file(2000+) but the problem stayed the same. 

 

If i look at all the missing ones i think it has something to do with sets above 1050 parts, the search stands on 0 to unlimeted.

But all sets with 1050+ are not shown anymore.

 

Is there something i'm doing wrong?

 

And i could not remove the advent calander(1076.1) from 1999 out the ignored sets, there are about 20 of them in my file.

 

Regards Marchel

Edited by Lucky-Ramses-2012

Share this post


Link to post
Share on other sites

Hi

 

Found another minor problem but when it happends you can't search anymore.

I moved the number of parts button complete to the right (Unlimeted parts) and then it is stuck under the button on the right sight.

I can't move it to the left anymore, it seems to be under the other one, and unreachable.

Second problem it saved itself so i can't use the search anymore, refreshing the page is not helping.

 

(Edit)

But after closing my browser(Vista-IE9.0/2011) and starting up again it is back to the begin situation.

Is this a IE9 problem?

 

Regards Marchel

Share this post


Link to post
Share on other sites

It's the exclude none lego parts that give an unexpected result. it works if I include them.

 

http://rebrickable.com/pick_set?p=1&partlist=1&s1=&s2=&s3=&c=1&ignore_nonlego=1&theme=0&numpieces=0&maxpieces=1050&year=1950&yearto=2014&inc_official=1&inc_bmodels=1&inc_accessory=1&sortby=0

 

THat isn't what I would expect for excluding none Lego parts that it starts ignoring whole sets. I thought it was strings,rubberbands,printed paper and cloth. Not whole boxes of parts.

This should be working now

Share this post


Link to post
Share on other sites

Could you explain how the site is handling "sets" as parts. Do I have it when I add it to my owned sets or is there no way to half own a collection. ie it's for marking multiple sets into your collection at the same time. ie if there is a set in collection X. and the set also exist standalone. I would technically need to own it twice to make them both as owned or these  part count would be messed up.

Share this post


Link to post
Share on other sites

Could you explain how the site is handling "sets" as parts. Do I have it when I add it to my owned sets or is there no way to half own a collection. ie it's for marking multiple sets into your collection at the same time. ie if there is a set in collection X. and the set also exist standalone. I would technically need to own it twice to make them both as owned or these  part count would be messed up.

 

Badly. If a set contains both sets and parts, or even just sets, they are not "parted out" when calculating build results. It basically treats the set as a single part. It's a problem I will have to tackle eventually.

Share this post


Link to post
Share on other sites

If i look at all the missing ones i think it has something to do with sets above 1050 parts, the search stands on 0 to unlimited.

But all sets with 1050+ are not shown.

Having the same problem...

Share this post


Link to post
Share on other sites

Having the same problem...

We all do, I don't own any set that big, so it's not critical for me. but it's a shame that all modular buildings are lost in the build results.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now