EDIT MODE

Over the last 6 months I have focused on rebuilding the majority of the backend code that deals with calculating parts reuse between sets (i.e. the main reason behind Rebrickable!). I am happy, and quite a bit relieved, to finally announce that this new functionality is now live and I am going to go ahead and call it v2 :)

Alternate Parts/Mold Variations

A major change included in v2 is that we now distinguish between the different mold variations of parts over time. In the past, I had force converted all of these parts into a single identifier so that the matching would give the best results. However, this resulted in set inventories that were not 100% accurate. It was never the original intention of Rebrickable to become yet another authority on inventories, but over time the need for it became clearer. In fact, Rebrickable now powers the inventories displayed on sites such as Brickset. Anyone else can get the same data via the API.

Similarly, we recently embarked on some parts renumbering and splitting up of minifigs (don't hate me!) for various reasons. More details in the forum.

So now that there are different molds of the same part throughout the set inventories, what happens to the build results? In v1 it could not tell the difference between eg a 4085c and 4085d so the % matches would suffer. That’s where Part Relationships come in, and the power of the v2 code comes into play.

Part Relationships

These relationships are what tie together the various molds of a single LEGO part design. In addition to alternate molds, there are other relationship types that can be defined:

  • Mold = Alternate mold which can be used as a functional drop-in replacement.
  • Print = Printed or Painted surface of the part.
  • Pattern = Marbled color, embossed or molded patterns. These are currently treated the same as Printed parts in all calculations, but are recorded separately.
  • Pair = One of a pair of related parts, e.g. tyre+wheel, left+right panels, etc.
  • Alternate = Similar part that can usually be used as a replacement, not necessarily functionally compatible.

Several months ago, I released this feature to the admin team to start populating. We now have over 20,000 relationships defined and it continues to be refined over time.

Some of the more common parts will have a hierarchy of relationships such as:

You can see some of the results of this effort on the part details popups and pages.

This has already made the submission of inventories far simpler than it used to be and this will continue to evolve to become simpler and transparent.

Building in v2

With those pre-requisite features out of the way, we can move onto the core of the v2 improvements – the Build Engine. This is the code that finds you sets you can build with your parts, provide your parts matching % on each set/MOC you visit, and suggest LEGO sets for you to buy to improve your matches. It now takes into account the part relationships, and offers some more flexibility in your build options such as ignoring printed/patterned part differences and mold variations.

To enable this, the Build page has had a bit of a redesign:

You'll notice that the 6 boxes that took Set IDs have been moved behind the "You may also enter some sets manually" link, and there are now only three boxes. Analysis of the usage shows this to be enough and I hope to encourage more people to sign up to use more anyway!

There are a number of new default settings now in use that you can change by clicking the "Show advanced options" link.

The first three options make use of the new part relationships.

  • Ignore printed and patterned part differences - Any printed parts from the same mold will be treated as if they were the same part for the matching calculations.
  • Ignore mold variations in parts - Any parts that are different molds of the same basic design will be treated the same. Note that these should all be 100% functionally identical and are usually things like strengthening changes in the mold.
  • Consider alternate parts that can usually be used as replacements, but are not always functionally compatible - This makes use of the Alternate Part relationships of which there aren't too many yet, but will increase later.

 

If you have all three options turned on, the Build Engine can for example equate a printed part from mold A with a non-printed part from mold B.

The next two options are used to filter out parts completely from any comparisons.

  • Exclude Minifigs and Minifig Accessory parts - Any parts of these types are taken out of the calculations. The idea being they usually don't play an important part of a build - however in some cases they might.
  • Exclude Non-LEGO parts such as stickers, gear, etc - Similarly, these types of parts are taken out of the calculations. I can't imagine why you would want these included, but left it an option anyway.

The next set of options are used to filter your search results.

They should all be pretty self-explanatory. The two new choices that weren't anywhere in v1 are the B-Models and Premium MOCs filters. You can also now search the output by a few different methods, although I don't see much value beyond the default of % match.

Another new feature here is the "Save these settings as your default" link which does exactly as it describes. The current values of all the advanced options are saved for reuse. They are used as the defaults next time you start a new build search, and for any automatic build calculations used on the site.

How does it work?

Let's follow an example. You are calculating the % of parts you have for some specific set. This set has part 3626cpr0001 in it, but you only have 3626b. You have chosen to ignore printed part differences so will this part be matched? No, because they are different molds. If you have chosen to ignore both printed part and mold differences, the two parts will be matched successfully.

Using the image of the 3626 Minifig Head relations tree above, the following rules would apply. There are a total of 1674 parts in the tree. With the old v1 code, that would result in 1674 parts that cannot be reused. With the new v2 Build Engine, we have:

  • Ignoring Molds = 1 pool of 4 parts (3626, 3626a, 3626b, 3626c) + 1670 various printed parts = 1670 part groups.
  • Ignoring Prints = 1 part (3626) + 1 pool of 3626a prints (16 parts) + 1 pool of 3626b prints (1238 parts) + 1 pool of 3626c prints (419 parts) = 4 distinct part groups.
  • Ignoring Molds + Prints = 1 pool with all parts in it.

 

Obviously, these calculations are now heavily influenced by the part relationships that have been setup by the admin team. This will continue to evolve over time, and a lot of the old v1 parts that were forcibly normalised will continue to be fixed.

So to calculate the percentage of matching parts in a single set for a single user, the Build Engine has to perform the following:

For every part in the set do:
    For every part in the user's list of sets and loose parts do:
        Lookup the two part's in the relationship tree and see if they are linked together via any number of 
        relationships that meet the criteria being used in this calculation.
        If they match, yay! If not, display it as a missing part.

If the set has 200 parts, the user has 10000 parts, and there are approximately 5 relationships per part, that makes 10 million database comparisons needed. A build search needs to perform this across every set in the database, so that equates to about 50 billion database comparisons. At any one time, there are between 30 and 100 users online running searches and build calculations simultaneously. And that doesn't include the complexity introduced by fuzzy colour matching, user's lost parts, etc.

And yet, the average build search time is under 1 sec :)

Of course, there are many tricks involved in making it do so much in such a short time, and these will remain the "secret sauce" of Rebrickable!

Next Steps

There are a few small tweaks to be done around the place to better make use of this new functionality, and these will slowly appear over time.

Since this massive change has been my main focus for so long, there is quite a backlog of other smaller fixes and features that need addressing. I will try to prioritise these and get to them soon.

Thank you to all my regular users who provide valuable feedback, even when I don't ask for it ;) These changes are a direct result of your suggestions so keep them coming!

24 COMMENTS

  • 2 years, 10 months ago Beleu (1812) MOC Designer
    Great upgrade, Thank you Nathan.
  • 2 years, 10 months ago Wolf01 (3407) MOC Designer
    Thank you for all the improvements which help to make this service more and more useful!
  • 2 years, 11 months ago zloy747 (61)
    <font><font>Привет, пишу на русском, огромное спасибо тебе за этот ресурс, успехов и ещё больше баз данных и пользователей. </font><font>Буду рад если приймешь во внимание мой комментарий - при выборе деталей для сборки из списка наборов сами иконки (пиктограммы) очень маленькие по размеру и не всегда понятно что за деталь необходимо выбрать из того или иного набора. </font><font>При нажатии кнопки использовать, весь перечень деталей уходит из общего списка на сайте, и при выборке 100% деталей, самое грустное, что нет конкретного списка, с большими иконками деталей, которые необходимо выбрать из набора что бы построить новую модель. </font><font>Спасибо что прочитал :-)</font></font>
    • 2 years, 10 months ago biodreamer (18115) MOC Designer
      Nathan, You will have to run this through google translate, he isn't just praising you he is giving you feedback and feature requests.
  • 2 years, 11 months ago mowikan (108)
    Thank you for your great work!
  • 2 years, 11 months ago Baseplate (307)
    Good changes. I like the way you give the user so many options for refining the search. You also explained the update very well.
  • 2 years, 11 months ago addyvandendoel (354)
    wow!, great job! love it! much more nice options. database guru!
    One question: The 58123, 9V IR Receiver 4 x 4 x 4 [Dark Bluish Gray Bottom], shows up in the missing parts list, even though I have 3 IR Receiver Unit V2 [Dark Bluish Gray Bottom / Black V2 Print] which are free to use. Shouldn't one of the options result in it disappearing this part from the missing parts list? Or is this work to be done in the parts database, creating relationships?
    • 2 years, 11 months ago Nathan (62037) Site Admin Pro
      I have added an "Alternate Part" relationship between these two now as I don't think they are 100% functionally the same. IIRC the v2 part can power larger devices a lot better than v1 and so might be the only option in some models. But this should fix your missing parts as long as you check the "Consider alternate parts" option.
      • 2 years, 11 months ago addyvandendoel (354)
        You are right. V2 can replace a V1, but the other way around,could result in a failure. one way relationship? :)
  • 2 years, 11 months ago RandoBogus (23)
    WOW! You are a database virtuoso. These are all pipe dream features that I never would have had the audacity to suggest, let alone expect. Moreover, the web interface is really well done and makes it very clear how the filters work. Totally impressed :D !!
  • 2 years, 11 months ago ukraineman (2020)
    The explanation almost melted my brain. Not really, though, well said :) I have thoroughly enjoyed the work you've put in here and this is my #1 go-to for Lego. The new build engine looks phenomenal...wonderful job!
    • 2 years, 11 months ago Nathan (62037) Site Admin Pro
      Thanks. I had a hard time trying to write that bit, and a much harder time trying to design it in the first place!
  • 2 years, 11 months ago biodreamer (18115) MOC Designer
    So totally broken,Nathan back to the drawing board, you still have gremlins to kill/tame. I get 100% on all CMF sets, All calendars and all set collections. so my 20 pages of matches are all filled with incorrect facts. or to be picky 99,5% inaccurate, There is one 100% set in there missing a minifig that I actually can build.
    • 2 years, 11 months ago Nathan (62037) Site Admin Pro
      Try ignoring accessory sets? If it's ignoring non-lego parts, why do you think the results are incorrect? A forum post explaining this would probably be easier than using these comments. EDIT: I see that you already did!
  • 2 years, 11 months ago dna2 (89)
    Thanks for all the hard work!
  • 2 years, 11 months ago DLuders (17140) MOC Designer
    Wow -- thanks for all of the great improvements! Rebrickable is very handy!
  • 2 years, 11 months ago Bobflip (196)
    Great stuff in this new update! Rebrickable is getting more powerful all the time! Congrats on getting it all rolled out :-)

    The 'fewest missing parts' sort order is definitely handy for me.

    I'm missing one feature from the old build search where you could enter a set you want to build (ok, ok... I mean 10179) and a set or two that you *might* buy in the manual entry fields, to see how much closer those would get you.

    Also, is it possible to use the part-matching options in the Part Search as well as the build search?
    • 2 years, 11 months ago Nathan (62037) Site Admin Pro
      I forgot to add a section explaining the changes in the build_set page. Once you have picked a set to build you can change any options via a link at the top, just for that set.
      • 2 years, 11 months ago Bobflip (196)
        Ah yeah, hadn't seen those new hidden options! They still don't allow me to change the manually entered sets though. Adding "&s1=7965-1" to the end of "http://rebrickable.com/build_set?s=10179-1&setlist=1&partlist=1"; updated the "Missing Parts" section correctly, but doesn't seem to be a way to do it from the web interface. The last sentence was about the "Part Search" page. If I select "Bricks Sloped" and do an empty search, I get lots of 3298 and 3297 variations - being able to disregard the printed parts would reduce these significantly.
      • 2 years, 11 months ago Bobflip (196)
        Incidentally, if you want me to take this feedback to a proper forum post to avoid cluttering up these comments let me know and I'll head over there :-)
  • 2 years, 11 months ago HuckleberryBrick (1952) MOC Designer
    Great update! My build list now includes a number of sets that I hadn't considered before.
  • 2 years, 11 months ago Bolbuyk (8529) MOC Designer
    This looks like a really good improvement!
    Must have been a huge lot of work. I had the impression you were really busy, so you did a great job!

Login to your account to write comments.