EDIT MODE

During July I focused on fixing a lot of the small nagging problems. I also spent some time on improving performance of many aspects of the site, as well as cleaned up a lot of code to make things easier to maintain.

Performance related changes:

  • Changed My Parts to only show one grouped section at a time so people with huge collections can actually use the page. I know this sucks for some others and I will restore it to the old behaviour later when I finish a rewrite of that page (for multiple parts list feature).

  • I improved my monitoring tools to be a better idea of where the slow parts are. This has provided some neat stats I'll show below.

  • Some basic web development 101 stuff to improve page loading speeds (javascript after page rendering, subdomains for images/static files, offloading embedded javascript code to external cached files). Yeah, I'm not a web developer so if you see something stupid let me know :)

  • Thumbnail images are now using much better compression to improve loading speeds via the neat pngquant tool. Plus, I reduced the resolution of the thumbnails which were stupidly being downscaled. What you see is exactly what you saw before but now the files are ~10 x smaller!

  • And of course, I performed a migration of the entire site to new hardware. This included upgrades to the latest versions of all the major components (OS, Database, Web Server, etc) as well as some nifty hardware upgrades (SSD disks, 2x RAM, faster CPUs). SSDs are awesome!

I love performance problems, they are a great optimisation challenge :) Here is a summary of the database query runtimes from one day this week after some of the above changes were made (Monday 28/07/2014).

RangeCountPercentage
0-1ms9,925,85088.77%
1-5ms867,5797.76%
5-10ms262,3092.35%
10-25ms97,9530.88%
25-50ms11,6510.10%
50-100ms5,2200.05%
100-500ms9,2950.08%
500-1000ms5650.01%
1000-10000ms7540.01%
> 10000ms90.00%

763 queries took longer than 1 sec... still some work left to do :) Of course the fastest query is the one that never has to run, and 95% of queries hit the cache instead of showing up in the above stats. This means Rebrickable is running about 220 million queries per day (2500/sec)!

Other boring fixit stuff:

  • BrickOwl store names/notes are now updated automatically when they change at BrickOwl.

  • Fixed quote encoding in moc descriptions (some images/links not working).

  • Fixed Brickset csv/txt file imports.

  • Fixed newsets RSS feed (twice!).

  • Fixed display of some duplicate part molds in part details pages.

  • Fixed display of some user avatar images.

  • Fixed Bricklink exports which were no longer mapping parts.

  • Fixed an issue with Bricksafe gallery not displaying sometimes, plus I reduced load on Bricksafe server which wasn't handling the Rebrickable levels of traffic very well.

  • Fixed display of duplicate molds on some part popups.

  • Fixed image overlays in the change log.

  • Fixed a stupid emailing issue. From 23rd to 30th July I didn't get any emails you may have sent to [email protected] Stupid firewall. I just thought everyone was happy for a change, it was so quiet :)

  • Fixed breadcrumbs link for MOCs.

  • Fixed save of premium MOC details.

  • Improved handling of different character sets within LXF files (MPD files were already handling this correctly).

  • Some refactoring of very bad Javascript into slightly less bad Javascript, as part of the initial work for adding multiple parts lists. Yes, I really am working on it but it's hard so be patient ;)

Plans for August

Sleep. Nahhh, that's boring :)

  • Multiple loose parts lists - there are a lot of components this touches so it's taking a while but is definitely coming soon.

  • API enhancements for various external sites and apps in development. This will include things like importing your parts orders from BrickOwl just like you can now with BrickLink.

  • Bricksafe - I need to spend more time getting it beyond it's initial v1 build and add features such as searching, paging through images, etc.

  • MOC Feedback - I want to add user ratings, maybe upload photos of fan builds, etc.

There's lots more planned but beyond August timeframe, and I'll be very lucky to get all of the above done anyway.

15 COMMENTS

  • 3 years, 1 month ago madlyR (2079) MOC Designer
    Great work! Performance is no longer an issue.

    "Plans for August - Bricksafe"

    May I suggest to add some kind of JavaScript gallery with prev/next buttons (touch screen compatible)? The easiest form - gallery in subdirectory.
    It would be great if such a gallery would be integrated into the MOC page in Rebrickable description, with ability to embed it on page with bigger than thumbnails form.
  • 3 years, 1 month ago PhyBuilder (284)
    It's great to have you understand the most underestimated part in many database-driven websites: storage performance. Although much is running in RAM, there are still many logs supplemented all the time, backup jobs running in the background (15 Minute increments?), etcetera, so storage does matter. Always! Thumbs up for the SSDs! Don't know about your hoster, but please be aware of the MTBF normal consumer SSDs have. They can't handle the huge number of write cycli a server runs on it.

    And: it's clearly visible that speed improved during different times of the day! But that has to do with the reduction of the thumbnail size at most I think. Good work.

    =====

    I have one small feature request / improvement proposal. When I want to check the inventory of a set, I now perform the following steps every single time:

    1) I search for the set and click on it
    2) Click on "Jump to Parts"
    3) Scrolling down a little because the button "Load Remaining Parts" doesn't fit on my Screen (1050 vertical pixels)
    4) Click on "Load Remaining Parts"
    5) Scroll down again because the page jumped up
    _____


    It would be really nice to reduce it to:

    1) I search for the set and click on it
    2) I click on a hardlydaretosaysoonavailablebuttonnamed "Show all parts"
    _____


    By not reducing it to only one step, but having that single click still included, you avoid to have a set inventory load and use resources when people don't need it at that moment. Therefore I think that the improvement can be made without performance penalties, but with a huge increase in usability.

    What's your opinion?

    Many thanks for this site, only using it for a few months but now cannot live without it anymore. It's useful without comparison! :)
    • 3 years, 1 month ago PhyBuilder (284)
      Today I discovered that the partlist on sets and socs loads completely upon opening the page. That's perhaps even more than one would need, but/therefore thanks! I now only need to scroll down and have all the parts visible immediately!

      It's really a great help, because often, one wants to see all the transparent/LBG/DBG coloured parts, which are down under in the list.

      Keep up the good work :)
      • 3 years, 1 month ago Nathan (63243) Site Admin ADMIN
        Yeah seemed the easiest option. I originally didn't want to show all of them to speed up the page load (plus comments were after the parts), but I don't think that's a problem anymore.
        • 3 years, 1 month ago PhyBuilder (284)
          Thanks again. Small change, great benefit!

          Nevertheless I'd strongly reccoment to have it under a button (not in my interest, of course). The site may be very fast now, but be optimistic: when the site gets more popular and you expand the functionality of it, you want to have every IOPS and every CPU cycle available ;-). Don't underestimate the potential of you own site, people just need to discover it and the power it has :-)
  • 3 years, 1 month ago Mestari (3344) MOC Designer
    Back to using My Parts, finally :) Thanks!
    Although some words on the page started looking silly after recent changes. I need to hover over for them to show proper words. Weird.
    • 3 years, 1 month ago Nathan (63243) Site Admin ADMIN
      I've noticed that too. I think its related to the fonts I use not getting loaded properly but don't know why any recent changes would affect that.
      • 3 years, 1 month ago Mestari (3344) MOC Designer
        Could be a coincidence then. But I have those funny looking fonts only since very recently. It was all good before.
  • 3 years, 1 month ago jantjeuh (60247)
    Thanks for the update, keep up the good work!
  • 3 years, 1 month ago biodreamer (19379) MOC Designer
    I am looking forward for the "Multiple loose parts" lists does it comes with multiple ignore lists, or do they atleast get their own?

    running both towards the same ignore list isn't very usefull, atleast if it's forced.

    can you explain the scope your going for here?
    • 3 years, 1 month ago Nathan (63243) Site Admin ADMIN
      I have no plans for multiple ignore lists, I don't see the point?
      • 3 years, 1 month ago biodreamer (19379) MOC Designer
        First because you don't want to build sets in your set list. these are only ignored when you build with your setlist. Secondly because I have over 300 items on my ignore list and if I do a new loose part list. the same set of ignored sets wouldn't be applicable, because it a subpart of the collection which should have a subpart of the ignorelist. Toggling the setlist use when build is also toggling if the sets are ignored or not so there is simply no way for me to control which sets I don't want to showup in the build result, without making ignore lists.
        • 3 years, 1 month ago biodreamer (19379) MOC Designer
          So either give me more ignore lists or a better way to control which sets that should be automatically ignored.
  • 3 years, 1 month ago jgw3000 (180) MOC Designer
    Much faster, thanks!
  • 3 years, 1 month ago DasMatze (1038) MOC Designer
    I'm really looking forward to the user rankings! Great work on the performance update. Thanks for continuing working on this great site!
to your account to add or reply to comments.