Lucky-Ramses

Search restrictions 500 is way to low

Recommended Posts

I did run into a bad search i can only search up to 500 parts in a catagory but minifigs acc is over 2000+ so the other 1500+ i can not see.

I get 496 a page and the second page only 4, so can you change that.

Can the search be set on 2500 is enough for all search behalve the minifigs themself.

Restricking it to a color does not work there are more then 500 returns but not shown again and a lott of parts can never be seen in the search because of the restriction.

 

This restriction is also if you search a color if i search black i get only 500 so if i search for a part i do not now how it is called the search is useless with a return of only 500.

Share this post


Link to post
Share on other sites

Admins are very aware of the problem.  Admins have been trying to get Nathan to break down the Minifig category into at least the 4 major body parts- hair, head, torso assembly, and legs and hips assembly.  Even that won't really solve the 500 limit problem, but it would be a start.  

Share this post


Link to post
Share on other sites
37 minutes ago, Lucky-Ramses said:

Solution is so simple set it on 2500 and all is solved, behalve the minifigs.

I just did a search for black minifig accessories.  I included unused and printed parts.  The results were only 477. I also checked some other colors that are used frequently and none were over 500.  So 500 is sufficient for this category.

 

Share this post


Link to post
Share on other sites

not if you don't now where to look, then you have to search in all 50 catagories and have to mark on paper were you are.

The easiest solution is just set it on 2500 just as on every websites search and no restriction, in the old v2 it was also higher if i'm right adleast 1000.

And time out problems are so often this will cause no more problems as there are now.

Error Page

This is the whole day, especialy when America is awake.

Share this post


Link to post
Share on other sites

I agree with both of you that for some categories the 500 parts limit is a big problem, but I also understand Nathans worries about the performance impact of a 2500 parts limit.

However, there is a solution that allows browsing all the minifig (literally no matter how many) with less performance impact than the current method.

Right now, RB is using a "closed search algorithm": the entire parts database is searched, the total number of matches is determined and used to create paging buttons (prev 1 2 3 next). This works perfectly for categories with less matches then the limit, but it is a huge problem when there are more matches then the limit.

Now, for minifigs (and, if needed, for other large categories), the system could switch to a "open search algorithm": the system then simply searches for the first 96/248/496 matches (depending on Parts/Page setting), it does NOT calculate the total number of matches nor the number of paging button, but simply displays a prev and an next button with links set up to continue the search from an entry point not at the start of the database but somewhere in the middle.

The only disadvantage is that the user cannot immediately skip to a particular page, but has to browse through all pages. However, one can always bookmark such searches and use those to quickly skip to the end.

Thought is was worth mentioning... -smile-
Take care,
Simon

 

Share this post


Link to post
Share on other sites

Thank you for you're answers it looks like i'm Always complaining but i only have problems with vieuwing pages and searching pages, with the release of V3.

The problem is my my collection i think, 2340 sets, 550000 parts in almost 21000 different types and colors then you run into big problems and get 10% of the time error pages that Rebrick is offline ore the responding time is to slow.

Also the plan restrictions gave a huge problem i did have to devide my spare parts over 3 seperate list by doing that and we did try it several times i lost over 3000 parts never to be found again, so now i'm still counting to sort out what was lost by doing that, ever counted over half a million parts!

By counting my collection i'm also on the same time giving a huge amount of change request ask Tobymac, and also missing pictures so it is on the other way for the rest of the users a huge advantage that it happend.

But my question is still why can Bricklink and also Brickset and many other websites handle these type of searches verywell, with brickset even returning 3000+ results for searching parts so it most be possible.

And Brickset handles my sets and parts also and no errors, the amount of data most be almost the same i think, only no moc's, but they have all the minifigs to show.

I'm not a programmer only a user but i most be possible.

I have much respect for the new website and all the work Nathan has done don't understand me wrong.

Share this post


Link to post
Share on other sites

Hai, Lucky,

well, I've been a programmer for 40 years, and, I am sorry, but half a million parts is going to be a problem.
Try loading 550.000 parts into brickstore on your local computer, or better, try loading a text file with 550.000 lines of 80 characters (42 MB).

Problem is: anything can be done, but it takes coins and hands. I'm pretty sure Bricklink has the coins, and both BL and BS, I imagine, have a team of developers. Nathan is working alone, which has advantages, he can make RB exactly as he wants it, but is has disadvantages too, and this might be one of them.

As to losing part lists - lesson 1 of programming: never trust a computer, or a harddisk, or a usb drive. Everything breaks down, it is just a matter of time. Do not make one copy, not even two copies, make, at least three copies of every important file your own. My LEGO data is on RBs server, and in csv on my local harddrive, and on another harddrive in another computer, and on a flashdrive in my car. And I probably also have a few CDs/DVDs with older copies.

To be sure, I have used RB quite a lot, over the last few weeks (from Holland, and mostly evenings) and I have never ever had problems with timeouts. And I also do a lot of searches.

So here might be a very simple solution for your problem.

Try creating a second userid and do your searches with that id. It might even be possible to open two browser screens and login to RB using different ids (not sure, but worth to try). If you login using your own userid, and perform a search, the system has to check your enormous collection for each set and part displayed, because it wants to add the number you own. With so many parts in your collection, that's asking for problems. However, if you search with another userid, and that userid doesn't own anything, your searches will be much faster, and you'll never encounter time-outs.

If the system doesn't allow you to add a second userid, you might ask Nathan to make an exception.

Take care,
Simon

Edited by Simon
typo

Share this post


Link to post
Share on other sites

PS.

If you re allowed to create a second userid, try working in batches. Take a box of parts, login with second id, search and add to a list. When ready, export the list as csv, logout, login with you primary id, and import the list. As long as you use your primary id for inventory, and your second id for searching and counting, I am sure, you'll be fine.
(additional advantage: backup (3 times) your batch csv's, and you create, in a sense, an incremental backup of your entire collection. If anything goes wrong, delete everything, restore your last full backup, and then upload all batch csv's that have been done after your last full backup)

Share this post


Link to post
Share on other sites

I work allready with 2 user id's 1 in Edge and 1 in Chrome(2 different emails accounts) but wihit the empty Chrome id same problems returning often the error page see above in a fromer post, the error pages begin in the afternoon when America is starting up so arround 1400 the worse is between 1500 and 1800 then it is getting better.

But 2 browsers is no solution and does not work to maintain and count you're collection.

Error Page

About the spare parts list download did go right splitting up did go right but the upload to rebrick not, he did remove everytime about 3000 part which gave erorrs without giving the numbers and amounts i could only see it on the total piece count that there was a differnece, and there is nothing in the world i can do about that, that was a problem out of the rebrickable upload module so bad luck to not join the plan.

I have 3 copies on running W10 machines and on 3 external HDD's

I'm also in the Netherlands near Hoek Van Holland provider caiway very fast account

 

Was typing when you're second message showed up.

Do to the enormous amount of rare parts i have and the problem with showing them if they are not in a set that is no sollution all rare parts will show up in the wrong color.

And i have to manualy correct them again and that takes again many hours.

Share this post


Link to post
Share on other sites

Well, as @Simon says (that sounds... weird :P), loading this much data can be quite demanding on the server. That being said, I still have the feeling that some pages on Rebrickable are not very well optimized and it should be perfectly possible to be able to load pages with a lot of data (e.g. large search results, big inventories etc.) a lot faster. I don't know how, I don't know if it's the software (code) or hardware (server) side of things, I don't know if I'm just completely wrong. I just got that feeling that there's room for noticeable improvement... :unsure: But let's not forget that @Nathan is working alone on the whole site, so let's not get too demanding ;)

Share this post


Link to post
Share on other sites

Only 1 note on this moment, when i look at all my parts i get a result back of almost 21000 parts so it is possible, with 496 on a page that are 43 pages.

When i go to page 2 and i look at the url i see a 2, and if i want to go to page 23 i only have to change the number in the url and it jumps to that page.

Share this post


Link to post
Share on other sites
23 minutes ago, Lucky-Ramses said:

Only 1 note on this moment, when i look at all my parts i get a result back of almost 21000 parts so it is possible, with 496 on a page that are 43 pages.

When i go to page 2 and i look at the url i see a 2, and if i want to go to page 23 i only have to change the number in the url and it jumps to that page.

But... What's your point with that? :unsure:

Share this post


Link to post
Share on other sites
1 hour ago, Lucky-Ramses said:

That it most be possible to set the search on 2500.

Oh, I see. Well, displaying items is something else than searching items. I think the 500 limit is not a display (number of pages) limit, I think the actual search result is limited at 500 (so, when the 500th result is found, it just stops going further).

Share this post


Link to post
Share on other sites

Greetings, All.

I've been talking privately about this with Lucky-Ramses, and I showed him this page I made:

https://bricksafe.com/files/Simon/Html/shortcuts.html

This is basically a single page with about 250 different Rebrickable searches, and each will open in a new window, keeping the shortcuts available for a next search. It is still a work in progress, I want to add another page for sets with all different themes and years, but for the parts, this page allows searches for each part category, each color, and 2 dozen text searches. It's not additional functionality, it just saves some time, not having to click on Search Parts, wait for the page to load, open the drop-down menu, scroll down to select a category of color, and click on Search.

All searches marked with a ^ sign are incomplete, showing only the first 500 items (still need to check some colors for this). I guess the most problematic are Duplo, Printed Bricks, Printed Tiles, and, of course, Minifigs and Minifig Accessories.

Which makes me wonder: want kind of additional options can we use in the search box. Can we use an AND function, OR, and NOT. Can "quotes" be used. Just trying to find out if there are other ways to work-around the 500 items limit.

Take care,
Simon

Share this post


Link to post
Share on other sites

OK, this is strange. If I search within category Tiles Printed for "round" (without quotes), I get what expect: 223 round tiles, 2x2 and 1x1.

However, if I search within category Tiles Printed for "1 x 1" (without quotes), I get 1 x 1 square tiles, then 1 x 1 round tiles, then a few 1 x 2 tiles, then 1 x 3, then 1 x 4, then all the other 1 x 2's.

If I use "1x1" without quotes, I get 23 items, which is wrong, because there are at least 80 1 x 1's in the database.

Am I doing something wrong?

Share this post


Link to post
Share on other sites
3 minutes ago, Simon said:

OK, this is strange. If I search within category Tiles Printed for "round" (without quotes), I get what expect: 223 round tiles, 2x2 and 1x1.

However, if I search within category Tiles Printed for "1 x 1" (without quotes), I get 1 x 1 square tiles, then 1 x 1 round tiles, then a few 1 x 2 tiles, then 1 x 3, then 1 x 4, then all the other 1 x 2's.

If I use "1x1" without quotes, I get 23 items, which is wrong, because there are at least 80 1 x 1's in the database.

Am I doing something wrong?

That is because not all the names are right on this moment i'm working on that but that takes time and have to send all these changes by word doc to Tobymac so that will take a few weeks to months before all names are right, and that search will work.

Share this post


Link to post
Share on other sites

Ahhh, now I understand (thanks!! -smile-). I am presuming the right description would be "1x1" without quotes. Then it all makes sense.

@TobyMac can I help? If yes, do you want word doc or html with links? I can co-operate with @Lucky-Ramses to make sure we don't work on the same parts.

Share this post


Link to post
Share on other sites

All catagories with the 500 problem on this moment

Bionicle 1121

Bricks Printed 763

Duplo, Quatro Primo 1574 (and there will be about 250 to 300 more if the catagorie other is emptied from all the Duplo parts)

Mini fig Acc 2095

Mini Figs 8873

Non Lego 2035

Other 1308

Plants & Animals 761

Tiles Printed 1287

And searching on every color is no solution there are over 125 colors(in my collection) that will give about 1000 search request

And with minifigs even there you will not get what you searching for more return then 500 so you will not see them all, hey are lost in the system.

4 minutes ago, Simon said:

Ahhh, now I understand (thanks!! -smile-). I am presuming the right description would be "1x1" without quotes. Then it all makes sense.

@TobyMac can I help? If yes, do you want word doc or html with links? I can co-operate with @Lucky-Ramses to make sure we don't work on the same parts.

 

No it will be 1 x 1 just as by all other parts there have to be spaces between them, but also the name have to be checket to the others in the serie and some to Bricklink a lott of work.

So that after changing the names they also will show up in a row and not everywhere between other tiles so as it now is.

I'm working on Bricks Sloped and will check that for a second time when my first 100 are changed.

Share this post


Link to post
Share on other sites

Now you have me confused... -smile-

The search function doesn't seem to have a quote feature, and, as @biodreamermentioned in another post, it neither has AND/NOT functionality.

This leaves an implicit OR: so if type "1 x 1" the system drops all spaces and searches for "1" OR "x", which returns all kinds of unwanted results.

No matter what changes are made to either part name or description, if the spaces stay in, then the search function needs to be changed. to enable searching for "1 x 1" with quotes, to prevent dropping the spaces..

Share this post


Link to post
Share on other sites

Simon Just send you a privat mail with some explenation.

We have to keep the naming a bit like that of Bricklink because most of the names are a direct copy/import of BL's names and parts.

And it is easier to search on both websites if the name is the same.

But by the import also all the wrong names are copied, on BL they are allready changed but not on rebrick i thought it were only 6 to 700 parts but after a recount last week it seems it are over 1250 parts that need name corrections.

Sometimes it is only by removing an extra not needed space ore just adding on to give a part the right name to show up right and indeed to lett it be found till that time a lott of parts will also not show up whit the suggested searches.

That is also what i ment with a lott of work to get it right but it has to be done, and a lott of work extra for the admins.

Edited by Lucky-Ramses

Share this post


Link to post
Share on other sites

the best would most likely be more clearly visible tag filters. ie you shouldn't need to know which tag that exist but be able to find them among the filters. so you can do a bit of "who is there" 

with the prints.

Edited by biodreamer

Share this post


Link to post
Share on other sites
On 9/8/2017 at 7:52 AM, Lucky-Ramses said:

I did run into a bad search i can only search up to 500 parts in a catagory but minifigs acc is over 2000+ so the other 1500+ i can not see.

I get 496 a page and the second page only 4, so can you change that.

Can the search be set on 2500 is enough for all search behalve the minifigs themself.

Restricking it to a color does not work there are more then 500 returns but not shown again and a lott of parts can never be seen in the search because of the restriction.

 

This restriction is also if you search a color if i search black i get only 500 so if i search for a part i do not now how it is called the search is useless with a return of only 500.

I have made some changes that should improve performance for large search results. For now I have increased the search limit to 1000 and will monitor for a while before increasing it further. Since it's heavily used, and most people don't need 1000 results, I'm hesitant to increase it further. Maybe I'll look into doing a "fetch next 1000 results" or something similar to Simon's suggestion.

Performance optimisation is an ongoing challenge for me, and something I am incrementally improving. Throwing extra hardware at the problem can only go so far and I'm not prepared to spend any more money on it at the moment.

On 9/9/2017 at 5:58 AM, Simon said:

Right now, RB is using a "closed search algorithm": the entire parts database is searched, the total number of matches is determined and used to create paging buttons (prev 1 2 3 next). This works perfectly for categories with less matches then the limit, but it is a huge problem when there are more matches then the limit.

Now, for minifigs (and, if needed, for other large categories), the system could switch to a "open search algorithm": the system then simply searches for the first 96/248/496 matches (depending on Parts/Page setting), it does NOT calculate the total number of matches nor the number of paging button, but simply displays a prev and an next button with links set up to continue the search from an entry point not at the start of the database but somewhere in the middle.

The only disadvantage is that the user cannot immediately skip to a particular page, but has to browse through all pages. However, one can always bookmark such searches and use those to quickly skip to the end.

The "open" system also means I can't generate the drill-down filters from search results. For most pages I've managed to keep the overhead from large search results to a minimum and most of the work is actually done to draw the items for the current page. 

10 hours ago, Simon said:

Which makes me wonder: want kind of additional options can we use in the search box. Can we use an AND function, OR, and NOT. Can "quotes" be used. Just trying to find out if there are other ways to work-around the 500 items limit.

No sorry. The search query isn't very flexible and on my list to investigate a better method.

Share this post


Link to post
Share on other sites

Thanks for the new limit of 1000, and normaly it would not matter that it is only 500, but i'm checking all the part names and photos and i can not reach the parts that were exeding the 500 limit and are not in my collection so now only minifig acc is a problem for me, have to search there every color to see them.

That is no problem only the returned searched in colors will be much higher then the basic 2095 mold parts and have to find a way to give no dubble change request.

Other option for me is to make a new temp part folder and ad every part that is in minifig acc and not in my collection, but then still i can not reach all of them.

I have to find out how i can do that 

Cat other has to be emptied to the right catagory then it will be under the 1000, and Bionicle has to be done by some one else who has the parts.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now