Peeron.....

[drdavewatford]

I've become incredibly dependent on the Peeron website - perhaps it's my obsessional side, but I seem to spend ages just looking through the pieces that make up each set, logging all my spares to make it easier to build MOCs etc., and of course logging my collection of sets.

Problem is, the site has become incredibly slow, even timing out and giving me error messages over the past few weeks. Sometimes it takes minutes just to refresh, even on a superfast broadband connection. I did e-mail the webmaster about this but got no reply.

I use Peeron so much that this issue is of concern...... Anyone else having problems ?

Dr. D.

[Big Cam]

I've become incredibly dependent on the Peeron website - perhaps it's my obsessional side, but I seem to spend ages just looking through the pieces that make up each set, logging all my spares to make it easier to build MOCs etc., and of course logging my collection of sets.

Problem is, the site has become incredibly slow, even timing out and giving me error messages over the past few weeks. Sometimes it takes minutes just to refresh, even on a superfast broadband connection. I did e-mail the webmaster about this but got no reply.

I use Peeron so much that this issue is of concern...... Anyone else having problems ?

Dr. D.

I have just barely starting using it, but I'm about ready to stop, it takes ages to search and I know for a fact it isn't my internet connection or computer, becasue other sites are just blazing fast.

I think they need some moeny for server upgrades.

[drdavewatford]

I think you're right, Cam.

It's a terrible shame because I love the site and it's the best "one stop shop" for set inventories AND logging your own sets and parts that I've found.

Perhaps we should have a whip-round ?

:-)

Dr. D.

[Rick]

I use it for all my sets at the moment, so the only wait for me is adding new sets. Since I don't buy in dozens, that's easily manageable. But the speed issues are getting quite...

[Whittleberry]

I absolutely love Peeron and find it very useful, but it usually takes longer to load than other sites. From past experience, I also think that it would be a much better resource if it could be edited by anyone with an account, like a Wiki. That way, there's less work for the admins of the site, updates happen quicker and errors can be quickly corrected.

[Peppermint_M]

I use it in conjunction with Bricklink and Brickset, when one hasn't got information, the other does. Maybe if it gets faster I can list all my sets there (I have rather a lot now, so to start on peeron would take hours) and see the benefits.

It is very useful when Bricklink's super strict search algorithms ignore my requests or can't find results...

[drdavewatford]

Brickset is a good alternative to Peeron and Brlicklink for logging actual sets and minifigs.

Are any of you aware of another site where I can log loose pieces as well ? I've had a look around, but Peeron is the only site I've found which offers this facility. I don't want to migrate away from Peeron, but it's so slow now that it's only borderline usable at certain times. If it gets any worse then I fear that won't have any alternative.

Dr. D.

[Matt Dawson]

I have been finding Peeron slow. I don't know why, maybe server problems etc. Though Peeron reminds me of a small glass case museum compared to the supermarket of Bricklink or the grand museum of Brickset!

[drdavewatford]

Nice analogy, Matt !

Thing about Peeron however is that it seems to be the only site which allows you to log all of your loose pieces, which is brilliant if you build MOCs (or even if you have a bit missing from a set and need to know if you have a spare).

Peeron's running a bit quicker tonight, incidentally - maybe they heard our call !

Dr. D.

[davee123]

Well, unfortunately, Dan Boger (the guy who coded and runs the site) has sort of drifted away from the LEGO hobby world, and isn't overly big on improving the site at the moment.

I've talked with him a few times about this, and he's narrowed it down to something strange with the webserver application itself-- not any capacity issues with the architecture it's running on. What we've learned is that upon occasion, there are so many requests that the server is throttled and won't accept new requests until it's finished with with all the pending requests. So it develops a backlog of queries that get hung; even though each individual request gets processed very fast.

I suspect it has to do with search engines combing the site, since there are a LOT of URLs in there for different sorting, viewing, etc, for EVERY single part and set. Dan wasn't sure that was the problem, but I don't really know why. I'll be sure to remind him when I see him again later this month...

DaveE

[Darth Legolas]

I use Peeron, Brickset, and oftentimes Bricklink for their parts list. All three are pretty good, Peeron is great for instructions, Bricklink for finding parts and minifigs, and Brickset for linking everything!

[prateek]

I used to use Peeron, but since it slowed down, I stopped.

[paul_delahaye]

I find brickset.co.uk bloody really slow as well as peeron :-(

[Big Cam]

Well, unfortunately, Dan Boger (the guy who coded and runs the site) has sort of drifted away from the LEGO hobby world, and isn't overly big on improving the site at the moment.

I've talked with him a few times about this, and he's narrowed it down to something strange with the webserver application itself-- not any capacity issues with the architecture it's running on. What we've learned is that upon occasion, there are so many requests that the server is throttled and won't accept new requests until it's finished with with all the pending requests. So it develops a backlog of queries that get hung; even though each individual request gets processed very fast.

I suspect it has to do with search engines combing the site, since there are a LOT of URLs in there for different sorting, viewing, etc, for EVERY single part and set. Dan wasn't sure that was the problem, but I don't really know why. I'll be sure to remind him when I see him again later this month...

DaveE

Has he any thoughts on selling the site to someone who is more involved with the LEGO community still and has the time to maintain it? Everyone is entitield to moe on with their lives, but to have such a great resource go down th drain would be sad. Not saying I could afford it. Or could I??

Either way, that's a thought. Maybe EB would buy it and incorporate it into the site. That would be so cool.

[davee123]

Has he any thoughts on selling the site to someone who is more involved with the LEGO community still and has the time to maintain it? Everyone is entitield to moe on with their lives, but to have such a great resource go down th drain would be sad. Not saying I could afford it. Or could I??

Either way, that's a thought. Maybe EB would buy it and incorporate it into the site. That would be so cool.

Honestly, I doubt that's realistic. I'm pretty sure he wouldn't give up the domain name, although I suppose there could be a chance that he'd give up the code and data, if offered some sort of magic "super deal". But in my experience, such "super-deals" don't really exist. LUGNET actually changed hands, but that was really to make sure that the site didn't simply disappear (not really so that it would continue to grow and be the uber-site it was meant to be). BrickShelf was looking to stop doing hosting, but couldn't find anyone willing to take over.

Ultimately, the problem is that there aren't a lot of true web developers out there in the hobbyist community. Sure, there are a slightly bigger bunch who know how to put together some PHP pages or write some JavaScript. But not that many with bona-fide build-it-from-scratch C++/Java/Perl development skills. And those that DO know how are either not really into the hobby anymore, or don't want to actively take on the hassle of taking on someone else's code-- they know how much work it is to do, and aren't interested.

The thing that I think COULD happen would be if someone decided to make a brand new site from scratch that mirrored Peeron's functionality. Then, I could imagine an import of Peeron data, which Dan might help facilitate (depending on the details). Anyway, that's really still pie-in-the-sky at this point.

DaveE

[drdavewatford]

Thanks, davee123 - great to see a posting from someone with insider knowledge of the situation.

Peeron is invaluable to me, and even a slow Peeron is better than no Peeron at all ! Please let Dan (and Jennifer) know how much I (and I'm sure others) appreciate the resource that they've provided.

Cheers,

Dr. D.

[Big Cam]

Honestly, I doubt that's realistic. I'm pretty sure he wouldn't give up the domain name, although I suppose there could be a chance that he'd give up the code and data, if offered some sort of magic "super deal". But in my experience, such "super-deals" don't really exist. LUGNET actually changed hands, but that was really to make sure that the site didn't simply disappear (not really so that it would continue to grow and be the uber-site it was meant to be). BrickShelf was looking to stop doing hosting, but couldn't find anyone willing to take over.

Ultimately, the problem is that there aren't a lot of true web developers out there in the hobbyist community. Sure, there are a slightly bigger bunch who know how to put together some PHP pages or write some JavaScript. But not that many with bona-fide build-it-from-scratch C++/Java/Perl development skills. And those that DO know how are either not really into the hobby anymore, or don't want to actively take on the hassle of taking on someone else's code-- they know how much work it is to do, and aren't interested.

The thing that I think COULD happen would be if someone decided to make a brand new site from scratch that mirrored Peeron's functionality. Then, I could imagine an import of Peeron data, which Dan might help facilitate (depending on the details). Anyway, that's really still pie-in-the-sky at this point.

DaveE

Thanks for the insight. I know you know him personally.

I wasn't suggest he give up the name, I was suggesting he sell the entore webiste, name and all. This happens all the time. The only reason I suggested it is because you said yourself he has kind of moved on from the LEGO scene and doesn't have the time to maintain the site. A new owner would probably have the ambition and money to hire a new designer to keep it up to speed. That's all I was suggesting.

Edited September 11, 2009 by Big Cam

[davee123]

I wasn't suggest he give up the name, I was suggesting he sell the entore webiste, name and all. This happens all the time.

It does? I guess I'm not aware of that having happened in the LEGO community, other than with LUGNET. It happens in the corporate world all the time, but they have money and employ teams of developers to take development in whatever direction they want. But in the hobby community, all the major developed sites I know of are owned, built and maintained almost exclusively by a single developer.

I suppose other *forum* sites could change hands pretty easily-- It doesn't take too much technical knowledge to run a phpBB site or something. That's really a "just-add-water" website. Peeron's been programmed from the ground-up, along with other sites like BrickShelf, LUGNET, BrickSet, BrickLink, and MOCPages. Most other LEGO websites (EuroBricks, FBTB, Classic-Castle, Classic-Space, Brothers-Brick, BZPower, BrikWars, etc) aren't really "developed" per se. They use forum/blog/gallery/etc software that someone else wrote, which usually comes with a handy-dandy interface for administrators already built in. So if you want to change how the site looks, all you need to do is know the admin password, and be able to click a few buttons and fill out some web forms.

By contrast, if you wanted to change Peeron, you'd have to go in and edit the code it's built on. You'd need to have an understanding of Perl, programming in general, UNIX, HTTP protocol, MySQL, JavaScript/AJAX, and other fun things. And on top of that, you'd then have to learn the way that Peeron's implemented in particular, and rewrite the necessary libraries to achieve the desired result. It's sort of like the difference between a kid who tells you "I built a webpage!" (they signed up for MySpace), and someone who wrote their page from scratch in HTML and has their own hosting service and domain name.

A new owner would probably have the ambition and money to hire a new designer to keep it up to speed.

You'd have to have a lot of money to burn if you were going to hire a developer, plus paying for the hosting. 10's of thousands of dollars a year, I'd guess. Not many LEGO fans I know have that much extra cash or are willing to throw it around. You could probably count them on one hand, or less. Generally, you want find someone who is a developer to take over. They're rare, but not as rare as a LEGO fan who's willing to find and hire a 3rd party developer.

Anyway, suffice to say, it's very unlikely. But if someone does step up to the plate, it's possible that Dan might be on board.

DaveE

Edited September 11, 2009 by davee123

[Big Cam]

It does? I guess I'm not aware of that having happened in the LEGO community, other than with LUGNET. It happens in the corporate world all the time, but they have money and employ teams of developers to take development in whatever direction they want. But in the hobby community, all the major developed sites I know of are owned, built and maintained almost exclusively by a single developer.

I wasn't specifically referring to LEGO websites, just websites or forums all together. Usually when someone is done or no longer wants to deal with it, they sell the enitre site to someone else.

I suppose other *forum* sites could change hands pretty easily-- It doesn't take too much technical knowledge to run a phpBB site or something. That's really a "just-add-water" website. Peeron's been programmed from the ground-up, along with other sites like BrickShelf, LUGNET, BrickSet, BrickLink, and MOCPages. Most other LEGO websites (EuroBricks, FBTB, Classic-Castle, Classic-Space, Brothers-Brick, BZPower, BrikWars, etc) aren't really "developed" per se. They use forum/blog/gallery/etc software that someone else wrote, which usually comes with a handy-dandy interface for administrators already built in. So if you want to change how the site looks, all you need to do is know the admin password, and be able to click a few buttons and fill out some web forms.

By contrast, if you wanted to change Peeron, you'd have to go in and edit the code it's built on. You'd need to have an understanding of Perl, programming in general, UNIX, HTTP protocol, MySQL, JavaScript/AJAX, and other fun things. And on top of that, you'd then have to learn the way that Peeron's implemented in particular, and rewrite the necessary libraries to achieve the desired result. It's sort of like the difference between a kid who tells you "I built a webpage!" (they signed up for MySpace), and someone who wrote their page from scratch in HTML and has their own hosting service and domain name.

I understand forums are set up much easier than a website from the ground up. I am the first to admit I have no idea on the coding your talking about. That doesn't mean that Dan is the only one in the world who does. As I said it was mearly a suggestion since apparently Dan doesn't have much of an interest in it anymore. Someone with a basic understanding of the coding could easily keep the site running as is.

You'd have to have a lot of money to burn if you were going to hire a developer, plus paying for the hosting. 10's of thousands of dollars a year, I'd guess. Not many LEGO fans I know have that much extra cash or are willing to throw it around. You could probably count them on one hand, or less. Generally, you want find someone who is a developer to take over. They're rare, but not as rare as a LEGO fan who's willing to find and hire a 3rd party developer.

Anyway, suffice to say, it's very unlikely. But if someone does step up to the plate, it's possible that Dan might be on board.

DaveE

The main problem is the speed, which would not cost 10's of thousands of dollars a year to host, a few thousand maybe but I know multiple forum owners and website owners and it can get expensive but it also depends on who you go through.

Again, I don't want to make this an argument or a challenge as to who can prove more points so I'll have this as my last post in here.

I was only suggesting a solution to the problem at hand.

Cheers

[davee123]

That doesn't mean that Dan is the only one in the world who does. As I said it was mearly a suggestion since apparently Dan doesn't have much of an interest in it anymore. Someone with a basic understanding of the coding could easily keep the site running as is.

That's true, but that's also what Dan's doing at the moment. He's maintaining Peeron the way it is, which turns out to be slow. If the idea is to *improve* Peeron (make it faster, more robust, etc), then that's where you'll need someone who's more active. It's not like Dan's abandoned it or anything, he just doesn't really have the drive to continue making improvements to the site.

But again, even finding someone to maintain the status quo is difficult. Being a developer myself, and having close ties to LUGNET and Peeron, I've been a part of or been close to a lot of various initiatives that people have had trying in vain to find willing developers in the Lego community. LEGO even had trouble finding developers in the fan community to work on their more ambitious projects.

The main problem is the speed, which would not cost 10's of thousands of dollars a year to host, a few thousand maybe but I know multiple forum owners and website owners and it can get expensive but it also depends on who you go through.

Well, that's correct, although I will point out that hosting something like a PHP site is cheaper than hosting an "anything-goes" setup. For Peeron (or other ground-up sites), you've got to do things like run your own custom scripts (Perl, C, etc), run a crontab, get shell access, have unlimited DB access and administration, as well as have 24/7 monitoring from your ISP (like Pair does). But even then you're right, hosting isn't the big cost-- an "anything goes" account isn't all THAT much more.

The big cost is in finding a developer who can understand the code. There's a big mess of code that takes days to wade through before you start to grasp what it's doing, let alone seeing the finer points of things. So getting a developer to continually (throughout the course of a year) log in and support the code is going to be the cost. It means they have to be available (be "on-call" so to speak), and have to be familiar enough with the code to be useful and respond in a timely manner to various issues that crop up.

Again, I don't want to make this an argument or a challenge as to who can prove more points so I'll have this as my last post in here. I was only suggesting a solution to the problem at hand.

My apologies if I sound argumentative, it's not intended-- my point was simply that while yes, it'd be a great solution to get someone who was willing and able to take on the site, it's just not likely. It's sort of a "wouldn't it be great if..." discussion. Like when someone suggests that LEGO should go back to making Black Falcons, or old gray for collector sets, or stopping production of low-quality pieces in China. Yeah, I agree it'd be great, but it's not all that likely to happen. The problems in this case would be finding a willing party (very difficult), and convincing Dan to give it up (I think there's at least some chance).

DaveE

[Badsneaker]

You may want to contact Jennifer Finch. She co-created PEERON.

http://www.bricklink.com/aboutMe.asp?u=LPBricks

I buy LEGO from her Bricklink store. She may have some insight, although I'm not sure if she is still involved in the site.

[jonwil]

If the issue is search engines combing the site then (unless having the site indexed by Google is important), maybe the answer is a robots.txt that tells the search engines what not to index.

[davee123]

If the issue is search engines combing the site then (unless having the site indexed by Google is important), maybe the answer is a robots.txt that tells the search engines what not to index.

That's what I suggested to him a few years ago, but yes, he wanted to keep his placement in the search engines. Part of Peeron's income is from BrickLink associated stores, which get better prominence if the individual parts and set pages are allowed to be scraped. I had also suggested that it might be alleviated by adjusting the max number of concurrent webservers in his Apache configuration-- although I'm not sure if that's controlled by the ISP or not. But regardless, like I said, Dan didn't seem to think that screen scraping was the problem; I'm not sure why. I don't know what he thought the problem might be, but he seemed to doubt that it was the web spidering.

All I really know is that even though you might wait 12 seconds for the page to load, the page gets generated in, say, 0.3 seconds or something. I don't know the exact numbers, although I think for a while he would put the page generation time on the webpages (now it seems like it's gone). So the excess time isn't the machine or the database being bogged down-- it's something probably with the webserver or the network. Something's queuing up HTTP calls, and isn't handing them off to the CGI's immediately to get built. Could be any one of a lot of things, I suppose, but I'm not familiar enough with network issues to know. I suppose it could be a limit to the number of concurrent processes allowed on the box, too, which would be a BSD configuration issue controlled by Pair. But that's just another shot in the dark.

DaveE

[Bonaparte]

All I really know is that even though you might wait 12 seconds for the page to load, the page gets generated in, say, 0.3 seconds or something. I don't know the exact numbers, although I think for a while he would put the page generation time on the webpages (now it seems like it's gone). So the excess time isn't the machine or the database being bogged down-- it's something probably with the webserver or the network. Something's queuing up HTTP calls, and isn't handing them off to the CGI's immediately to get built. Could be any one of a lot of things, I suppose, but I'm not familiar enough with network issues to know. I suppose it could be a limit to the number of concurrent processes allowed on the box, too, which would be a BSD configuration issue controlled by Pair. But that's just another shot in the dark.

About 1,5 years ago we've moved Eurobricks from a regular webspace provider to a VPS (Virtual Private Server) running Linux. Since that moment we've been managing our own server. We've been writing some scripts (scheduled via crontab) to take automatic backups and such. The first months after moving to our own Linux server, the biggest problem was related to slowness and slow loading of our site. I've been reading a ton of articles on webserver tuning which (combined with some server upgrades) fixed the problem. I've learned a lot about tuning an (Apache) webserver for heavy use (we're having a lot of simultaneous people browsing EB) and I won't mind taking a look at the Apache configuration file of the Peeron.com server (I assume it's running Apache). If I'm also provide with hardware specification of the server (RAM, dynamic RAM, ...) I might have some suggestions for making it run smoother. Maybe someone already looked into all of this, in which case you can forget I wrote this

[drdavewatford]

Great - all we need is for someone to put Bonaparte in touch with Dan Boger and we have lift-off (maybe !)

Incidentally, someone mentioned Peeron co-founder Jennifer Finch in an earlier posting. I realised quite coincidentally that I'd actually bought some parts off her Bricklink site a few days ago and so I just had to e-mail her back and thank her for giving Peeron to the world !! I also discovered why the site is called Peeron.....

Cheers,

Dr. D.