Photo Camel Not Available Today?

[mikej]

So what happened to the Camel today? It was looking like it might be a crash rather than a problem with the ISP and I was afraid you were going to have to rebuild. Or was it a crash and you had some good backups?

Anyway, glad to see it's back up and running.


I just took a look and it appears you did have to recover from backups from a few days ago. I see a couple of posts I put in aren't there now. At least you had backups if that's the case.


Mike

[Cool Camel]

Originally Posted by Mike Jordan

* So what happened to the Camel today?

Mike,

First of all, to those of you who registered in the last four or five days, please re-register. Your accounts were not deactivated on purpose.

The server on which PhotoCamel is hosted went down for at least 8 hours on 8/12/2005 and 8/13/2005, taking this site and others with it. Even worse, we appear to have lost about four days' worth of threads and accounts. The host was supposed to be keeping daily backups of all sites, but claims that the drive that held the daily backups crashed. I was just about to institute my own daily backup plan in addition to the host's. This will now happen faster.

> > >

Rest assured that it's my number one priority to keep PhotoCamel running 24/7 and to ensure that your posts do not go missing. I'll do whatever it takes to make that happen. It breaks my heart to see recent posts like SpeedChallenge blown away. You can bet I'll be working in the background to see to it that we're not affected this way again.*

All I can ask and hope is that you take the time to try to recreate whatever posts you had made over the past few days.

Jay

[mikaelwardhana]

yes a couple of posts are missing...but yeah...its lucky that you've got the backup there.
Well, computers do fail sometimes. Reliability of a commercial server is one of the main concern when selecting your host.

[mikaelwardhana]

wow...some of the new members have been gone as well...they will have to re-register then...
It has got until 670 members yesterday... :-[

[Paul Shields]

Oh dear - looks like some issues at the hosting center http://forums.alwayswebhosting.com/s...?t=1852&page=1. Looks like they've lost a few days data...

If I was the photocamel admin I would be a little bit annoyed...

[Cool Camel]

Originally Posted by Paul

Oh dear - looks like some issues at the hosting center http://forums.alwayswebhosting.com/s...?t=1852&page=1. Looks like they've lost a few days data...

If I was the photocamel admin I would be a little bit annoyed...

More than a little.

>

[mikej]

Before people get up in arms at the ISP, take a few pictures in their shoes. I deal with computers and critical servers every day (and sometimes the nights as well). I'm the IT manager of a 24/7 call center that provides access to interpreters for hospitals, emergency rooms, law enforcement, posion control, etc. across the US. If our servers or PBX goes down, it can ruin more than a few people's days. And I don't care how well you prepare, how much redundency you have, how good the people are, there are a million and one ways for a system to find a way to crash. As the saying goes... it's not IF it happens, but WHEN.

I don't know how big the hosting site is that Photo Camel is on, but if it was a major crash (or it could have been a hack job as that seems to happen a lot amoung the hosting sites) then it's not that the site went down, but how fast they got the system back up and running so the sites like Photo Camel could get back up. Yes, it's a pain and yes it can be very annoying to all of us. But just like childbirth, the pain is soon forgotten..

Jay, hopefully your hosting site will be able to prevent what happened this time from happening again. Having your own backups is a good idea, at least for the structure and configuration of the forum. To be really effective you will have to keep it off the hosting site though. You might find it a good idea to find a seperate hosting site that is cheap but has enough disk space so you can store a tar file of the forum directory and a database backup. Or depending on what size it is (forums really aren't that big file size wise) just download it to your computer. The forum configuration and structure is the number one important piece followed by the database. You can have the database, but to be able to recover the database, you need the structure to be the same to dump it into. And unless you do a dump every night, if there is another crash you will probably still find yourself days behind. But if the hosting site does lose the backups as well, at least you will be able to rebuild without having to start from scratch.

In a couple of days, if there are no more problems, we'll be back to normal.

Thanks Jay.

Mike

[mikej]

I just got done reading the thread posted above. I know something like this is a real pain, but frankly, from the way some of those users moaned about how it was affecting their business, I have to think they are real idiots if their business was so critical that they intrusted their whole business at a single source of failure. Why wheren't they doing their own backups and maintaining a backup site? Or why wheren't they hosting their own site with their own equipment and at their facility? If they did host it themselves, they might find out just how easy it is for problems to crop up, even with the best emergency plans in place.

For those that don't understand database backups, it's common practice to have the SQL database backup to it's own server on a different physical drive every few hours. This is less impact on the database and all of the people that are using it. Then those backup files that the SQL database creates should be backed up to tape. There should also be replication to a second SQL server every few minutes or a bit longer set up so that if the primary SQL server goes down, the replicated server can be bought up as primary and take over as database server. It doesn't sound like they were doing this part though.

But if places like the US Government, Yahoo, Amazon, IBM, Microsoft, etc., can have failures, even with all their equipment, money and high level techs working there, how can some of those users expect a small hosting site to stay up 100% of the time? And be as cheap as they are? Some people just can't see beyound their own noses sometimes.

Mike

[Paul Shields]

Mike, you make sensible points - but I wouldn't excuse the hosting company just because it's a 'cheap' service. I too manage systems for a very large ISP/network provider in the UK, and there are certain basic pre-requisites that you would follow if offering a commercial hosting service - that's what the site owner is paying for after all These basics include not storing backups on the same server as the database :-\

Anyway, I do agree with your suggestions, but in this case, hindsight's a wonderful thing

[mikej]

Paul, I'm not excusing them. I don't know what happened and how much they could have prevented with better system and software management. Also, I don't know how big they are or what they charge for their service.

What I am saying is that people shouldn't get into a feeding freenzy of accusations against the hosting company just because they went down. It's easy to start throwing rocks at them (as it seems the users on the hosting site's message forum are doing) just because they did have problems since they are the biggest target.

But how many of those people are putting their "critical" customer business on a hosting site because of cost? So many people only want to pay $9 a year for 5 star service rather than finding a 5 star site and paying the monthly fee that that's going to cost. It takes money to have backup servers, redundent backup systems, spares sitting on the shelf, and the people with the level of expertise needed to keep it all running. Even if a site has 10,000 users, if they are each paying a few dollars a month for the service, the ISP is not going to be able to provide the service that those people expect. Not unless that ISP has another source of income or the owner is rich and is doing it for the fun. When I look for a hosting site, I don't base it on my price... at least price is not at the top of my list. I do look at what they are charging people though and stay away from those ultra cheap (and even just cheap) hosting sites.

Anyway, I'm not excusing them, but I'm not condeming them either. I've been in situations where when our system was down, we were costing people thousands per minute and I'm on the phone with VP's wanting to know when will it be up, how did it happen, what am I going to do for them about the loss, etc. . Now that will raise the blood pressure a bit and tax anyone's deoderant...


Mike

[Cool Camel]

Originally Posted by Mike Jordan

* Before people get up in arms at the ISP, take a few pictures in their shoes. I deal with computers and critical servers every day (and sometimes the nights as well). I'm the IT manager of a 24/7 call center that provides access to interpreters for hospitals, emergency rooms, law enforcement, posion control, etc. across the US.* If our servers or PBX goes down, it can ruin more than a few people's days.* And I don't care how well you prepare, how much redundency you have, how good the people are, there are a million and one ways for a system to find a way to crash.* As the saying goes... it's not IF it happens, but WHEN.*

* I don't know how big the hosting site is that Photo Camel is on, but if it was a major crash (or it could have been a hack job as that seems to happen a lot amoung the hosting sites) then it's not that the site went down, but how fast they got the system back up and running so the sites like Photo Camel could get back up.* Yes, it's a pain and yes it can be very annoying to all of us.* *But just like childbirth, the pain is soon forgotten..

* Jay, hopefully your hosting site will be able to prevent what happened this time from happening again.* Having your own backups is a good idea, at least for the structure and configuration of the forum. To be really effective you will have to keep it off the hosting site though.* You might find it a good idea to find a seperate hosting site that is cheap but has enough disk space so you can store* a tar file of the forum directory and a database backup.* Or depending on what size it is (forums really aren't that big file size wise) just download it to your computer.* The forum configuration and structure is the number one important piece followed by the database.* You can have the database, but to be able to recover the database, you need the structure to be the same to dump it into.* And unless you do a dump every night, if there is another crash you will probably still find yourself days behind. But if the hosting site does lose the backups as well, at least you will be able to rebuild without having to start from scratch.* *

* In a couple of days, if there are no more problems, we'll be back to normal.

Thanks Jay.

Mike

Mike, thanks. I will definitely be keeping daily backups (and more) off the hosting site. In fact, I already have backups; unfortunately, none was more recent than the four-days-ago backup that the host had. I'll be setting up a daily backup that will spool the database to a server I own. So this will be a third backup of the site every day.

As for the host company, they are promising that measures are being taken so that this never happens again. Whatever, it's my job to make sure things are covered no matter what they do.

[ellette]

So you want us to re-post posts like the two I had in the suggestions section?

Sara

[hartcons]

Well, at least we didn't have to start over from scratch. A good reminder about the importance of backups for all of us (I wonder how many pictures are lost each year because of hard disk failures?) I've read about folks who immediately make a copy of their uploaded pictures right after reading from the card just to be on the safe side.

Supposedly RAID systems can withstand the failure of a single disk but we had one real estate client that had a crash *because* of the RAID system itself. So, no system is perfect.

We have one client who was faithfully backing up all of his desktop icons (just the little icon pictures, not the applications they pointed to) to cd for a long time before we came onboard.

[Cool Camel]

Originally Posted by ellette

So you want us to re-post posts like the two I had in the suggestions section?*

Sara

You can repost anything you feel that you'd like to have here. Thanks.

J

[Alice Morrison]

Originally Posted by hartcons

Well, at least we didn't have to start over from scratch. A good reminder about the importance of backups for all of us (I wonder how many pictures are lost each year because of hard disk failures?) I've read about folks who immediately make a copy of their uploaded pictures right after reading from the card just to be on the safe side.

Supposedly RAID systems can withstand the failure of a single disk but we had one real estate client that had a crash *because* of the RAID system itself. So, no system is perfect.

We have one client who was faithfully backing up all of his desktop icons (just the little icon pictures, not the applications they pointed to) to cd for a long time before we came onboard.

What we all need are RAID systems for personal use, but I just never seem to have the time to set this up or learn how to. Whenever you rely on a traditional backup, there's always data that gets lost forever when something bad actually happens. It's only by writing everything twice immediately that you have a chance, but then that means buying twice as much storage as you really need at all times. It gets costly.

[AlbertP]

Originally Posted by th0t

There's new hard drive technology coming to market in about a year that will boost storage capacity by a factor of about 10. Large RAID arrays will become even more affordable. Although you can purchase a terabyte array for under US$1000 today---which is pretty amazing.

Nice explanation.

What is this new drive technology? 10 times the amount of storage I now have sounds great. I also see the hard drive as the bottleneck in my current computer, so faster couldn't hurt, either.

[Patzt]

Thanks for the PM Jay. Horrible for this to happen but looks like you have everything under control if this should happen another time (hopefully never).

[AlbertP]

Server outage haiku

My server hates me.
My backups flip me the bird.
PhotoCamel smiles.

--AlbertP

Haiku: A Japanese poem composed of three unrhymed lines of five, seven, and five syllables. Haiku often reflect on some aspect of nature.

[hartcons]

Originally Posted by Alice Morrison

What we all need are RAID systems for personal use, but I just never seem to have the time to set this up or learn how to. Whenever you rely on a traditional backup, there's always data that gets lost forever when something bad actually happens. It's only by writing everything twice immediately that you have a chance, but then that means buying twice as much storage as you really need at all times. It gets costly.

I've been using an iomega rev drive for photo backups. 35gb per cartridge uncompressed and 90gb compressed.

[hartcons]

Originally Posted by th0t

I've used a lot of Iomega gear over the years. Bernuelli (sp?) drives, Jazz drives, almost all forms of their ZIP drives. I've never liked their removable hard drive cartridges because they were so fragile and had a high failure rate. So I gave up on them. I hope that their current cartridge system is better.

Anyone remember Syquest drives? I've never had a problem with an Iomega product (and have also used most of them) but have heard the horror stories. I've mostly used them for backup purposes not for actively working on projects. So far so good with the rev drive; they supposedly did some things with the tapes this time to make them more reliable. I've avoided adding extra internal hard drives for backup purposes mainly because they tend to add heat/noise to the computer. We have several external firewire hard drives and have had a number of strange things happen with them (they start clicking and have to be power-cycled and/or things mysteriously get lost and/or locked) especially when using products like Adobe Illustrator in conjunction with them.