Neurox Media Network: | | IDF Webmail  
Neurox Media's American Idol 10, 9 & 8 / So You Think You Can Dance 8, 7 & 6 / Glee / X Factor Forums & Message Boards
Neurox Advertising   The banner image below is an ad.

Welcome Guest ( Log In | Register - It's quick & free! | Resend Validation )    

( Star this topic Star this topic | View starred topics )    Closed TopicStart new topic
> UPDATED: Database recovery - Sep 6/7 - missing threads, posts and accounts
post Sep 7th 2009, 12:03 AM
Post #1

buzz live!
Group Icon

Group: Neurox Media Administrators

Posts: 4,554
Gender: Male
From: SF Bay Area
Fav. AI3: Jasmine Trias
Fav. AI4: Carrie Underwood
Fav. TV show: 24

Our database server suffered a major power outage at 10:07PM EST on Sun, Sep 6. Upon powering back up, we experienced some database corruption and we initiated data recovery at 10:47PM. The recovery process finished at 12:47AM.

After preliminary inspection, we seemed to have lost perhaps about 1 hour to 4 hours of posts and other data just prior to 10PM. However, we are unsure of posts that have been edited during that timeframe, so we advise that you check recent and older postings, especially those that are long and not easily reproducible (ie: thread listings, artwork, etc.)

We have a recent backup that we can use to recover such postings, if you give us the full URL to the posting. This link can be found by clicking on the "Post #" link to the right of the posting date inside a thread.

We're sorry for any inconvenience this has caused. Our current budget doesn't give us a whole lot of power backup time. When power fails, we generally have about 15 minutes before servers begin powering down. The servers generally save as much data as possible before losing power but we didn't get much of a heads up with this.

UPDATE: For those of you experiencing the phenomenon where a large chunk of posts reappear within a thread when making a new post, it's because the topics data and the posts data aren't in sync. Making a new post will trigger a resync for the thread.

It won't bring back everything but it does retrieve quite a bit. I know this isn't a great approach, but let's just say that I was literally holding my breath for about 2 hours hoping that I won't need to go back to last Thursday's backup. This could've been much worse. ermm.gif

UPDATE 2: I can bring whole threads back from the dead too. So far I've resurrected 3 threads (they had 791 posts, 11k posts and 21k posts, respectively). All i need is the original link to the thread and I can try to recover as many posts as I can. Would also be nice to have following info as well: topic title, topic description, number of views, forum it belonged to.

UPDATE 3: Same goes for accounts. I'll need a link to a post where the user has posted. Then I can attempt to recover. So far got 4 accounts raised from the dead!

Also, if you have lost a thread and don't have a link. Try the Search feature by Google on the IDF homepage. Try searching for words and if you see it in the search results, copy the link and I can work with that. Also look in your browser history, in case you might've visited the thread recently. It will be a dead link now but I still need the link url.

UPDATE on Sep 7 regarding 2nd outage: Well, I'm fairly annoyed. Apparently our hosting facility didn't actually experience a power outage yesterday. Apparently, it was fan failure inside our server. The server runs on 2 CPUs and the fan on one of them died. Without a fan, the CPU runs super hot. To prevent the CPU from overheating, the server generally powers down after reaching a certain temperature.

Today, it happened again as soon as the traffic started increasing. The server didn't lose simply powered down to prevent hardware damage. Luckily, there is no additional data loss this evening. Some of the tricks I put into the server last night seemed to have paid off and data is saved on an hourly basis now. So even if it happens again (which it shouldn't, since we simply removed the 2nd CPU for now until we can buy a new fan), we shouldn't experience more than an hour's worth of data loss.

Sorry again for the outage. Unfortunately, hardware failure is a reality, no matter what kind of site you're running. We'll keep trying until we get it right. Thanks to everyone who've reported missing threads and accounts from last night's outage. If you notice threads missing, please PM me with the necessary info above. No need to feel rushed...the post data from missing threads will always be there. Even a year from now, we'd still be able to revive vanished threads.

buzz  IPB Image
neurox media
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

Closed TopicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

Lo-Fi Version 81 Time is now: Feb 23rd 2018, 9:32 AM

We now accept check/money order donations in addition to online donations via PayPal!  Help us pay for our servers, donate to Neurox Media!  Thanks!
Links: Reality TV Links - American Idol | SirLinksalot: American Idol

Powered by: Invision Power Board v2.1.7 © 2018  IPS, Inc.
Licensed to: Neurox Media