Reply to topic  [ 13 posts ] 
Site Outage - May 4 - 5 
Author Message
Forum Admin
User avatar

Joined: Sun Jan 26, 2003 3:35 am
Posts: 5233
Location: San Jose, CA
Post Site Outage - May 4 - 5
Hello all,

I wanted to give everyone some insight into why we were down for 24 hours, and where we stand now.

We currently host with MediaTemple (mt) on their GridServer Service (gs). I chose MediaTemple for several reasons, one being that they have a tremendous reputation for the absolute highest level of customer service (which having dealt with them a few times I can completely vouch for). Additionally, I chose to use (gs) because it was built from the ground up to withstand huge amounts of traffic and to avoid the general danger of hosting 200 sites on 1 machine--the "bad neighbor" syndrome. That is, if one of the sites on the particular box you are hosting on gets a hit from Digg or some such--you get screwed.

Unfortunately, building a system as complex as (gs) does not come without growing pains and difficulties of its own and we (along with ~2000 other (mt) clients) felt that over the past 24 hours.

The quick and dirty:

MediaTemple is aware that their original design for storage on the grid sucks--which is why they are in the process of rolling out a completely new storage architecture.

Our storage segment failed; all told 980 terabytes of data became inaccessible from (mt) client sites.

(mt) worked around the clock to get people back online. We came in about the middle, even as I type this about 900 more sites are still offline.

The Good News

We've been moved! We have been moved to (mt)'s completely redesigned storage architecture on an entirely new storage segment. This should mean that in the day-to-day, the site should be significantly faster.

Sorry again for the downtime--

Further reading:
http://weblog.mediatemple.net/weblog/20 ... -the-grid/
http://weblog.mediatemple.net/weblog/20 ... -and-more/
http://weblog.mediatemple.net/weblog/20 ... s-updates/

Here's a full account of this incident:
http://weblog.mediatemple.net/weblog/ca ... ilability/

_________________
Nalthien
Currently playing: Guild Wars 2 - Nelthian, Warrior
BbB Officer in DAoC, WoW, WAR, Aion, Rift, SWTOR, and GW2


Tue May 05, 2009 9:42 pm
Profile WWW
BbB Member
User avatar

Joined: Wed Feb 19, 2003 4:28 am
Posts: 1520
Location: tulsa oklahoma
Post Re: Site Outage - May 4 - 5
Well atleast they fixed it fast :) 24hours isn't to bad considering what happened :)


Tue May 05, 2009 10:09 pm
Profile YIM
BbB Officer
User avatar

Joined: Sat Jan 25, 2003 9:39 pm
Posts: 1832
Location: Jacksonville, FL
Post Re: Site Outage - May 4 - 5
Site is super fucking speedy now.

I signed up for their gs hosting service and used bbbguild.com as a referral. So if you get any goodies from them, it was me.


Wed May 06, 2009 9:37 am
Profile
Forum Admin
User avatar

Joined: Sun Jan 26, 2003 3:35 am
Posts: 5233
Location: San Jose, CA
Post Re: Site Outage - May 4 - 5
I'll keep my eye open for anything. :)

Yeah, the new storage architecture is a huge improvement!

_________________
Nalthien
Currently playing: Guild Wars 2 - Nelthian, Warrior
BbB Officer in DAoC, WoW, WAR, Aion, Rift, SWTOR, and GW2


Wed May 06, 2009 2:38 pm
Profile WWW
BbB God
User avatar

Joined: Mon Jan 27, 2003 11:26 am
Posts: 4803
Location: Minnesota
Post Re: Site Outage - May 4 - 5
the new forums are still allowing spam to come thru

_________________
Lodi
Guild Leader
Brotherhood by Blood


Wed May 06, 2009 4:58 pm
Profile
Forum Admin
User avatar

Joined: Sun Jan 26, 2003 3:35 am
Posts: 5233
Location: San Jose, CA
Post Re: Site Outage - May 4 - 5
Quote:
See also:


viewtopic.php?f=68&t=16926

_________________
Nalthien
Currently playing: Guild Wars 2 - Nelthian, Warrior
BbB Officer in DAoC, WoW, WAR, Aion, Rift, SWTOR, and GW2


Wed May 06, 2009 8:11 pm
Profile WWW
BbB Member
User avatar

Joined: Sat Jan 25, 2003 10:52 pm
Posts: 5052
Location: Kansas City, MO
Post Re: Site Outage - May 4 - 5
I have noticed that pages are loading noticeably faster than they used to.


Tue May 12, 2009 12:50 pm
Profile YIM WWW
BbB War
User avatar

Joined: Sun Jan 26, 2003 3:41 am
Posts: 527
Location: Columbus, Ga
Post Re: Site Outage - May 4 - 5
[quote="Unkowen"]I have noticed that pages are loading noticeably faster than they used to.[/quote]

same.


Wed May 13, 2009 10:36 pm
Profile
Forum Admin
User avatar

Joined: Sun Jan 26, 2003 3:35 am
Posts: 5233
Location: San Jose, CA
Post Re: Site Outage - May 4 - 5
Good news--I get 1 year of free web hosting out of the deal.

_________________
Nalthien
Currently playing: Guild Wars 2 - Nelthian, Warrior
BbB Officer in DAoC, WoW, WAR, Aion, Rift, SWTOR, and GW2


Thu May 14, 2009 10:01 am
Profile WWW
BbB Officer
User avatar

Joined: Sat Jan 25, 2003 9:39 pm
Posts: 1832
Location: Jacksonville, FL
Post Re: Site Outage - May 4 - 5
From me or cause the site was down?


Thu May 14, 2009 11:39 am
Profile
Forum Admin
User avatar

Joined: Sun Jan 26, 2003 3:35 am
Posts: 5233
Location: San Jose, CA
Post Re: Site Outage - May 4 - 5
Airmoros wrote:
From me or cause the site was down?


For the downtime. Since it was the second time in as many months that sites on Cluster.02 were down for over 24 hours. Last time they gave a 1 month credit--this time it was a 1 year credit.

_________________
Nalthien
Currently playing: Guild Wars 2 - Nelthian, Warrior
BbB Officer in DAoC, WoW, WAR, Aion, Rift, SWTOR, and GW2


Thu May 14, 2009 3:43 pm
Profile WWW
BbB Officer
User avatar

Joined: Sat Jan 25, 2003 9:39 pm
Posts: 1832
Location: Jacksonville, FL
Post Re: Site Outage - May 4 - 5
Good stuff


Thu May 14, 2009 4:28 pm
Profile
BbB Member
User avatar

Joined: Sat Jan 25, 2003 10:52 pm
Posts: 5052
Location: Kansas City, MO
Post Re: Site Outage - May 4 - 5
[quote="Nalthien"][quote="Airmoros"]From me or cause the site was down?[/quote]

For the downtime. Since it was the second time in as many months that sites on Cluster.02 were down for over 24 hours. Last time they gave a 1 month credit--this time it was a 1 year credit.[/quote]

That's good customer service.

Granted i'm sure they aren't loosing much in the deal, but gaining your satisifaction.


Thu May 21, 2009 1:06 pm
Profile YIM WWW
Display posts from previous:  Sort by  
Reply to topic   [ 13 posts ] 

Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware for PTF.