I’m a Six Apart customer and I think they are getting a raw deal

You know, I’m a Six Apart customer (Shel and I host our book blog over at http://www.nakedconversations.com) and I join Brent Simmons in offering some moral support.

I, too, worked at UserLand and have seen the inards of a business that worked to host lots of people’s blogs. It’s not easy work. I watch the teams inside Microsoft work to keep things up. Hotmail, for instance, gets more than a billion spam messages a day (and those are the ones that it blocks from getting to users).

I’ve talked with Matt Mullenweg (the guy who runs WordPress) about this, and I’ve gotten to know the folks at F5 that make data center infrastructure equipment (we recently got a tour of F5 that’s pretty interesting). This stuff is not easy to keep up — even when you have the money.

I’ve been watching the commentary on how bad TypePad is. I know many of the folks there. They are working around the clock to get our sites back up and are doing the best they can.

Sorry, sometimes technology goes south even in the best-planned out systems.

How many of you have built systems that get hit with millions of visits per day? There aren’t many people who’ve done that in the world.

Yes, it’s frustrating when your business is down. I am feeling that frustration too (our book is launching in a few weeks and our traffic has been going up lately) but it doesn’t help to make ad hominem attacks that I’ve seen on some sites.

Plus, saying “I’m going somewhere else” isn’t real rational either. Why? Because who else has a professionally-run data center? Oh, you gonna host on your own servers? Yeah, let me know how that works when San Francisco is hit by the next big earthquake and you need your site to be up or let me know what happens when your server’s drive or networking card fails. Oh, you gonna put it on a cluster so that point of failure isn’t possible? Right. Got it. And when the box that your cluster is connected to goes down? Or when a crew digs up the fiber optic cables that go to your datacenter by accident (I’ve seen that happen, believe it or not).

Anyway, what we need is more transparency from the companies we’re betting our businesses and lives on. We need to know how many places our data is copied to. We need to know how redundant their systems are. And, in times of failures, we need more communication (especially with video and audio that’ll help let us know what’s going on). Although even here I loved getting TypePad’s updates. Made me feel better, even as I was nervous about our blog’s data.

This is one reason, though, that I’m happy having blogs on multiple services (my son’s is on Google’s Blogger, my wife’s is on MSN Spaces, mine here is on WordPress and I keep my Radio Userland blog around, and our book blog is on Six Apart’s TypePad).

None of these services is perfect, by the way. They all have their pros and cons. I’m still recommending TypePad to most businesses, though. I bet that they’ll get through this rough patch and go onto build a great business.

Oh, and to any of the technical folks, if you ever need help during such situations, don’t hesitate to contact me. We work with a ton of huge businesses (one of the largest Web sites in the world will annouce at www.mix06.com that they are switching to Windows, by the way). I know a few folks who keep Microsoft.com running and can martial technical help that can prove valuable in these situations.

Advertisements

33 thoughts on “I’m a Six Apart customer and I think they are getting a raw deal

  1. Sorry if this double posts, the first didn’t take and was cut off…
    >> one of the largest Web sites in the world will annouce they are switching to Windows

    Like

  2. Sorry if this double posts, the first didn’t take and was cut off…
    >> one of the largest Web sites in the world will annouce they are switching to Windows

    Like

  3. I find it really unfair that you tell people what they have to expect from a business. If they expected their blogs to be available 24/7 then that’s their expectations. If they haven’t been told by Six Apart that they’re not delivering on their expectations and never mentioned what type of service they plan on delivering to their customers, that’s their problem really.

    I know that for the price they’re paying these expectations are absolutely unrealistic and most of the people complaining will probably not pay for more availability.

    And I’m sorry, but to say “sometimes technology can go south” is not helping either. – You know, there are ways to make technology “not go south”. It’s just a matter of money.

    Ever noticed an airline shutting down because of computer problems for a few days? – Ever seen a bank not being able to process your transactions? – Ever heard of a major stock exchange not opening for a day?

    Sometimes there are glitches that are not covered by currently deployed infrastructure, but the last time an airline was unable to take off or a stock exchange not opening/closing at the right time you can bet that the infrastructure people were under a lot of stress to not let that happen ever again. – And yes, downtime is a problem and needs to be fixed. This is not “a part of life” that you have to accept.

    And about the largest sites switching to Windows… I’ll not get into a discussion with you about that. 🙂

    Like

  4. I find it really unfair that you tell people what they have to expect from a business. If they expected their blogs to be available 24/7 then that’s their expectations. If they haven’t been told by Six Apart that they’re not delivering on their expectations and never mentioned what type of service they plan on delivering to their customers, that’s their problem really.

    I know that for the price they’re paying these expectations are absolutely unrealistic and most of the people complaining will probably not pay for more availability.

    And I’m sorry, but to say “sometimes technology can go south” is not helping either. – You know, there are ways to make technology “not go south”. It’s just a matter of money.

    Ever noticed an airline shutting down because of computer problems for a few days? – Ever seen a bank not being able to process your transactions? – Ever heard of a major stock exchange not opening for a day?

    Sometimes there are glitches that are not covered by currently deployed infrastructure, but the last time an airline was unable to take off or a stock exchange not opening/closing at the right time you can bet that the infrastructure people were under a lot of stress to not let that happen ever again. – And yes, downtime is a problem and needs to be fixed. This is not “a part of life” that you have to accept.

    And about the largest sites switching to Windows… I’ll not get into a discussion with you about that. 🙂

    Like

  5. The problem is with the offering and the support you receive. Most businesses imply they’ll be up 24/7, most think they can do it without much work. That’s not true, but it can be done. It needs money and resources. The best thing is to be upfront with your customers. If you are going to hold their data you have to be responsible for it, specially if you are getting paid to do so… This is something I have done in banks, I know what it implies… Don’t get into it without knowing how much it will cost.

    Like

  6. The problem is with the offering and the support you receive. Most businesses imply they’ll be up 24/7, most think they can do it without much work. That’s not true, but it can be done. It needs money and resources. The best thing is to be upfront with your customers. If you are going to hold their data you have to be responsible for it, specially if you are getting paid to do so… This is something I have done in banks, I know what it implies… Don’t get into it without knowing how much it will cost.

    Like

  7. Hi Scoble,

    You’re being an apologist for 6A. If this was the first time I’d agree. But they’ve had too many problems over the past several months. Smells to me like something major is wrong internally at 6A. And as far as your comment about technology “goes south” sometimes, remember you work for Microsoft – so your expectations are automatically set low.

    Like

  8. Hi Scoble,

    You’re being an apologist for 6A. If this was the first time I’d agree. But they’ve had too many problems over the past several months. Smells to me like something major is wrong internally at 6A. And as far as your comment about technology “goes south” sometimes, remember you work for Microsoft – so your expectations are automatically set low.

    Like

  9. Mike: you should know that 6A doesn’t run on Microsoft technologies. But, my expectations with technology have been low for years. Remember in college? We rebooted our Macintoshes EVERY HOUR in order not to lose work (Quark XPress would crash after being run for more than an hour).

    That said, when you have built a reliable system that has millions of publishers and tens of millions of visitors, let me know. I’d love to hire you.

    Like

  10. Mike: you should know that 6A doesn’t run on Microsoft technologies. But, my expectations with technology have been low for years. Remember in college? We rebooted our Macintoshes EVERY HOUR in order not to lose work (Quark XPress would crash after being run for more than an hour).

    That said, when you have built a reliable system that has millions of publishers and tens of millions of visitors, let me know. I’d love to hire you.

    Like

  11. Hi Scoble,

    What expectations should 6A build relative to performance? And for that matter any Web 2.0 or on-demand service that hosts your data and becomes part of your business model?

    Like

  12. Hi Scoble,

    What expectations should 6A build relative to performance? And for that matter any Web 2.0 or on-demand service that hosts your data and becomes part of your business model?

    Like

  13. Mike: my expectations are always going to be higher than the business can deliver. But, this is a marketing opportunity. Those that are going to be more transparent (and who are going to invest more in infrastructure) are going to be the winners long term.

    Like

  14. Mike: my expectations are always going to be higher than the business can deliver. But, this is a marketing opportunity. Those that are going to be more transparent (and who are going to invest more in infrastructure) are going to be the winners long term.

    Like

  15. “Plus, saying “I’m going somewhere else” isn’t real rational either. Why? Because who else has a professionally-run data center? Oh, you gonna host on your own servers?”

    Well, yeah…Bobby. 6A isn’t in the only datacenter in the biz and this last outage wasn’t due to a hosting infrastructure (pipes/power/environmental) issue, but a hardware upgrade. The last big one was due to them running out of power and space in the datacenter. I see this as more a story of do they have the right people, in the right places with the adequate resources to host a pay service on this scale? This is to the point where it needs Amazon/Google/eBay/Yahoo sorts of scalability and failover and it’s not clear that 6A is in a position to do just that at this point.

    As for hosting your own, it’s much easier than one would think. Even a moderate two way box with enough RAM and a suitable array can serve millions of pages a month. The 6A issue isn’t with a single user or users, it’s with the size of the system as a whole. If you are a more or less regular Joe with a blog of less than a million page views a day (which at this point is most everyone except for you esteemed A listers), you can get something at Serverbeach or Rackspace (or the like) that offer some pretty good value. Even during the boom we were able to get space in what was then Exodus (now Savvis)for a few grand a month with enough power and pipes to serve a few million pages a day. The 6A folks are likely doing tens of millions of pages a day to millions of uniques but that doesn’t change the fact that when you hit that level, that’s the price of poker. The more 9s you want, the more it costs and right now it looks like 6A only has one 9. And that’s too bad… they seem like nice people in a bad spot.

    Like

  16. “Plus, saying “I’m going somewhere else” isn’t real rational either. Why? Because who else has a professionally-run data center? Oh, you gonna host on your own servers?”

    Well, yeah…Bobby. 6A isn’t in the only datacenter in the biz and this last outage wasn’t due to a hosting infrastructure (pipes/power/environmental) issue, but a hardware upgrade. The last big one was due to them running out of power and space in the datacenter. I see this as more a story of do they have the right people, in the right places with the adequate resources to host a pay service on this scale? This is to the point where it needs Amazon/Google/eBay/Yahoo sorts of scalability and failover and it’s not clear that 6A is in a position to do just that at this point.

    As for hosting your own, it’s much easier than one would think. Even a moderate two way box with enough RAM and a suitable array can serve millions of pages a month. The 6A issue isn’t with a single user or users, it’s with the size of the system as a whole. If you are a more or less regular Joe with a blog of less than a million page views a day (which at this point is most everyone except for you esteemed A listers), you can get something at Serverbeach or Rackspace (or the like) that offer some pretty good value. Even during the boom we were able to get space in what was then Exodus (now Savvis)for a few grand a month with enough power and pipes to serve a few million pages a day. The 6A folks are likely doing tens of millions of pages a day to millions of uniques but that doesn’t change the fact that when you hit that level, that’s the price of poker. The more 9s you want, the more it costs and right now it looks like 6A only has one 9. And that’s too bad… they seem like nice people in a bad spot.

    Like

  17. Can’t agree with you on this one Robert. When your business depends on a service it gets way more serious than you’re implying. I can live with outtages from time to time. It happens. But not for 18 hours with no access to the DB to do a flip and kewep your business running.

    Would you have said the same thing if the outtage had occurred the day after the book launch? I doubt it. And that’s where disaster recovery thinking should start.

    The way it was described to me -it ‘sounds’ and I don’t have the benefit of 1st hand knowledge so I’m happy to be proven wrong – but it sounds – like the kind of mistake a novice DB would have made.

    What worries me more than anything is that reports suggest companies in the ‘web 2.0’ category are not taking scale seriously. Some have suggested open warfare between execs and investor/advisors in this very point. Which I didn’t know before now.

    Rather than expend $$$ on getting the infrastrcuture right, they’re throwing money at features. That’s all very well if you’re making widgets – not if you’re a commercial platform or are creatin infrastructure.

    In regard to 6A generally, I don’t see their execution as robust enough to carry ‘business bloggers’ forward – at least not for the moment. That’s not to say they will never get it done – I sincerely hope for everyone that has an intrerest in seeing this current wave move forward that they do get it decisively resolved.

    But 6A’s had more than its fair share of ‘episodes’ on a bunch of stuff that looks to me like a lack of understanding around the general area of ‘scale.’ Either that or management isn’t up to the job.

    That’s why ‘I’m outta here.’

    It’s not personal, but what is it they say in the US…3 strikes and you’re out?

    Like

  18. Can’t agree with you on this one Robert. When your business depends on a service it gets way more serious than you’re implying. I can live with outtages from time to time. It happens. But not for 18 hours with no access to the DB to do a flip and kewep your business running.

    Would you have said the same thing if the outtage had occurred the day after the book launch? I doubt it. And that’s where disaster recovery thinking should start.

    The way it was described to me -it ‘sounds’ and I don’t have the benefit of 1st hand knowledge so I’m happy to be proven wrong – but it sounds – like the kind of mistake a novice DB would have made.

    What worries me more than anything is that reports suggest companies in the ‘web 2.0’ category are not taking scale seriously. Some have suggested open warfare between execs and investor/advisors in this very point. Which I didn’t know before now.

    Rather than expend $$$ on getting the infrastrcuture right, they’re throwing money at features. That’s all very well if you’re making widgets – not if you’re a commercial platform or are creatin infrastructure.

    In regard to 6A generally, I don’t see their execution as robust enough to carry ‘business bloggers’ forward – at least not for the moment. That’s not to say they will never get it done – I sincerely hope for everyone that has an intrerest in seeing this current wave move forward that they do get it decisively resolved.

    But 6A’s had more than its fair share of ‘episodes’ on a bunch of stuff that looks to me like a lack of understanding around the general area of ‘scale.’ Either that or management isn’t up to the job.

    That’s why ‘I’m outta here.’

    It’s not personal, but what is it they say in the US…3 strikes and you’re out?

    Like

Comments are closed.