Ahh, Valleywag demonstrates one more time that they’ll print any damn thing sent to them without any care whatsoever about whether or not such a thing is true or not.
First of all, if an employee wanted to take down a datacenter, they wouldn’t need to “f___ up a lot of stuff” to do it.
There’s a big red button right by the door in every data center I’ve been to. It’s an emergency power off button. It has a huge sign next to it saying that if you push it without cause you’ll be prosecuted. Why? Cause it turns off the power to everything in the datacenter. Doing that would require admins to spend hours bringing back up their equipment (like it took today).
But it’s ridiculous to say that someone could “mess up” a rack. These things are all bolted to the floor — if they weren’t a small earthquake would cause major heck. A single human being could NOT push them over, even if you were using all of your weight.
Even if someone were able to push over a rack, it hardly would take out the entire datacenter, either. And, most employees who get access to datacenters don’t have access to racks anyway. In every colo center I’ve been in (three different ones in my career) all the machines are inside locked cages. So, how would an employee get access to a cage to do enough damage to bring down at least six companies? Hint: they couldn’t.
I gotta get myself a fake email address so I can send bulls**t into Valleywag and get it printed. Jason Calacanis told me about how he got free publicity for Mahalo that way.
Anyway, I love Valleywag. The stuff that people get printed there is *funny.* Just don’t confuse it for the truth.
Scobleizer 
http://www.canix.ca/a0_news.php?langue=en
BeerCo has an exclusive deal with a large cage owner at the Canix datacenter in downtown Montreal. We can set you up with Gigabit and or 10/100 Megabit connections at low enterprise prices for data reciprocation, and duplication purposes.
We can set you up with a network switch that will automatically shuttle network traffic from your San Fran center up to your duplication servers in Montreal seemlessly with no percieved downtime.
We can set up a light duplication system at Canix so that only a light version of your services run while your other center gets back up and working. So you do not have to cache all the data in duplication. We can do this at a very small enteprise class cost, and know the people at the center well.
We will set up all the software and hardware for you, for Linux OR windows, so that you only need to set up a duplication client on your side to shuttle the data in secure SSL. We will make a custom and highly scalable solution for you, no matter the time it takes.
Can you afford not to?
I deliver.
Thanks everybody.
LikeLike
http://www.canix.ca/a0_news.php?langue=en
BeerCo has an exclusive deal with a large cage owner at the Canix datacenter in downtown Montreal. We can set you up with Gigabit and or 10/100 Megabit connections at low enterprise prices for data reciprocation, and duplication purposes.
We can set you up with a network switch that will automatically shuttle network traffic from your San Fran center up to your duplication servers in Montreal seemlessly with no percieved downtime.
We can set up a light duplication system at Canix so that only a light version of your services run while your other center gets back up and working. So you do not have to cache all the data in duplication. We can do this at a very small enteprise class cost, and know the people at the center well.
We will set up all the software and hardware for you, for Linux OR windows, so that you only need to set up a duplication client on your side to shuttle the data in secure SSL. We will make a custom and highly scalable solution for you, no matter the time it takes.
Can you afford not to?
I deliver.
Thanks everybody.
LikeLike
Oh care went into it, they have journalists. But pot calling kettle black, you blog whatever that play-dough brain of yours happens to oatmeal out.
LikeLike
Oh care went into it, they have journalists. But pot calling kettle black, you blog whatever that play-dough brain of yours happens to oatmeal out.
LikeLike
And to pre-answer the question, NO, you obviously can’t run your own nameservers. I suggest UltraDNS which we use for our SN site, as we can make a CURL based php application to go in an automate changes to the DNS for an instant and un-noticable switch to Montreal from San Fran.
No one will realize what happened, and we keep all customers 1000% confidential.
LikeLike
Robert, remember the old “post first/fast”, “correct later” mantra. Ya, I know, it’s never been right but was/is practiced by many. Ring a bell?
LikeLike
And to pre-answer the question, NO, you obviously can’t run your own nameservers. I suggest UltraDNS which we use for our SN site, as we can make a CURL based php application to go in an automate changes to the DNS for an instant and un-noticable switch to Montreal from San Fran.
No one will realize what happened, and we keep all customers 1000% confidential.
LikeLike
Robert, remember the old “post first/fast”, “correct later” mantra. Ya, I know, it’s never been right but was/is practiced by many. Ring a bell?
LikeLike
During regular uptime BeerCo pumps beer through their Gigabit pipes:-)
LikeLike
During regular uptime BeerCo pumps beer through their Gigabit pipes:-)
LikeLike
Robert,
I’ve worked in MANY data centers in my career. If an employee with full access (never give full access to anyone) to everything wanted to really throw the cat in among the pigeons, it wouldn’t be hard.
Here are a few things that could be done if someone was to go rogue with too much leverage.
– On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises. Kill VRRP. All the IP space could be advertised so the router(s) are targets for bots and worms, etc. Blackhole/null routes could be put in place for entire IP ranges. Routers should be watched and the logs looked at everyday by qualified people.
– Firewall rules could be altered/established to allow bad ports and services pointed at mission-critical machines.
– TTL could be set on the DNS server entries to the maximum setting so dynamic websites do not update when content is updated. Push changes out to the root servers.
In a data center, jobs SHOULD be compartmentalized. Firewall guys work firewalls, router guys work routers, DNS guys do DNS. It’s been this way everywhere I’ve ever worked. No one ever had the keys to the Lamborghini, so the speak. Good security is more than just firewalls, it’s a process, procedure, it’s limited access.
Worse, most data centers allow their people to have SSH access to servers, routers, and firewalls. Kerberos, SSH, and RSA keyfobs offer no security against inside threats. Believe it or not, quite a few threats happen from WITHIN an organization. From its own people.
Trust but verify. EVERYTHING.
LikeLike
Robert,
I’ve worked in MANY data centers in my career. If an employee with full access (never give full access to anyone) to everything wanted to really throw the cat in among the pigeons, it wouldn’t be hard.
Here are a few things that could be done if someone was to go rogue with too much leverage.
– On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises. Kill VRRP. All the IP space could be advertised so the router(s) are targets for bots and worms, etc. Blackhole/null routes could be put in place for entire IP ranges. Routers should be watched and the logs looked at everyday by qualified people.
– Firewall rules could be altered/established to allow bad ports and services pointed at mission-critical machines.
– TTL could be set on the DNS server entries to the maximum setting so dynamic websites do not update when content is updated. Push changes out to the root servers.
In a data center, jobs SHOULD be compartmentalized. Firewall guys work firewalls, router guys work routers, DNS guys do DNS. It’s been this way everywhere I’ve ever worked. No one ever had the keys to the Lamborghini, so the speak. Good security is more than just firewalls, it’s a process, procedure, it’s limited access.
Worse, most data centers allow their people to have SSH access to servers, routers, and firewalls. Kerberos, SSH, and RSA keyfobs offer no security against inside threats. Believe it or not, quite a few threats happen from WITHIN an organization. From its own people.
Trust but verify. EVERYTHING.
LikeLike
They knew it was BS when they printed it:
“We’re sure 365 Main will deny that such a thing could ever happen. And, conveniently, the neighborhood is having power troubles, too.”
The absurdity of that last line is phenomenal…”Yes, the suspect was conveniently dead at the time of the crime, but how else can you explain…”
My daughter is in law school and Valleywag has come up in class discussion. They’ll probably be sued out of existence one day when they cross the line and the wrong party.
LikeLike
They knew it was BS when they printed it:
“We’re sure 365 Main will deny that such a thing could ever happen. And, conveniently, the neighborhood is having power troubles, too.”
The absurdity of that last line is phenomenal…”Yes, the suspect was conveniently dead at the time of the crime, but how else can you explain…”
My daughter is in law school and Valleywag has come up in class discussion. They’ll probably be sued out of existence one day when they cross the line and the wrong party.
LikeLike
“- On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises.”
What are routers?
Unix right. Take the /etc/shadow and toss it through the dictionary. With a powerful enough cipher throughput, it shouldn’t take very long.
No miscreant is going to sit there and bother with a 32 char password. If you know what you’re doing, nobody is going to be able to seize your infrastructure.
“During regular uptime BeerCo pumps beer through their Gigabit pipes:-)”
If your company pays, we’ll find a solution to whatever IT issue you may throw at us, and we’ll do it with a smile. Beer through gigabit pipes included. We’ll say yes.
LikeLike
Interesting story. I wonder how many bloggers print BS on a daily basis? For that matter, I wonder how many print reporters write pure propaganda?
There is something to say for information literacy.
LikeLike
“- On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises.”
What are routers?
Unix right. Take the /etc/shadow and toss it through the dictionary. With a powerful enough cipher throughput, it shouldn’t take very long.
No miscreant is going to sit there and bother with a 32 char password. If you know what you’re doing, nobody is going to be able to seize your infrastructure.
“During regular uptime BeerCo pumps beer through their Gigabit pipes:-)”
If your company pays, we’ll find a solution to whatever IT issue you may throw at us, and we’ll do it with a smile. Beer through gigabit pipes included. We’ll say yes.
LikeLike
Interesting story. I wonder how many bloggers print BS on a daily basis? For that matter, I wonder how many print reporters write pure propaganda?
There is something to say for information literacy.
LikeLike
Or if the router is ROM boot only and you don’t have a prompt, you can simply reset it, and restore the backup of the settings you professionally and regularly backed up to secure media and put in a safe place.
LikeLike
Or if the router is ROM boot only and you don’t have a prompt, you can simply reset it, and restore the backup of the settings you professionally and regularly backed up to secure media and put in a safe place.
LikeLike
Chris,
Infrastructure doesn’t need to be seized. A dedicated outsider or even insider could use a bot network and DDoS your routers into oblivion.
Long passwords are rather useless in any event. I’m not advocating using names or dictionary words. I use nothing but passphrases on all my machines.
For example, you could take a lyric from a song and turn it into a passphrase. Just make sure the passphrase isn’t an actual word.
Mary had a little lamb, it’s fleece was white as snow.
Take the first letter from every word…
Mhallifwwas. Could be made better…
Mh@1lifWwaS
Can get better with a little thinking…
The thing to do on servers is to create several root passwords and give each responsible admin a separate one. This way you know who did what.
SSH, Kerberos, RSA keyfobs. etc., all need to be used. Together. Two or three-factor authentication. What you know (ssh passwd), what you know (kerb passwd), what you have.
Set kerberos’ time clock skew to 1 minute. Make sure employess’ laptops for remote access are synchronized with the kerberos time server clock. Make it a thinking man’s game. The more employees have to think, the less mistakes they make.
LikeLike
Chris,
Infrastructure doesn’t need to be seized. A dedicated outsider or even insider could use a bot network and DDoS your routers into oblivion.
Long passwords are rather useless in any event. I’m not advocating using names or dictionary words. I use nothing but passphrases on all my machines.
For example, you could take a lyric from a song and turn it into a passphrase. Just make sure the passphrase isn’t an actual word.
Mary had a little lamb, it’s fleece was white as snow.
Take the first letter from every word…
Mhallifwwas. Could be made better…
Mh@1lifWwaS
Can get better with a little thinking…
The thing to do on servers is to create several root passwords and give each responsible admin a separate one. This way you know who did what.
SSH, Kerberos, RSA keyfobs. etc., all need to be used. Together. Two or three-factor authentication. What you know (ssh passwd), what you know (kerb passwd), what you have.
Set kerberos’ time clock skew to 1 minute. Make sure employess’ laptops for remote access are synchronized with the kerberos time server clock. Make it a thinking man’s game. The more employees have to think, the less mistakes they make.
LikeLike
One more VC: I’m a journalist, by training, too. That doesn’t mean that what you read here is journalism.
Whenever I’ve been the subject of Valleywag stories I’ve found they NEVER call for my reaction, or to give my side of the story. They only reprint stuff that’s emailed into them from readers. Most of the time it turns out to be false because they can’t tell whether they are being hoaxed or not. They just like to be entertaining. That they certainly are, but don’t say it’s “journalism.” They even admit it’s a gossip magazine. I guess you think the National Enquirer is “journalism” too.
Sigh.
LikeLike
One more VC: I’m a journalist, by training, too. That doesn’t mean that what you read here is journalism.
Whenever I’ve been the subject of Valleywag stories I’ve found they NEVER call for my reaction, or to give my side of the story. They only reprint stuff that’s emailed into them from readers. Most of the time it turns out to be false because they can’t tell whether they are being hoaxed or not. They just like to be entertaining. That they certainly are, but don’t say it’s “journalism.” They even admit it’s a gossip magazine. I guess you think the National Enquirer is “journalism” too.
Sigh.
LikeLike
Whether you’re a journalist, a blogger or a gossip monger, you can still be sued for libel. Of course, Robert knows this, but I’m surprised by the number of people who think it’s only “journalists” that are beholden to defamation laws and that “freedom of speech” means you can publish anything you want to without consequence.
LikeLike
Whether you’re a journalist, a blogger or a gossip monger, you can still be sued for libel. Of course, Robert knows this, but I’m surprised by the number of people who think it’s only “journalists” that are beholden to defamation laws and that “freedom of speech” means you can publish anything you want to without consequence.
LikeLike
“But it’s ridiculous to say that someone could “mess up” a rack”
Not really. “A rack” generally refers to the physical rack itself as well as all of the equipment housed on it. Slace through the neatly organized twisted pair cables, damage the rack fans, and a few other things I can think of doing with a screw driver, would pretty much “mess up a rack” for a day or two at least.
I’m not so sure about the big red button. That would be disruptive for sure, but most modern hardware, particularly of the server variety, is designed to survive having the plug yanked.
The Valleywag story is obviously wrong, but interestingly, PG&E still says the cause is unknown.
LikeLike
“But it’s ridiculous to say that someone could “mess up” a rack”
Not really. “A rack” generally refers to the physical rack itself as well as all of the equipment housed on it. Slace through the neatly organized twisted pair cables, damage the rack fans, and a few other things I can think of doing with a screw driver, would pretty much “mess up a rack” for a day or two at least.
I’m not so sure about the big red button. That would be disruptive for sure, but most modern hardware, particularly of the server variety, is designed to survive having the plug yanked.
The Valleywag story is obviously wrong, but interestingly, PG&E still says the cause is unknown.
LikeLike
@13 “I’m a journalist, by training, too”
Why? Because you took some classes at SJSU? I’m pretty sure they want you to stop admitting that.
“I guess you think the National Enquirer is “journalism” too.”
Uh..by your past definitions, it is,
LikeLike
@13 “I’m a journalist, by training, too”
Why? Because you took some classes at SJSU? I’m pretty sure they want you to stop admitting that.
“I guess you think the National Enquirer is “journalism” too.”
Uh..by your past definitions, it is,
LikeLike
>>The absurdity of that last line is phenomenal…
Dawn, it’s the absurdity that ensures that Valleywag will NOT be sued out of existence. They know what they’re doing.
Talk to your daughter again after she graduates from law school and practices First Amendment law a few years… then she’ll know what she’s talking about. Law students don’t know much of anything, and I say that from direct experience.
Not defending Valleyway Robert, just saying. Old lawyers die hard. 🙂
LikeLike
>>The absurdity of that last line is phenomenal…
Dawn, it’s the absurdity that ensures that Valleywag will NOT be sued out of existence. They know what they’re doing.
Talk to your daughter again after she graduates from law school and practices First Amendment law a few years… then she’ll know what she’s talking about. Law students don’t know much of anything, and I say that from direct experience.
Not defending Valleyway Robert, just saying. Old lawyers die hard. 🙂
LikeLike
You’re wrong Scoble. I spend all day in data centers (multiple ones, well known big ones and smaller ones too) and an employee with the right set of keys could wreak havoc in a matter of minutes. You could walk up, unlock racks and start jerking out cables (power, network). Each rack would takes less than 30 seconds to completely hose for a while (not total destruction, just chaos). Network cables would be worthless, power downs, etc. Give me 5 minutes with a master key (oh yes they exist) and I could ruin the weeks for several admins.
LikeLike
You’re wrong Scoble. I spend all day in data centers (multiple ones, well known big ones and smaller ones too) and an employee with the right set of keys could wreak havoc in a matter of minutes. You could walk up, unlock racks and start jerking out cables (power, network). Each rack would takes less than 30 seconds to completely hose for a while (not total destruction, just chaos). Network cables would be worthless, power downs, etc. Give me 5 minutes with a master key (oh yes they exist) and I could ruin the weeks for several admins.
LikeLike
Brian, I know that libel cases are next to impossible to win, but there are people out there, people like Donald Trump as an extreme example, who will sue them for the sheer principle of it.
The National Enquirer has been sued a lot and there have been huge settlements out of it and tons of legal costs. Does Valleywag have that kind of money? Seems to me that unless they have very deep pockets, they can’t sustain a high level of exposure forever.
LikeLike
Brian, I know that libel cases are next to impossible to win, but there are people out there, people like Donald Trump as an extreme example, who will sue them for the sheer principle of it.
The National Enquirer has been sued a lot and there have been huge settlements out of it and tons of legal costs. Does Valleywag have that kind of money? Seems to me that unless they have very deep pockets, they can’t sustain a high level of exposure forever.
LikeLike
btw, Brian, my daughter would be the very first one to agree with you that she doesn’t know anything.
LikeLike
btw, Brian, my daughter would be the very first one to agree with you that she doesn’t know anything.
LikeLike
UPS and generators take care of most power issues, data centers all over the impact area that kept on humming, so CLEARLY something else is a factor, incompetence or internal sabotage. The smoke hasn’t cleared yet, so Valleywag’s theory is still very much in play. 365Main has a history of blowouts, so I am leaning towards incompetence, as reports are falling in that their generators didn’t kick in. Regardless, I suspect the real truth will never come out.
LikeLike
UPS and generators take care of most power issues, data centers all over the impact area that kept on humming, so CLEARLY something else is a factor, incompetence or internal sabotage. The smoke hasn’t cleared yet, so Valleywag’s theory is still very much in play. 365Main has a history of blowouts, so I am leaning towards incompetence, as reports are falling in that their generators didn’t kick in. Regardless, I suspect the real truth will never come out.
LikeLike
Agreed with 15. Having been in multiple datacenters (mostly private due to the nature of my profession) but way more than Robert has, I can say that taking down a data center isn’t actually very difficult. Especially a rack. As long as you know what you’re doing, it wouldn’t take too long. In fact, there’s one data center that I know of that used to not even have redundant routers. Pull the plug on the the single point of failure, and the whole place falls.
It really depends on how the datacenter is setup and how detailed anyone would know about the internal workings and design. Not defending Valleywag, since I don’t read that … uhh… gossip, but really now…
You don’t really need to push the red button to do some serious damage if you knew how they designed their layout. We used to joke about switch techs tripping over the power cable in one of the data centers due to the design.
LikeLike
Agreed with 15. Having been in multiple datacenters (mostly private due to the nature of my profession) but way more than Robert has, I can say that taking down a data center isn’t actually very difficult. Especially a rack. As long as you know what you’re doing, it wouldn’t take too long. In fact, there’s one data center that I know of that used to not even have redundant routers. Pull the plug on the the single point of failure, and the whole place falls.
It really depends on how the datacenter is setup and how detailed anyone would know about the internal workings and design. Not defending Valleywag, since I don’t read that … uhh… gossip, but really now…
You don’t really need to push the red button to do some serious damage if you knew how they designed their layout. We used to joke about switch techs tripping over the power cable in one of the data centers due to the design.
LikeLike
I, on the other hand, am defending Valleywag. It is awesome. Well, it used to be. But seriously folks, it’s fucking Valleywag. I hope they make billions of dollars, cause hundreds of heart attacks and never print a true word.
LikeLike
I, on the other hand, am defending Valleywag. It is awesome. Well, it used to be. But seriously folks, it’s fucking Valleywag. I hope they make billions of dollars, cause hundreds of heart attacks and never print a true word.
LikeLike
Valley wag reminds me of one of my Fathers maxims:
“Never let the truth stand in the way of a good story”.
I tend to view their articles like reading Private Eye you know they only basis they have in facts is that a journalist wrote an article.
LikeLike
Valley wag reminds me of one of my Fathers maxims:
“Never let the truth stand in the way of a good story”.
I tend to view their articles like reading Private Eye you know they only basis they have in facts is that a journalist wrote an article.
LikeLike
No. Wait a second. Data centers have serious power backup systems. Power grid failures happen all the time. But not data center outages. Particularly not failures at data centers like 365Main, which has an industrial strength UPS system, batteries, and redundant diesel generators. Something’s up here, something that isn’t routine, and customers should be asking these guys some very hard questions.
LikeLike
No. Wait a second. Data centers have serious power backup systems. Power grid failures happen all the time. But not data center outages. Particularly not failures at data centers like 365Main, which has an industrial strength UPS system, batteries, and redundant diesel generators. Something’s up here, something that isn’t routine, and customers should be asking these guys some very hard questions.
LikeLike
LayZ: well, seeing that the journalism department at San Jose State University has asked me to speak there not once, but three times, tells me that you have no clue about what they think of me.
LikeLike
LayZ: well, seeing that the journalism department at San Jose State University has asked me to speak there not once, but three times, tells me that you have no clue about what they think of me.
LikeLike
yeh But you know that those dodgy urine testing companies that frighten/bully ompanies into doing bad stuff are going to have a field day plugging testing at work for even more slaves ^h^h^h^h^h workers now 😦
LikeLike
yeh But you know that those dodgy urine testing companies that frighten/bully ompanies into doing bad stuff are going to have a field day plugging testing at work for even more slaves ^h^h^h^h^h workers now 😦
LikeLike
re chriss of hosing cisco
“On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises.”
”
Err recovering a bricked router is somthing your learn very early on in the CCNA course – they should have a complete set of configfiles for every router and switch ready to reload.
LikeLike
re chriss of hosing cisco
“On routers, the routes could be changed then enable password(s) changed to something terribly difficult to guess. Remove all IOS software from premises.”
”
Err recovering a bricked router is somthing your learn very early on in the CCNA course – they should have a complete set of configfiles for every router and switch ready to reload.
LikeLike
Something’s up here, something that isn’t routine
Exactly. Plus, the truth is usually on the other side of a Scoble analysis. So Valleywag might not be as outlandish as it may seem.
LikeLike
Something’s up here, something that isn’t routine
Exactly. Plus, the truth is usually on the other side of a Scoble analysis. So Valleywag might not be as outlandish as it may seem.
LikeLike
One more VC: if it really was a rogue employee there would have been LOTS of witnesses who would have talked. Have you ever been inside a datacenter during the day? I have. There’s always TONS of people inside and not just people who work for the colo either.
And, let’s say there wasn’t any witnesses. Then there’d be tons of things left out in the open when things started going wrong. You’d see unplugged wires, destroyed machines, etc.
They wouldn’t have been able to clean that up within minutes (which they’d have needed to do).
Also, if someone hit the power button I’m pretty sure that works like most fire alarms and would have sprayed colored ink all over the person who pushed it, so they would have been easily identifyable.
But, yet, no data has come out corroborating the “drunken employee” story.
This story was made up. Pure and simple.
And Valleywag printed it without going on scene and checking out the facts.
Valleywag does this all the time. They write shit about me and never bother to check the facts out and never bother getting MY side of the story.
They just love being entertaining. Valleywag +is+ entertaining, that’s for sure. But if you believe it then you’re full of it too.
Of course, why am I arguing with a guy who isn’t a VC and won’t identify himself. I might just be arguing with Nick Denton himself here. Or maybe you’re Fake Steve Jobs? Ahh, I see your IP address doesn’t come from Boston, so that counts that out.
LikeLike
One more VC: if it really was a rogue employee there would have been LOTS of witnesses who would have talked. Have you ever been inside a datacenter during the day? I have. There’s always TONS of people inside and not just people who work for the colo either.
And, let’s say there wasn’t any witnesses. Then there’d be tons of things left out in the open when things started going wrong. You’d see unplugged wires, destroyed machines, etc.
They wouldn’t have been able to clean that up within minutes (which they’d have needed to do).
Also, if someone hit the power button I’m pretty sure that works like most fire alarms and would have sprayed colored ink all over the person who pushed it, so they would have been easily identifyable.
But, yet, no data has come out corroborating the “drunken employee” story.
This story was made up. Pure and simple.
And Valleywag printed it without going on scene and checking out the facts.
Valleywag does this all the time. They write shit about me and never bother to check the facts out and never bother getting MY side of the story.
They just love being entertaining. Valleywag +is+ entertaining, that’s for sure. But if you believe it then you’re full of it too.
Of course, why am I arguing with a guy who isn’t a VC and won’t identify himself. I might just be arguing with Nick Denton himself here. Or maybe you’re Fake Steve Jobs? Ahh, I see your IP address doesn’t come from Boston, so that counts that out.
LikeLike
Valleywag is entertaining, and, it is a genre of journalism that predates the printing press. The notion that only hard news is journalism is a shallow one. What people like Dawn and Robert would criticize, if they remotely got the issues involved, is Valleywag’s failure to emphasize that it is usually publishing opinion, not fact. (Not necessarily honestly held, opinion, either.) But, being a clever person himself, Nick Denton likely expects his audience to figure that out for themselves.
Ditto Brian on the rather reckless tossing around of claims of libel and slander around here. Denton knows he is on solid ground because the targets of his snarkiness (okay, lies, if you prefer) are public figures. Short of him publishing, that say, Robert Scoble buggered the bursar of San Jose State while podcasting at 365 Main, taking calls on his N95 all the while, Denton is not going to lose a libel or slander case if anyone is foolish enough to bring one. (Actually, he might get away with that example.) The flip side of being an attention seeker is that you might not like some of the attention you get.
I have no idea what caused the problems at 365Main, but find the suggestion of a drunken employee going postal amusing. An inebriated gourmand rat having eaten through various cables would be even more entertaining. The truth will probably be boring.
LikeLike
Valleywag is entertaining, and, it is a genre of journalism that predates the printing press. The notion that only hard news is journalism is a shallow one. What people like Dawn and Robert would criticize, if they remotely got the issues involved, is Valleywag’s failure to emphasize that it is usually publishing opinion, not fact. (Not necessarily honestly held, opinion, either.) But, being a clever person himself, Nick Denton likely expects his audience to figure that out for themselves.
Ditto Brian on the rather reckless tossing around of claims of libel and slander around here. Denton knows he is on solid ground because the targets of his snarkiness (okay, lies, if you prefer) are public figures. Short of him publishing, that say, Robert Scoble buggered the bursar of San Jose State while podcasting at 365 Main, taking calls on his N95 all the while, Denton is not going to lose a libel or slander case if anyone is foolish enough to bring one. (Actually, he might get away with that example.) The flip side of being an attention seeker is that you might not like some of the attention you get.
I have no idea what caused the problems at 365Main, but find the suggestion of a drunken employee going postal amusing. An inebriated gourmand rat having eaten through various cables would be even more entertaining. The truth will probably be boring.
LikeLike
wreck, the bottom line is you should back up all your settings, no matter what router config or router hardware you use.
I believe at Canix datacenter they do that.
In our office we have a Cisco router, and a Netgear FVL328 for VPN and office routing, and the settings are backed up.
The only thing a person could do at a datacenter before getting tackled would be to kill the master routers in the control room. If they started going into every single cage messing everything up, they would be stopped before they got very far.
So basically that’s just 1 or 2 routers on the master to restore.
What BCS could do for enterprise, yes, or any other site with critical apps is get some low cost 1Us from cybertronpc.com and set you up with a coloc rack in a cage at Canix on 100Megabit or gigabit. This rack will run mirrors of all your critical services, say you have XML RPC services running through Axis or another B2B. We can make a light backup of those service to keep distributers running even when your main servers are down with switching through UltraDNS.net, or even a dedicated 3rd party DNS located somewhere super safe.
The mirrors in Montreal will be fed with only diff data from your master in San Fransisco fed via SSL secure data transfer.
When your datacenter in San Fran hits the floor, the software will detect it and switch the seperate DNS of all your critical business services, including but not limited to HTTP web to the Montreal location.
While doing this for HTTP and websites only is more trivial, our service can do it for Web services like a big manufacturer that feeds datasheets to customers via ICS 650 format, xls or SOAP.
What happens when distributers need to check the stock on items from a manufacturer and the datacenter went down?
Those are orders for several thousands of dollars.
Can people really afford to deal with downtime?
We can help.
LikeLike
wreck, the bottom line is you should back up all your settings, no matter what router config or router hardware you use.
I believe at Canix datacenter they do that.
In our office we have a Cisco router, and a Netgear FVL328 for VPN and office routing, and the settings are backed up.
The only thing a person could do at a datacenter before getting tackled would be to kill the master routers in the control room. If they started going into every single cage messing everything up, they would be stopped before they got very far.
So basically that’s just 1 or 2 routers on the master to restore.
What BCS could do for enterprise, yes, or any other site with critical apps is get some low cost 1Us from cybertronpc.com and set you up with a coloc rack in a cage at Canix on 100Megabit or gigabit. This rack will run mirrors of all your critical services, say you have XML RPC services running through Axis or another B2B. We can make a light backup of those service to keep distributers running even when your main servers are down with switching through UltraDNS.net, or even a dedicated 3rd party DNS located somewhere super safe.
The mirrors in Montreal will be fed with only diff data from your master in San Fransisco fed via SSL secure data transfer.
When your datacenter in San Fran hits the floor, the software will detect it and switch the seperate DNS of all your critical business services, including but not limited to HTTP web to the Montreal location.
While doing this for HTTP and websites only is more trivial, our service can do it for Web services like a big manufacturer that feeds datasheets to customers via ICS 650 format, xls or SOAP.
What happens when distributers need to check the stock on items from a manufacturer and the datacenter went down?
Those are orders for several thousands of dollars.
Can people really afford to deal with downtime?
We can help.
LikeLike
Sorry folks, valleywag cracks me up, their head lines read like the weekly world news that you get in the check out at the store next to all the people magazine types and gossip rag types. At least they have made themselves into a natural niche market. And they are still funny when they are not being too snarky.
LikeLike
Sorry folks, valleywag cracks me up, their head lines read like the weekly world news that you get in the check out at the store next to all the people magazine types and gossip rag types. At least they have made themselves into a natural niche market. And they are still funny when they are not being too snarky.
LikeLike
Perhaps I missed the point of this attempt at a humorous poke at a PR company. A datacenter could be downed without having to resort to knocking over a server rack and there is essentially a big red button effect. The person would have to have access of course but every data center I have been in has a reliance on power and tons of cables. There is usually a fire suppression system that can take a center down if hit and would subject the datacenter to a gas suppression. Imagine how long it would take to figure out what patch cable goes to which if a drunk/hostile/malicious person ran through a datacenter in less than 60 seconds pulling those easily accessed cables. It could cause days or weeks of outage unless you have a cohosted location.
Just another thing to worry about… 😦
LikeLike
Perhaps I missed the point of this attempt at a humorous poke at a PR company. A datacenter could be downed without having to resort to knocking over a server rack and there is essentially a big red button effect. The person would have to have access of course but every data center I have been in has a reliance on power and tons of cables. There is usually a fire suppression system that can take a center down if hit and would subject the datacenter to a gas suppression. Imagine how long it would take to figure out what patch cable goes to which if a drunk/hostile/malicious person ran through a datacenter in less than 60 seconds pulling those easily accessed cables. It could cause days or weeks of outage unless you have a cohosted location.
Just another thing to worry about… 😦
LikeLike
Scoble – holy smokes man maybe it’s because I’ve just started reading the comments but wow dude are you trolling your own comments just to create attention? Maybe you’ve only been in the giant MS/Google/Monster colo DCs and so your picture of the inside of a data center is different.
Please listen to those of us that work in data centers day in and day out. One employee could completely destroy a DC with not a lot of effort.
Not all DCs have an EPO switch, or at least one that’s readily available.
Not all DCs have racks that are bolted to the floor. If they’re not bolted to the floor, you’re right it’s quite difficult to push them over, but if you open the front door and slide out a couple of servers at the top of the rack, physics does the rest.
There aren’t always a tonne of people inside a DC – I’ve been inside a lot of different DCs at various times of the day where I am completely alone. Even if there were people, do you know many folks that would step in to physically restrain someone on a rampage? Most people shrink away from violence – I’m not sure if I would be the one to take some punches if I saw one of my coworkers obliterating racks in the DC in a drunken rage.
Most major sites/applications/enterprises have disaster recovery plans which should is ideally instantaneous, but can take minutes. It would be entirely possible that they would be up in a couple of minutes after a major disaster and they enact their DR plan.
I’m not suggesting that the article is correct in any way, but it certainly is plausible.
LikeLike
Scoble – holy smokes man maybe it’s because I’ve just started reading the comments but wow dude are you trolling your own comments just to create attention? Maybe you’ve only been in the giant MS/Google/Monster colo DCs and so your picture of the inside of a data center is different.
Please listen to those of us that work in data centers day in and day out. One employee could completely destroy a DC with not a lot of effort.
Not all DCs have an EPO switch, or at least one that’s readily available.
Not all DCs have racks that are bolted to the floor. If they’re not bolted to the floor, you’re right it’s quite difficult to push them over, but if you open the front door and slide out a couple of servers at the top of the rack, physics does the rest.
There aren’t always a tonne of people inside a DC – I’ve been inside a lot of different DCs at various times of the day where I am completely alone. Even if there were people, do you know many folks that would step in to physically restrain someone on a rampage? Most people shrink away from violence – I’m not sure if I would be the one to take some punches if I saw one of my coworkers obliterating racks in the DC in a drunken rage.
Most major sites/applications/enterprises have disaster recovery plans which should is ideally instantaneous, but can take minutes. It would be entirely possible that they would be up in a couple of minutes after a major disaster and they enact their DR plan.
I’m not suggesting that the article is correct in any way, but it certainly is plausible.
LikeLike
Whatever happened, I always love it when big companies look stupid.
LikeLike
Whatever happened, I always love it when big companies look stupid.
LikeLike
Six graders are known for ganging up on one or two classmates with lies and gossip because “it’s funny.” Woe to the kid who goes to a teacher for help, or even cries foul. He’s supposed to just sit there and take it and not disturb the fun around him.
Nick Denton caters to people who still enjoy six grade bullying technique.
LikeLike
Six graders are known for ganging up on one or two classmates with lies and gossip because “it’s funny.” Woe to the kid who goes to a teacher for help, or even cries foul. He’s supposed to just sit there and take it and not disturb the fun around him.
Nick Denton caters to people who still enjoy six grade bullying technique.
LikeLike
@26 “San Jose State University has asked me to speak there not once, but three times, tells me that you have no clue about what they think of me.”
However, it does tell me that reputation is of no importance to them then.
LikeLike
@26 “San Jose State University has asked me to speak there not once, but three times, tells me that you have no clue about what they think of me.”
However, it does tell me that reputation is of no importance to them then.
LikeLike
When I read the story it said nothing about the drunken employee pushing over racks; it just said the drunken employee had messed up some racks. Hitting the big red button is more likely than pushing over the racks.
LikeLike
When I read the story it said nothing about the drunken employee pushing over racks; it just said the drunken employee had messed up some racks. Hitting the big red button is more likely than pushing over the racks.
LikeLike
LayZ, your reputation must be important to you, else you wouldn’t post here anonymously.
Given that the fake “LayZ” is your true voice, I can’t help but wonder what fake voice your real persona uses.
LikeLike
LayZ, your reputation must be important to you, else you wouldn’t post here anonymously.
Given that the fake “LayZ” is your true voice, I can’t help but wonder what fake voice your real persona uses.
LikeLike
One person on their own could certainly create a significant amount of havoc in a DC even if it would not be plausible to create a total systematic disaster.
http://mikeelliottsblog.wordpress.com
LikeLike
One person on their own could certainly create a significant amount of havoc in a DC even if it would not be plausible to create a total systematic disaster.
http://mikeelliottsblog.wordpress.com
LikeLike
Dear Chris from BeerCoSoftware:
You are a moron. Please switch careers; used automobile salesman is more up your alley.
“What BCS could do for enterprise, yes, or any other site with critical apps is get some low cost 1Us from cybertronpc.com and set you up with a coloc rack in a cage at Canix on 100Megabit or gigabit.”
Let me give you a hint, chump: anyone interested in “enterprise, yes, or any other site with critical apps” isn’t going to accept your lowest-bid small-time crap from illbuildyourpcinmybasement.com. You don’t know the first thing about DRR or managing a datacenter.
“We can set you up with a network switch that will automatically shuttle network traffic from your San Fran center up to your duplication servers in Montreal seemlessly with no percieved downtime.”
O RLY? What happens when that “automatically shuttling network switch” runs out of power?
Your shilling here has gotten way, way beyond reasonable. Scoble’s weblog is not your billboard for advertising your joke of a consulting company and I honestly wish he would take out your trash.
LikeLike
Dear Chris from BeerCoSoftware:
You are a moron. Please switch careers; used automobile salesman is more up your alley.
“What BCS could do for enterprise, yes, or any other site with critical apps is get some low cost 1Us from cybertronpc.com and set you up with a coloc rack in a cage at Canix on 100Megabit or gigabit.”
Let me give you a hint, chump: anyone interested in “enterprise, yes, or any other site with critical apps” isn’t going to accept your lowest-bid small-time crap from illbuildyourpcinmybasement.com. You don’t know the first thing about DRR or managing a datacenter.
“We can set you up with a network switch that will automatically shuttle network traffic from your San Fran center up to your duplication servers in Montreal seemlessly with no percieved downtime.”
O RLY? What happens when that “automatically shuttling network switch” runs out of power?
Your shilling here has gotten way, way beyond reasonable. Scoble’s weblog is not your billboard for advertising your joke of a consulting company and I honestly wish he would take out your trash.
LikeLike
Our big, red button by the door is quite critical. Without it, the door won’t open, and we couldn’t leave.
LikeLike
Our big, red button by the door is quite critical. Without it, the door won’t open, and we couldn’t leave.
LikeLike