When Rob La Gesse, Graham Weston, and Lew Moorman hired me at Rackspace (the world’s largest web hosting company) they said they expected me to challenge them to do new things. I haven’t done enough of that, but last week sitting in the audience listening to Twitter’s API Chief Ryan Sarver got me excited and that’s what this post is about (I haven’t talked with anyone inside Rackspace about this yet, so this is NOT something Rackspace is officially doing — it’s just something I would love to see them do and, because Twitter is a public service I figured it’d be interesting to have a conversation about this idea out in public rather than try to keep it behind closed doors).
What did he do? He announced that Twitter was going to make the real time firehose available to everyone (you can watch video of his speech here). That was the biggest news at LeWeb, even though most people didn’t understand the impact.
What is the firehose feed? It’s what businesses like Seesmic, Tweetdeck, Peoplebrowsr, and FriendFeed get from Twitter: it is the full stream of Tweets (more than 10gigs a day). The problem is that to deal with it you need to be expert at dealing with large, changing databases, and if your business gets popular you must be able to build scalable databases that can serve out millions of queries per day. That’s beyond most developer’s skill sets, but that shows there’s opportunity for new kinds of businesses to build on top of Twitter. Which is why I wrote this note.
I’ve been watching a new kind of business evolve thanks to Twitter’s real time stream. One example is StockTwits. A few months back I interviewed Howard Lindzon, founder. The problem is that he had to hire developers, setup servers, and all that. It cost him a lot more to start that business than $100 and took a lot longer than 15 minutes.
Let’s say you have a brain fart in the middle of the night like I did last night. I’d love to start a micro business that tracks discussion of Half Moon Bay Restaurants. I’d love to watch for Tweets that are actually happening inside and around our few restaurants. But that kind of business isn’t that profitable, so I can’t afford to hire a developer and raise investments to make it happen.
But what if I could go to Rackspace Cloud, click “start a Twitter business” and start filling out a wizard. Something like:
Track Tweets from these locations: [ ] < -Import a list of restaurant locations, or hand enter them.
Track Tweets with these keywords: [ ] <-Import a list of restaurant names, like Pasta Moon, plus a few other keywords, like "restaurant" or "eating."
Join these Tweets with APIs from these services [ ] <-Here's how the business will make money, we'll get people to reserve tables with OpenTable, which will get us a kickback.
Track sentiment using these phrases: [ ] <- Things like "service sucks" or "great meal" or "having fun" or other phrases you come up with.
Build page using this design [ ] <- Give me some choices of how the page will look, and give me some options for changing colors, background image, header images, etc.
Display on our directory [ ] <- Of course we'll have a directory, wouldn't we? Something like an app store where you can add other people's Twitter apps to your page, with revenue sharing, of course!
I'm sure there's lots of other details such a business will need to enter too, but this gives you a taste of what a new Twitter business could do and how easy it could be to build such a business.
Now what would Rackspace need to implement to make 15-minute $100 microbusinesses happen?
1. A clonable database of the Twitter firehose feed. After all, if you have a brainstorm about a new business you want to see results immediately upon starting up your business and you'd want to iterate it and see what the effects of changing various things are. In the example above, for instance, maybe if you are too strict with the geolocations you won't get any results, so you'll have to be looser about what your page will accept. You'll need to see the results on a few months' worth of data and you'll need to see your changes in real time. If you have to wait a few days to see results you'll lose interest and won't finish building a micro business like this.
2. Needs to be "cloud computing" simple. If you go to the Rackspace Cloud you can start up a new server in minutes just by entering in your credit card data. It's simple and anyone can do it. Same needs to be here.
3. Needs to have an upgrade path. If 100,000 of these simple businesses get created a few will get popular, like StockTwits has. That's when you will need to hire real developers to add some other features and build out iPhone apps and all the rest of the things a modern business needs now.
4. Needs to have Rackspace's famous service. Run into trouble, or need some help building something out, even at 2 a.m. on a Saturday? Gotta have the same business service Rackspace is famous for.
What kinds of things will Rackspace need from Twitter to make this happen (if it happens at all)?
1. A decent service agreement that will let Rackspace host Twitter's firehose real time stream and clone it many, many times. Also, needs to have a way to keep a database of several months' worth of Tweets (I'd prefer to have a database of all Tweets, but I understand that Twitter doesn't want to enable shadow Twitters or real-time search engines).
2. Access to all the metadata around Tweets. The real value in Twitter is no longer just the Tweets themselves, but new businesses will need access to what lists people are on, how many times the tweet was retweeted, geolocation data, and more.
3. If Twitter comes out with a "supertweet" (a new display surface/metadata payload that gets included with each Tweet) then we'll need access to that and we'll need a clean revenue-sharing agreement for any ads or affiliate networks placed on this supertweet.
4. Need an easy to understand, and affordable, licensing agreement for the firehose feed.
Anyway, what do you think of this idea? Would you use it if it were built? What other kinds of things would you like to do with Twitter's real-time firehose feed?
Scobleizer 
This is a very cool concept. I've been wanting to build a small twitter based business for a long time and would sign up for $100 with RS in a heartbeat. Love to see them grab on to this.
LikeLike
the skeptic in me says that relative to other things going on at Rackspace, this may be considered a niche opportunity.but I'd love to see it — there are certainly a lot of business opportunities in this.
LikeLike
Twitter was once considered a niche opportunity. That was only three years ago. Now it is valued at more than a billion.
LikeLike
Don't get me wrong – I'd use the heck out of this service….. but wouldn't Twitter be mighty pissed about this? I mean this would more or less deprecate their public API with a third-party API-like interface that gives out for free what Bing and Google paid a pretty penny for, a firehose feed.I can't put my finger on it just now, but it sounds like something that'd make Twitter a little nervous, or at least it should, since it takes a lot of control from their hands.
LikeLike
Businesses the size of Rackspace are collections of many many niches. Tailoring services and packages to serve each niche is part of what keeps them relevant and growing. I don't see it as an impediment to it getting done.
LikeLike
Well, Twitter is releasing the real time firehose to the public. What is Google and Microsoft paying for? The right to use it on a search engine. Like Ryan said to me last week, there are a lot of details still to come. So, I'm sure the terms of service will be mighty interesting to read over.
LikeLike
Hell yes I would use it. Hope RS steps up to the challenge!
LikeLike
Right.But you do see the need for nervousness on Twitter's part, right? The control over the API and firehose is a major deal for them. If anyone can pick it up willy nilly all over the web… well, it smacks of that open and federated thing that Twitter has up to now refused to do.
LikeLike
Wow, what's so interesting about this is it's a business taking a first stab at becoming a value creation network. Essentially, that form you're filling in is a dialogue between business and customer about how exactly they want to create value for themselves.The infrastructure in the background makes this a working example of scalable value creation (value unique to each customer). I think one way or another, this is increasingly going to become the way to compete for customers.
LikeLike
I'm still not convinced Twitter is going to be releasing the fire hose — or at least not in the way you are assuming.They're now making over $1m/month from their fire hose deals with Google and Microsoft/Bing. I don't see why they would suddenly stop that.
LikeLike
They won't. They will just write into their terms of service that you can not use the firehose feed as part of a generalized search engine. That will protect their deals with Google and Bing. They did just that with FriendFeed, for instance. They kept FriendFeed from displaying all tweets and only let FriendFeed display Tweets from people who were registered on the system (because FriendFeed was a search engine).
LikeLike
I think what they actually did with FriendFeed is what they call the garden hose – where you define which users you want to obtain a feed for and it sends them to you 'firehose style'.That's great for a lot of applications, but it doesn't necessarily help you with many of your business ideas. It's not going to help you get a constant feed of people taking about restaurants if you have to define the scope on a user-name by user-name basis.
LikeLike
Scobes, Let me play devil's advocate here. What is the competitive advantage of the $100 company if all they do is specify some parameters and give Rackspace the job to work on the content from Twitter? What stops the other guy that has $100 to play with to specify the same parameters. Remember, the 'business owner' neither owns the content nor the infrastructure.
LikeLike
Seems my comment went into spam 😦
LikeLike
How many businesses would track comments in Half Moon Bay restaurants? Not many. There are thousands of microbusinesses like this that would not have any competition and by the time they got competition they would be more established anyway. Hey, did any of us switch from Twitter to FriendFeed? I did, but not many of you did, even though FriendFeed had more features. Same for microbusinesses.
LikeLike
Paul Buchheit told me they had access to the firehose feed and that FriendFeed's servers got all Tweets, not just ones that were registered to them, but that their license enabled them to only display Tweets from registered users.
LikeLike
Yes. That's why when I was back stage with Ryan Sarver I peppered him with questions which he refused to answer, saying that they needed to still work out the details. The devil is in the details and I hope they are liberal rather than conservative with their terms. We will have to wait and see. I've offered Ryan an interview so he can explain the terms in depth.
LikeLike
From a technical standpoint this seems feasible. I read somewhere that during the Obama inauguration they handled about 5000 tweets a minute. That gives us about 83.3 tweets a second, or 12ms to process each tweet. Given that most tweets will be tossed away, I think filtering 100 a second is doable.I would also be interesting to see the firehose(or other data stream) available on an internal IP address so it could be accessible from a cloud server without having to pay for the external bandwidth.(disclaimer: I'm not a Racker)
LikeLike
Pls delete the extra comments. This and the one below.
LikeLike
My horns are coming out now :). If it is so niche how profitable do you think it will be? Secondly, what stops Rackspace or any other big company from creating a free blogspot/wordpress kind platform for things like that; ala specialtweet.rackspace.com and have a larger pie. thirdly, what stops Twitter themselves for creating such a platform themselves?
LikeLike
Great idea. Put a Yahoo Pipes type of front-end to it where people can share their filters and you've got the makings of a really interesting ecosytem.
LikeLike
Great idea. Put a Yahoo Pipes type of front-end to it where people can share their filters and you've got the makings of a really interesting ecosytem.
LikeLike
Thinking about ubiquitous coherent/synchronous state of user shared info…If and when a real time protocol becomes totally public and mainstream, it won't be able to be bottled by any single location. The sum of all people creating and exchanging information in real time doesn't lend itself to easy cloning/mirroring.There are several candidates that could spear head a truly open protocol: Twitter, Facebook, laconica, opensocial, rsscloud, etc. Or we could write our own tool that parses the state of the web and updates it in real time.The state of the real time web has only some real estate going through twitter. Don't forget what happens outside of the twitter pipes.
LikeLike
Like the idea, but you should be on Twitter's payroll. Why would they do this deal with Rackspace when they are looking to build products that they can monetize?
LikeLike
Very cool and a huge opportunity for anyone with some serious top end programming talents. Simply having access to that much information is interesting in and of itself. I can see the industries popping up now.
LikeLike
Rackspace already hosts lots of Twitter businesses. I don't see why a revenue share with Twitter couldn't be worked out, either. After all, it's their feed.
LikeLike
Not sure if my original comment was clear enough. Your idea would be a good one for Twitter to implement and go to market. My assumption is that Twitter is looking desperately for products to sell to businesses and consumers directly instead of selling through partners.
LikeLike
An important feature, like you said, is the ability developers will have to be able to attach their own enclosures to the Hosebird stream to create their own context of the tweetsIf, right now, rackspace were able replicate/update the twitter stream and give developers the ability to add their own enclosures to it, very powerful (and meaningful) exchange apps in real estate, commodities, news, supply chain etc will surface almost instantly.We desperately need context control. Rackspace should consider this seriously
LikeLike
sign me up! As a digital marketer, I've got lots of ideas, but not a lot of technical knowledge. I'm behind you RS!@onehalfamazing
LikeLike
I love this… but why stop at just Twitter? I would love to see this kind of service work across news, blog, and real time social media data.
LikeLike
I see this the other way around. The sheer volume of tweets is enormous, and growing exponentially. Something like 80% are retweets. In order to make useful sense of it someone has to analyse that raw data and produce something approaching genuine insight. And it will require language analysis, not simple word aggregation. This will not be Twitters skill set, so they have little choice but to make it available to have someone else do it.
LikeLike
Same question of mine Scobes did not reply to above 🙂
LikeLike
Actually, I thought more or less no-one had the firehose right now; most services (Seesmic, Tweetdeck) don't need it, creative use of all the other APIs available are enough for quite a lot of use cases. I do wonder if opening up the firehose is foolish; it's clearly going to be a strain on Twitter to provide for 'anyone' and there are going to be plenty of people that just /assume/ they want the firehose when actually they can use the track or follow services to just as good effect.Bit off-topic I know.
LikeLike
I would be on this like a fat kid on a cupcake….so to speak. YES!!!
LikeLike
thx robert. the firehose is cool, but they cant truly do real time reliably for the financial markets so we would still need to build our own platform like we did. this is a great move though for twitter i agree.
LikeLike
I would use this in a heartbeat
LikeLike
20091221c1ygy wearugg Classic Mini bootsis considered a kind of a fashion trend, this trend has affected the whole world. If you have not tried, then it is regrettable. With the advent of winter, classic short boots online become popular up a lot of people can not wait to experience the feeling of wearing ugg prices ugg some people appear to be high, ugg boots online above ugg style good, the price is not expensive. To meet your needs, make haste to experience the feeling of the trend.Purchased this classic short boots last September, thinking priciest was best!Rain cover split from velcro on the first use! shopping basket is too small and not easy access at all. Now eight months down UGG Nightfall Boot on sale the line…?
LikeLike
Although I'm not sure about your specific idea, I think you have hit on an important point that RS needs to think… and maybe act upon. How does a very successful hardware centric firm start adding in services? As a RS customer one of the tools I really love is their simple case support tool. Last year we moved our customer service off of SalesForce and built a tool into our own application based on RS's model. I tried to get RS to show us the support agent side, but they wouldn't. However, I may have considered it if they offered it as a service.
LikeLike
Fall Sale – Save 20% Off Your Purchase – 2 Months!Just image getting a pair of Ray-Ban sunglasses with less than 40 US dollars! What are you waiting for?We offer quality Ray-Ban sunglasses including Ray Ban Wayfarer, Ray Ban Aviator, Predator, Polarized, Highstreet, Outsiders, Casual Lifestyle, Iconic, Leisure Sport sunglasses.During Nov.1 to Dec.31, the price of all the items on our website will be cut down at most 20% to celebrate the Christmas day! And you will enjoy the most comfortable shopping experience on our web store:100% satisfaction guaranteeHigh quality productsTimely customer serviceRefund serviceWhy wait? Come and buy right now since a special discount of 20% will only be offered for 2 months!For further information please visit our website at: Topbuyrayban.com http://www.human-wigs.comhuman hair wigs,lace wigs,full lace wigs,front wigs,human wigs – human-wigs.comhuman hair wigs,lace wigs,cheap lace wigs,full lace wigs, front wigs,human wigsWelcome to human-wigs.com, this is a professional online lace wigs store. Happy shopping for human wigs,full lace wigs,Front Wigs, party wigs, Black hair wigs… Fall Sale – Save 20% Off Your Purchase – 2 Months!Just image getting a pair of Ray-Ban sunglasses with less than 40 US dollars! What are you waiting for?We offer quality Ray-Ban sunglasses including Ray Ban Wayfarer, Ray Ban Aviator, Ray Ban Predator, Ray Ban Polarized, Highstreet, Outsiders, Casual Lifestyle, Iconic, Leisure Sport sunglasses.During Nov.1 to Dec.31, the price of all the items on our website will be cut down at most 20% to celebrate the Christmas day! And you will enjoy the most comfortable shopping experience on our web store:100% satisfaction guaranteeHigh quality productsTimely customer serviceRefund serviceWhy wait? Come and buy right now since a special discount of 20% will only be offered for 2 months!For further information please visit our website at: Topbuysunglass.com Fall Sale – Save 20% Off Your Purchase – 2 Months!Just image getting a pair of Ray-Ban sunglasses with less than 40 US dollars! What are you waiting for?We offer quality Ray-Ban sunglasses including Ray Ban Wayfarer, Ray Ban Aviator, Ray Ban Predator, Ray Ban Polarized, Ray Ban Highstreet, Ray Ban Outsiders, Ray Ban Casual Lifestyle, ray ban Iconic, Leisure Sport sunglasses.During Nov.1 to Dec.31, the price of all the items on our website will be cut down at most 20% to celebrate the Christmas day! And you will enjoy the most comfortable shopping experience on our web store:100% satisfaction guaranteeHigh quality productsTimely customer serviceRefund serviceWhy wait? Come and buy right now since a special discount of 20% will only be offered for 2 months!For further information please visit our website at: Topbuysunglasses.com Fall Sale – Save 20% Off Your Purchase – 2 Months!Just image getting a Gucci fine jewelry with less than 40 US dollars! What are you waiting for?We provide high quality Gucci jewelry, including Gucci Necklaces, Gucci Pendants, Gucci Bracelets, Gucci Earrings and Gucci Rings.During Nov.1 to Dec.31, the price of all the items on our website will be cut down at most 20% to celebrate the Christmas day! And you will enjoy the most comfortable shopping experience on our web store:100% satisfaction guaranteeHigh quality productsTimely customer serviceRefund serviceWhy wait? Come and buy right now since a special discount of 20% will only be offered for 2 months!For further information please visit our website at: Topbuygucci.comFall Sale – Save 20% Off Your Purchase – 2 Months!Just image getting a Gucci fine jewelry with less than 40 US dollars! What are you waiting for?We provide high quality Gucci jewelry and Links of London jewelry, including Gucci Necklaces, Gucci Pendants, Gucci Bracelets, Gucci Earrings and Gucci Rings.During Nov.1 to Dec.31, the price of all the items on our website will be cut down at most 20% to celebrate the Christmas day! And you will enjoy the most comfortable shopping experience on our web store:100% satisfaction guaranteeHigh quality productsTimely customer serviceRefund serviceWhy wait? Come and buy right now since a special discount of 20% will only be offered for 2 months!For further information please visit our website at: Topbuyjewelry.com Fall Sale – Save 20% Off Your Purchase – 2 Months!Just image getting a Gucci fine jewelry with less than 40 US dollars! What are you waiting for?We provide high quality Gucci jewelry, including Gucci Necklaces, Gucci Pendants, Gucci Bracelets, Gucci Earrings and Gucci Rings.During Nov.1 to Dec.31, the price of all the items on our website will be cut down at most 20% to celebrate the Christmas day! And you will enjoy the most comfortable shopping experience on our web store:100% satisfaction guaranteeHigh quality productsTimely customer serviceRefund serviceWhy wait? Come and buy right now since a special discount of 20% will only be offered for 2 months!For further information please visit our website at: guccisalestore.com
LikeLike