Many of you will know that Twitter unexpectedly cancelled it’s contract to allow DataSift to resell Twitter data to 3rd parties. I read the declarations by industry analysts on Twitter that this was “proof that you can’t build a business on somebody else’s platform” and perhaps DataSift should have known better.
This misunderstands the situation so I want to clarify things a bit. DataSift was never built on a single platform and never desired or expected to be Twitter’s re-syndication provider as its sole business.
Let’s start with the most important fact that wasn’t discussed. DataSift was selected as the topic data supplier for Facebook, which allows companies to analyze a data feed that is > 20x larger than the entire Twitter feed and creates privacy-safe insights from a network of 1.4 billion people.
We never built our product on one partner’s data platform but on many (20 primary data sources plus 100+ smaller ones) and Twitter’s move doesn’t “shut down DataSift” it simply takes 1,000+ customers who consume Twitter data through DataSift and makes their life more difficult. We plan to work with Twitter to migrate these companies to work directly with Gnip (Twitter’s in-house data supplier) and for those that want value-added processing on top we plan to ingest Gnip data for our customers who request this.
But of course customers will have to do some technical work to migrate products and will lose DataSift functionality that Twitter / Gnip does not possess directly. They will of course try to plug these technical gaps over the next 24 months but with pressures on other parts of Twitter (ad monetization, ad syndication, UI improvements to stem customer churn) – business integration products will of course have to compete for scarce resources as happens with any company that is trying to serve consumers & businesses, ad products & data products as well as 3rd-party developers.
Never mind that Twitter in writing specifically asked us to build this re-syndication product with them and that every step of the way encouraged us to build out the service. We’re big boys and we knew that as Twitter grew in prominence they would want more control. They of course had these conversations directly with us and even though we were surprised at the unexpected timing since we were negotiating a continuation with each other in good faith – they have privately made it clear to us over the past year what their long-term desires are. We know and like many on the Twitter management team and product teams and recognize that their strategy has shifted and so must we.
As you can imagine, every single board meeting for the past 3 years has held major discussions on
data sources and building big data tools for enterprises to process their own private data (which is larger than any third-party data stream). Simply put: We never intended to build a business on one company’s platform and despite market perceptions – we never did! 90% of DataSift customers ingest data from 2 sources (most ingest multiple streams) and select from > 20 large data sources of which Twitter is one.
We know from our customers that they specifically wanted an independent aggregator of data because they don’t want to figure out how to ingest large, complex data feeds from each individual data provider and they know that even though Twitter rose in prominence over the past 5 years that there will continue to be new data sources they need to process and make sense of. The top three datasets that are requested of us? Facebook, LinkedIn and increasingly YouTube. Again, no big surprise.
And in international markets they care about Sina Weibo, WeChat, KakaoTalk, Line and many others. We live in a global and rapidly changing world. We, and our customers, know that there won’t be one data feed to rule them all.
We also built a product called VEDO which allows customers to ingest any large dataset and draw insights using Machine-Learning based algorithms. We built this specifically to allow our customers to ingest their own internal datasets (customer records, order entry trends, network outage data, whatever) and draw conclusions. Public, social data plus private internal data = real corporate intelligence.
If anything you should think about DataSift as a natural successor to products like Autonomy or as a next generation Informatica. These are the markets we play in – making sense of large volumes of structured and unstructured data – not Twitter reselling. If you want to watch a short explanation of DataSift’s VEDO it’s embedded below or click the previous link.
Still. We did make money helping customers process Twitter data. Just how much?
This article on TechCrunch states that Twitter only gets a 20% cut on DataSift reselling its data almost making you believe this is why Twitter wants to cancel its deal. Let me make it easy on you – this number is wrong. DataSift passes 95% of the data revenue back to Twitter. Ninety five. This is not about us making money off of Twitter data. We make money by helping customers reduce complexity in data streams and they pay us SaaS licenses and volumetric processing fees to do so.
This isn’t about money between DataSift & Twitter – it’s about control. Twitter wants to control the way its customers ingest its data and that’s understandable. Until last Friday we had been discussing with Twitter in good faith our desire to hand over all customer contracts for their data directly to them (allowing them control and to charge whatever they wanted) but still allowing us to provide our customers with an integrated data feed across vendors.
Without notice Twitter informed us on Friday that they intended to communicate with the market in less than 24 hours that they would no longer work with us or any third-party re-syndication partner.
We know directly that many customers don’t want to cede control of how they process data streams to Twitter. Some evidence?
Twitter announced its acquisition of our competition Gnip almost a year ago. During this year Twitter required us to inform any new customers that signed up to ingest Twitter that our contract had an expiration date in it and yet record numbers of customers still signed up with DataSift. You might ask yourself why?
Our product isn’t simply a data reselling product. We help organizations make sense of very large data sets in realtime and it turns out this isn’t that easy to do. Some specific things that have won customers over for DataSift. These are just some quick examples to help you understand the differences
- DataSift allows end-users to filter data before receiving it which allows customers to have more manageable datasets vs. an entire stream and it vastly reduces costs because customers don’t need to buy the entire data stream. You could imagine why Twitter didn’t like this, but what about customers?
- DataSift augments the data in realtime allowing insights about demographics allowing insights into facets such age, gender, location and favorability on topics or brands. The “sift” part of our brand stands both for reducing data volume complexity and allowing insight and not raw data.
- DataSift crawls the links that are shared and categorizes the upstream information providing exponential insight. Why does this matter? If I Tweet “Check out this amazing article www.link.com” then reading the Tweet provides no context. When you do semantic analysis on the article shared you would find out that this is an article trying to persuade people that shale oil drilling in the US is important for energy independence. The former is data, the latter is insight.
- For larger organizations we’ve built integration tools to allow companies to insert their data directly into their existing business intelligence platforms. It turns out that people actually care about simplifying how this ingested data feeds into the products their already using for customer research and analysis.
Of course there are many more examples but since DataSift is a realtime, big-data processing company most of the examples would go over the heads of many of my readers.
My inbox was filled yesterday with emails with a mixture of consternation and anger from some customers who are being asked to migrate directly to Twitter and who don’t want to. We’re explaining that we believe it’s in their interests to do a peaceful migration and we want to enable that. Of course we’re also showing them the power and size of Facebook data along with our dozens of other sources. We will still serve our customers’ needs even if we have to have a less powerful version of the Twitter data stream.
I also a series of messages from others who have tried to build relationships with Twitter over the years with a mix of “I told you that you couldn’t trust Twitter” and many ex Twitter employees expressing sympathy because they knew the history of our relationship with Twitter, were involved in building the tools and processes and know the promises that were made to us.
None of this really matters.
We went into our relationship with Twitter knowing they would eventually shift from just selling ads to trying to build a broader business. We planned for this with VEDO and tools to allow customers to ingest data directly. We planned for this by integrating many datasets and in particular Twitter’s much larger competitor, Facebook. And our hope is that other large data players like LinkedIn or YouTube will want an independent party to help deliver their anonymous data to large corporates rather than these companies partnering with Twitter (which of course isn’t likely to happen).
In the end we’ll stay focused on serving our customers. If their voice continues to request independence, multi-data-stream integration with a single tool, public & private data insights – then we’ll do just fine. We’re not going to let a single setback interfere with our business imperative to simplify & add insights to the growing volume of realtime data.
The post Can You Build Your Business on Somebody Else’s Platform? appeared first on Bothsides of the Table.