Much has recently been written about Klout, the latest darling of social scoring that aims to measure social clout, an individual's ability to influence others. In fact some may say we are all klouted out. Isn't it time to stop railing and leave the developers to fix the known issues?
Alas, market opportunity does not always wait for engineering.
Like many emerging technologies (after all, we are at the dawn of social scoring), Klout has some problems. Even as some opine that it's premature to arrive at an Influence number, the groundswell from the marketing and advertising elite see too much promotional value in Klout to hold their deals at bay. Disney, Virgin America and other marquis names are already building their buzz on line using Klout, handing out perks, airline and hotel upgrades (and yes, social media consulting contracts) based on an individual’s Klout score . Mercedes Benz just chose contestants for their "Tweet Race" challenge for the Super Bowl based on their tweet score. Even lay twitterers have their appetites whetted for an industry accepted social score to help pave the way to paid tweets and sponsorships.
The K-Bomb has exploded.
Even as many of the pundits, even non-pundits, have weighed in on their technical reservations, and even as Klout CEO Joe Fernandez has been extraordinarily attentive in tracking and answering their criticisms, there’s still that shock wave advancing. I suspect Joe Fernandez now has an excellent grasp of the issues. But does the rest of the online community?
If Klout is here to stay (and it is) and serve as a "Neilsen rating for Social Media”, before marketing people leave the table to turn on “Klout-juiced-up” sales, perhaps it's time for one last look over our community shoulder , taking stock of some of the criticisms published. Are there points of consensus on key fallibilities of the scoring system? Is there any truth to the view that only low Klout scorers are upset? Most important to my line of thinking, are the significant problems easily fixed or are some fixes light years away?
So I’m going to risk rudeness and engage in the impertinence of keeping the conversation open a bit longer. Toward that, I took a look at some 23 blog posts published between Aug 2010 and Jan 17, 2011 which expressed technical and/or application reservations with the Klout metric. (My point here is not to point the finger at Klout as ignoring these, nor to suggest that there are more critics than supporters out there. Some of these criticisms apply as well to competing one-number social metrics which strive to a person/entity’s influence. )
I approached the review much as a product manager for Klout might, looking for which categories were most frequently cited. (A few caveats here: While I made good effor to find as many blogs posts covering Klout’s issues as possible and made effort to read each carefully to understand and identify the repeating refrains, I apologize if I omitted any significant posts and/or might have collaped a unique POV into a general category. But you can judge for yourself by checking out my reading list.
What the Critics Were Concerned About
Below I list the identified problem areas in order of how often it’s mentioned. Along the way, I cite a few of the sources which originally made the point or captured it well through example or experimentation.
1. “The Warren Buffett Problem” : The current Klout algorithm takes little to no account of a person’s offline influence.
As Klout has acknowledged itself, for Warren Buffett to get a low score within their 100 point system, is a failing. Some of the wild variability in assessing offline influence is captured by Tom Webster of BrandSavant,
I think my biggest question is this: is it even possible to measure online influence, divorced from offline influence? If a Klout score is truly focused solely upon online behavior, then Oprah’s Klout score should be far less than 65, since she has only 134 Tweets and follows just 19 people. Clearly her offline influence, not her online behavior, is solely responsible for her higher Klout score. Yes, she has well over four million followers, but certainly not by dint of her Twitter ability!
If offline influence plays a role here, then surely someone so influential as Malcolm Gladwell should pull better than a Klout score of 25? (Update: He's now up to 43.) Jason Keath correctly noted that Gladwell doesn’t really have “online” influence, and from a strictly are-you-good-at-Twitter perspective, that is certainly true. Yet, we were all talking about his recent “the revolution will not be Tweeted” piece online just over a week ago, and I would argue that his thinking is extraordinarily and demonstrably influential online. If Klout is working towards capturing that kind of influence – including citations, searches, trackbacks, etc – then they are building something very impressive indeed. SkyNet should be nervous. Jason Keath is right, however – Malcolm Gladwell is not good at Twitter. But is Oprah?
2. Klout only taps into the Twitter, Facebook and LinkedIn portions of the entire online ecosystem.
Klout today does not count your presence on YouTube, Digg, StumbleUpon and the many other social services and networks which exist. As Mark Krynsky observed, Klout currently does not take into account your blogging activity, whether its in terms of RSS subscribers, blog traffic, and comment activity.
Update: According to Clickz, CEO Joe Fernandez said he will use part of the recent $8.5 million in funding to add 20 services to that list by the end of 2011, including Foursquare, YouTube and Quora.
3. Klout is Gameable
One of the key readings for anyone seriously wanting to understand a key hurdle for any standard of influence is Adriaan Pelzer’s excellent piece, Klout is Broken. Based on a series of tests done by building Twitter bots, Pelzer was able to show that a bot could attain a highly respectable Klout score of 50 in 80 days. As he pointed out, “The fact is, though, no matter how you look at it, unless Klout updates this aspect of their algorithm, in another 80 days Bot 1 could very well have the same Klout Score as @scobleizer!”
4. Klout provides a single number to rank all users, not a number within an industry.
As Danny Brown wrote,
When it comes to influence, the folks that matter to us are the ones that are in our industry, or affect the industries of our customers and clients. That’s what influences our business and its success (or lack of it), not someone who’s in an industry that has little to no relevance to us.
A good down-to-earth articulation of the “industry knowledge” problem was provided by Frank Woodman Jr., a tax accountant, commenting in Jason Keath’s post.
As, for example, in my case being and tweeting in an area that doesn't cause that much viral interest. (Face it tax accounting and business consulting aren't exciting subjects to most people.) And thus the whole issue of re-tweets becomes of less importance than the spread of information of a specific nature that is not really subject to a lot of re-tweet interest. So naturally my re-tweet percentage isn't what some teen stars' site is. And so my over all Klout score will show that in a negative way.
(BTW Jason’s "Can Klout Really Measure Influence?” provides one of the best balanced overviews of Klout, detailing the over 30 measures Klout takes into account.)
5. Volume and frequency of tweeting and status updates is over-weighted compared to quality of content.
As Chuck Hemann wrote,
Yes, someone that’s relatively inactive on a particular channel would likely not be considered an influencer, however there have been enough analyses by people showing how important frequency is to your Klout score. I’d much rather contribute a smaller amount of content and have it be useful to people then just be a chatter box on Twitter. Check out my man Kasey Skala’s post from Monday about the inflated importance of volume on influence if you don’t believe me.
Even with these technical issues still quite alive, two of the most remarked upon failings apply to all emerging one-number scores of influence:
- Use of Klout as a “Universal Influence” Indicator . Check out Jason Goldborough’s post for a good discussion of this.
- The emerging trend to use Klout as a key metric for hiring decision. Michelle Tripp’s post gives a vivid portrayal of the problem. More recently, Eric Peterson, creator of social metric Twitalyzer, weighed in , after learning in a discussion with Shel Israel that a social media consultant had been passed over in a hire due to a low Twitalyzer score. Eric wrote,
To use Twitalyzer (or Klout for that matter) to make any decision about an individual other than broadly how they use Twitter as a tool is a mistake and does disservice to the individual, Twitter, and our analytics platform.
Why is this misuse occurring? As I said in my opening point: Marketing opportunity does not always wait for engineering to catch up.
Before moving onto the slippery slope of one party controlling your "social net worth" let's put one question to rest.
Did the Criticisms Come Mostly from People with Low Klout Scores?
One Klout-defending blogger expressed,
Many objections have already been raised – and not surprisingly many of those objections appear to be originating from social ‘experts’ and ‘gurus’ with high Twitter follower counts and not quite-so-high Klout scores .
Looking at the distribution of Klout scores for the authors in the sample, the average score was 55.
For context, I’ve also shown two data points from Brian Solis’ study of “the top 50,000 influential people on Twitter” (average Klout scores of 54 -56) as well as his random 100K sample from within the Twitterverse (average Klout scores of 31-34, depending on sex).
The critics' scores are far above what one would expect from “sour grapes” low scorers, but smack dab on Solis' "Influential people" average score.
Now I wouldn’t expect a lot of A-lister bloggers in here: High Klout scorers have little reason to rock the boat: through the Klout Perks program (the ethics of which were early on questioned by Geoff Livingston) , A-Listers are getting their hotel and airline upgrades, and, more importantly, Super Bowl consulting contracts. Personally, I found reading some of the commentary on Klout like trying to understand someone talking with a mouth of marbles. This is all the more reason to take our hats off to the few and the brave who continue to ask the difficult questions.
So let's kick the door open a bit further.
As I decribed at the start, Klout has acknowledged most of these issues and voiced their commitment to resolve the problems.
And yes – many of these technical issues seem imminently fixable. The company has already indicated it will be expanding to include a broader swath of social platforms. Further, in a recent interview , Joe Fernandez indicated that they will also be addressing that core offline influence problem by Q2 of next year. .
But yes- there will be intense gaming of the system. The importance of a social influence number, particularly as a “Page Rank of the Web” has the potential to roil the very quality of our online ecosystem: Whether we seek to avoid an “online world run by bots” (as Pelzer wrote) or “where authority itself is purchased as happened with link farms.” (from Dan Cristo), we should have our eyes wide open that the the signal/noise ratio on the web is declining.
But there's a stronger reason we may wish to avoid complacency, the "It's inevitable" shoulder shrugging.
After all, we don’t walk around with our FICO, net worth and IQ scores on our sleeves. If you hand over the keys to Klout or any other company to control the Nielsen rating for social media influence, you’re handing over a great deal of your social currency. In fact, you're handing over your “social net worth” to a proprietary and closed system. At least with FICO, IQ, Black-Sholes and other life-influencing metrics, you can backtrace what components led to the final score and how they were weighted.
A Call for Openness & Transparency
What gives me particular worry is that some of the technical issues are dreadfully complex and not very tractable. Taking down the bear of “The Warren Buffett” problem in a couple of quarters, seems overly optimistic. What's more, on the bot spamming front, CEO Fernandez has admitted that this will not be easily addressed.
In fact, even as the company integrates data from more social networks and services swiftly, the complexity of their data set increases. Brandon Prebynski called this,
Combining qualitative and quantitative data from various sources (social networks) that serve different purposes for the individuals who are active on the networks in order to achieve one “influence score” is a task that I believe cannot be achieved. As you can see, the idea of combining unlike data from unlike sources (social networks, for this case) is flawed in itself.
When it comes down to it, my primary concern regards the likelihood of the engineering timeline catching up and keeping pace with the Klout marketing juggernaut.
What gives me particular pause is memory of another company using machine learning: NetFlix with its $1 Million NetFlix Prize was certainly able to get improvement on their movie recommendation engine. And their problem was centered on a much more delimited, homogeneous data set. It was also a problem not complicated by interference from public bot abuse. A ten percent improvement. Over two years. With the help of over 55,000 crowdsourced engineers.
Perhaps,as Eric Berto calls for, it’s time for more science. Whether Klout or some other social scoring company does it, my bet is on for the company that kicks open the door, inviting in a much larger crowd of experts and scientists.
Oh yes. You can view my Klout score here. Meh, Not bad for someone living in a mud hut with no electricity. ;-)
Image 1 credit: Artwork from The Higher Critical Review