« Internet Fame and The True Impact of Influence | Main | Why Mobile Marketing is Way Hotter than Social Media Marketing »
Thursday
Jan202011

Klout & Critics: Time to Close the Door or Kick it Wide Open?

Much has recently been written about Klout, the latest darling of social scoring that aims to measure social clout, an individual's ability to influence others.  In fact some may say we are all klouted out. Isn't it time to stop railing and  leave the developers to fix the known issues?

Alas, market opportunity does not always wait for engineering. 

Like many emerging technologies (after all, we are at the dawn of social scoring), Klout has some problems.  Even as some opine  that  it's premature to arrive at an Influence number, the groundswell  from  the marketing and advertising elite see too much promotional value in Klout to hold their deals at bay. Disney, Virgin America  and other marquis names are already building their buzz on line using Klout, handing out perks, airline and hotel upgrades (and yes,  social media consulting contracts) based on an individual’s Klout score . Mercedes Benz just chose contestants for their  "Tweet Race" challenge for the Super Bowl based on their tweet score. Even lay twitterers have their appetites whetted for an industry accepted social score to help pave the way to paid tweets and sponsorships. 

The K-Bomb has exploded.

Even as many of the pundits, even non-pundits,  have weighed in on their  technical reservations,  and even as Klout CEO Joe Fernandez has been extraordinarily attentive in tracking and answering their criticisms, there’s still that shock wave advancing.  I suspect Joe Fernandez now has an excellent grasp of the issues. But does the rest of the online community?

 If Klout is here to stay (and it is) and serve as a "Neilsen rating for Social Media”, before marketing people leave the table to turn on “Klout-juiced-up” sales,  perhaps it's time for one last look over our community shoulder , taking stock of some of the criticisms published.  Are there points of consensus on key fallibilities of the scoring system? Is there any truth to the view that only low Klout scorers are upset? Most important to my line of thinking, are the significant problems  easily fixed or are some fixes light years away?

Let’s Review

So I’m going to risk rudeness and  engage in the impertinence  of keeping the conversation open a bit longer.  Toward that, I took a look at some 23 blog posts published between Aug 2010 and Jan 17, 2011 which expressed technical and/or application reservations with the  Klout metric. (My  point here is not to point the finger at Klout as ignoring these, nor to suggest that there are more critics than supporters out there.  Some of these criticisms apply as well to competing one-number social metrics which strive to  a person/entity’s influence. )

I approached the review much as a product manager for Klout might,  looking for which categories were most frequently cited.  (A few caveats here: While I made good effor to find as many blogs posts covering  Klout’s  issues as possible and made effort to read each carefully to understand and identify the repeating refrains, I apologize if I omitted any significant  posts and/or might have collaped a unique POV into a general category.  But you can judge for yourself by checking out my reading list.

What the Critics Were Concerned About

Below I list the identified problem areas in order of how often it’s  mentioned.  Along the way, I cite a few of the sources which originally made the point or captured it well through example or experimentation.

  1. “The Warren Buffett Problem” : The current Klout algorithm takes little to no account of a person’s offline influence.

 

As Klout has acknowledged itself, for Warren Buffett to get a low score within their 100 point system, is a failing.  Some of the wild variability in assessing offline influence is captured by Tom Webster of BrandSavant,

I think my biggest question is this: is it even possible to measure online influence, divorced from offline influence? If a Klout score is truly focused solely upon online behavior, then Oprah’s Klout score should be far less than 65, since she has only 134 Tweets and follows just 19 people. Clearly her offline influence, not her online behavior, is solely responsible for her higher Klout score. Yes, she has well over four million followers, but certainly not by dint of her Twitter ability!

If offline influence plays a role here, then surely someone so influential as Malcolm Gladwell should pull better than a Klout score of 25? (Update: He's now up to 43.) Jason Keath correctly noted that Gladwell doesn’t really have “online” influence, and from a strictly are-you-good-at-Twitter perspective, that is certainly true. Yet, we were all talking about his recent “the revolution will not be Tweeted” piece online just over a week ago, and I would argue that his thinking is extraordinarily and demonstrably influential online. If Klout is working towards capturing that kind of influence – including citations, searches, trackbacks, etc – then they are building something very impressive indeed. SkyNet should be nervous. Jason Keath is right, however – Malcolm Gladwell is not good at Twitter. But is Oprah?

   

2.  Klout only taps into the Twitter, Facebook  and  LinkedIn  portions of the entire online ecosystem.

Klout today does not count your presence on YouTube, Digg, StumbleUpon and the many other social services and networks which exist. As Mark Krynsky observed, Klout currently does not take into account your blogging activity, whether its in terms of RSS subscribers, blog traffic, and comment activity.

Update: According to Clickz, CEO Joe Fernandez said he will use part of the recent $8.5 million in funding  to add 20 services to that list by the end of 2011, including Foursquare, YouTube and Quora.

 

3.  Klout is Gameable

One of the key readings for anyone seriously wanting to understand a key hurdle for any standard of influence  is Adriaan Pelzer’s excellent piece, Klout is Broken.  Based on a series of tests done by building Twitter bots, Pelzer was able to show that a bot could attain a highly respectable  Klout score of 50 in 80 days.  As he pointed out, “The fact is, though, no matter how you look at it, unless Klout updates this aspect of their algorithm, in another 80 days Bot 1 could very well have the same Klout Score as @scobleizer!”  

 

4. Klout provides a single number to rank all users, not a number within an industry.

As Danny Brown wrote,

When it comes to influence, the folks that matter to us are the ones that are in our industry, or affect the industries of our customers and clients. That’s what influences our business and its success (or lack of it), not someone who’s in an industry that has little to no relevance to us.

A good down-to-earth articulation of the “industry  knowledge” problem was provided by Frank Woodman Jr., a tax accountant, commenting in Jason Keath’s post.

As, for example, in my case being and tweeting in an area that doesn't cause that much viral interest. (Face it tax accounting and business consulting aren't exciting subjects to most people.) And thus the whole issue of re-tweets becomes of less importance than the spread of information of a specific nature that is not really subject to a lot of re-tweet interest. So naturally my re-tweet percentage isn't what some teen stars' site is. And so my over all Klout score will show that in a negative way.

.

(BTW Jason’s "Can Klout Really Measure Influence?”  provides  one of the best balanced overviews of Klout, detailing the over 30 measures Klout takes into account.)

5. Volume and frequency of tweeting and status updates is over-weighted  compared to quality of content.

As Chuck Hemann wrote,

Yes, someone that’s relatively inactive on a particular channel would likely not be considered an influencer, however there have been enough analyses by people showing how important frequency is to your Klout score. I’d much rather contribute a smaller amount of content and have it be useful to people then just be a chatter box on Twitter. Check out my man Kasey Skala’s post from Monday about the inflated importance of volume on influence if you don’t believe me.

Even with  these technical issues still quite alive,  two of the most remarked upon failings  apply to all  emerging one-number scores of influence:  

  • Use of Klout as a “Universal Influence” Indicator . Check out Jason Goldborough’s post for a good discussion of this.
  • The emerging trend to use Klout as a key metric for hiring decision. Michelle Tripp’s post gives a vivid portrayal of the problem. More recently, Eric Peterson, creator of  social metric Twitalyzerweighed in ,  after learning in a discussion with Shel Israel that a social media consultant had been passed over in a hire due to a low Twitalyzer score. Eric wrote,
  •  

To use Twitalyzer (or Klout for that matter) to make any decision about an individual other than broadly how they use Twitter as a tool is a mistake and does disservice to the individual, Twitter, and our analytics platform.

 

Why is this misuse occurring? As I said in my opening point: Marketing opportunity does not always wait for engineering to catch up.

Before moving onto the slippery slope of one party controlling your "social net worth" let's put one question to rest.

Did the Criticisms Come Mostly from People with Low Klout Scores?

One Klout-defending blogger expressed,

Many objections have already been raised – and not surprisingly many of those objections appear to be originating from social ‘experts’ and ‘gurus’ with high Twitter follower counts and not quite-so-high Klout scores .

 Looking at the distribution of Klout scores for the authors  in the sample, the average score was 55. 

 

For context,  I’ve also shown two data points from Brian Solis’ study of “the top 50,000 influential people on Twitter” (average Klout scores of 54 -56) as well as his  random 100K sample from within the Twitterverse (average Klout scores of 31-34,  depending on sex).

The critics' scores are far above  what one would expect from “sour grapes” low scorers, but smack dab on Solis' "Influential people" average score.

Now I wouldn’t expect a lot of A-lister bloggers in here: High Klout scorers have little reason to rock the boat: through the Klout Perks program (the ethics of which were early on questioned by Geoff Livingston) , A-Listers  are getting their hotel and airline upgrades, and, more importantly, Super Bowl consulting contracts.  Personally, I found reading some of the  commentary on Klout  like trying to understand   someone talking with a mouth of marbles.  This is all the more reason to take our hats off to the few and the brave who continue to ask the difficult questions.

So let's kick the door open a bit further.

 

The  Slippery Slope of One Party Controlling Your "Social Net Worth"

As I decribed at the start, Klout has acknowledged  most of these issues and voiced their commitment to resolve the problems.

And yes – many of these technical issues seem  imminently fixable. The company has already indicated it will be expanding to include a broader swath of social platforms.  Further, in a recent interview , Joe Fernandez indicated that they will also be addressing that core offline influence problem by Q2 of next year.  .

But yes-  there will be intense gaming of the system.  The importance of a social influence number, particularly as a “Page Rank of the Web”  has the potential to roil the very quality of our online ecosystem:  Whether we seek to avoid  an “online world  run by bots” (as Pelzer wrote) or “where authority itself is purchased as happened with link farms.” (from Dan Cristo), we should have our eyes wide open that the  the signal/noise ratio on the web is declining.

But there's a stronger reason we may wish to avoid  complacency, the "It's inevitable" shoulder shrugging.

After all, we  don’t walk around with our FICO, net worth and  IQ scores on our sleeves.  If you hand over the keys to  Klout or any other company to control the Nielsen rating for social media influence, you’re handing over a great deal of your social currency. In fact, you're handing over your “social net worth” to a proprietary and closed system. At least with FICO, IQ, Black-Sholes and other life-influencing metrics, you can backtrace what components led to the final score and how they were weighted.

 

A Call for Openness & Transparency

What gives me particular worry is that some of the technical issues are dreadfully complex and not very tractable. Taking down the bear of  “The Warren Buffett” problem  in a couple of quarters, seems overly optimistic. What's more, on the bot spamming front, CEO Fernandez has admitted that this will not be  easily addressed.

In fact, even as the company integrates data from more social networks and services swiftly,  the complexity  of their data set increases.  Brandon Prebynski called this,

Combining qualitative and quantitative data from various sources (social networks) that serve different purposes for the individuals who are active on the networks in order to achieve one “influence score” is a task that I believe cannot be achieved. As you can see, the idea of combining unlike data from unlike sources (social networks, for this case) is flawed in itself.

When it comes down to it, my primary concern regards the likelihood of the engineering timeline catching up and keeping pace with the Klout marketing juggernaut.

What gives me particular pause is memory of another company using  machine learning: NetFlix with its $1 Million NetFlix Prize was certainly able to get improvement on their movie recommendation engine.  And their problem was centered on a much more delimited, homogeneous data set.  It was also a problem not complicated by interference from public bot abuse.  A ten percent improvement. Over two years.  With the help of over 55,000 crowdsourced engineers. 

Perhaps,as Eric Berto calls for, it’s time for more science.  Whether Klout or some other social scoring company does it, my bet is on for the company that kicks open the door, inviting in a much larger crowd of experts and scientists.

Oh yes. You can view my Klout score here.  Meh, Not bad for someone living in a mud hut with no electricity. ;-)

 

Image 1 credit: Artwork from The Higher Critical Review

 

References (1)

References allow you to track sources for this article, as well as articles that were written in response to this article.

Reader Comments (17)

Lisa, a terrific overview. Absolutely excellent. I've spent a lot of time talking with Klout's CEO Joe Fernandez about these issues and I'm impressed by his continued desire to be open and transparent, and to acknowledge the many improvements that still need to happen to get Klout where it needs to be.

Like you, I agree that one source controlling or measuring influence is disturbing - and I've also been more than a little disturbed by not all the gaming that's transpired, but by the ridiculousness of it all. A-Listers (or people who are, at least in their own minds, A-Listers) attending events, test driving cars, flouting their Klout are, in my estimation, probably driving little to NO brand awareness and/or actually compelling people to take action and make a purchase. And that's where this system fails. Influence that doesn't do anything - well, I think it's pretty worthless. Whether it's online or off.

Some of the people with high Klout scores got them by tweeting a lot, RT other folks' content, riding the social media "bandwagon" of their own greatness and high follower numbers -- but some of those very same people are barely successful in their own businesses. Or, better yet, HAVE no business, but the business of being online and trying to appear influential.

I call that a fail.

That said, I have every confidence that Klout will figure themselves out. And so will the other measurement products and solutions that are out there. And just like we use tools like Forrester, Gartner and Nielsen analyses, we'll probably likewise be using these tools as we move into an age where influence DOES matter. But figuring it out in a reliable manner is a journey. One that I, quite honestly, am delighted to be a part of.

Thanks again for a fabulous post!

Shelly
@shellykramer
http://v3im.com

January 20, 2011 | Unregistered CommenterShelly Kramer

Excellent analysis. Hopefully the folks at Klout hear you. They are usually pretty good about listening for their name online and engaging in the conversation.

I would add that there's one more broken process: what happens if you want to dispute something? I have been waiting for an explanation from @klout for a few weeks now as to why one of their scores for my "true reach" plummeted 5k+ the day after Christmas. I was told "we'll look into that!" and now they won't answer me at all.
There seems to be no remedy if something bizarre like that occurs other than to "hope" they agree to fix it. I can't remove my profile from Klout, just my access. The first time it comes down to a "but your Klout score says..." that it costs someone money or a job and there's something inaccurate? I suspect we'll see our first Klout lawsuit.

January 20, 2011 | Unregistered CommenterLucretia Pruitt

Lisa, I commend you on researching and writing this post. Far too often in life we jump on the bandwagon without really understanding what we're doing. We see this behavior more and more in the social media space today.

I have used Klout since its very early days (I'm always intrigued with the new shiny toy and it was that once) and have enjoyed getting quick email responses to my questions. Having said that, I share your concern about one party controlling one's social net worth and think posts like this one are vital as we move forward.

Thanks!

January 20, 2011 | Unregistered CommenterAllen Mireles

Lucretia, interesting point you make. With such volatility in the Klout score, and a large base of job-seekers whose Klout scores might be looked at as a basis for hiring, these are - well - scary grounds. I think it's time to look at the Terms of Service for Klout. As a developing platform, I hope they have this covered.

January 20, 2011 | Registered CommenterLisa Thorell

Shelly,

Thank you for your commentary. I'm truly honored to have your attention. Would that Klout was just another metric of the Forrester-Neilsen-Gartner crowd, speaking to markets and industries. ( I suppose I'm genetically related as ex-Dataquest-Gartner stock ;-))

I am not sure that people realise that, now that "The Media is You", Klout, or any other metric that purports to be 'The Standard of Influence' is another phrase for "Social Net Worth". And when we state it in those terms, more like FICO or IQ, it becomes more dear to them -- perhaps worthy of protecting. It certainly is worth asking at this point whether the machine learning algorithms that determine a score which affects our ability to get a job, a lucrative social media consulting contract, or get a discount on our weekend at Atlantis are really trust-worthy. When we see profound variability in the scores of well-known experts that don't accord with common sense. when we recognize that these are the very problems (aka Influence) that PhD candidates in Socio-Cognitive Psychology still wrestle with today, why should such deterministic forces operate without more transparency, more open-ness as to the methods of computation?

January 20, 2011 | Registered CommenterLisa Thorell

Thank you so much, Allen! Really- I was just worried that people had expended their due diligence and were moving on. In complete deference to Klout for focusing and moving forward the conversation, Klout or some other "standard of Influence" is an extremely important metric as we move forward into social e-commerce. It isn't the usual Forrester- Neilsen-Gartner stuff: These are the scores of real-life people, some of whom do not have PR, Marketing and Legal departments to substantively counter the short-term effects of their score, even as it goes through its refinements.

January 20, 2011 | Unregistered CommenterLisa Thorell

Really, really good post. I hadn't heard the thing about the only people who complain about Klout have low scores...I have a decent score and complain about it plenty! Although, weirdly, while my number is similar to (or higher than) friends who get Klout perks, I haven't ever gotten one. So there are still more weird variables involved in the whole numbers game.

And good point by Lucretia...if Klout is indeed going to potentially cost people jobs or cause them to lose out on opportunities when the accuracy of the system is questionable, that's a huge problem.

January 20, 2011 | Unregistered CommenterMaggie McGary

Wow, awesome post Lisa. This is as close to a description of the day to day challenges we are working on here as I have seen.

I think this line sums it up really nicely:

"my primary concern regards the likelihood of the engineering timeline catching up and keeping pace with the Klout marketing juggernaut."

As CEO this is what scares me the most. We are way less mature as an organization and product than where the press and public wants us to be. We are working incredibly hard here though to get ahead of this and deliver against these challenges.

Also, I am personally taking a look at Lucretia's account to see what issues there might be.

Thanks!

January 20, 2011 | Unregistered CommenterJoe Fernandez

Lisa & Maggie: they are on the front of a wave - it's one of those 'surf or crash' kind of things. As the Internet as primary business mode matures along with the business models all of the legal stuff starts cropping up. Legislation is nowhere near this decade even, let alone this year. Case law is going to be crucial to shaping a lot of it as will regulatory bodies (the FTC's interest in blogger disclosure is just the tip of the iceberg.)

Joe: thank you for proving my point that you and your team are always listening. I cannot think of a company the size of yours that does a better job of monitoring their own online mentions and dealing with issues straight-forwardly. That counts for a lot in our world. You are correct that the rest of the world expects you to be far ahead of where you can reasonably be... it's the fast-food syndrome. We forget that quality still takes time.

January 20, 2011 | Unregistered CommenterLucretia Pruitt

Lucretia & Maggie, Thank you both for your commentary. In trying to wrangle down my long-form style - i actually edited out a small paragraph on the high volatility of the Klout score. What Lucretia describes, a Plus or minus 5000 in True Reach, is a significant factor to a marketing and social media consultant. If this blog post's done nothing else, I am totally happy that Lucretia got attention on this.

Maggie- not getting the Klout perks when your score is higher than those who do? Geez - I would be calling San Francisco and asking for the Perks Program Manager to get the scoop. It sounds like an oversight to me -- or at least the makings of a good blog post ;-)

January 20, 2011 | Registered CommenterLisa Thorell

Joe,

Wow! I second Lucretia's point on your recird fir off-the-charts attentiveness & monitoring Klout online. Welcome.

I am very sympathetic to the "surf or crash" phenom that Lucretia describes, having been a VP of Marketing in an internet startup in Seattle where we went from 40 to 200 employees in one year. And I'm afraid i was one of those marketing execs that is guilty of leaping up the stairs to engineering at 7 pm at night and saying, "Hey, we just promised this to <marquis name company X> so now go build it." In retrospect, I know i could have made more leaps up those stairs for more companies if our architecture had been more open. I dont have to tell you that the problem Klout is onto, that Influence number, is so significant that the more computer scientists, psychologists, machine learning experts you can pull into it, the better.

Should you open key parts of the API, or share data sets, to make advances on the True Reach, Amplification or Network components - leveraging the larger research community's knowledge of these immense abstractions which underlie the even mega mega abstraction called Influence - I'd be leading the 'stand on the chairs" clapping for your folks. ( I for one would love to hear what @timoreilly in your neighborhood has to say about Klout and Influence. )

But independent of taking on a more open strategy (which i know is nontrivial) - I see Klout the company as really having a profound sense of rewards psychology - a hand you've played extraordinarily well. So I am going to look forward to how you further strategically deploy that - even as it might be used to increase the engineering pace.

January 20, 2011 | Registered CommenterLisa Thorell

Great, balanced post.

I like Klout as a theory or idea - and kudos, as you got Joe to respond to your issues, but they never addressed mine (http://pop-pr.blogspot.com/2010/12/numbers-dont-add-up-popularity-doesnt.html).

My issue with Klout is this: the people they are upholding as influencers aren't really influencers - they might be prolific Twitters but is there really any real world influence there? Likely (well, more than likely) no. And that's just part of the issue - companies are using Klout as a drive-by engagement with no real life engagement.

I hope they work it out, but I have a feeling they won't. They're making money (I'm guessing), getting funding and getting clients - they don't need to really fix it unless one of those three drops off.

January 20, 2011 | Unregistered CommenterJeremy Pepper

Thanks for posting your great points. Joe seemed to resonate with my point on the disparity between the speed of the marketing machine behind Klout (and the resultant tsunami wave of adopting it as as "Standard of Influence) with the ability of engineering to really make it a stable, believable measure of influence. I totally appreciate his cutting right to the meat of the matter and not denying it.

Marketing and PR folks have used important numbers in the past, eg. the Neilsen ratings, Arbitron, etc., with not this type of blowback that the Klout score is getting. But we always knew those were measures of reach. We were never so bold (or cavalier?) to call it "influence".

I'm excited that Klout and others (PeerIndex, Twitalyzer, etc.) are working on this problem, specifically, that there's an attempt to automate it, calculate it. Klout takes into account many measures but combining so many diverse inputs from many social services/platforms has an inherent danger as Mark Prebynski wrote of "apples vs.oranges" combinations. I for one am concerned when I read ( ClickZ's Jan 19th post) that Klout regards a National Weather Service bot, emitting storm warnings, as an example of a valid high influence Klout score. Does that sound right, really? Such a bot, if was verified, branded and certified as really coming from the NWS (do we have those checks today?) has authority but I wouldnt call it "Influence" in the sense of an article written by Mossberg. Influence is too high a level of abstraction to assign to a non-interactive, emitting signal beacon.

I like that your post brought up the fact that Klout does indeed have an transparent policy when it comes to the Klout Perks. That's well and good on the marketing front. I will point out that when i get distressed about lack of transparency I am referring to the inability to check the exact variables, their weightings - really, what is the algorithm being used - that is underneath the hood of my Klout score. It just seems that as long as a human's Klout score, what the marketing world of large advertising sponsorships, hiring companies, etc. increasingly regard as a measure of social net worth, - where that score is indistinguishable from that of a transmitting weather bot, we know we are talking about apples vs oranges. Hopefully there will be more Adriaan Pelzer-style experiments soon!

January 21, 2011 | Unregistered CommenterLisa Thorell

I believe the entire exercise is silly and counter-productive.

There are many good writers with something to say out on the 'net. The problem is finding them.

It may seem old-fashioned to say that popularity is hardly an indicator of quality, but apparently this is a radical concept right now. That's a serious problem at the heart of every social media "ranking" system. Any type system is completely "gameable" because it will always favor a visceral "I like!" kind of response over serious discussion.

If the 'net is going to live up to its promise of connecting people and ideas it cannot have such a short-term focus. All of these efforts encourage a High School atmosphere that is very counter-productive to the long term health of this medium.

January 22, 2011 | Unregistered CommenterErik Hare

Thanks for weighing in, Erik. The never ending quest for popularity is potent juju. The quest to buy WOM by brand advertisers is also potent market juju. Even as i write this as the streams of "According to Klout..." stream by in my Twitter public timeline. Soon come the "Boost My Klout" bots. It's working and we can expect a lot more, degrading the online ecosystem. The only thing good is eventually through market competition, better numbers will arrive. Eventually.

January 22, 2011 | Registered CommenterLisa Thorell

Really good post: nicely summarizing some of the problems.

These issues are important: as I note in a post today on my blog, bad metrics can lead to bad behavior!

Thanks for a good article!

Gary

February 16, 2011 | Unregistered CommenterGary

Thanks for weighing in, Gary! As you wrote in your post, it seems the bad behaviors and misapplications have already begun. (BTW- thanks for helping me find Martijn Linssen's observant post on this too!)

February 16, 2011 | Registered CommenterLisa Thorell
Comments for this entry have been disabled. Additional comments may not be added to this entry at this time.