Information source evaluation has always been part of the research and information professional's job. As information formats and delivery channels evolve, so must the approaches that we take to evaluate sources. The recent explosion of social media tools offers a unique and appealing opportunity in the evolution of source evaluation approaches.
In our particular organisation, we are consistently challenged to find a methodology for evaluating the importance and influence of media sources, regardless of format or delivery channel. For example, how does the New York Times website rank in comparison to The Huffington Post or Newsweek magazine? After some investigation, we realised that social media tools can provide pertinent data to answer just this type of question.
Subsequently, we developed a framework that leverages a variety of social media tools to evaluate the importance and influence of media sources. The resulting methodology is data-driven, easy to replicate, and applicable whether we are evaluating a list of blogs, podcasts, twitter accounts, or more traditional media publications.
Creating the Evaluation Framework
Our first step in developing this methodology was to identify available metrics that could be indicators of the influence of various information sources and build a list of relevant social media and traditional research tools. In compiling our list of influence metrics, we tried to look beyond the traditional applications of specific social media tools to find underlying metrics.
For example, BlogPulse is a blog search engine that is geared toward general keyword searches, but its Trend Search feature maps the percent of the blogosphere referencing a topic by day. By searching for a website's URL using this Trend Search, we were able to identify a data point for the peak day when blogs linked to the URL in the past x months. This hidden data point was added to our list of metrics.
Here is a list of metrics identified in our preliminary research:
INFLUENCE METRICS
RESOURCE/TOOL
METRICS
Alexa
Alexa traffic rank
Sites linking in
Time spent on site
BlogPulse
BlogPulse Profile rank
Citations
Trend Search
Compete
Compete rank
Unique visitors
Visits
Digg
Number of Digg hits
Content with 100+ Digg hits
Quantcast
Quantcast Rank
Estimated monthly US visitors
Traffic frequency - Addicts, Regulars, Passers-by
Audience demographics
Retweetrank
Retweet rank
Technorati
Authority
Twitter
Followers
Following
Listed
TwitterCounter
Twitter rank
Current followers
Predicted followers in 30 days
YouTube
Views
Star ratings
Comments
In addition to these 3rd party resources, we also identified common metrics that could be collected from the media source being evaluated, such as self-reported circulation or web traffic data, audience profiles, or number of comments.
As a final step in building our list of metrics, we researched the real-world data ranges for each metric, using a subset of media sources as a test group, and normalised the data on a five-point scale. This gave us a consistent way to quickly compare influence across multiple metrics and also facilitated the creation of a composite influence ranking system. The table at right shows the process for assigning normalised scores for Alexa Traffic Rank.
ALEXA TRAFFIC RANK
RAW DATA RANGE
NORMALISED SCORE
5000 +
1 (low influence)
3001 - 5000
2
1001 - 3000
3
501 - 1000
4
1 - 500
5 (high influence)
Once we completed this preliminary research, we began grouping metrics into categories based on how the data defined or illustrated influence. We ultimately identified five Influence Attributes: Reach, Buzz, Engagement, Content and Audience. Dividing the metrics into these attributes added structure to our methodology and provided us with more flexibility for presenting the results. The following table describes the five Influence Attributes in more detail:
INFLUENCE RANKING FRAMEWORK
ATTRIBUTE
DESCRIPTION
ASSOCIATED METRICS
Reach
direct readership or subscriber base (traditional metric)
Unique visitors per month
Site Ranks
Twitter followers
Buzz
secondary readership reached through social media channels
Popularity metrics
Inbound blog linking
Retweets
Engagement
reader participation or dialogue with content creators and other readers
Average time spent on site
Average number of comments
Number of @Replies, #Hashtags
Content
relevance of the source content to topic of interest
Frequency of relevant content
Depth of relevant content
Audience
reader groups targeted or reached by the source
Audience indicators in content
Reader demographics
For each Influence Attribute, we created a score based on the normalised data for each underlying metric. These scores could then be combined into a composite score, allowing us to rank lists of media sources by total influence or by the individual Influence Attribute, depending on client or project needs.
Applying the Evaluation Framework To illustrate the application of the Influence Ranking framework, we collected sample data from blogs that cover the eReader or eBook industries. Although we focused exclusively on blogs to simplify this example, the framework could be applied to multiple content channels, including Twitter, podcasts, vlogs, print or online media.
1. The first step was to collect data for each blog. The table below shows selected metrics under the Buzz Attribute.
BUZZ MEASURES: EREADER/EBOOK INDUSTRY
SOURCE
DIGG HITS
BLOGPULSE LINKS
TECHNORATI AUTHORITY
booktwo.org
1
0.001
36
CrunchGear
293
0.013
2574
Engadget
1551
0.067
7301
if:book
1
0.001
173
Teleread
18
0.014
331
O'Reilly TOC
2
0.001
81
2. Next, we normalised the data for each metric on a five-point scale.
3. We calculated a score for each Influence Attribute, again on a five-point scale (1 = low influence, 5 = high influence). Weights could be applied to Attribute scores at this stage, though none were used in this example. The table below shows the Attribute scores for each blog as well as a composite Total Influence score based on a sum of the Attribute scores.
INFLUENCE RANKING: EREADER/EBOOK INDUSTRY
SOURCE
CONTENT
AUDIENCE
REACH
BUZZ
ENGAGEMENT
TOTAL INFLUENCE
booktwo.org
4
4
1
1
3
13
CrunchGear
2
2
5
4
4
17
Engadget
1
2
5
5
5
18
if:book
4
4
2
2
1
13
Teleread
5
5
2
3
1
16
O'Reilly TOC
3
4
3
2
3
15
When presenting Influence Ranking results to clients, we include a table similar to the one shown above as well as breakdowns by Influence Attribute and media type, as needed. Our eReader media list combined general technology blogs with niche industry blogs. The general technology blogs ranked highest in terms of Total Influence despite low Content and Audience scores. Separating the rankings by blog type would provide an opportunity to explain the differences between these blogs and recommend targeted actions.
Summary
Our organisation has been successfully applying the Influence Ranking framework in research requests for over a year. In that time, it has significantly improved our ability to create robust media lists for our clients. The Influence Attribute scores have provided starting points for further discussion of the strengths and weaknesses of specific sources. And the consistent, replicable methodology has opened up opportunities for further research into media trends.
While we developed this framework in response to an industry-specific business need, we hope that the explanation of our process can help other information professionals develop similar systems to consistently evaluate information sources.
and Samara Omundson
Emily Wheeler is a Research Manager at Waggener Edstrom Worldwide, a global public relations firm. She has been with Waggener Edstrom for four years and specializes in secondary research and media analysis.
Samara Omundson is Research Director at Waggener Edstrom Worldwide, a global public relations firm. She has a decade of experience in media-related research and analysis. She is a past President of the Oregon Special Libraries Association.
The FreePint Family is a family of resources to help information workers be more effective, raise the value of information in their organisations and contribute to success.
'FreePint... provides most of my professional development because it won't come through work and [other resources] just don't cut it.'
FUMSI Forum: Do you have a research question? Post it to the FUMSI Forum, where professionals share Q&A and useful tips on how to Find, Use, Manage and Share Information. It's free.
![[Reuse options]](http://license.icopyright.net/images/icopy-w.png){kind=small}
and Samara Omundson
Subsequently, we developed a framework that leverages a variety of social media tools to evaluate the importance and influence of media sources. The resulting methodology is data-driven, easy to replicate, and applicable whether we are evaluating a list of blogs, podcasts, twitter accounts, or more traditional media publications.
FreePint Shop:
VIP LiveWire:
