E-MAIL | PRINT | SAVE | LICENSE  

Written by Arthur Weiss.

Ten years ago, searching the Web for competitive intelligence was a hit-or-miss affair. Although by then most quoted companies had websites, private companies were just starting their love affair with the Web. Finding information depended on the meta-tags included on the site - and the top search engines were Alta Vista, Hotbot and Northern Light. Yahoo! was a search directory and Google only started in the final quarter of 1998.

Corporations were rushing to put up websites, and many gave little thought to the sensitivity or confidentiality of the information being made publicly available. Sometimes, marketing was hardly involved at all as the Web was seen as an IT issue. Companies would list customers, suppliers, future pricing strategy and other information that should never have been in the public domain. In many ways, these were glory days for competitive intelligence (CI) analysts who knew how to search the Web - and many didn't. (There was even an article published in the CI press saying that searching the Web was a waste of time and had no CI value). The only drawback was that too few companies had a substantive Web presence.

Today the opposite is the case - most companies will have a Web presence. In fact it's almost impossible for a commercial body not to have some presence on the Web - if not via its own website, then through its employees' Web activities. The problem now is finding the nuggets of information that can give clues to what the company is doing.

Planning the search

Before undertaking any competitor research, research objectives need to be specified. These will guide the research process. In other words what information is wanted? Ideally the researcher should also know:-

• Why it is wanted?
• What sort of decision will be made from gathered intelligence?
• How will the intelligence be used?

Next, it is useful to identify what (if anything) is already known about the competitor? For example:-

• Is the company publicly quoted?
• In which countries do they operate?
• Who are the key people in the competitor?
• What are their products and brands?
• Who are their customers, suppliers or other stakeholders?

Answering these questions can guide the search approach and also avoid wasting time searching for information that isn't needed. For example, if the objective is to only understand financials, then the areas of a corporate website that will hold this information are the investor pages, and often the press / news pages. Searching for technical data or product data or any of the other information that could be available would be a waste of time (and ultimately money).

In contrast, if the objective is to compile a full competitor profile, a more in-depth analysis will be needed, of both the competitor's own website and sites that link to the competitor, from it or even just mention it. The remainder of this article examines how to look at a competitor in depth, using the Web.

Finding Competitor Websites

Generally the main competitor website will be known - although sometimes companies maintain a number of sites, in different countries or for key brand-names. For instance Pepsi has www.pepsi.com  and www.pepsico.com - the former being the brand site, and the latter the corporate site. There are also multiple local country sites - www.pepsi.co.uk, www.pepsi.cn and many more, as well as some local country corporate sites e.g. www.pepsico.co.uk and www.pepsico.fr . A global profile on Pepsi's operations would need to look at each - as restricting to just the main .com sites could miss important intelligence and clues to what Pepsi is doing. A simple Google search (or even guesswork) will generally find the main competitor website, and local country versions can be found by substituting the .com for the local country variant.

Sometimes - and especially with large corporates - there will be numerous other sites linked to the company. In the case of Pepsi, there is also www.pepsiworld.com, www.pepsistreetmotion.com , www.pepsilime.com, www.pepsi.yahoo.com and many more. Some of these will have limited value while some may indicate or gives clues to what Pepsi is doing. One way for finding such sites is the SearchDNS tool on the Netcraft site (http://searchdns.netcraft.com). This allows you to enter in a keyword - and a list of sites containing that word is listed. In the case of domains starting with Pepsi, 214 domains are listed - of which many are relevant to Pepsi.

Another approach is to use a reverse IP lookup - as provided by www.domaintools.com. This looks at sites sharing the same IP address. Often these alternative names forward to the main site, but not always. For example, www.pbsg.com  forwards to www.pepsico.com but www.maxyourteam.com  doesn't forward to www.pepsi.co.uk even though both share the same IP address.

Checking each site can sometimes yield unexpected nuggets such as future promotions, products or even internal company websites that are not password protected. In the latter case, such sites would almost certainly not be found using a Google search, as they will be protected using the Robots metatag, a Robots.txt file or other search engine blocking techniques. Search engines respect the Robots.txt protocols and will not index the content of these pages. (In fact, it can be useful to check the top-level of a competitor site in case there is a robots.txt file. If there is, look for any directories, preceded with a ‘disallow' command. Such directories will not be indexed - and so could prove valuable for competitor research.)

Examining the competitor website

In a sense this seems straightforward - just navigate through the site to look at the various pages. Financials will usually be on the investor pages, corporate history and key personnel will be on the About Us pages and so on.

Often, however, there is more that can be found. Check for the existence of a Robots.txt file - and see what's there. Use the search box - and look for relevant material that may not be in site index. Even consider putting in terms such as ‘confidential' or ‘for internal use only' in case a page has been loaded by mistake. (For more hints on such ‘mistakes' made by companies on their publicly available pages see: . This lists further ways of taking advantage, legally, of such errors).

Do the investor pages include Acrobat PDF files, or PowerPoint presentations? The latter can be a goldmine if they contain graphs embedded into the presentation. Looking at the document properties of the downloaded file will show this. If the source data for graphs is an Excel spreadsheet or Microsoft Graph then, at the least, you will have found the raw data behind the chart, just by clicking on it. The background programme - Graph or Excel will open up. Sometimes even more can be found.

Until recently, a quoted UK company in the security industry included the PowerPoint files used in their annual results presentations on their investor pages. These included a results graph that linked to an Excel spreadsheet with multiple worksheets, including the budgeted and forecast sales of all their clients: several UK police forces, court services and custodial services. Their business was protecting human beings - not their own data!

Another approach is to search for hidden links. These are links that have no text associations and are often just a single pixel image in size. The aim is to put up pages that only those in the know can find. These may be used as a test site linked to a home page or other key page. A famous example can be found on the Amazon site directory ( ) at the bottom of the page just below the © 1996-1998. This directs you to a homage page in honour of David Risher. You can find such links using the Document or Page Info options on browsers - checking where each link leads and the text associated with them.

Looking further afield

As well as an examination of the company's own website, check who links to the company - and also who the company links to. In both cases, Google is not the best tool. Although Google has a link: field code allowing searches linking to a particular page, Yahoo! has a linkdomain: field code (the Yahoo! site explorer) that allows you to check all links to a particular domain. Microsoft's Live service has a linkfromdomain: field that allows you to check all links going out of the company domain. In both cases, unless the list is very large with 1000s or in- or out-links, check each one. Some may provide key information on the competitor - suppliers, customers or other stakeholders with an interest in the company.

Also look at some of the networking sites - LinkedIn.com, Xing.com, Facebook.com, Myspace.com, etc. as well as people search tools such as Zoominfo.com for company mentions and also competitor employees. Such sites will say what people below board level are doing, their background and interests. Information like this can be invaluable when compiling a list for primary research - as it enables contacts to be found easily, including, in many cases, ex-employees. Sites such as LinkedIn.com and Xing allow you to see where people used to work as well as where they are now. The networking sites can also be searched using tools such as yoname.com and wink.com.

Finally, what else can be found?  This is where the traditional search tools - Google, Yahoo!, Ask.com, Live, etc. should be used. Put in the company name, product names, the names of key people, etc. Add words such as ‘case study' or ‘strategy' to the competitor name to see what else comes up. Limit searches to Acrobat files with the filetype: fieldcode - and also use specialist search tools such as Omgili.com for finding forums, or Technorati for browsing blogs.

This just gives a brief overview of some of the approaches that can be used for finding competitive intelligence on the Web. I've not discussed foreign language searching (e.g. using Google's translate search at http://translate.google.com), focused searching using Exalead, podcast and video searches or advanced search techniques on the main search engines. These extend the repertoire of the competitive intelligence collection process even further.

Arthur Weiss is the founder and managing partner of AWARE (www.marketing-intelligence.co.uk), founded in 1995. AWARE provides competitive intelligence consulting, training and research - with a focus on the UK and online research. Arthur also teaches Marketing Research for the CIM Certificate & Diploma courses at Thames Valley University and has lectured and led workshops on various aspects of competitive intelligence globally. Previously Arthur worked as Marketing Information Manager with Dun & Bradstreet and was the general editor for Croner's Marketing - a Practical Management Guide. He has a BSc in Biochemistry and an MBA, and can be contacted at a.weiss@aware.co.uk. 

Related FUMSI Articles

DIY Detection: Competitive Intelligence for SMEs: http://web.fumsi.com/go/article/manage/2483

Vigilant Due Diligence: http://web.fumsi.com/go/article/manage/2498

Research: A Diligent Approach: http://web.fumsi.com/go/article/find/3125

Finding Competitive Information for Growing Companies: http://web.fumsi.com/go/article/find/2968

Develop Your Independent Business: Using Information Strategically: http://web.fumsi.com/go/au/2780

Finding People Information: Free Sources and Web 2.0: http://web.fumsi.com/go/af/2773

Click here for copyright permissions!

You may also be interested in:

• Subscribing to FUMSI »
  
  
• Browse the articles database for more Find articles »
  
  
• Search the articles database for more Find articles »
  
  
• Tools and reports for Finding Information »