Shrinking the haystack to find the needles you want.

Selecting and Connecting Sources to Your Intelligence Portal.

Let’s start by imagining your sources as a huge haystack of unstructured big data. Next, imagine that you must quickly search through it to find the golden information needles that you urgently need. Finally, picture the haystack as not only growing in size every time you see it, but also having new haystacks appear in other places. It becomes harder and harder to find the needles, right? Yet, this is the information world we live in today. We believe that the mysticism and confusion surrounding sources is largely due to challenge posed by “big data,” i.e. the enormous quantity of available sources.

The challenge before us is: how do we deal with the huge volumes of secondary information flows today? This article addresses a very straightforward question: what types of sources are there and how do we make the best use of them? It is of course beyond the scope of this article to provide a holistic and detailed coverage of all available sources for all industries. So, we will aim at providing:

1. A structural model for how to segment your particular set of sources
2. How to think in terms of source mixes with regards to available budgets
3. How to connect your sources to an Intelligence Portal

1. SEGMENTING SOURCES

Gathering information is not a challenge these days. Information is ubiquitous in most industries. The challenge is to gather relevant information by making a conscious selection and managing the sources to narrow down the target information into a digestible volume of useful content. In other words trying to shrink the haystack you are looking at.

For a start, let’s segment the entire source landscape into two very generic categories, with two subcategories each:

• External Sources

– Open sources such as web sites, news sources, social media, etc.

– Commercial sources such as analysis companies, market research, news agencies, etc.

• Internal Sources

– Observers, aka your own employees/colleagues throughout your company

– Content produced by your employees, such as shared drives, SharePoint sites, CRM systems, e-mail, etc.

The way one choses to mix these sources has a significant impact on output, as well as on required work volume and costs, so let’s begin by a brief overview of each category.

Open Sources

Open sources are publicly available sources that are often provided for free. Open sources have seen a never-ending increase in popularity over the recent decades. “It’s all free on the internet,” has become a dogma that justifies an information overload situation that few companies can cope with these days.

Today, e-mail alerts from any magazine, Google alerts, Social Media, and web spider services are examples of free, or near free, information sources. RSS (Rich Site Summary) is a popular format for delivering regularly changing web content. Many news-related sites, weblogs, and other online publishers syndicate their content as an RSS Feed to whoever wants it.

However, from an intelligence perspective, it is very important to consider the reliability and accuracy of the content. Open sources clearly do not provide any quality control whatsoever on the free content. With regards to social media, it has also started to become apparent that most of these kinds of sources are significantly biased in different directions as the typical web behavior is that you only voice your opinion if you really have a strong point, and as such, if it’s mainly negative. The few social media monitoring strategies that really seem to work is when the company itself “controls” the media and can use it to probe large user groups or customer groups. Is this to say that open source content is not to be used at all? No, definitely not, but make sure to select it carefully in order to avoid being drowned in a tsunami of irrelevant Internet links.

Commercial Sources

The “Quality before Quantity” method is probably the best way to summarize the argument for commercial sources, i.e. only retrieve the most relevant and trusted external sources to start with. Be prepared to pay for high quality content and then utilize technology to automatically filter and categorize the selected sources according to your view of the world. The return on investment is usually obvious to those companies that have embarked upon this route. Also, there are different types of commercial sources; pure aggregators that do not really produce content themselves all the way to specialized industry consultants with a content delivery practice as part of their services. This is also typically where the quality services of primary source intelligence are to be found. It is very worthwhile to investigate this for your industry and learn how to optimize the use of a mix of such commercial sources. However, the commercial sources market is huge and it can be helpful to segment it further to make it more manageable. It can be divided into four different vendor categories:

• Aggregators of free (open source) content: these are typically web crawlers, so-called “agents,” some Google services, etc. Pricing ranges from almost nothing to fairly pricy services, depending on the level of services available to support customers in both the selection and usage of such content.
• Aggregators of commercial content: major examples in this category are Dow Jones/Factiva, Reuters, LexisNexis, etc. The vast majority of their sources are professional news bureaus or other providers of copyright protected content.
• Pure originators of content: companies whose entire business is to provide subscription based information services but with very limited room for customer specific deliveries, one example is the Economist Intelligence Unit (EIU).
• Consulting companies with a content delivery component: these range from small niche boutiques to major consulting corporations and the selections is particularly needs specific.

It is important to realize that these different vendor categories have remarkably different business models. Understanding these business models will make it significantly easier for the sourcing manager to craft an optimal mix of sources with a reasonable budget.

Internal Sources – Observers

People are the eyes and ears of any company and yet are dismally used for intelligence purposes. With little effort any company can develop a wide and distributed network of sporadic collectors in which all employees are encouraged to participate. By doing this the company is developing a tremendous capability to identify a wide range of market signals. It’s easy! Enable any employee to report & contribute via web forms, mobile forms, or directly by e-mail. Add to this a simple touch of gamification and your employees will participate as part of the daily fun. The challenge with this source type is to keep it going.  Feedback loops are needed and a clear sense of quality control raises the “status” of being quoted, requested for further information, etc. Hence, this source is much more sensitive than the others to what happens to the content once it has been fed into the intelligence apparatus.

Internal Sources – Content

In most companies, this source of intelligence is by far the most underutilized and least leveraged source of all; your organization’s internal information. This is the only content, with the obvious exception of Observer input, that you can be sure of is NOT in the hands of competition. Often all competing companies possess the same external information, but it is the capability of drawing the best conclusions with respect to the company’s own strategy that selects the winner of the battle. The winners use a mix of external and internal information as a major advantage over those relying on external information only. Unfortunately, this source base is often ill managed and if there is no clear Knowledge Management initiative (or alike) the intelligence apparatus tends to be left in the dark. Making use of a global company’s entire content base is a very challenging task, unless there are approved internal tools for doing so.

2. CREATING THE RIGHT SOURCE MIX

Once you have understood the different types of sources available, you need to narrow that down and create the right source mix for your organization. The sources you select obviously have a huge impact on the result of the intelligence work you do, so you need to consider this very carefully. The sources you select should reflect the needs of your particular organization, i.e. what “needles” are you looking for in your haystack?

To help you do this we recommend that you use the
following information structure:

1. Gather the information we know we need
2. Gather the information we do not know we need, but do need
3. Avoid all information we know we do not need

So, what do we mean by “need”? In very broad terms it could be specified as:

1. Information supporting the current strategy
2. Information challenging the current strategy

Now, if we start combining these statements we quickly see that the “easy part” is gathering the information we know we need about the current strategy deployment needs (e.g. general market information, competitor watch, customer analysis, pricing support, product benchmarks). This is the “classical” CI scope, based on which intelligence analysis is to be performed in support of company operations.

On the other end of the scale we have the information we do not know we need and that is supposed to challenge the current strategy. History shows that the failure to gather and act upon such information is one of the key reasons companies go out of business, so the importance cannot be underestimated. Yet finding sources for this kind of information is perhaps the most difficult. Therefore, we need to balance the types of sources in the needs categories, in order to enable an efficient selection of sources.

With all the segments in place, it is possible to match the different source types with the different needs segments. One of the major challenges is often to filter out all the content that you do not need. Remember that external commercial sources are usually easier to start with since you can pay them to deliver exactly what you want. Internal sources, such as your Observers, often have the most valuable information, but it can be difficult to motivate people to share what they have.

3. IMPLEMENTING THE SOURCE MIX IN AN INTELLIGENCE PORTAL

With all your sources now at hand, how do you make best use of this vast amount of information? OK, that was a rhetorical question, since there is no other way to deal with such a flow of content than utilizing software tools. You need to use some form of an IT tool, such as an Intelligence portal, to manage and automate retrieval and distribution of all your sources.

Intelligence portals are a subsection of the so-called Content Analytics/Enterprise Information Access family of enterprise software platforms. The key purpose of an Intelligence portal is to provide a single point of access to both external and internal sources for all users and contributors of intelligence in your company. All the sources should be automatically structured and filtered in such a way that each individual perceives an optimal service to support his/her specific mandate of decisions.

Connecting your sources to your Intelligence Portal requires 5 key activities:

1. Set up your portal with an automated taxonomy that matches your intelligence needs and supports your team in the structuring of the unstructured information collected.
2. Make sure your portal has a simple mobile interface so your Observers can easily provide their insights at leisure.
3. Define a well-structured model for access rights to different content.
4. This serves three key purposes:
5. Manage copyright matters with your commercial content providers
6. Manage confidentiality matters with your internal contentSupport your users in filtering content based on their needs
7. Connect your external sources to retrieve your information.
8. Ask your IT department to help to provide access to the internal content you need.

Continuously Tuning Your Source Mix

Voila! You have your sources and they are all connected to your Intelligence Portal, so you are all set for the time to come, right? Wrong! You have indeed taken a huge leap forward in terms of intelligence capabilities, but sources and portals need to be continuously monitored and tuned in line with how your business and market evolves. This is however not as complex as it might sound, it just needs to be recognized.

There are two factors that need to be revisited and these are the taxonomy of topics and the sources. This should be done regularly, e.g. every quarter or at least once a year. The process can be illustrated as in the image below.

In summary:

• With the help of your intelligence portal, monitor the market evolvement and refine the taxonomy accordingly.
• When the taxonomy is refined, monitor the content development within the changed/new parts of the taxonomy and assess if your current sources provide sufficient volume of relevant content. If not, revisit the source/needs matrix and initiate dialog with your content providers and possible recommunicate your needs to your active observers.

4. CONCLUSION

Selecting sources for an enterprise intelligence operation is a complex and time consuming task. It is however worth every minute invested in the process since the outcome is usually a vastly better informed and insightful company. To summarize; the key factors are to define the different needs and disciplines and to match these against all types of sources. Having done that, you let it all come to life in an Intelligence Portal. And if you do all of this well, you will make it much, much easier to discover the “golden intelligence needles” by shrinking the haystack that you search in!

Jesper & Gabriel