By Dr. David James Kritz | 02/21/2024
Almost all intelligence collection begins with open source intelligence (OSINT). According to researchers Riccardo Ghioni, Mariarosaria Taddeo, and Luciano Floridi, approximately 80-90% of all intelligence comes from open source intelligence.
However, the U.S. Intelligence Community (IC) has long been entrenched in the belief that its strength lies in secrecy. Until recently, open source intelligence was considered a lesser-status intelligence discipline compared to:
- Geospatial intelligence (GEOINT)
- Human intelligence (HUMINT)
- Measurement and signature intelligence (MASINT)
- Signals intelligence (SIGINT)
Open Source Intelligence Has Been Criticized in the Past, But Not Now
One of the positive aspects of the global COVID-19 pandemic was a shift in the mindset that intelligence collection could not occur outside a sensitive, compartmented information facility. As new analysts and collection managers enter the Intelligence Community workforce, this mindset is changing and the stigma of using open source intelligence has decreased with each passing year.
In reality, open source intelligence has always been important to U.S. intelligence collection efforts. In the 1940s, for example, President Roosevelt formed a government unit, the Foreign Broadcast Monitoring Service, to monitor, translate, and analyze propaganda messages to the U.S. These messages were meant to undermine the U.S. and its citizens, and analysts working for this service had doctoral degrees that focused on social or political science to be as effective as possible
Today, open source intelligence is much more than doing a Google search. It is erroneous thinking that just because open source intelligence is collected from open sources that anyone can do open source intelligence. Analytic tradecraft always applies, regardless of the intelligence discipline used to collect information on a target.
Often, it takes an intelligence failure and reform to force how the IC collects relevant information and provide analysts with information they need to address priority intelligence requirements. Open source intelligence obtained reinforcement following the tragic events of 9/11, so that future terrorist attacks and intelligence failures would be mitigated.
Data collection from open source intelligence has become highly important to intelligence and national security senior leaders. In the 2022 National Security Strategy, President Biden’s goal to improve the IC included “enhancing integration of open source material.”
According to the Office of the Director of National Intelligence (ODNI), the first goal of the 2023 National Intelligence Strategy is to position the IC for intensifying the use of strategic communication. This intelligence document states, “The IC will improve its ability to provide timely and accurate insights into competitor intentions, capabilities, and actions by strengthening capabilities in language, technical, and cultural expertise and harnessing open source, ‘big data,’ artificial intelligence, and advanced analytics.”
Today, open source intelligence continues to evolve from gathering publicly available data from the printing press and the internet to artificial intelligence, natural language processing, and ChatGPT.
The Intelligence Community’s Customers
Open source intelligence from relevant sources is only one stream that the Intelligence Community’s customers consume. For intelligence collectors, it pays to stay abreast of events covered by the news and other open sources to understand topics, anticipate inquiries about current events, and understand the historical context of those events.
According to ODNI, the the IC’s customers include:
- The President
- The National Security Council
- The department heads of federal agencies in the executive branch of government
- The Chairman of the Joint Chiefs of Staff
- Senior military commanders
- Congress
During my four years living in the National Capitol region, I had opportunities to have discussions with members of Congress. During these discussions, I would always ask them, “How the IC can better support you?”
A common answer was, “When I ask a question, just tell me what you know on the subject. I do not have time to wait for a polished product.”
Government decisions can be made with or without an IC assessment. Open source intelligence is essential in this situation, because that information is collected in the open and allows for broader dissemination and distribution when it is used as the sole discipline for analysis.
In addition, using open source intelligence ensures that customers receive information in a more timely manner. open source intelligence is unclassified and less likely to be scrutinized by intelligence classification managers.
The value of finding the missing pieces of intelligence gaps to address priority intelligence requirements covers a wide range of mediums. There are many useful open source intelligence tools, including:
- The internet and search engines
- Social media platforms
- Grey literature
- Advertisements, brochures, and pamphlets
- Foreign press and media
- Mapping tools
- Patents
- Public records
The Internet and Search Engines
The internet provides bountiful opportunities to gather information from search engines on the surface web. There is also information in the deep web and the dark web, but that information is more difficult to access.
There are a considerable number of internet users, and a large amount of data is uploaded daily. In January 2024, there were almost five billion internet users globally and 147 zettabytes of data added daily, according to Finances Online.
Domo’s Data Never Sleeps depicts how much internet content is produced every minute of every day:
- 571 new websites are created
- WordPress® users publish 347 new blogs
- The mobile web receives 217 new users
- YouTube® users upload 500 hours of video
- Google® receives over 5.9 million search queries
- Meta® (formerly Facebook) users share 1.7 million pieces of content
- Instagram® users post 66 thousand photos
- X® (formerly Twitter) 347.2 thousand tweets
- Viewers spend 1 million hours streaming
Social Media Platforms
Personal and sensitive information can be collected by gathering data through publicly accessible social media sites. Social media websites such as Meta, X, Instagram, TikTok®, YouTube, and LinkedIn® provide a treasure trove of information for security professionals, because people freely upload personal information in the open for everyone to see. Open source intelligence from social media platforms includes:
- Addresses and telephone numbers
- Contacts, friends, and other relationships
- Job history
- Narratives of events with intentions to visit
- Pictures (sometimes with geolocations)
- Videos
Social media websites are often connected to each other to form social media networks that offer direct access to viewers. For instance, a Meta account may be used to display videos or images from Instagram, TikTok, and YouTube.
Grey Literature
Grey literature – information published outside of traditional publishing environments like academia and commercial publishing companies – can be highly useful. However, gathering intelligence from grey literature may be challenging for security professionals to find.
Imagine going to war in another country and being unsure if your military equipment will be effective in a different terrain. However, it's possible to gather raw data on soil composition and other variables in another country from the Library of Congress or other public archives.
Grey literature commonly includes:
- Annual reports
- Blogs
- Conference materials
- Podcasts
- Policy documents
- Government reports
- Think tank reports
- Unpublished studies
It is important to remember that some publications can be affected by publication bias. Sometimes, publishers may consider information that is contrary to established ideas and does not contribute to a discipline is not worthy of publication. But unlike academic literature, where the significance of a study’s results may matter, grey literature is unaffected by this publication bias.
Advertisements, Brochures, and Pamphlets
Information not easily found about military equipment can be found from various companies at private-sector tradeshows. The availability of advertisements, brochures, and pamphlets enable intelligence professionals to gather data on products' future capabilities and company or government intentions.
At such events, open source intelligence collectors can determine useful information such as:
- What companies produce specific resources
- How those resources may be applied to give countries an advantage over enemies
- Who are the current and potential customers
Foreign Press and Media
In the past, I was an intelligence collection manager who aided analysts and used foreign newspapers to determine the intentions of nation-state leaders. Translation was also necessary: articles often had to be translated from a native language to a second language and then to English.
This type of open source intelligence was essential, because it showed what a layperson might determine about a topic. Subject matter experts with a deeper understanding of the nuances of when/how certain words are used and the context/implications of word choices might see the same material as propaganda or misinformation.
Mapping Tools
Open source also includes mapping tools that can be used as graphical link analysis tools. For instance, you can use a graphical link analysis tool such as Google Maps to see locations, find distances between points, and see images of buildings and backgrounds.
To a certain extent, mapping tools allow users to visualize terrain and topography. Various websites such as NASA Earthdata Search and Maxar Open Data Program also provide free image for intelligence collectors. Similarly, NASA Worldview is an excellent open source tool for people seeking information on natural disasters.
Patents
Imagine a scenario where a new technological tool was created, and analysts were tasked with identifying who developed the new technology. Conducting a patent search would be one way for open source intelligence collection managers to provide information to analysts.
Intelligence collection managers may rarely think of patents, but patents can be important for closing intelligence gaps. Patents provide information on who has the exclusive rights to an invention and are commonly used to prevent unauthorized use or selling of the product, so it's possible to gather intelligence from patent-related documents. Furthermore, patents provide insight into who did something in a new way or developed a new technical solution.
Public Records
Information can also be gained from searching through the public data of public records. Examples include:
- Agendas
- Birth records
- Construction plans
- Contracts
- Court records
- County records
- Criminal records
- Government reports
- Grants
- Marriage licenses (published in newspapers)
- Meeting notes
- Obituaries (published in newspapers)
Economic Resource Consideration
It is not a secret that the U.S. needs robust economic resources to keep people safe. The total 2024 U.S. intelligence budget is $72.4 billion, according to ODNI. While this budget has steadily increased since the 9/11 terrorist attacks, it could change at any time.
While this budget has steadily increased since the 9/11 terrorist attacks, it could change at any time.
You might be surprised at how much information is outside commercial databases and is freely available. Unlike GEOINT and MASINT that rely heavily on research and design, these other intelligence disciplines are far more expensive than open source intelligence. For instance, open source intelligence practitioners are unlikely to be concerned about a failing satellite and its resulting costs.
HUMINT is expensive because the safety of intelligence personnel is a priority. But in the open source intelligence field, the event of death by murder is next to nil.
Unlike the other intelligence disciplines that primarily deal with collecting raw data, gathering actionable intelligence comes from information that was already in the open. Open information is abundant and the ability to store mass information or big data is a cost-effective way to retain information. This information can be placed in a repository that can be useful for storing past or current intelligence information.
The Weaknesses of Open Source Intelligence
Like the other intelligence disciplines, open source intelligence has its weaknesses. Open source intelligence is susceptible to deception by misinformation, disinformation, and propaganda, so that is why analytic tradecraft is always necessary.
Now that we live in the age of big data, information continues to grow exponentially. As a result, analysts need to be concerned about the timeliness of the raw data they receive.
Why Is Open Source Intelligence So Useful?
Open source practitioners are living in exciting times as the information age continues to evolve in terms of how information is generated and disseminated. Open source intelligence almost always acts as the beginning point to address priority actionable intelligence requirements and generate actionable intelligence. It is used in conjunction with other types of secret or classified intelligence to fill in knowledge gaps.
One example is North Korea’s secret underground nuclear tests. Open source intelligence was used to tip the world that a seismic event occurred, and MASINT provided refined details of the nuclear tests. Open source intelligence can also be applied to provide details on a target for HUMINT endeavors.
In addition, open source intelligence can be used to refine the quality of information. For instance, now that X is assessed as valid, what are the implications for Y and Z? What other hypotheses should be explored and why should those hypothesis be explored? This type of analysis leads to more informed decisions about threats to and opportunities for the U.S.
The growth opportunities for open source intelligence are almost limitless. At the individual level, people continue to put personal information on social media sites. At the global level, nations continue to face public pressure to become more transparent about their activities.
Today’s technology allows all of us to daily contribute information in the open at an unprecedented rate that continues daily. The capability of having the correct information at the right time from secondary sources allows intelligence professionals to offer quicker information dissemination to their customers.
WordPress is a registered trademark of the WordPress Foundation.
YouTube is a registered trademark of Google, LLC.
Google is a registered trademark of Google, LLC.
Meta is a registered trademark of Facebook, Inc.
Instagram is a registered trademark of Instagram, LLC.
X is a registered trademark of the X Corporation.
TikTok is a registered trademark of Bytedance, Ltd.
LinkedIn is a registered trademark of the LinkedIn Corporation.
