Using AI to unleash the power of unstructured government data
Applications and examples of natural language processing (NLP) across government
Government agencies are awash in unstructured and difficult to interpret data. To gain meaningful insights from data for policy analysis and decision-making, they can use natural language processing, a form of artificial intelligence.
Tom is an analyst at the US Department of Defense (DoD).1 All day long, he and his team collect and process massive amounts of data from a variety of sources—weather data from the National Weather Service, traffic information from the US Department of Transportation, military troop movements, public website comments, and social media posts—to assess potential threats and inform mission planning.
While some of the information Tom’s group collects is structured and can be categorized easily (such as tropical storms in progress or active military engagements), the vast majority is simply unstructured text, including social media conversations, comments on public websites, and narrative reports filed by field agents. Because the data is unstructured, it’s difficult to find patterns and draw meaningful conclusions. Tom and his team spend much of their day poring over paper and digital documents to detect trends, patterns, and activity that could raise red flags.
1 Tom is a hypothetical example. View in article
William D. Eggers