Human-Machine Collaboration in Intelligence Analysis: An Expert Evaluation

In this paper we illustrate how novel AI methods can improve the performance
of intelligence analysts. These analysts aim to make sense of — often conflicting or incomplete — information, weighing up competing hypotheses which
serve to explain an observed situation. Analysts have access to numerous visual analytic tools which support the temporal and/or conceptual structuring
of information and collection, and on the other hand other tools can be used to
evaluate alternative hypotheses. We believe, however, that there are currently
no tools or methods which allow them to combine the recording and interpretation of information, and that there is little understanding by analysts about how software tools can facilitate the hypothesis formation process. Following the identification of these requirements, we developed the CISpaces (Collaborative Intelligence Spaces) decision support tool in collaboration with professional intelligence analysts. CISpaces combines multiple AI-based methods including argumentation theory, crowdsourced Bayesian analysis, and provenance recording. We show that CISpaces is able to provide support to analysts by facilitating the interpretation of di↵erent types of evidence through argumentation-based reasoning, provenance analysis and crowdsourcing. We undertook an experimental analysis with intelligence analysts which highlights three key points. (1) The
novel, principled AI methods implemented in CISpaces advance performance in
intelligence analysis. (2) While designed as a research prototype (at TRL 3),
analysts benchmarked it against their existing software tools, and we provide
results suggesting intention to adopt CISpaces in analysts’ daily activities. (3)
Finally, the evaluation highlights some drawbacks in CISpaces. However, these
are not due to the technologies underpinning the tool, but rather its lack of
integration with existing organisational standards regarding input and output
formats. Our evaluation with intelligence analysts therefore demonstrates the
potential impact that an integrated tool building on state-of-the-art AI techniques can have on the process of understanding complex situations, and on how such a tool can help focus human e↵ort on identifying more credible interpretations of evidence.