Surf the Data Tsunami

The great information surge offers the Navy the opportunity to replace situational awareness with situational understanding. 

To make sense of all this video and sensor data today, human analysts have to look at it or use a computer algorithm to decide if it requires further action. Error rates, even by experienced operators, and the large numbers of automatic system-generated false alarms, show that neither approach works effectively. The military needs to combine the two in an innovative, flexible way. Perhaps Watson, IBM’s artificial intelligence (AI) machine famous for winning Jeopardy!, can provide some clues.

LEARNING TO SURF

Algorithms are great at analyzing video and image content quickly, but they need to be trained by humans on what to look for, and when the stakes are high, humans need to play a role in the final analysis.

The entertainment and AI worlds collided recently when Watson created the first AI-directed movie trailer, for the film Morgan (an apt choice given its subject, an artificially created human).3 The trailer is the result of a true collaboration between AI and humans. Breaking down the workflow shows the handoffs between Watson and the human creative team and demonstrates how algorithms combined with people power can change the process of video analysis—and even how new things can be created from it. The Morgan trailer is a good example of the “algorithms + human cognition” approach needed to ride the tsunami.

The first step is for people to train the algorithms. The IBM researchers taught Watson what types of scenes typically show up in a horror-movie trailer by selecting 100 trailers and then choosing video segments the AI could analyze. A number of different algorithms detected people, objects, and scenery, and each scene was tagged with a label, such as “eerie,” “frightening,” and so on. Then the researchers gave Watson the full movie from which to select scenes that would fit into a typical horror movie trailer. Watson selected ten based on the inputs and tagging. Finally, a human editor chose a few of Watson’s selections, arranged them, and added music and title cards. The process only took about 24 hours, weeks less than the time typically required.

The human part is easy to comprehend, but the algorithms sometimes seem to be magical. If a simple Facebook algorithm can tag your friends in a photo without human intervention, we should expect the limitless world of computer algorithms to be able to deal with all of the data being generated by today’s tactical systems. Yet for all their apparent omniscience, computers remain nothing more than direction-following machines, and an algorithm is just a series of instructions to them. Even the seemingly autonomous unmanned vehicles the Navy uses simply carry out a series of instructions.

The key is to have the right algorithms. Currently programmers load a blank algorithm with known information, known as “ground truth.” This “blank algorithm” approach is the basis of machine learning—teaching a machine by using ground truth data. It learns and starts to draw conclusions about items, making correlations between things that are similar. This is familiar to anyone who has seen an e-commerce scrollbar saying, “People buying your item also bought . . . .” People who buy bicycles have a higher-than-average probability of needing helmets, gloves, reflector vests, and so on. These things are simple to predict. But an algorithm also can tell you when average ibuprofen and bandage consumption increases unpredictably, and not just that it has increased but that it did so by 17 percent. It will learn that a woman in her teens or 20s who is buying bigger clothes and lotions not normally associated with her personal customer profile has a higher probability of being pregnant.4

The more specific the problem-set to which an algorithm is applied, the higher probability it has of success. The trick is to make it as specific, and free of edge cases (problems that occur only at extrema), as possible. When algorithms go wrong, it can have embarrassing consequences, as Google found out to its dismay when some people were tagged as gorillas in a facial-recognition algorithm, or possibly fatal consequences in a tactical situation such as a missed weapon detection.5 On the other hand, the more ground truth you include in an algorithm, the more complex the algorithm needs to be, adding enormous complexity that will require substantial processing time to produce results. Another difficulty is that separating the correct results from the incorrect ones takes additional time and, in most cases, human involvement. Last, training each algorithm on one highly specialized toolset specific to a problem requires a large investment but cannot produce useful results for a different problem. Every time the problem-sets change, analysts start over again using the same blank algorithm. Applying what was learned during the development of the previous algorithm can shorten the time to build the next one, but each starts with new ground truth. This solution is untenable.

We envision a world of algorithms and microalgorithms that do one small thing, or series of things, very well, working in concert to help arrive at an answer. Programmers, researchers, and businesses are developing millions of these algorithms; the algorithmic job of the future will be to determine which of those millions should be applied to “my” problem. Algorithms take cues from people, and people take over where algorithms leave off, so humans will have to remain in the loop. Human-machine teaming is the only thing that makes a practical solution possible. This is the essence of the Department of Defense’s “Third Offset Strategy.” Its successful application will depend on combining humans and algorithms in a way that leads to better decisions.

BUILDING THE SURFBOARD

Human-machine teaming is fundamentally different than either designing a better man-machine interface (MMI) or using automation. Just as surfers could not have conquered monster waves without SkiDoo tow-in and advanced board technology, warfighters need new approaches to overcome the big wave of data challenges. By fundamentally rethinking and redesigning the way warfighters work, they will be able to work seamlessly and collaboratively with technology. The good news is that the technology exists for a new approach to the vast amounts of unstructured data-heavy streams such as turning real-time video into knowledge.

Human operators make sense of their world by collecting data and information, comparing that data to what they already know, making inferences, and drawing conclusions so they can make decisions. Data and information come from many sources and can have different meanings depending on the context or frame of reference. For humans to make a decision when risk is involved, they need to extract the right information and determine its meaning with respect to the particular goal or outcome. Do the risks outweigh the benefits? What are some other options? How can risks be mitigated? How much risk exists? What information is missing? Should more data be collected? These questions usually play out quickly and unspoken in the minds of the decision makers.

A “friendly” vessel may not be so friendly if it is 440 meters long, weighs 220,000 tons, and is sitting where a submarine wants to surface. It is obvious to the submarine’s crew that an ultralarge containership represents a real danger in this context.

Automating the derivation of knowledge from data with AI begins by identifying the problem or the issue to be resolved. The next step is to develop a structure for the data that underlies that knowledge. A structure that allows human knowledge and machine speed and efficiency to be combined is called an “ontology.” It does this by describing the relationship among the data, objects, actions, and other critical elements that drive decision-making. The very act of performing an ontological analysis clarifies the structure of that knowledge.

As an example of this approach, sonar tactical displays can be analyzed ontologically as a way to detect and classify unknown contacts more quickly and accurately. By treating the tactical system’s outputs (e.g., sonar displays and audio signals) as algorithm inputs, we can take advantage of machine learning to analyze those outputs ontologically.

A sonar operator could highlight particular acoustic features present on the sonar display; the algorithm then would compare those features to the ground-truth library to generate a list of possible contacts (along with confidence levels) for presentation to the human. If additional features were to be highlighted, the list and confidence levels would be updated. The operator would then use intuition and knowledge of the tactical situation to make decisions. This is different from the operator having to keep track of everything and make all the comparisons in his head. Dr. William D’Angelo at the Office of Naval Research is currently funding a project to compare operator performance under both conditions using archived sonar data.

HANGING TEN

The ontological approach has great advantages for both the operator and the system developer. With it, operators will make better decisions. Because this type of system is completely scalable, the ontological structure can handle a great number of different inputs and input types. An analyst can work from a single sensor or a combatant commander’s theater-level inputs, as when the Joint Interagency Task Force searches the Gulf of Mexico for drug runners’ semisubmersible submarines. The approach also avoids the limitations of current datamining techniques applied to traditional database structures.

In addition, analyzing the data in this way indexes the data according to the ontological structure. This allows the data to be searched, played back, and analyzed in ways never before possible. The decision maker has the ability to search the stored data semantically—not merely play it back like an old video cassette—and perform in situ an-alyses that are not time-feasible in the current systems. This creates the ability to use near-real-time data to modify tactics and increase tactical or strategic advantage. Think of a professional football team analyzing video data during the game to adjust and modify their play-calling—waiting for the next game or season to gain lessons learned and improve is not an option.

Using an ontological framework offers new and creative ways to present the data. Decision makers can better engage with the data and information and gain new insights that reduce the time to make a decision. These combined advantages allow the warfighter to move from situational awareness to “situational understanding.”

For developers, such a knowledge-based system can be produced as a standalone box that requires only a connection to a video stream from a sensor or output from non-sensor data—e.g., intelligence messages. This separation reduces integration complexities and limits impact on current operations and manpower while it greatly increases capability.

One major problem with current approaches for getting a good common-operating picture is that data from different combat systems vary widely, among not only ship classes but even different flights of the same class, as well as from different services. Attempts to use common data interfaces or structures have not solved the problem. The semantic framework of ontology greatly strengthens system interoperability by allowing non-proprietary sharing of data among previously incompatible systems.

It also addresses the reuse problem of many sets of algorithms. The approach can be applied in different situations by changing the ontology structure when moving from one problem set to another. These advantages allow system architecture to evolve from cobbled-together “systems of systems” to a truly integrated worldview.

SITUATIONAL UNDERSTANDING

If the Navy does not address the data tsunami, the fleet will be engulfed by it. But by adopting the “algorithms + human cognition” approach, the service will go far beyond handling this burgeoning data problem. It will solve interoperability challenges that have befuddled acquisitions and operational programs for years and, more important, will offer commanders an entirely new way of understanding their situations. If radar and sonar brought commanders situational awareness that exceeded what lookouts posted on mastheads could do, ontological sensor analysis will bring about an understanding that will change the way fleets act and fight profoundly.

Notes:

1. Jesse Sloman and Sean Cate, “Operating under Constant Surveillance,” U.S. Naval Institute Proceedings, 142/5/1 (May 2016), 59.

2. “SkySat 3, 15,” Gunter’s Space Page.

3. Amelia Heathman, “ IBM Watson Creates the First AI-Made Film Trailer—And It’s Incredibly Creepy .” 

4. Kashmir Hill, “ How Target Figured Out a Teen Girl Was Pregnant before Her Father Did. ” 

5. Jessica Guynn, “ Google Photos Labeled Black People ‘Gorillas.’ ”