US Military using AR in combat

Microsoft has won a $480M contract with the US military to provide its HoloLens technology as part of an augmented reality technology package that the Army intends to use for training as well as in combat missions.

Background

Microsoft’s HoloLens was first released in 2016. Microsoft’s initial focus with HoloLens was on corporate and academic uses, from medical schools looking at hologram cadavers to architectural firms tweaking building plans in 3D. The company started shipping orders for the $3,000 developer edition of the goggles in March 2016. Unlike Google and its ill-fated Google Glass, Microsoft has kept HoloLens firmly positioned as a developer and business option as opposed to quickly pushing the product into the hands of consumers. This has limited the reach of the hardware and any near-term ambitions Microsoft might have otherwise had around augmented reality or, ‘mixed’ reality as it is termed in Microsoft. Microsoft has reportedly sold over 50,000 HoloLens to-date.

In 2016, the Israeli army’s C2 Systems Department purchased two HoloLens glasses from Microsoft Corp. The commander of the head programming department, Major Rotem Bashi, intended to develop the technology to improve battlefield strategy and train field personnel. He planned for HoloLens to be used on active duty within months. The U.S. Army has been using Microsoft’s HoloLens during training missions since at least 2016, providing soldiers with live information and locking onto targets.

Analysis

Microsoft has won a $480 million deal to supply more than 100,000 augmented-reality HoloLens headsets to the US Army. The Army plans to use the headsets for combat missions as well as training. The technology will be adapted to incorporate night vision and thermal sensing, offer hearing protection, monitor for a concussion, and measure vital signs like breathing and “readiness.” AR firm MagicLeap also bid for the contract.

HoloLens is used for training by the US and Israeli military already, but this would be the first time it’s been used for live combat. It’s another example of how AR is being adopted far more enthusiastically by organisations than consumers.

The military is planning to use Microsoft’s augmented reality goggles to “increase lethality by enhancing the ability to detect, decide and engage before the enemy,” according to a government description of the program. The Army is also planning to enhance the capabilities of the HoloLens by adding night vision, thermal sensing, live vital signs monitoring, and hearing protection.

In 2016, the Israeli army created a software program for the HoloLens that allowed commanders to manipulate military terrain models and intelligence data to monitor troop positioning from enemy vantage points. Battlefield maps are superimposed on top of the real terrain, streamed in via satellite, to create a blend that can be interacted with via sight, voice and hand gestures. The team has found ways to allow HoloLens-wearing medics to operate on wounded with simultaneous instruction from trained surgeons, and combat soldiers to fix equipment malfunctions. Other plans range from streamlining security checks at roadblocks by installing face recognition software to controlling robots and even drones through gesture.

Counterpoint

There are still issues to iron out: The device works better in a closed room than in sunlight, and communication - conducted via Skype - must be secure.

Earlier this year, a group of Microsoft employees published a letter on Medium, asking the company not to bid on the Joint Enterprise Defense Infrastructure (JEDI) contract, over moral concerns that employees were being asked to develop tools that would be used for waging war. In this case, the linkages are even more explicit. JEDI is a program to develop cloud services. This new HoloLens contract directly calls for sophisticated warfighting capabilities to be implemented in hardware with the goal of locking onto targets more efficiently.

Microsoft’s response to its own employees’ concern over JEDI was to promise that individuals who did not wish to work on the project would have the opportunity to transfer to other areas of the company. Presumably, the company will make a similar offer here, especially as the Pentagon contract fundamentally changes the nature of what HoloLens is for the employees who will work on the military version.

Assessment

Our assessment is that with the tremendous opportunity presented by new technologies, it is essential to understand how to develop software for a device such as the HoloLens, and find ways in which it can impact the military. However, we believe it is essential to invest even more heavily into securing these devices, as the digital space is a veritable battleground in its own right, and military devices are especially vulnerable targets.

India Watch

ShootAR is an Augmented Reality simulator developed for the Indian Army’s combat training. It is an immersive environment that enables the soldier or trainee to view and interact with a mixed rendered environment of the real world with the virtual world. The viewing hardware used is a see-through head-mounted display device similar to Google Glass. The simulator uses a sensor controlled gun prototype. The trainee sees a simulation of virtual terrorists moving around in an AI controlled fashion, in a real physical room with real people. The gun position and trainee's head orientation is tracked in real time to analyse the trainee's actions like trigger press on the gun to calculate a hit on the virtual terrorist.