More Reactive Camera Helps Warfighter Identifying Incoming Attacks

Behind the extensive work of Dr. Akhilesh Jaiswal, the warfighter might soon have a more reactive camera to leverage in identifying incoming attacks from enemy combatants.

Dr. Jaiswal, an assistant professor in the Department of Electrical and Computer Engineering at the University of Wisconsin, is working to develop a camera that doesn't just record images but understands them in real time. That's the promise of the IRIS (Integrated Retinal Functionality in Image Sensors) camera, a bio-inspired innovation being explored by the Air Force Research Laboratory (AFRL).

“Many [Department of War (DoW)] applications require cameras,” said Dr. Jaiswal, who has collaborated with Dr. Gregory Schwartz (Northwestern University) and Dr. Maryam Parsa (George Mason University) on the project. “In a typical application, cameras remain ‘dumb.’ They simply convert light intensity into electrical signals. All critical decision making is offloaded to other systems outside the camera or at times far away in [the] cloud. This leads to slower decision-making speed and excessive power consumption in moving millions from pixels from their source (cameras) to decision-making hardware."

“We believe many commercial and [DoW] use cases, specifically those involving high-speed platforms like drones or weapon systems, could benefit from IRIS cameras. For example, if an object is about to hit the platform under interest, an IRIS camera in real-time can identify it is under attack and also predict the future trajectory of the attacker.”

The inspiration came from a unique place.

“This specific project started at USC (University of Southern California),” he said. “I ended up reading some news articles about interesting circuits neuroscientists discovered in the eye. I started reading about the capabilities of animal retinas. In the last decade, research has shown the retina is a very sophisticated computer. I dug more into it because I felt that if these capabilities could be put into a camera system, we could tackle a lot of problems with autonomous cars, and specifically for very high-speed moving platforms like drones or fighter jets.”

That interest led to the eventual development of the collaborative project with George Mason and Northwestern universities.

“As opposed to state-of-the-art cameras, animal eyes are not just converters, they are sophisticated spatiotemporal computers,” Dr. Jaiswal said. “They have evolved over millions of years to extract critical features about objects, their size, shape and motion in real-time to elicit stereotypical attack and escape responses. Through a truly interdisciplinary team involving retinal experimental and computational neuroscientists, hardware design experts and algorithm designers, our project aims to create new class of spatiotemporal cameras that can mimic the feature extraction functionality of animal eyes.”

Focusing on the retina allows the camera to bypass what Dr. Jaiswal refers to as an “energy bottleneck,” in which the camera isn’t transmitting enormous amounts of data to do computations.

“In the retina, we can compress the data immediately,” Dr. Jaiswal said. “As signals flow through the retina layers, it starts extracting features, like, ‘is there an object that's about to hit me?’ Rather than sending millions of pixels, I'm sending very specific information, maybe 10-to-20 features, each encoded as voltage spikes. Rather than transmitting millions of pixels, where each pixel doesn't have a lot of information, and it's all those pixels together that create the information, we can transmit meaningful information. You don't need a million pixels to figure out that you need to run away. The end goal is to get rid of this energy and latency bottleneck and make the systems more responsive.”

In looking to advance the project, Dr. Jaiswal was fortunate to connect with Dr. Robert Ewing, director of the Center for Innovative Radar Engineering at the AFRL. They got together through Tech Connect.

“It was great to get connected to Dr. Ewing and his team,” Dr. Jaiswal said. “We're now connected to the right people who can tell us if what we're developing has a real application or a real need. Making that connection was great, and the fact that it didn't take a lot of time to fill out the details didn't discourage us. I took the chance, and I'm pretty happy it helped.”

Dr. Ewing sees potential in IRIS.

“This technology enables a new class of low-cost, secure and networked EO (electro-optical) imaging systems for autonomous platforms, including automotive applications,” he said. “Its impact lies in embedding advanced functions, such as deep-learning inference, adaptive processing and motion prediction, at the pixel level. This is achieved through biologically inspired circuit elements like gap junctions and biphasic filters, which emulate retinal processing mechanisms." 

“They also extend to NASA (National Aeronautics and Space Administration) areas such as planetary missions, earth remote sensing and space optical communications and also into more general fields such as fiber communications, 3D LIDAR imaging and tomography, night vision, quantum entangled encryption and broader homeland security and medical applications.”

Working with Dr. Ewing and Tech Connect has been essential in establishing a path forward for IRIS.

“One of the questions I had was, ‘how do I get more traction from the [DoW] and get people more interested in this technology,’” Dr. Jaiswal said. “His feedback was that if you could show that you can implement a specific problem, automatic target recognition, there would be more takers for the technology. As we're planning our silicon prototype effort, that's one of the central themes we're trying to address. ‘Can we have some aspect of target recognition or target tracking embedded in our first prototype demo?’ That would help us sell it better to end-users in AFRL and other defense agencies.”

About AFRL

The Air Force Research Laboratory, or AFRL, is the primary scientific research and development center for the Department of the Air Force. AFRL plays an integral role in leading the discovery, development and integration of affordable warfighting technologies for our air, space and cyberspace force. With a workforce of more than 12,500 across nine technology areas and 40 other operations across the globe, AFRL provides a diverse portfolio of science and technology ranging from fundamental to advanced research and technology development. For more information, visit www.afresearchlab.com.

About Air Force and Space Force Tech Connect

The Air Force and Space Force Tech Connect website provides access to current, open opportunities, meet-up events, other Department of the Air Force science and technology enterprise connectors and a way for anyone to share an idea. The Tech Connect team, comprised of AFRL personnel, connects quality, relevant ideas/technologies with Department of the Air Force subject matter experts. The team will review ideas/inquiries, provide feedback on innovative ideas and establish a dialogue with potentially interested Air Force and Space Force programs. For more information, visit: https://airforcetechconnect.org/.