AI Can Re-create Video Games Just By Watching Them

There are many striking differences between a fence post and a human being, but one may prove particularly useful to robotic vehicles: temperature.

At least that’s what established thermal imaging firm FLIR and the Israeli startup Adasky think. Both companies are building new thermal sensors for use in autonomous cars, and believe that the extra data could be used to spot hazards in adverse conditions, particularly humans and animals on or near the road that create far more heat than their surroundings.

Some high-end automakers, including Porsche, BMW, and Audi, have been fitting vehicles with thermal imaging sensors from FLIR for several years. The vehicles use them to alert a driver to the presence of animals or people on the road at night, which may not show up within reach of the car’s headlights. Those sensors offer an image 320 pixels wide by 240 high.

Now, both FLIR and Adasky hope to achieve a similar result in autonomous cars using new sensors that they’re both building.

Adasky today announced a new far infrared sensor, called Viper, that offers better resolution (640 pixels by 480) at refresh rates of 60 frames per second, and can detect temperature differences as small as 0.05 °C. Adasky’s Dror Meiri tells MIT Technology Review that such sensitivity allows the device to not just spot something like an elk on the road, but also easily discern things like black ice on the surface of a highway. He says the device should be in mass production by 2020.

The video above shows Adasky’s sensor in use on a country road. On the left is a visible light camera, on the right its thermal sensor. Cyclists, pedestrians, and animals certainly show up far more sharply on the right. And the sharp contrast with surroundings would make it easier for a car to approach those living hazards with caution in a way that lidar, which provides only ranging information, may not.

FLIR is also developing a VGA device for use specifically in self-driving cars. Paul Clayton, director of automotive at FLIR, says that sample units of the sensors are being shipped to some manufacturers for testing this year and that mass-produced sensors could cost “a couple of hundred bucks.”

If you’re wondering why another sensor may be required in a driverless car, both companies point to an important, if uninspiring word: redundancy. “It’s an augment, a somewhat redundant sensor that helps classify in bad lighting conditions or weather,” explains Clayton. We’ve explained in the past that multiple sensors can allow autonomous cars to make better sense of their surroundings, and the sharp contrast of thermal imaging may enable, say, a glimpse of a human to provide extra context that helps avoid a dangerous incident.

Even so, an extra sensor does provide another data stream to process, and because thermal imaging is less widely used on the roads compared to visible light cameras and even lidar, the machine learning systems that discern details from footage will have to catch up. Both firms seem to understand that: Adasky has built its own AI to process data for its customers, and FLIR is putting together a thermal image database that clients can use to train their own neural networks.

Adoption of thermal imaging in autonomous vehicles will likely come down to budget. Some manufacturers may balk at the expense of yet another sensor, but until cheap lidar sensors offer the quality that automakers might like, the devices could add a useful layer of safety to the proceedings.