Platform for Scalable
Testing of Autonomous Vehicle Safety
From University
of Illinois College of Engineering
October 25, 2019
-- In the race to manufacture autonomous vehicles (AVs), safety is crucial yet
sometimes overlooked as exemplified by recent headline-making accidents.
Researchers at the University of Illinois at Urbana-Champaign are using
artificial intelligence (AI) and machine learning to improve the safety of
autonomous technology through both software and hardware advances.
"Using AI
to improve autonomous vehicles is extremely hard because of the complexity of
the vehicle's electrical and mechanical components, as well as variability in
external conditions, such as weather, road conditions, topography, traffic
patterns, and lighting," said Ravi Iyer
"Progress is being made, but safety
continues to be a significant concern."
The group has
developed a platform that enables companies to more quickly and
cost-effectively address safety in the complex and ever-changing environment of
autonomous technology. They are collaborating with many companies in the Bay
area, including Samsung, NVIDIA, and a number of start-ups.
"We are
seeing a stakeholder-wide effort across industries and universities with
hundreds of startups and research teams, and are tackling a few challenges in
our group," said Saurabh Jha, a doctoral candidate in computer science who
is leading student efforts on the project. "Solving this challenge
requires a multidisciplinary effort across science, technology, and manufacturing."
One reason this
work is so challenging is that AVs are complex systems that use AI and machine
learning to integrate mechanical, electronic, and computing technologies to
make real-time driving decisions. A typical AV is a mini-supercomputer on
wheels; they have more than 50 processors and accelerators running more than
100 million lines of code to support computer vision, planning, and other
machine learning tasks.
As expected,
there are concerns with the sensors and the autonomous driving stack (computing
software and hardware) of these vehicles. When a car is traveling 70 mph down a
highway, failures can be a significant safety risk to drivers.
"If a
driver of a typical car senses a problem such as vehicle drift or pull, the
driver can adjust his/her behavior and guide the car to a safe stopping
point," Jha explained. "However, the behavior of the autonomous
vehicle may be unpredictable in such a scenario unless the autonomous vehicle
is explicitly trained for such problems. In the real world, there are infinite
number of such cases."
Traditionally,
when a person has trouble with software on a computer or smart phone, the most
common IT response is to turn the device off and back on again. However, this
type of fix is not advisable for AVs, as every millisecond impacts the outcome
and a slow response could lead to death. The safety concerns of such AI-based
systems has increased in the last couple of years among stakeholders due to
various accidents caused by AVs.
"Current
regulations require companies like Uber and Waymo, who test their vehicles on
public roads to annually report to the California DMV about how safe their
vehicles are," said Subho Banerjee, a CSL and computer science graduate
student. "We wanted to understand common safety concerns, how the cars
behaved, and what the ideal safety metric is for understanding how well they
are designed."
The group
analyzed all the safety reports submitted from 2014-2017, covering 144 AVs
driving a cumulative 1,116,605 autonomous miles. They found that for the same
number of miles driven, human-driven cars were up to 4000 times less likely
than AVs to have an accident. This means that the autonomous technology failed,
at an alarming rate, to appropriately handle a situation and disengaged the
technology, often relying on the human driver to take over.
The problem
researchers and companies have when it comes to improving those numbers is that
until an autonomous vehicle system has a specific issue, it's difficult to
train the software to overcome it.
Further, errors
in the software and hardware stacks manifest as safety critical issues only
under certain driving scenarios. In other words, tests performed on AVs on
highways or empty/less crowded roadways may not be sufficient as safety
violations under software/hardware faults are rare.
When errors do
occur, they take place after hundreds of thousands of miles have been driven.
The work that goes into testing these AVs for hundreds of thousands of miles
takes considerable time, money, and energy, making the process extremely
inefficient. The team is using computer simulations and artificial intelligence
to speed up this process.
"We inject
errors in the software and hardware stack of the autonomous vehicles in
computer simulations and then collect data on the autonomous vehicle responses
to these problems," said Jha.
"Unlike humans, AI technology today
cannot reason about errors that may occur in different driving scenarios.
Therefore, needing vast amounts of data to teach the software to take the right
action in the face of software or hardware problems."
The research
group is currently building techniques and tools to generate driving conditions
and issues that maximally impact AV safety. Using their technique, they can
find a large number of safety critical scenarios where errors can lead to
accidents without having to enumerate all possibilities on the road -- a huge
savings of time and money.
During testing
of one openly available AV technology, Apollo from Baidu, the team found more
than 500 examples of when the software failed to handle an issue and the
failure led to an accident.
Results like these are getting the group's work
noticed in the industry. They are currently working on a patent for their
testing technology, and plan to deploy it soon. Ideally, the researchers hope
companies use this new technology to simulate the identified issue and fix the
problems before the cars are deployed.
"The safety
of autonomous vehicles is critical to their success in the marketplace and in
society," said Steve Keckler, vice president of Architecture Research for
NVIDIA. "We expect that the technologies being developed by the Illinois
research team will make it easier for engineers to develop safer automotive
systems at lower cost. NVIDIA is excited about our collaboration with Illinois
and is pleased to support their work."
No comments:
Post a Comment