And Google, of course, is not the only company developing driverless cars. It's an industry-wide project, yet many automakers have an appalling disrespect for public safety: they fought seat belts, set private detectives to harass and intimidate consumer advocate Ralph Nader, and lobbied against air bags for years, delaying a mandate until 1998.
Meanwhile, also in the 1990’s, over 200 people died in crashes linked to failure of Firestone tires, but they weren’t recalled until 2000. Consumer advocates charged that both Firestone and Ford (in whose vehicles a majority of the deaths occurred) covered up the problem and failed to inform NHTSA.
Present-day behavior is no better: In March of this year, Toyota agreed to pay a record $1.2 billion criminal penalty stemming from charges that it intentionally hid information about safety defects from the public and made deceptive statements to protect its brand image. That was in addition to an earlier $48.8 million civil penalty and a litigation settlement of over a billion dollars. All were in connection with unintended acceleration: ultimately, in two separate recalls in 2009 and 2010 totaling nearly 8 million vehicles Toyota acknowledged that its gas pedals could get trapped under floor mats or that the pedals themselves could simply stick, leading to out of control vehicles. Deaths and injury were the result.
Even more ominously for driverless cars, a jury in 2013 found Toyota liable for unintended acceleration stemming from a third cause, alleged software errors. An expert on embedded systems (real-world machinery that incorporates software), Michael Barr, charged that he found 80,000 violations of software reliability programming rules in Toyota’s code (pdf, p. 29), even after an investigation by NASA (on behalf of NHTSA) had found no software cause for unintended acceleration but not ruled out the possibility either (pdf, p. 17; pdf, p. 13); a NASA engineer charged that its team was taken off the investigation prematurely. The jury awarded $3 million in compensation and was set to consider punitive damages when the case settled.
Toyota is not the only automaker with unintended acceleration problems. Just last weekend, NHTSA disclosed that it’s investigating 360,000 Nissan cars for the same problem, in this case allegedly caused by improper design of a trim panel. Last month, Ford recalled 1.4 million vehicles, for defects including loss of power steering and (as with Toyota) floor mats that can interfere with the accelerator.
That followed a recall by Ford of about 700,000 vehicles for software bugs that could cause airbags to delay inflation and – a separate problem on the same vehicles – doors that could come open while the car is moving. Also last month, Chrysler recalled almost 500,000 SUVs because cracks in a circuit board could lead to a faulty signal that caused the transmission to shift by itself from Park into Neutral. Toyota too had a software recall, in 2010, involving anti-lock brake software and another last month, involving airbag electronic control unit software.
Also on the airbag front, chemical and mechanical defects have been disclosed in the last several weeks in airbags from a company called Takata, resulting in recalls or halted sales by eight car companies: Ford, Chrysler, Honda, Mazda, Nissan, Toyota, BMW – and GM, a company which this year has recalled over 28 million cars in about 30 separate recalls.
The most noted of those recalls, of course, have been for an ignition switch problem. In the last five months the company has recalled about 14.8 million vehicles (6.6 million in the last several months plus 8.2 million yesterday) from model years 2003-2011 because the ignition switch can turn off while the car is in motion, disabling the power steering, power brakes and airbags. According to the government, GM knew of the problem since at least November 2009, but didn’t tell NHTSA for almost 4-1/2 years. The New York Times reports that GM’s knowledge goes much further back: a GM engineer approved the faulty design in 2002 before the car was released, even though the part maker said the switch didn’t meet specifications. GM investigated later complaints, but closed the inquiry in 2005 because “none of the solutions represents an acceptable business case.”
Four months later came the first death tied to the ignition switch.
The following year, the same GM engineer allegedly signed off on a redesigned switch intended to be safer, but there was no recall for another eight years. Last year, the engineer claimed under oath in a wrongful death suit that he had not authorized the change (notwithstanding a signed document to the contrary); he told Congressional investigators in May that he didn’t remember.
Also last month, the company paid a record $35 million civil penalty even as the Secretary of Transportation said “what G.M. did was break the law.” According to NHTSA, GM employees were actually trained in how to obscure safety problems. Documents released last week by a Congressional panel reportedly show that a current top GM executive knew of the problem as far back as 2005. Criminal, congressional, SEC and state investigations are ongoing. Hundreds of lawsuits have been filed, and a compensation plan was unveiled yesterday that could cost the company billions.
In addition, a second GM ignition switch problem has emerged: in many of the same cars with switches that could turn off while driving, it was also possible to remove the ignition key while the car wasn’t in Park or while the engine was running. A GM report said that the company knew about this in the early 2000’s but didn’t recall the cars until April of this year. Meanwhile, although NHTSA received the report in April, it wasn’t posted on the agency’s website until a reporter asked for it in June.
These are the sorts of companies that would self-certify their driverless cars.
What About the Regulators?
Not only did NHTSA delay posting the safety report, it seems to have fumbled the GM ignition switch issue from the beginning. The agency received the first complaint in 2003 but didn’t propose an investigation until 2007. Even at that, no recall occurred until 2014. Instead, despite receiving an average of two complaints per month, NHTSA repeatedly responded that “there was not enough evidence of a problem to warrant a safety investigation,” according to the New York Times. Now Congress is investigating NHTSA as well as GM.
NHTSA doesn’t only handle recalls; it’s the federal agency that issues automobile safety rules. Under California’s self-driving car law, any NHTSA regulations on the subject will preempt whatever the DMV is doing now.
Commenting on the agency’s approach to software, embedded systems expert Barr said, “NHTSA … needs to step up on software regulation and oversight. For example, FAA and FDA both have guidelines for safety-critical software design within the systems they oversee. NHTSA has nothing.” (See also pdf, p. 44.) There are techniques to reduce automotive software risks, but NHTSA doesn’t require them.
And the agency is not currently regulating driverless cars at all.
That leaves the field to the less technically-experienced, more resource-poor state DMVs. It’s not hard to play one state off against another, and that looks like what Google did with California. Although a Stanford report says that self-driving cars were “probably legal” already, the company presumably wanted more certainty, so it got Nevada to pass a driverless car law to its liking in 2011, then Florida, and then used those statutes to incite California legislators to act. Sacramento, fearful that Google’s car operation would decamp to the home of legalized gambling, passed legislation requiring the DMV to issue regulations by January 1, 2015, legalizing the cars on California roads. Gov. Jerry Brown signed the bill at Google headquarters in September 2012.
But a section of the law, CVC 38750(d)(2), requires that the regulations include “any testing, equipment, and performance standards ... that the department concludes are necessary to ensure the safe operation of autonomous vehicles on public roads, with or without the presence of a driver inside the vehicle.”
The only way to even have approximate confidence in the safety of a complex embedded system is for a knowledgeable independent third-party certification authority to review (under confidentiality agreement) the software and firmware source code, custom chip (ASIC) designs, hardware specs, design documents, flowcharts, and other technical documentation.
The Problem with Road Testing
If you’re not a programmer, you may still be wondering why road testing is inadequate for driverless cars. Once again, the FDA explains, “One of the most significant features of software is branching, i.e., the ability to execute alternative series of commands, based on differing inputs. This feature is a major contributing factor for another characteristic of software – its complexity. Even short programs can be very complex and difficult to fully understand.”
It is hard enough to test garden variety software (say, Microsoft Word), whose operation is deterministic and does not require sophisticated judgments, involve real-time sensor inputs or perform safety-critical functions. A word processing program is comparatively straightforward, yet MS Word, for instance, still manages to be buggy.
A robocar’s software is non-deterministic, in that (a) it makes high level decisions (brake? accelerate? swerve? turn? etc.) in real time and (b) it does so in response to huge quantities of extremely complex sensory inputs representing dozens or hundreds of stationary and moving objects that are never the same twice. No matter how precisely you try to replicate a road test, something different will always happen:
● One time the sun will glint off the windshield of an opposing car and blind a sensor, perhaps causing the car the mis-steer – or maybe the sun will be at a slightly different angle that day and the problem will remain latent, ready to affect an unwary consumer.
● Maybe a splash of mud mixed with oil and gravel will fly up past the radar in the front bumper, leading the car to swerve away from an imagined obstacle – or maybe the road will be dry that day, or the mix of oil, mud and gravel just different enough that the software properly processes and ignores it.
● Maybe a journey will be long enough and the number of objects encountered large enough that a heavily-used area of memory referred to as the “stack” will overflow, with potentially catastrophic consequences. Stack overflows can be hard to debug or predict. But perhaps the road test will be too short to trigger the bug, again leaving latent a problem that will only show up when a car crashes in real-world operation.
● Perhaps one day the car will pass by a strong electrical field – say, from a power plant or even a downed wire – that evokes an electric current that modifies memory or triggers a false sensor reading, again leading to a dangerous malfunction not detected during a DMV road test.
● How vulnerable are the car’s sensors to snow, ice and mud? A fair-weather road test won’t answer that question – and we already know from the crash of Air France Flight 447 off the coast of Brazil just how deadly the consequences of sensor malfunction can be.
● Maybe one day the car will hit exactly 32 mph – a binary number that looks like this, 100000 – at precisely the same time a car in the next lane is decelerating and hits 16 mph (10000) and that confluence of zeros and velocity changes triggers some quirky bug that only appears if GPS signal is momentarily lost and a train crosses in front of the car. Of course this is a contrived example (here’s one from the real world), but the point is that software can fail for strange reasons in weird and sometimes catastrophic ways.
As the DMV assistant general counsel conducting the driverless car sessions, Brian Soublet, pointed out, there are a large number of traffic maneuvers – in his words probably too many to test (video). Even worse, there are an unbounded number of driving scenarios – combinations of maneuvers, pedestrians, cars, trucks, motorcycles, cyclists, road geometry, stop signs, traffic lights, weather conditions and more. You can’t road test all of them or even any large fraction because the test space is effectively infinite. The only way to probe them is to examine and audit the source code, and also to run the software through simulators – under the watchful eyes of DMV or third-party experts.
As a teenager, I managed to halt a “crash-proof” computer by directing it to take input from the display screen and send output to the keyboard, both of which were obviously impossible. It was a simple prank that the programmers somehow hadn’t anticipated. What pranks and scenarios will driverless car companies overlook as they turn their machines loose?
Cybersecurity and More
Yesterday’s pranks are today’s cybersecurity threats, of course, and these cars will be vulnerable. Even today, cars are at risk. Yet, John Tillman of Mercedes, Chrysler’s Good and VW’s Wendling all urged the DMV not to regulate driverless car cybersecurity (video). They didn’t feel that DMV was the right agency for the task, but had no alternative proposals.
There will be a lot of pressure for driverless car software upgrades to be transmitted wirelessly over the air (OTA), but this should only be allowed with extraordinary security. OTA mechanisms represent enormous threat vulnerability. Moreover, even under the best of circumstances, software updates are problematic, especially when the initial code is complex. It’s easy to “break something” when you’re in the process of fixing something or adding capabilities. The FDA found that 79 percent of software-related recalls in medical devices were due to defects introduced in updates, not in the original code.
There are also legal issues, because adhering to the Vehicle Code isn’t always straightforward. Soublet highlighted a left turn issue which Google acknowledged ruefully (“I can’t believe you’re bringing that up,” said Medford; video). Soublet also mentioned that obeying the speed limit on a freeway isn’t always the right thing, since it can lead to vehicles traveling too slow for traffic conditions. Here again, the decisions should be made by the DMV, not Google – and certainly not Google alone.
(Other legal issues – who’s liable for accidents, who gets the ticket and the points when a self-driving car breaks a traffic law, how will insurance work – were discussed at the DMV hearings but don’t relate to the point of this article.)
Adding to the seriousness of all of these issues – safety, ethics, privacy, cybersecurity, legal – is software monoculture, or the presence of the same software in many vehicles. This means that failures will affect numerous people, just like a bug in MS Word affects millions of people.
Can we trust Google or car companies to protect our privacy, ensure our safety, guard against bugs, prevent cars from being weaponized into self-driving human-free suicide bombs, make ethical decisions when crashes are unavoidable, and all the rest? No, not without rigorous oversight and meaningful independent certification.
Finally, we also have to put these cars in context. They're the first safer-yet-dangerous autonomous robots the public will encounter. There will be more -- eventually Honda's ASIMO robot will be released from the lab, ATLAS will come untethered and BigDog and other Boston Dynamics (i.e., Google) robots will be set loose, whether as helpers, packbots, monitors, security bots or more. Will we get the same answer then -- "this technology is a net good, regulators are self-interested, auditors wouldn't catch the bugs, so let the company self-certify and be done with it"? Google plays a long game and a large one. They're aware that they may be setting a precedent for the way robots are regulated, or not.
The California DMV regulations – which may set a national and even international template (video) – will be out soon. Hopefully, they’ll be adequate to the task. Click here to tell the DMV what you think (and cc me).
Jonathan Handel (jhandel.com) is an entertainment/technology attorney and journalist in Los Angeles. He does not have clients related to self-driving cars. The opinions expressed here are his own. Images via Wikipedia and do not imply endorsement or affiliation.