Archive for the ‘Safety and By-Wire Systems’ Category

Should we go back to mechanical systems to ensure the safety of cars?

March 10, 2010

A reporter asked me yesterday whether there are any ‘independent’ organizations or groups that test the safety of car electronics.   An independent entity would be one that does not work with any carmaker, automotive supplier or plaintiffs in a car accident.   Unfortunately, I had to answer ‘No’.   The reason for this absence of independent entities who can offer “unbiased” feedback is simple: how will they support themselves?   Automotive electronics is complex; one needs the services of experts in mechanical engineering, electrical engineering, control systems, electronics hardware, embedded real-time software, fault-tolerant systems, sensors, actuators,  EMI and ESD.  There are hundreds of models sold *every* year.   The cost of sustaining such a testing operation will be enormous, and unless one has a service contract with one of the automakers, or looking at specific issues for a plaintiff, it is very difficult to sustain the operation.  Let’s look at the landscape and how we can help the situation.



Edmunds and the Crowd-Sourcing Competition

March 3, 2010

Edmunds is planning to announce a competition with a prize of $1 million to detect and propose a solution to the sudden and sustained unintended acceleration of Toyota cars.   Crowd-sourcing has worked in many other contexts, so why not here?   (Those of us who posted possible theories about the cause of the acceleration problem may feel like chumps right about now 😉

It is important to keep in mind some aspects we know of the situation :

  1. The acceleration problem does not seem to happen in every car and even in cars where it does happen, it does not seem to happen always.  These facts imply that the problem is not fully deterministic.   Look for intermittent failures.   (There may be permanent failures which manifest themselves as a component/subsystem that needs replacement).
  2. If Edmunds  is looking for a solution to the sudden unintended acceleration problem  “once and for all”, it needs to be pointed out that such a silver bullet is highly unlikely to exist.   If there are a thousand models with unique electronic throttle control system designs out there, there can be (many) more than thousand unique ways for these systems to fail.
  • Engine control modules are, simply put, complex.  They take care of engine control including fuel injection, sparking, cruise control, throttle control, ignition control, traction control, etc. etc. with an array of fail-safe mechanisms.   The design (and implementation) space is correspondingly complex and very large.  These are multi-dimensional systems:- cost, reliability, functionality, maintainability, safety and initial cost of investment are but a few of the dimensions that are traded against each other.   There can be core technological limitations in the electronics but how one integrates a fail-safe design around such limitations can be as varied as the human imagination.  Things could go wrong on the mechanical, electrical, electronics, software and human interaction fronts and/or interactions among these.   So, there is no silver bullet.  These are systems designed by a multi-disciplinary cadre of engineers with different experiences and foundations.   Yes, they can also make mistakes and/or incorrect assumptions once in a while.
  • Blaming SUA on a driver “always” would be meaningless when any system has a non-zero rate of failure and must be based on assumptions about the environment in which it is working.
  • Blaming the SUA problem on the electronics “always” would be equally meaningless since humans, ahem, will always be humans and occasionally err.
  • All entrants into this competition should be aware that these are safety-critical (i.e. life-critical) systems.  Don’t tinker with the system while it is running unless you really know what you are doing.  At the end of the day, trust me, your life is worth more than a million dollars.  So, please do exercise caution.

All that having been said, it is within the realm of possibility that some out-of-the-box idea can reproduce the problem repeatably.    Finding it could be very easy (sometimes the right imagination and creativity goes a long way), or could be very complex (just watch Toyota asserting in many different ways through many different people including the CEO and grandson of the founder of Toyota that nobody has been able to show that there is a problem in the electronics).

May the best idea win.   Just be safe.

Mandatory brake override? Sure, but which one?

March 2, 2010

The Department of Transportation is actively considering whether brake overrides should become mandatory on all vehicles.   If a driver is pressing on both the gas pedal and the brakes, a brake override will give priority to the brakes and just ignore the gas pedal positions.   Given the history of sudden intended acceleration incidents, this mandate may seem like a slam dunk.  If the vehicle accelerates without driver input, just hit the brakes and all is well.  If only safety-critical systems were that simple…

The question is not whether there should be a brake override mechanism, but what kind of brake override should be in place.   A fully electronic override may not even be recognized if implemented in the wrong subsystem within an automobile.  A mechanical override on the other hand will have independent failure characteristics.   I have made this argument in a previous blog posting here.

Consider Toyota’s ETCS-I, which has been in the news a lot lately.   There are those of us who believe that there is a problem in the electronics that can cause sudden and sustained unintended acceleration (SSUA).  If when the brake is pressed, this ETC system is supposed to receive a message from the brake sensor(s), override the gas pedal position completely and close the throttle.   But suppose the problem is indeed in the ETC system which is unable to close the throttle resulting in SSUA.  (We anticipate that this could just may be end up being the source of the SSUA problem.   The electronics that controls the throttle, for example, could have been damaged intermittently or permanently).   Under this condition, the brake override will no nothing.  Nada. Zilch.

In other words, a  fail-safe mechanism that is completely independent of the ETC system is needed to close the throttle.

Brake override?  Yes.  But the right one.

Addendum:  To provide a more balanced perspective, suppose the problem in the ECTS-I is the one that Prof. Dave Gilbert (SIU) points out  – where if the two acceleration sensors are shorted to the 5V supply, the vehicle takes off and there is no DTC.   In this case, an electronic brake override will indeed work fine – the accelerator values will be ignored.  So, an electronic brake override by the ECUs in the ETCS-I is better than having nothing.

If something went wrong in the electronics control, …

March 2, 2010

… and no Diagnostic Trouble Code (DTC) were registered, did an error really occur?

Descriptions of the Toyota ETCS-I indicate that in some cases, a DTC is registered only if a sensor (like the Acceleration Position Pedal Sensor which has a redundant copy) AND an ECU (there are two redundant copies of these Electronic Control Units) fail.  If a sensor fails, but an ECU does not fail, a DTC is not registered.   The failure condition of the sensor ought to provide substantial information but the condition is not recorded resulting in substantial loss of valuable information.   If the DTCs are meant for repair and maintenance purposes, a secondary log that contains intermediate error states would be very useful as debugging aides.  This secondary log should also augment a more detailed event data recorder (“black box”).

The Toyota webinar I would like to see

February 28, 2010

On its website,  Toyota has a webinar describing highlights of its electronic throttle control system.  (For my comments on that webinar, please see a previous blog posting).   In that webinar, Dr. Paul Williamsen of the University of Toyota and Kristen Tabar, General Manager of Electronics Engineering at Toyota Technical Center in Michigan present key features of the ETCS-I and how it is tested for electro-magnetic interference (EMI).

A Toyota webinar I would like to see will comprise of the following:

  1. A similar presentation made by Toyota engineers who were deeply involved with the design of the ETCS-I and with the testing on vehicles that exhibited udden unintended acceleration.   Real-time translation can be used if core designers are not fluent in English – but there are certainly engineers at Toyota who are well-versed in both English and Japanese.   (No dis-respect to Dr. Williamsen or Ms. Tabar – their presentations were very clear and informative.  But they were clearly not involved in the design and long-term testing of the ETCS-I, and cannot speak to all aspects of the system).
  2. Media representatives may be present but the webinar is attended by reputed experts (a) from academia, government and the private sector, (b) with in-depth knowledge in mechanical engineering, embedded hardware, embedded software, sensors, actuators, automotive electricals, diagnostics, and (c) with backgrounds in other safety-critical real-time systems including automobiles, avionics, nuclear power plants, aerospace systems, and defense systems.
  3. There is an open-ended Q&A session with the experts in attendance after the formal presentations.

Can it happen? Sure.  Will it happen?

Toyota says ETCS-I is infallible – Convincing?

February 27, 2010

The YouTube Video Clip and More

Toyota has posted an informative video clip on YouTube that explains how the electronic throttle control system (ETCS-I) on their vehicles works. The fail-safe mechanisms that are built into the system are illustrated pretty well using animations and informative diagrams. If one accelerator pedal position sensor fails, the throttle is closed. If the throttle is blocked by a mechanical object, the engine is shut down. If there is a signal coming from outside the ECM (engine control module), the engine (supposedly) shuts off. If one accelerator or throttle sensor fails, the system detects it. It can switch to ‘limp-home’ mode or ‘idle’ mode depending upon the severity of the failure conditions.  In the worst-case scenarios, the engine is shut down completely.

This is a good video clip that is presented cleanly and produced well. (Where can I get one of those desktop ETCS-I simulators?)  Toyota is trying to educate and convince its customer base that it’s still safe to buy and operate Toyota vehicles. An extended version of this clip is on Toyota’s website in webinar form with Q&A from media representatives. In this clip, additional presenters also describe EMI (electro-magnetic interference) and ESD (electro-static discharge) testing.

The Flip-Side

So, what’s not to like? It’s the assumptions behind the design and the arguments.

  • Any safety-critical system has to make assumptions about what will fail, how it will fail and what the failure rate will be. For example, there are two ECUs (processors) in the ECM (Engine Control Module). The typical assumption one makes (and Toyota makes it as well) is that they will fail independently.  This is called the independent failure assumption. But can both fail together? This is called a common mode failure. Both processors can fail together, for example, if there is a flaw in the processor design or implementation (perhaps they got too hot, there is a hardware bug that manifests itself only so often, solders were not correct for a chip, …). Now, what?
    • For the technically knowledgeable, Toyota also does not answer the important question “Is there a single point of failure in the ETCS-I”? Also, Toyota does not present a complete fault-tree analysis. I would presume that such an analysis exists but it is not presented. Toyota also does not talk about permanent and intermittent failures. Most descriptions of SSUA (sudden and sustained unintended acceleration) seem to allude to intermittent failures.  These are failures that disappear when power is turned off and the system is reset.   This property of the problem is what makes it so hard.
  • The presenters emphasize that a DTC (Diagnostic Trouble Code) would have been stored if a failure has happened and a fail-safe mechanism kicked in. However, every error condition is not stored as a DTC. For example, if one of the application pedal position sensors (APPS) fails but both processors are working, a DTC will not be stored. Such conditions may be exactly when a sequence of events that eventually lead to SSUA gets triggered.
  • A DTC code that is stored is erased after a “few” cycles of engine start/stop. They do not quantify “few” – knowing the exact number(s) would help people who experienced SSUA whether their DTC was indeed stored but subsequently erased. (Remember – the failures we see are intermittent failures).
  • Toyota does not even bring up the topic of software (wisely). According to the Sustained Computing Consortium, there can be 20-40 bugs in every 1000 lines of code. Even if Toyota or its suppliers did a spectacular job with the ETCS-I, it would still be very hard if not impossible to prove that the code is correct. The questions on this front are two-fold:
    1. Are there bugs in the software? Why not?
    2. Have there been bug fixes in various releases of the ETCS-I firmware? What were the fixes? When were they made?
  • The response to the question “Where is the ‘black box’?” is very lame. Toyota released ETCS-I in 1998, deployed it widely across its popular models since 2001/2002 and has had it on all 100% of its vehicles for the past couple of years. In other words, its investment in the technology is huge. SSUA complaints have surged (by a factor of about 10x) since its introduction in 2001. Toyota claims that nobody has been able to show that electronics is a cause for these complaints. The implication is that drivers press the gas pedal when they mean to press the brake pedal. If that were indeed true, Toyota can instantly exonerate itself if a black box (event data recorder) shows this unambiguously. So, why no black box even now after 10 years?   Why not install one and make it directly accessible to consumers and law-enforcement officials, as GM and Ford have done? Why the delay given that it would be in Toyota’s interest to have it be there?
  • The discussion about EMC is not very convincing either. They should really talk about how/whether the ETCS-I fail-safe mechanisms work under the EMI and ESD scenarios tested. From what is said, it feels like only normal working conditions are tested but not failure conditions.  For example, have one sensor fail (note that no DTC will be stored) and then let EMI/ESD crop up.

In summary, these are good technical presentations by Toyota and a good PR step to boot.  But ultimately they are unconvincing.

What Should Toyota/NHTSA Do?

February 24, 2010

We know the following.  Toyota itself has acknowledged that it cannot rule out problems in its electronics throttle control system as a cause behind the sudden acceleration of some of its cars.  NHTSA is in the spotlight to ensure public safety.   What should they do?

The problem in the Toyota electronics could be due to

  • hardware design issues (assumptions made regarding failures that are violated),
  • hardware implementation issues (such as sensors, actuators or computers with manufacturing defects that result in higher than normally very low failure rates),
  • electrical issues including ESD (electro-static discharge), grounding and EMI (electro-magnetic interference) problems,
  • software bugs that kick in only under abnormal conditions,
  • architectural integration with each subsystem doing the “correct” thing but the completed system having an integration issue, or
  • An insidious combination of the above.

What should Toyota/NHTSA do now?

  • Each should assemble an independent team of experts – mechanical engineers, electrical engineers, embedded hardware engineers, embedded software engineers, control engineers, sensor/actuator experts, reliability engineers and safety engineers.
  • Get all documentation and internal communications available at Toyota.
    • In particular, get access to Toyota’s internal investigations and results from vehicles that have had known acceleration problems – these are critical materials to study and will narrow down the search space dramatically.
  • Evaluate the fail-safe mechanisms in ETCS-I and their effectiveness against a wide spectrum of possible failure points.   What if this sensor fails?  What if the throttle control mechanism fails? What if this board fails? What if the ECU (electronic control unit) fails? What if the output there get stuck?  What if an input here fluctuates all the time?  What if there is a voltage spike at an input? At an output?  What if any these two failures happen simultaneously? What does the software do if an unexpected event happens?   These questions can be asked (and hopefully answered) very methodically and rigorously.
  • Work backwards from the sudden acceleration event.  That is, suppose that sudden acceleration has occurred.  What events and what sequences can lead to that outcome?  What could have gone wrong (however improbable) for that outcome to be produced?   To paraphrase Sherlock Holmes, if you have eliminated all possibilities but one, then you have found the culprit, however improbable it may seem.
  • Study all architectural and design documents, then pore into the code, the wiring diagrams, and the internals of all the sensors, actuators and ECUs.

If you think that a businessperson/manager should also be on the team, think again.   It would serve Toyota much better over the long term to find the problem in its electronics (there is one!) and fix it.  Else, this cloud would hang over the company, its customers and its shareholders causing more damage.   Bite the bullet, figure out the source, fix it and move on.   The short-term pain will be much less than the long-term impact on the company (and the Japanese economy perhaps).

Addendum: Any data that Toyota has in their “black boxes” should also be studied.  As I have argued in a previous blog, it could help exonerate them too.   Any studies they may have of why drivers may press the wrong pedal (more so than in other cars) should also be looked at.

Toyota: Recalls May ‘Not Totally’ Resolve Issues

February 24, 2010

Many (including me) have pointed out that the floor mat and ‘sticky pedals’ issues do not fully address the nature of many of the complaints in the NHTSA ODI database.   Specifically,

  1. Many complainants have vouched that the accelerator pedals have not been stuck and in several cases, the floor mat was not present.
  2. There has been a significant spike in the number of SSUA (sudden and sustained unintended acceleration) incidents since Toyota introduced the ETCS-I across multiple models in 2001/2002.   Even the 2010 Toyota Camry, which has been barely on the market for a few months, has 18 complaints in the vehicle speed control category already.   The complaint count since 2002 is way disproportionate in relation to market share when compared complaints filed on other carmaker models.

Now, the US Toyota Chief has acknowledged that Toyota cannot rule out problems in electronics, and that the floor mat and sticky pedal fixes may not fully resolve the SUA issue.    Given earlier emphatic denials from Toyota that electronics was not a problem behind SUA, does the burden of proof of safety now shift unequivocally to Toyota?

Prof. Dave Gilbert’s Findings on SSUA

February 24, 2010

ABC News yesterday (Feb 22, 2010) had a video segment on findings made by Prof. Dave Gilbert of the University of Southern Illinois regarding SSUA (Sudden and Sustained Unintended Acceleration) on Toyota vehicles.

Two findings of Prof. Gilbert stand out:

  1. It is indeed possible to force a Toyota vehicle with ETCS-I into sudden and sustained unintended acceleration.    No fail-safe mechanism kicks in.  Recall that there is no brake override (Toyota has promised an update on recent-year models but not earlier-year models), and the driver has to switch to neutral to bring the vehicle to a safe stop.
  2. When the vehicle does go into SSUA, no corresponding diagnostic code is registered in the vehicle.  So, if this happens to a driver and they take it to the mechanic, there will be no trace of the event happening.  Then, it’s your word against the diagnostics!

Both these findings offer very useful insights for those studying whether electronics problems are behind SSUA.

Additional thoughts:

  1. Prof. Gilbert manages to put the vehicle into SSUA repeatably by shorting two pins that go into the Engine Control Module of ETCS-I (Electronic Throttle Control System with Intelligence).    Prof. Gilbert points out that moisture, corrosion and wear can cause such a short to happen.   SSUA has been known to happen even in 2010 Toyota models (for example, there are 18 complaints in the vehicle speed control category of the NHTSA database for the 2010 Toyota Camry).   It would seem unlikely that such new cars have already undergone dramatic wear and corrosion to cause a short.
  2. Toyota’s description of the ETCS-I mechanisms point out the following.  For example, a diagnostic error code is registered only when one of the acceleration pedal position sensors (or the throttle position sensors, both of which have redundant sensors) fails AND one of the redundant ECUs (computers) fails.  If only one fails, no diagnostic code is recorded.  So, it is known that only subsets of conditions are recorded as diagnostic error codes.

What Prof. Gilbert has found is that even a (dramatic) occurrence of SSUA goes unrecorded.  Toyota could perhaps argue that continued acceleration at high speed could be an intended action on the part of the driver, and therefore it should indeed go unrecorded.   If the driver is trying to override the acceleration though, the situation becomes dramatically different.  So, this matter goes back to the issue of ETCS-I in several Toyota vehicles not having a fail-safe mechanism.

Is ESD (Electro-Static Discharge) the Culprit behind Toyota ETCS-I?

February 23, 2010

Note: This post is a subset of a posting titled “Possible Electronics Causes for Sudden Unintended Acceleration“.

Electro-Static Discharge (ESD) Issues

Electrostatic discharge (ESD) is the name given to the sudden and short-lived electric current that flows between two objects at different electrical potentials (voltages) caused by direct contact or induced by an electrostatic field. These currents, while short-lived, are unwanted that may cause damage to electronic equipment.

The simplest ESD example that people see in practice is the very brief spark that happens when during winter you  touch a metallic object, get a ‘shock’ and see a spark. (Charge develops on your body as you walk across a carpet, for example, and it gets discharged when you touch a conducting material).   More than 1KV can be generated albeit for a very short time!  At home, one often uses voltage surge suppressors to connect sensitive electronics like TVs and computers to wallpower.  Without such surge suppressors, voltage spikes like lightning can enter your electronics and cause permanent damage. Lightning, therefore, is another classic example of ESD.

Due to the damage that ESD can cause to electronics, there are military, industrial, automotive and international standards to deal with the issue. Popular consumer electronics like camcorders, mobile phones, and digital cameras have built in voltage shunts that trap these spikes from reaching the core of the electronics and damaging them.

ESD damage to electronics, which worsens over time, can fall into different categories. One, the damage can be permanent and the device fails. This is often referred to as a hard fault. Two, the damage seems to reset itself and function correctly (for a while) when the device is shut down and restarted. This is often referred to as a soft fault. Some standards even define finer distinctions.   SSUA on Toyota vehicles seems to correlate to soft faults in many cases, but in cases where a vehicle was totally wrecked, the problem could have been a hard fault.