Toyota plays (hard)ball

March 9, 2010

Today, Toyota had a webcast and press conference to ‘debunk’ Dr. David Gilbert’s demonstration of an Avalon undergoing SUA.   In specific, Toyota pointed out that  (a) Dr. Gilbert had manipulated the vehicle’s electronic system and that (b) the errors Dr. Gilbert injected could not happen in practice.  The whole episode received a lot of attention in the media.  Toyota even had a Stanford University professor speak in support of Toyota.  Exponent filed a report pointing out that they could reproduce the same problem in vehicles from other carmakers.

Let’s look at what Dr. Gilbert actually said in his report and his congressional testimony.  (a) He specifically said that he had manipulated the vehicle’s acceleration pedal position sensor subsystem.  He even gave a detailed diagram showing the interior of the breakout box he used to conduct the experiment.  He further enumerated the list of capabilities the breakout box allowed him to do.   (b) Dr. Gilbert pointed out how moisture, corrosion and wear could lead to the  fault scenario.  He however did not say that this WILL happen or how likely it was for it to happen.  Many including me pointed out the relative impracticality of this scenario.   So, there was absolutely zero news on what Toyota said today.   It seemed like Toyota wanted to do a PR blitz and that was that.

Dr. Gilbert during his demo mentioned that he was unable to repeat the problem in an American car (GM I think but could have been Ford).  Exponent had been able to repeat Dr. Gilbert’s experiments on a Subaru, Honda, BMW, Mercedes and a Chrysler.  In other words, they did not find any problem with Dr. Gilbert’s conclusion.   It’s all about perception management, one is forced to conclude.

Toyota Playing Its Version of Russian Roulette

March 8, 2010

The number of SUA (sudden unintended acceleration) complaints in Toyota vehicles that have been ‘fixed’ keeps rising on a daily basis.   The House Energy and Commerce Committee has asked Toyota for its documents showcasing their internal tests of throttle control electronics and for access to Toyota engineers who worked on it.   Given all this, what is Toyota doing?  They have decided to aggressively launch a marketing campaign to sell its cars to Toyota loyalists.   What is going on?

Here’s my interpretation. ..  (All numbers are within a factor of about 2. )  There are 1000-2000 complaints of SUA  over the past 9 years in the US with about 20 million Toyota vehicles.  This translates to a failure rate of approximately 1 in 10,000.   This rate could go up as more complaints are filed in part due to an increased awareness of SUA.   (There will also likely be an increase in the reported tally due to human errors being reported as vehicular problems.)    Overall, however, there appear to have been a few tens of deaths and a few hundreds of injuries that could be attributed to SUA.   Taking a legal perspective, even if Toyota loses all these lawsuits, the total liability is perhaps less than $500 million in the US.    Toyota may have just made a business decision that it can always settle injury/death cases in court, and be aggressive in the market as always.    Traditional business model, you say?   If electronics is indeed shown to be a problem later, Toyota will immediately lose face, alienate its heretofore loyal customer base and be liable for potentially billions of dollars.    A bunch of executives within Toyota must be hoping that they have calculated these odds correctly.   The odds they may have calculated could indeed be the 1 in 10,000 failure rate.  But one may  be able to show a trouble-making sequence in the electronics of a vehicle that did experience SUA before.   The odds of somebody showing such a sequence, I believe, are (much) better than 1 in 10,000.  The House Committee can also  follow through and find additional information.

Let me re-emphasize: Toyota can demonstrate to a lot of customers the correctness of its system by installing a “black-box” (event data recorder) in vehicles that have exhibited SUA (and even more so on vehicles which have exhibited multiple SUA incidents).  If/when the driver claims SUA, the data in the black box would show where the fault lies – with the driver or with the carmaker.  We have not seen this testing happen – and its marked absence only reinforces the skepticism of many people like me.

A Clash of Cultures and Its Consequences

March 7, 2010

Today’s New York Times describes how Toyota has been able to not make any recalls  in its home-base, Japan.  In both Japan and the United States, big corporations unfortunately have a major say in policy-making such as the (non-)establishment of regulations and how they are (not) enforced.  Think Toyota in Japan and the banks in the US.   Old-timers in the US may recall how the Big Three of the past became over time the Detroit Three, by advocating reduced investments in public transportation infrastructure back in their golden age during the 50s and 60s, and then adopting ill-advised market-growth techniques that came to be collectively referred to as “planned obsolescence”.

The lack of accountability to customers is something that may define how this whole episode of concerns over recalls plays out.  We need to watch how extensively Toyota responds (or not) to the request from the House Energy and House Committee for test documents and interviews with technical experts.   If Toyota is able to bring to bear political pressure not to bring to light its internal documents, that would be a pity.  The harsh spotlight shining on Toyota, which some attribute to “typical” hype from the press, has nevertheless put immense pressure not just on Toyota but also on other carmakers, who have rushed to announce recalls that may not have happened otherwise.  A good outcome, hopefully, will comprise of

  • An added public awareness of both the many benefits and risks of electronics in safety-critical systems,
  • An increased concern over consumer safety among the carmakers, who will otherwise pay for it with reduced market share and diminished reputation, and
  • More industry standards and products that favor the use of independent fail-over mechanisms.

Cars for the people!

Toyota at the Crossroads

March 5, 2010

The context: More drivers are beginning to complain of sudden unintended acceleration after the recent fixes in response to massive recalls by Toyota.   Some of these complaints may not be legitimate but others likely are.   As the Toyota recall story was building to a crescendo leading up to the congressional hearings, many owners of Toyota vehicles have taken the following position in recent weeks.   “I have owned one or more Toyota vehicles for a long time. They have been very reliable.   I can understand that something can go wrong once in a while, but Toyota will find the problem and will fix it.   Meanwhile, if I can find a Toyota at a good price, that’s not a bad deal, is it?”   Very reasonable.    This argument also explains the less-than-expected drop in Toyota sales in February 2010 (about 9.5% when some analysts expected a bigger drop).

Now, if Toyota begins to be perceived as being unable to find and fix a problem that is safety-critical (who wants to encounter SUA with your family in the car?), the tables can turn very rapidly.  Hondas, Nissans and Hyundais will sell well, while many others will look at vehicles made by the Detroit Three.  And many are likely to be pleasantly surprised that the quality level, designs and prices of those other cars are not so bad after all.     Credibility can be lost very quickly and rebuilding reputation can take a long time.

It’s precarious times for Toyota from that perspective.   With about 20 complaints of cars with “fixes” reporting SUA, Toyota has said “This represents only a ‘tiny fraction’ of the cars” reporting the problem.    I am no PR expert, but this does not sound like a good defense in the public sphere of consumers.   (These numbers are just from the last 2-4 weeks).   Loyal Toyota owners are watching.

On a related note, the US House Energy and Commerce Committee is now asking Toyota to

  1. Submit internal documentation and detailed reports from the various tests that Toyota conducted on their ETCS electronics.
  2. Provide names of Toyota engineers and technical experts who worked on and tested the ETCS – so that they can be interviewed.

Both are steps that I have advocated (read this blog posting for example).

Black Boxes on the move

March 4, 2010

Yesterday, Toyota seems to have given 3 “black boxes” to NHTSA to see if SUA is happening in its vehicles.    This represents some movement on this issue, as I have argued for some weeks.  The black box records vehicle data on the fly and can be used to see what was happening in the vehicle before an event of interest (like SUA) occurred.   3 is more than 1, and certainly larger than 0, which was the status quo earlier.   So, this is progress.   Now, calculate the costs of the recall even ignoring the impact of perception on Toyota sales and image.   Could they do only 3 such boxes?  Hopefully, there are more on the way.

If I got such a black box, I would calibrate it.  For example, run the vehicle at high speed, low speed, apply the brakes, change the gear, rev the engine etc. and see what the data say.   Apply the brakes and press the gas pedal together in a safe location.  Then, I would even try doing an experiment like SIU Prof. Gilbert’s – even if somewhat forced, the black box logger should record the sequence that results.  In other words, the black box should not judge the logic and just report the data.    Then, off one goes looking for SUA and the results.   The data ought to be very revealing.

Accelerations ‘r Us

March 4, 2010

We do not believe sudden unintended acceleration because of a defect in our E.T.C.S. has ever happened,” said Takeshi Uchiyamada, an executive vice president for Toyota.    May be, just may be, Toyota engineers have never been able to confirm an error in the electronics that led to SUA.   But given everything that has happened, with now people complaining of SUA after the “fixes” resulting from the recent major recalls, is this what Toyota should be saying?  So, are all the Toyota drivers reporting SUA at fault?

Should they not be saying something along the following lines?

From the time these SUA complaints started several years back, we have investigated the incidents diligently and have been trying to reproduce the problem.  We keep looking.  We just cannot find an electronics problem.   We have built new tools, new hardware and we keep looking for more and more information from the system but we still cannot see our electronics causing SUA.  We have developed lots of new tools.   In fact, here on the table in front of you is some new hardware we have built – anybody with recurring problems, please install such a unit in your Toyota vehicle.  It can record/observe what is going on.    All of you, not just us, can observe what happens then.  Each unit will record accelerator pedal position, throttle position, brake position, vehicle speed, vehicle acceleration, cruise control usage, engine rpm, and more.  The last  15-30 minutes of driving data will always be recorded when the engine is on.  Our customers can look at the data directly – the software to do this can be downloaded right from the homepage on our website.  If SUA happens, stop, do not restart the engine, have the car towed and look at the data yourself.   If SUA happens due to electronics, the data will show it.  If SUA is not due to electronics, the data will also show it.  Please bear in mind that this box is not fully crash-proof and can get damaged in the case of a big collision.  But we still expect that in most if not all cases, if the driver switches to neutral avoiding a collision, the data will be accessible to you and be very informative.   If you report the incident to police, you can ask them to look at the data.  You can have your Toyota dealer show you the information.    Let us know after you have seen the data. Call NHTSA and us. We are right now looking for volunteers. We will be happy to work with NHTSA and others to screen these volunteers, install the units and have you collect more data for detailed observation.  The first preference will be given to those who have documented SUA incidents before.  The more observations you can make, the more everybody will know.    We have 1000 units available right now in the US.  It’s a minor expense given the magnitude of our recalls.  If more units are needed, we’ll make more real soon.   And then, as more drivers see that our electronics is not at fault, all our customers can feel safe knowing that the electronics in our vehicles is not malfunctioning.   Toyota proudly stands by its customers.  We always will“.

Why is Toyota not saying the above?   Frustrating.

Edmunds and the Crowd-Sourcing Competition

March 3, 2010

Edmunds is planning to announce a competition with a prize of $1 million to detect and propose a solution to the sudden and sustained unintended acceleration of Toyota cars.   Crowd-sourcing has worked in many other contexts, so why not here?   (Those of us who posted possible theories about the cause of the acceleration problem may feel like chumps right about now 😉

It is important to keep in mind some aspects we know of the situation :

  1. The acceleration problem does not seem to happen in every car and even in cars where it does happen, it does not seem to happen always.  These facts imply that the problem is not fully deterministic.   Look for intermittent failures.   (There may be permanent failures which manifest themselves as a component/subsystem that needs replacement).
  2. If Edmunds  is looking for a solution to the sudden unintended acceleration problem  “once and for all”, it needs to be pointed out that such a silver bullet is highly unlikely to exist.   If there are a thousand models with unique electronic throttle control system designs out there, there can be (many) more than thousand unique ways for these systems to fail.
  • Engine control modules are, simply put, complex.  They take care of engine control including fuel injection, sparking, cruise control, throttle control, ignition control, traction control, etc. etc. with an array of fail-safe mechanisms.   The design (and implementation) space is correspondingly complex and very large.  These are multi-dimensional systems:- cost, reliability, functionality, maintainability, safety and initial cost of investment are but a few of the dimensions that are traded against each other.   There can be core technological limitations in the electronics but how one integrates a fail-safe design around such limitations can be as varied as the human imagination.  Things could go wrong on the mechanical, electrical, electronics, software and human interaction fronts and/or interactions among these.   So, there is no silver bullet.  These are systems designed by a multi-disciplinary cadre of engineers with different experiences and foundations.   Yes, they can also make mistakes and/or incorrect assumptions once in a while.
  • Blaming SUA on a driver “always” would be meaningless when any system has a non-zero rate of failure and must be based on assumptions about the environment in which it is working.
  • Blaming the SUA problem on the electronics “always” would be equally meaningless since humans, ahem, will always be humans and occasionally err.
  • All entrants into this competition should be aware that these are safety-critical (i.e. life-critical) systems.  Don’t tinker with the system while it is running unless you really know what you are doing.  At the end of the day, trust me, your life is worth more than a million dollars.  So, please do exercise caution.

All that having been said, it is within the realm of possibility that some out-of-the-box idea can reproduce the problem repeatably.    Finding it could be very easy (sometimes the right imagination and creativity goes a long way), or could be very complex (just watch Toyota asserting in many different ways through many different people including the CEO and grandson of the founder of Toyota that nobody has been able to show that there is a problem in the electronics).

May the best idea win.   Just be safe.

Toyota Recall Fixes Not Helping?

March 3, 2010

BusinessWeek reports that some Toyota vehicles, which were fixed as part of the recent massive recalls, have continued to exhibit sudden unintended acceleration.   Unfortunately, not very surprising, is it?    *IF* Toyota knows more than what it has admitted publicly, this may be a good time to bite the bullet and address the situation.   (The first principle of modern PR is to get all the bad news out as quickly as possible).   Else,  if Toyota continues to be at a loss as to what is happening, we and they could be in for a long and harrowing ride (no pun intended).

Update: The storyline continues here (sadly).

Mandatory brake override? Sure, but which one?

March 2, 2010

The Department of Transportation is actively considering whether brake overrides should become mandatory on all vehicles.   If a driver is pressing on both the gas pedal and the brakes, a brake override will give priority to the brakes and just ignore the gas pedal positions.   Given the history of sudden intended acceleration incidents, this mandate may seem like a slam dunk.  If the vehicle accelerates without driver input, just hit the brakes and all is well.  If only safety-critical systems were that simple…

The question is not whether there should be a brake override mechanism, but what kind of brake override should be in place.   A fully electronic override may not even be recognized if implemented in the wrong subsystem within an automobile.  A mechanical override on the other hand will have independent failure characteristics.   I have made this argument in a previous blog posting here.

Consider Toyota’s ETCS-I, which has been in the news a lot lately.   There are those of us who believe that there is a problem in the electronics that can cause sudden and sustained unintended acceleration (SSUA).  If when the brake is pressed, this ETC system is supposed to receive a message from the brake sensor(s), override the gas pedal position completely and close the throttle.   But suppose the problem is indeed in the ETC system which is unable to close the throttle resulting in SSUA.  (We anticipate that this could just may be end up being the source of the SSUA problem.   The electronics that controls the throttle, for example, could have been damaged intermittently or permanently).   Under this condition, the brake override will no nothing.  Nada. Zilch.

In other words, a  fail-safe mechanism that is completely independent of the ETC system is needed to close the throttle.

Brake override?  Yes.  But the right one.

Addendum:  To provide a more balanced perspective, suppose the problem in the ECTS-I is the one that Prof. Dave Gilbert (SIU) points out  – where if the two acceleration sensors are shorted to the 5V supply, the vehicle takes off and there is no DTC.   In this case, an electronic brake override will indeed work fine – the accelerator values will be ignored.  So, an electronic brake override by the ECUs in the ETCS-I is better than having nothing.

If something went wrong in the electronics control, …

March 2, 2010

… and no Diagnostic Trouble Code (DTC) were registered, did an error really occur?

Descriptions of the Toyota ETCS-I indicate that in some cases, a DTC is registered only if a sensor (like the Acceleration Position Pedal Sensor which has a redundant copy) AND an ECU (there are two redundant copies of these Electronic Control Units) fail.  If a sensor fails, but an ECU does not fail, a DTC is not registered.   The failure condition of the sensor ought to provide substantial information but the condition is not recorded resulting in substantial loss of valuable information.   If the DTCs are meant for repair and maintenance purposes, a secondary log that contains intermediate error states would be very useful as debugging aides.  This secondary log should also augment a more detailed event data recorder (“black box”).