January 11, 2007

Perhaps They Should Have Tested More - NASA

I may be jumping the gun on this, but it appears a software update for the Mars Global Surveyor was buggy.

I'll try to watch this story and see if NASA comes to any different conclusions.

Software Glitch May Have Doomed Probe

Irene Klotz, Discovery News

Jan. 11, 2007 — NASA is investigating whether human error caused the loss of Mars Global Surveyor, a 10-year-old probe credited with finding signs of liquid water on the planet's surface.

The spacecraft fell silent in November after a problem with one of its solar array panels. Preliminary findings indicate software that was uploaded to Global Surveyor earlier in the year may have contained an error that set off a chain of events leading to the probe's demise.

Global Surveyor had already lasted four times longer than its design lifetime, but was about to embark on what might have been its most critical observations yet.

Scientists had hoped to fly the spacecraft over parts of Mars previously imaged to look for topographical changes. It was not just wishful thinking: Scientists had already found two locations showing signs of very recent water flows on the planet's surface.

It will be up to NASA's new spacecraft, the Mars Reconnaissance Orbiter, to pick up the surveys, as engineers have virtually no hope of recovering Global Surveyor.

"We still listen for it, but that's really the extent of it," NASA's Mars Exploration Program director Doug McCuistion told Discovery News.

The agency has set up a investigation board to try to figure out what happened to Global Surveyor. McCuistion said the software glitch is among the possible scenarios.

"Something may have occurred that we may never know," he added.

The board also will be looking into how the faulty software slipped through NASA's safety and quality control reviews. The program was intended to improve Global Surveyor's computer, but it had an error that may have caused the probe's solar array panel to jam.

The spacecraft then went into an automated emergency standby mode awaiting instructions, but engineers believe the battery, which was facing the sun, overheated and failed.

NASA used MRO as well as one of its rovers on the planet's surface to try to locate Global Surveyor, but the spacecraft could not be found.

Even with Global Surveyor's demise, Mars is still well-populated with robotic probes: two rovers have been scouring opposite sides of Mars' equator for signs of past water and three orbiters, including Europe's Mars Express, are circling the planet searching for water-bearing minerals and water-carved geologic formations.

NASA's next Mars mission is a lander named Phoenix that will touch down on the planet's north pole to study if the icy terrain could have — or possibly still does - support life. The Phoenix launch is scheduled for August.


April 14,2007 - an update...

It sure seems as if the bug fix made things worse.

"A software 'fix' sent up to NASA's Mars Global Surveyor spacecraft in June 2006 ultimately led to its failure five months later, a new preliminary report unveiled by the space agency has revealed.

The report has also said that the existing procedures used by the mission team were not thorough enough to catch the resulting problems, and that reductions in budgets and staff might have played some role in the loss of the mission."

"Incidentally, the incorrect June 2006 software upload was originally meant to fix a problem that arose in September 2005.

That is when two different engineers updated "in slightly different ways", two redundant control systems for the pointing of the high-gain antenna in 'error' mode, which led to a discrepancy in the computer's two memories, which the June 2006 upload was meant to rectify."


"Mission operators were not required to double-check the information they sent or test how it would affect the spacecraft, Perkins says. "Obviously, had these procedures been more rigorous … then perhaps this wouldn't have happened." "


"The review also said the spacecraft's onboard fault protection system failed to respond to the errors. Instead of protecting the spacecraft, the programmed response made it worse."


"We are making an end-to-end review of all our missions to be sure that we apply the lessons learned from Mars Global Surveyor to all our onging missions,” said Fuk Li, Mars Exploration Program manager at NASA’s Jet Propulsion Laboratory. "These lessons include the need for more thorough reviews of all non-routine changes to stored data before uploading as well as the need to evaluate contingency modes for overheating risks."