Thursday, February 5, 2004
Spirit gets back on its feet, while Opportunity begins to explore...
We're now up to sol 32 on Spirit. Wow! We've been living on Mars time for a month now. By this time on Pathfinder, the Sojourner
team had rebelled against Mars time, but we're still going strong.
6:07 am pst
The Spirit science team has been impatiently waiting to resume their science observations. I don't blame them. Science has
been forced to take a back seat to the engineering troubleshooting necessary to getting the Spirit rover fully operational
again. (I personally would rather be planning science sols rather than tracking the engineering sequences we're using to
investigate and correct the anomaly.)
The problems on Spirit have been traced to memory use in the rover's RAM as the number of distinct files stored in its "flash"
memory increases. When the RAM fills up, the rover stops working. By deleting a number of files, we were able to get the
rover to operate normally.
Today (sol 31) we told the rover to take it easy, shutting off by 1:30 in the afternoon and not waking up overnight to talk
to us. The idea was to get the batteries fully charged, and to let the rover electronics cool off as much as possible.
This is all in preparation for re-formatting our onboard "flash" memory tomorrow. Once we got the rover stable,
we were able to determine that some of the flash memory had been corrupted by the sol 18 anomaly. So, just like you might
do on your own computer, we're going to erase and reformat our memory to clean things out. Since this process will take many
hours and be monitored closely from the ground, we're going to get an early start on sol 32, waking the rover up at 6 am local
Mars time. By cooling off the rover on sol 31, we ensured that Spirit could stay powered up all day on sol 32 without overheating.
The reformat went without a hitch! Spirit is healthy and ready for the rest of its mission.
Let's get back to doing science!
Meanwhile, on the other side of the planet...
Opportunity also checked out as very healthy after its bouncy landing. The only nagging problem is a heater for one of the
joints of the rover's robotic arm, one of several heaters used to warm up the arm if we want to operate it in cold nighttime
temperatures. Unfortunately, this one heater is coming on overnight even when we don't want it to, and it's running down our
batteries a bit. This isn't really an issue early in the mission, but as dust piles up on the solar arrays and the seasons
change, we will wish we had that extra energy to keep the inside of the rover toasty at night. The team will keep looking
for a solution.
Opportunity drove off the lander on sol 7. Even being cautious not to stumble into Spirit's memory woes, egress preparations
went much more smoothly for our second rover. Due to Opportunity's side-petal landing, all of the airbags in the forward
direction of travel were fully retracted out of the way.
Once Opportunity had driven off the lander, people on the team started to say "12 wheels on dirt!" to symbolize
the two MER rover successes. I reminded them that there is one more rover on Mars, one that's been waiting since 1997. While
everyone was drinking champagne to celebrate, I raised my glass and said, "18 wheels on soil!"
Since Opportunity managed the unlikely achievement of landing inside a small crater, the horizon is a bit higher than if we
had landed on a flat plain. And with the very sandlike terrain, the image mosaic of the post-egress photos creates an optical
illusion, with the rover and now dead lander apparently sitting on a Little Prince-sized planet.
I just saw a fantastic picture! It was taken with the rover's Navigation cameras back on sol 4, but we just got it transmitted
back to Earth today (sol 12). The image was taken while opportunity was stood-up on the lander, at the rover's hight elevation
for many sols. The rover was standing up high enough at this point to see some of the terrain beyond the walls of the crater
we landed in. And out there, to the southwest, there appear to be two objects. One is rather dark and perhaps rectangular
or conical. A little ways to the right of this object is a flatter, longer white shape. Most people on the team who've seen
the photo think we're looking at spacecraft backshell (separated from the lander in the final minute before landing) and its
attached parachute. It's tantalizing...we'd love to get a higher resolution image using the rover's Pancam cameras, but now
that we've driven off the lander, the rover isn't tall enough to see over the lip of the crater we're sitting in. We'll just
have to wait until it's time to drive out of this crater, and try to get another shot then...
Sunday, January 25, 2004
Ups, Downs...and Landing on Mars--Again!
It's been awhile since I've had time to post anything. The past week has been a rollercoaster of activities and emotion.
4:06 pm pst
Spirit began its post-egress explorations of Mars on sol 12, analyzing the soil immediately in front of the rover. Everything
was working great. We did our first drive to a rock the scientists had named "Adirondack;" we were going to analyze
the rock with our instruments, and then RAT the rock, using our Rock Abrasion Tool, which should grind away a coin-shaped
depression in the rock's surface. When this was done, we'd apply our robotic arm-mounted instruments to the hole.
Then, on sol 18, we couldn't get most of our command sequences to the rover. At first we thought that the thunderstorms at
our transmitter in Australia were getting in the way of our commands. But as the day wore on, our ability get commands to
the rover and see the right responses seemed to deteriorate. Sometimes the rover didn't answer when expected. The rover
would miss communications sessions, and the team would try to develop an explanation. Then we'd get a report that lots of
data was coming down from Spirit through one of our relay orbiters; the team would cheer with relief, only to discover minutes
later that the content of the data was meaningless. What was going on? There had been no warning of a problem onboard.
Over the next sol or two we commanded Spirit to send us "beeps" that would prove it was listening to us. Sometimes
After one such attempt, we got no signal. The mood in the control room collapsed. The team forced itself into planning what
to do next.
A few minutes later, their was a somewhat tentative, incredulous voice on the network: "Uh. Flight. Telecom. Station
63 is reporting carrier lock." Engineers around the room looked up in surprise. "They're reporting symbol lock...We've
got telemetry lock." Spirit was back! The data coming down was garbled, but our girl was at least babbling at us...on
her own. The mood in the room had transformed again.
From what little data we could get, it seemed that Spirit was booting up and reseting over and over, never fully waking up,
never completely shutting down. If the rover was coming up long enough to begin communicating, it might reset in the middle
of talking to us. And whatever was causing the resets was preventing Spirit from doing the rest of its tasks, like preparing
data to be sent. The power and thermal engineers were particularly worried: Spirit was designed to shutdown each night.
In its current state, it might never turn itself off. The result might be that the rover was overheating and draining its
batteries down to nothing. On sol 20, an attempt to command the rover to shut itself down for the night failed. Spirit might
be listening to us, but wasn't often doing what it was told.
I came back on shift as the anomaly Tactical Uplink Lead on the night of sol 20. The team decided that the objective for
sol 21 would be to make Spirit shutdown. The software team had been meeting all day, coming up with scenarios to explain
the rover's bizarre behavior. Their best notion was that the rover's main memory was corrupted, causing the rover to reset
every time it tried to access data in that memory. So, the software guys proposed forcing the rover to wake up without using
the main memory, like booting a computer without using its hard drive.
To get Spirit to boot up this way, the rover would have to accept a single command telling it what to do the next time it
reset. So on the morning of sol 21 we sent this one command several times. We then commanded a reset (which seemed a bit
redundant given what Spirit had been doing on its own), and after a while commanded Spirit to communicate with us. The rover
started talking to us on schedule. We were getting data that made sense again! The rover appeared to be stable. The telemetry
confirmed that we were in a low-power state. We sent a command to tell the rover to go to sleep until the next morning.
This would give Spirit time to recharge its batteries. All the indications were that Spirit had listened to us this time.
We still have a lot of work to do. As it stands, we'll have to send the special command every morning to get Spirit out of
its strange state and stable again. But at least we can now begin tracing the problem on a stable spacecraft.
In the meantime, Opportunity has been falling toward Mars. On Saturday night, those of us working on Spirit paused long enough
to watch the landing events unfold. Waiting for a Mars landing is always nerve-wracking. (After Pathfinder and Spirit, this
was my third time.) But everything went as perfectly as could be. Data came back from Opportunity pretty much the whole
time, during descent, landing, and even bouncing across the Martian surface! And a few hours later, Odyssey relayed photos
of our new landing site back to Earth that were amazing, even for Mars. There are no rocks! It looks like we rolled to a
stop at the bottom of a bowl (probably a small crater). The soil is a grayish red, except where we've disturbed it with
our airbags; there it looks like a deep pure red. There are imprints of the airbags in the soil in several places. And while
there are no individual rocks, we seem to be partly encircled by a rock outcropping--bedrock. No one has seen this on Mars
before. And it's only yards away. We'll probably spend weeks exploring it.
And thanks to landing on our side, and then needing to rotate the lander to be right-side up, there is no airbag at all in
the way of driving forward off the lander. It appears that we have a perfectly clear path off the lander into an unexpectedly
Thursday, January 15, 2004
Sol 12: Six Wheels on Dirt!
Spirit is now free of the lander!
11:09 am pst
The Vice President of the United Stated visited JPL yesterday to congratulate our Mars team on our success so far. Those
of us working the Mars night shift to get the command sequence updates ready for driving off the lander had to keep working
while the rest of the team and JPL employees listened to his speech. If we had taken a break for an hour or so, we wouldn't
have been ready for the next Mars day's activities!
Mars time continues to be disorienting. During another sequence planning meeting for Spirit, a Congressman came by and was
introduced to all of us. All I could think was, "What's he doing here in the middle of the night?" Well, it was
two in the morning-Mars time. Only after the Congressman left did I remember that it was mid-afternoon Pacific time...
After delivering the commands for sol 12-drive off day-I took a break, tried to nap, and came back to hear the poll of the
daytime team for "GO FOR EGRESS." When every subsystem reported "go," all of us crowded into the mission
control area cheered and applauded. The Flight Director played the theme from "Rawhide"-and everyone applauded
Of course, we would now have to wait another hour and a half to hear back from Spirit how well egress from the lander had
gone. All the rehearsals indicated it should go fine, but we were all tense anyway. Engineers are professional worriers.
We imagine all the ways things can fail, so that we can design to prevent those failures from occurring. But even when we've
done our jobs, and considered all the alternatives we can come up with, there is always some doubt...
I went back to my office and tried to sleep for an hour. I really should have gone home and to bed for the night, so I'd
be fresh for my Mars-time shift tomorrow, which would start just after noon PST (but at 11 pm Mars-time). But I wasn't going
to miss egress. So I slept on the floor-and I did sleep. Exhaustion can be conducive to rest.
I woke up again, and it was less than fifteen minutes until we should get a signal. I wandered back to the mission control
area. Steve Squyres, our Principal Investigator, was finishing off an explanation of what our long term science plans were
likely to be. It was now only a few minutes until we should hear again from Spirit. There were several different types of
telemetry that would tell us if things had gone well. We weren't sure which would be the first source to arrive. The Flight
Director pointed out that getting any signal at all would at least confirm that Spirit was still healthy.
Carrier lock! There was a signal. We waited for data to come through that signal. One subsystem reported that the rover
had been tilted as much as 30 degrees during part of the past hour-just what you'd expect from driving down the ramp. People
cheered. The mobility team said their data showed that the rover had rolled more than two meters. Applause. Then low-resolution
pictures started to appear. There was the lander?behind us! We could see tracks in the dirt where Spirit had driven off.
The front cameras showed nothing but soil Martian soil under-and on-our wheels. We were off! Engineers were cheering, applauding,
and hugging each other. People were shaking my hand, telling me I now had a lot of work to do. The mission had just shifted
from "Impact to Egress" to normal surface operations, and the operations process I had designed was about to kick
Time to go home. I have a big day tomorrow.
Sunday, January 11, 2004
Sols 6 and 7: Spirit Stands Up!
The day before yesterday, my wife came home with a new license plate frame for my car: "My other cars are on Mars."
It replaces the one I've had since Mars Pathfinder that said, "My other car is on Mars." Between Sojourner and
Spirit, I'm now up to two. I'll keep my fingers crossed Opportunity will land just as successfully.
2:49 am pst
I?ve been working Mars time for the past four days, and now finally have a couple of days off. The Mars day (called a "sol")
is just a bit longer than an Earth day, at twenty-four hours and thirty-nine and a half minutes. Since the rover is solar
powered, and wakes with the sun, its activities are coordinated with the Martian day. And so are the activities of the MER
operations team. Part of the team works the Martian day, interacting with the spacecraft, sending commands, and analyzing
the results. But those of us who build new command sequences for the rover work the Martian night, while the rover sleeps.
Since the rover wakes up about 40 Earth minutes later every morning, so do we. It seems like sleeping later every day would
be easy, but it can be disorienting. It's very easy to lose track of what time it is on Earth...
We've been struggling for the past few days to get the lander's airbags retracted far enough to drive straight off the lander
onto the Martian surface. While it's possible that the rover could get tangled in airbag material during its egress from
the lander, the bigger concern seems to be that the airbag might contact one of the rover's delicate solar arrays, tearing
it or otherwise causing damage. The airbags all have internal cords that can be drawn in to the lander using airbag retraction
motors. We've tried "lift and tuck" maneuvers with the lander, leaning back one petal to create clearance to make
it easier to pull in the airbag under another petal. Some of the airbags have cooperated, but the one in our way is immobile.
We've decided to turn 120 degrees to the right and drive off that way.
For the voyage to Mars, the rover was folded intricately into the volume of the lander, crouched down with its front wheels
almost touching each other. Standing up the rover is a multi-step process--described by our Principal Investigator as "reverse
origami--with careful checking all along the way. If any part of the standup is incomplete, and we go on to the next step,
we could find ourselves unable to recover. On sol 6, we used the "rover lift mechanism" to raise the rover high
enough that we could then rotate the front wheel "rockers" up and out to their full extension. The space is so
tight that we have to steer the wheels out of the way during the rotation to keep from striking the solar array. After the
wheels were deployed, the lift mechanism lowered the rover onto the wheels. The pictures came back and showed the wheels
in the correct position. To confirm that the rockers were latched in place, the rover was raised again. There was no indication
that either of the latches had failied...but...the telemetry didn't look exactly as the mechanical team expected.
So, on sol 7, we ran another version of the rocker latch test, just to be sure. Results looked good. So we extended Spirit?s
rear wheels. With that completed, the rover is fully stood up on the lander! Once we cut the one remaining cable umbilical
(the one that let's the rover move the petals of the lander around), Spirit will be completely independent of the lander and
ready to drive off.
Egress should now be only about three days away. It's almost time to go exploring!
Wednesday, January 7, 2004
Starting to Live on Mars Time
The night of Sol 4-5
10:03 pm pst
I just finished working the Martian night (which today happened to be daytime at JPL), planning Spirit's activities for Sol
5, the rover's fifth day on the surface. The uplink team is doing a great job of getting a lot done in a short amount of
time. The rover seems to be getting more sleep than we do...
We've had lots of firsts the past few days, as we prove to ourselves that Spirit survived its bumpy landing in good shape.
We've demonstrated that both orbiter relay links--MGS and Odyssey--are working, giving us multiple paths to get images and
data down to Earth. Amazingly, both relays worked the first time we tried them out! Then we proved that we could relay commands,
if the need arose, through the Odyssey orbiter. And we've been commanding the rover--and getting data back--every sol through
the "low gain" antenna.
Spirit seems very healthy. Some of our team has been worried about our "high gain" antenna. This is the antenna
on the rover that looks like a lollypop and tracks the Earth, sending back far more data than possible with the low gain.
The first time we used the high gain antenna, it looked like one of the motors was noisier than it should have been--and
working harder than it should be--but only when turning the antenna in one direction. We stopped using the antenna, while
we studied the problem. But the antenna seems to be working fine now. The most likely theory is that there was some debris
or maybe lumpy lubricant that got pushed out of the way the first time we swung the antenna around, and now everything is
The rover has also been getting warmer than expected, forcing us to do a little bit less to avoid overheating. Spirit was
designed to hold in the heat it generates, to avoid freezing overnight: Mars is a cold place, often getting down to -100 degrees
C. But the weather at the landing site is balmy--at least for Mars. The lander seems to be insulating the rover from the
cold ground better than we thought, and the "thermos bottle" we keep our electronics in may be even more efficient
than expected. When we started developing Spirit and Opportunity, we knew that managing the thermal state of the rovers would
require care throughout the mission. The fact that we're warm now may mean that we will survive for a long time as things
get colder later...
The science team seems interested in investigating "Sleepy Hollow," what looks to be a small impact crater only
about 50 feet from the lander. That may be our first destination once we finally driver off the lander, after Spirit is fully
checked out and deployed. Then we may literally head for the hills. The hills might be a kilometer or more away, but in
this terrain, we may be able to travel fast. We'll certainly be cautious at first. Time will tell...