If you fly two thousand hours over eleven years you will see some equipment failures (and some pilot failures). This was the first time I had a failure and didn’t know it.
Friends who follow me on Facebook are familiar with these two pictures:
As I was flying Dexter from Santa Monica, California to New Orleans, Louisiana for SICB’s annual conference we stopped for fuel (of human and engine both) in Austin, Texas. We took off into a night sky. I could spot traffic coming off Houston’s main airport more easily and watched the passenger jets coming in from the north heading for a long downwind leg to land.
A good pilot is always scanning everything. I take photos and on long, daylight legs will read a book, but I don’t read more than a sentence or two before I glance at all the important information. My ears are tuned to the hum of the engines and when they cough or mumble I am instantly alert and double-scanning everything. More often with the twins the propellers will get out sync, with the left engine at 2090 RPMs and the right at 2100 RPMs. A little nudge on the throttle and they are back humming evenly again. But I am always checking the groundspeed, comparing to the true airspeed, the tailwinds, the estimated time en route (ETE) to our next navigation point, and then a glance to check on the passenger (snoozing).
Glance, glance, glance, then a peek at the night landscape down below. Glance, glance, whoa! the forty knot tailwind has started to go wild. I am seeing it build up to 74knots, then swing around to the south at over a hundred knots. The amazing autopilot dutifully mashes the rudder and gets the plane sliding along the GPS track to New Orleans airspace. I listen on the frequency thinking any moment I will hear another pilot comment on the winds aloft. All is quiet. Within four minutes the wind spins all the way around the compass until it is coming out of the north at ninety-three knots instead. And then, after another few minutes, it gathers behind us and slows down to the previous forty knots. I sat with my hand on the control stick the whole time, with my thumb poised on the autopilot disconnect button.
In my head I tried to construct what could possible have caused that sort of wind. It seems like I had flown through something that should cause terrifying wind shear, but there wasn’t a ripple of turbulence. I pondered it a little, but I was flying through unfamiliar airspace at night, and into an airport that until recently had a weather forecast for low visibility in mist. So I was studying approach charts, arrival procedures, looking at the runway and taxiway diagrams and, always, scanning the instruments. All the way to New Orleans, a two hour flight, there was never another hiccup.
Even in a twin, I was a little anxious flying over what I assumed were the lightless swamps to the west of the city. It was the combination of the total blackness and distance to any sort of civilization. It looked like a bad place to get forced down. I stayed a little higher than I would have during the day, descending for the ILS to runway three six left at New Orleans Lakefront airport. As you are landing, the runway points directly out onto Lake Pontchartrain, and even with almost seven thousand feet of asphalt there’s an illusion prompting me to think, “What if I slide off into the water?” We were stopped before two thousand feet. It has been a long day of flying and Dexter goes in to summon an Uber while I empty the plane and check in with the FBO counter.
Two nights later I am back at the plane. I have the slightest bit of unease, but it is difficult to tell if that’s leaving Dexter behind on his own in a strange city, leaving my brother knowing he is headed to the Middle East for a few weeks, or that I was taking off into the teeth of a wind that was calculated to turn our 8.5hr flight east into a 14.5hr flight west. After a silly mistake (probably due in part to that distraction), I am motoring along for my longest time aloft, sitting in my perch above an overcast layer. Yes, there’s a thirty-seven knot headwind, but I love the landscape of the clouds and the new prescription aviator sunglass (progressive bifocals! perfect for spotting traffic and reading charts). I don’t have music playing, I am enjoying listening to the approach controller talking to the professional and military pilots. “Well, I’d like to help you out with that, but I’m all alone this morning and busier than a one-legged man in a butt-kicking contest.” You don’t hear that very much on frequency in SoCal.
Then I heard a bong that dumps adrenaline for me, no matter how often I hear it for expected events. It sounds when the auxiliary fuel tanks are emptied as I pump the fuel from the inboard tanks out to the wings (I know, I was trying to empty them). It sounds when I turn off the pitot heat even though I am purposefully turning it off, “PITOT HT OFF.” This is the first time I have seen a yellow dialog box in the center of the Primary Flight Display along with the voice: “CHECK ATTITUDE.”
I’m fine, thanks. A little blue about leaving behind my son… oh, the plane’s attitude. Looks rock solid, wings level just like it was before the bong. But… uh, this is strange. The horizontal situation instrument (HSI) is slowly turning counter clockwise and the winds are climbing like I’m going to follow Dorothy to Oz. Since I was above an overcast layer I had no reference for our course. Were we still following it. Here’s where I tapped the home button on my iPad mini, my backup to my backup, and with one more tap I had a duplicate set of instruments, including a GPS course track and synthetic vision. Thank you, ForeFlight. It showed that my HSI had gone nutty somehow, but I wasn’t quite sure how. What could be failing that would show the wrong heading information, and why would the wind be so confused? The aircraft on Foreflight’s display was slowly turning off course so I disengaged the autopilot and flew by hand while trying to troubleshoot and document the problem.
It’s a little different to be just managing the systems and scanning for trouble versus flying by hand, scanning for more trouble, and managing the systems which might be failing. At the same time I was talking with air traffic control because they were handing me off to the next controller and I was debating whether I needed to land. If I did, the overcast layer was not that thick and I was sure that I could use the iPad to get down through it on a vector from ATC. And as I flipped through some nearby airports and considered the options, the HSI sluggishly spun back so that I was headed west, and the winds settled back to a forty knot headwind. I tentatively engaged the autopilot again. It happily follow the magenta line west, canted slightly for a two knot crosswind from the north. I sat and watched it like a hawk. A very anxious, somewhat puzzled hawk, gradually calming down because of the usual steady hum of the engines, the chatter on frequency and the beautiful view through the canopy. Not a hiccup. So we motored on at 8,500 feet (sometimes a little higher when the cloud deck was irregular and I needed to climb in order to remain VFR).
A couple hours later I had decided there was some sort of magnetic anomaly over east Texas that messed with the G1000 suite of avionics somehow. I couldn’t get it to misbehave, but continued to watch it like a toddler with an ice cream cone (the intentions are good but the potential for disaster…). And sure enough just short of San Angelo, Texas the HSI started to rotate. I popped off the autopilot and already had the iPad on Foreflight’s gauges up to fly by. This time you can’t see the slightest hiccup in my course. I asked the controller if there was scheduled airline service out of San Angelo and out of Midland, about an hour ahead. He said there was at both. Peering at the Foreflight pages for both airports it looked like Midland might have just a little better service on the field for a little plane, but I wasn’t certain. I decided an hour west was an hour closer to home. If I had the fuel I would have pressed on the El Paso, since I was pretty sure that would be a non-stop commercial flight home.
Now the attitude indicator and the HSI just blacked out entirely. Finally, the G1000 knew that it was dead in the water. Er, dead in the sky. Over the next forty minutes it rebooted itself three times, slowly deteriorating and folding up after about ten minutes on duty. Oddly, approaching the airport it was perfectly happy. But I wasn’t fooled. I couldn’t continue to fly the plane west, chasing the sun. I would be crossing the last bit of the Rockies in the dark and the current weather projected for Santa Monica the next day included some clouds. I had no problem hand flying in the daylight, in the wide open sky. Doing anything at night with just my backup seemed like a bad idea. And, technically, the ForeFlight-Stratus2s-iPad solution was not certified for instrument flight.
The G1000 is complex and most shops don’t service them regularly. Chasing an intermittent failure, even with help from the Diamond and Garmin teams, would be difficult. I knew I’d have to head home. I steered on, glad that the overcast below me had dispersed.
It would be my longest flight in the plane so far. Five hours and fifty minutes after leaving New Orleans I was lined up for runway zero-four at Midland. A gusty crosswind that tried to push me off the centerline, but I could not have been more focused and the wheels kissed the thousand foot markers without a bump, the nose wheel settling right on the centerline. I was pretty wiped out and the ground controller gave me a progressive taxi instruction to get to Signature.
The shock was when I opened the canopy. There was a thirty mile an hour wind with 33F degrees, which meant my fingers were numb and red before I had the plane locked up. I hustled into the FBO with my bag, and pulled up my Kayak travel app. It was 5:07pm. There was an American flight departing at 6:10pm that would get me to LAX at 10:25pm with a single stop. Everything else was an eight hour or thirteen hour ordeal. I made a made dash across a wide parking lot to the terminal. I somehow triggered extra attention from the TSA, so I can say I’ve gotten my groin felt up in Midland. Texas. All in all I spent less than two hours in that lovely city and I wish it wasn’t quite that much.
Now I need to plan the repair and the pickup. If they can’t get it repaired on the field I will have to fly it back VFR and bring it to my own excellent mechanics.
The Ferry Flight Home
Initial detailed report of the failure, events on January 3 and January 5: n972rd-ahrs-failure (PDF download).
On January 12 I returned to Midland, Texas to fly the plane home.
Here is a video about two hours into the flight home:
The MFD showed the plane sliding along the GPS track (I was hand flying, our actual track was west). I had the VOR tuned and it showed us a little off that track, but it also showed us on the wrong heading on the HSI.
1:30pm We took off KODO (Odessa, adjacent to Midland) at 1:30pm.
2:43pm near El Paso, Texas we had the recorded failure. We hand flew.
3:04pm Attitude fail. Complete failure of the AHRS with a red X over the PFD. Slowly reboots.
3:15pm Compass is thirty degrees off, steady.
3:53pm System seems to recover and is fine for the remainder of the flight.
Landed Phoenix. Took off 6pm and flew home via Palm Springs and the Banning Pass. No further failures for the 3.5hr flight home.
Community Troubleshooting
Suggestions from the active Diamond forum online:
[P]lug-in connections on the backplane (or elsewhere) are the most common failure point for this type of gear. I would remove all of the boxes, clean all of the connectors as well as I could, and then treat them with something like Stabilant 22 (http://www.stabilant.com/appnt17.php). I had a noisy NAV/COM in my aircraft where simply cleaning and replugging the box did not cure the static, but one application of this stuff did (for > 1 year, so far).
I had a “progressive” failure actually with similar AHRS data drop out that was down to bad crimps between the GMU and AHRS. The failure was progressive as it was temperature related: the crimps concerned were right on the limit and as the temperatures increased the connections got gradually worse/became more intermittent, so more failures. Found by pulling the cables (gently) at the crimps. So as well as the G 1000 chassis suggest that it may be worth a gentle tug test on the cables.
I had the same problem on my 2005 DA-40, I got a big red X on the screen. A Garmin service center serviced the aircraft and with the supervision of Diamond Service in London, they look at all the connectors clean, retighten some connectors, and the problem remained. We are 3 owners in the DA-40. Some pilot do their own thing; I never have the AHRS problem on my flights, but when the 2 other pilots fly they have the issue… the difference between them and me: they always turn on the navigation light during the day and when I fly during the day, I never turn it on… That was a clue. The Garmin service center performed a magnetic field check on the aircraft with the G1000 in board self-testing mode, and they fail the test, on the right navigation light… So they start to look there, the round collar which holds the strobe light navigation light assembly to the round tubular was the problem. They put rubber tape between the collar and the tube. Inside the wing you have a long tube for the wing electrical cable, they pass behind the AHRS, so they find the tube is now the grounding source or something and amplified the magnetic field… In the end, we chose to put the wingtip LED conversion for the Whelen Orion 600… We saved 5 pounds in total and all problems disappeared.
Failures Persist
As per a visit to the shop, five bent pins were discovered, and I am sure those were good to have repaired. But two recent Angel Flights revealed the AHRS failures persist.
On Feb 6 2017, departing Santa Monica at 3:01pm we had an hour flight to San Luis Obispo. KSMO – KSBP. The refueling took half an hour and just after takeoff at 4:45pm the AHRS failed, X’ing out the HDG box and then the AI, but rebooting as we flew downwind and was online until we were landing at 5:40pm, when it lost the synthetic vision, then lost the AI and heading entirely. It rebooted as we were taxiing.
And on Feb 9 2017 we departed Santa Monica at 9:53am. We flew up to Santa Barbara, but were unable to land due to fog. We turned around and headed home. At 10:53am there was an AHRS failure where the compass spun. 11:08am as we were approaching to land there was an ADS-B failure (yellow warning text).