How to Avoid Fatal Product Flaws with Agile and Scrum? Lessons from Boeing’s 737 MAX 8 Program

Why did Boeing’s development of the 737 MAX 8 effectively avoid Agile principles? Agile foundations are part of Boeing’s toolset. Boeing could have maintained a focus on quality and safety for the aircraft’s overall development program, and its key avionics systems, had they been followed.  

Employees of Boeing, and others with an understanding of avionics systems and the business of airplane manufacturing, suggest the recent 737 MAX 8 crashes are rooted in Boeing’s business priorities and design processes for the aircraft. Both have been aimed at avoiding re-certification, re-training costs, and delays for the Boeing 737 Max, an aircraft that differed markedly in aerodynamic and avionic terms from its predecessors.

Agile Foundations at Boeing

The loss of life from recent 737 MAX 8 crashes is incomprehensible.  Concerning and confounding is that the principles we now call Agile were foundational to the success of Boeing’s 777 development program.  From 1990-1995, across a complex international context of 10,000 engineers, Boeing engineering executives promoted such Agile-informed principles as:  

• quality and/is schedule (quality is a critical ingredient);
• panic early (fast fail);
• plan for zero overtime (sustainable pace);
• weekly design, build, test (DBT) reviews (weekly scrums);
• make decisions faster (act and learn fast).

With Agile principles, and related quality assurance practices having been deployed with success at Boeing in the early nineties, why was an Agile-underscored focus on quality abandoned during the development of the 737 MAX 8?  In Agile terms, why were extrinsic quality (value in meeting the needs of end-user pilots for trust, transparency, and training) nor intrinsic quality (the myriad of internal aircraft design and manufacturing criteria) genuinely valued and pursued diligently?

Cultural and Engineering Failings:  Sales Before Safety?

In its flagship publication, IEEE Spectrum, the Institute of Electrical and Electronics Engineers (IEEE) offers an analysis that make a strong case that business considerations for the 737 MAX trumped values of quality and safety, resulting in a flawed aircraft design that cost hundreds of lives.  

In “How the Boeing 737 Max Disaster Looks to a Software Developer:  Design shortcuts meant to make a new plane seem like an old, familiar one are to blame,” long-time pilot and software engineer Gregory Travis details, in aerodynamic and avionic terms, insights as to why this “737” is a very different aircraft than its predecessor. He summarizes that Boeing called the plane a 737 for reasons of cost, avoidance of regulatory scrutiny and avoidance of training that a new aircraft would require.

Underplaying the aerodynamic differences between the avionics systems of the 737 MAX 8 and its predecessors underscores the serious failures in the development of the 737 MAX 8’s software systems:  

1. Developers with insufficient aviation domain experience;
2. Poor oversight compared to that for hardware and overall aerodynamics;
3. Lack of understanding or commitment to Agile as a framework for quality software systems development.

Some Thoughts on Culture at Boeing.

A few years ago I toured the Everett, WA plant where 787s are built, and my impression was that it was a huge factory, yet it was very quiet for mid-week, not the level of activity I would expect on a factory floor. A free workshop we facilitated showed it took 3 months to organize and complete a (in-situ) component test, however the actual time needed for testing was 1 to 4 hours. We had lots of advice on how to reduce that cycle time so testing could be completed faster. My impression was that the advice was ignored. Personally I was fascinated at the cultural differences between the plant producing the large aircraft in Everett, a northern suburb of Seattle, and the plant producing the small 737 aircraft in Renton, a southern suburb of Seattle. Like two different companies, the Renton plant seemed more fast paced, competitive and continuously improving, while the Everett plant was, well, quiet.

I know there are many great people at Boeing. They have some fantastic and safe products that I use almost every week, such as the previous generation of 737 planes. I use Boeing’s ability to produce the previous generation of 737s in Renton (a plane every 10 days per line, 42 per month) as an example of how dedicated Lean thinking can produce a fantastic manufacturing process and a great product. The people building the current 737s were not the people directing the development of the new aircraft with all the problems.

While social media has its negative points, one benefit is the transparency it can create, often illuminating problems organizations would prefer remain hidden. In this spirit of transparency, the last comments on culture I will leave to a blunt developer who worked for 1.5 years at Boeing.

“First, Boeing’s corporate culture is the worst I have ever experienced. All large corporations have a lot of internal issues and problems but nothing like the Lazy B. It was like working in a company designed by Kafka. I signed up at Boeing as a programmer. When I showed up at my first day of work, the first words out of my supervisor’s mouth were, “I don’t know why you are here, we have no need for programmers.” The Boeing interview process is done so that at no point, do you ever have contact or communication with the team you will be working with.”

Read his full comments are available on Reddit. His comments were regarding this Vox Video: The real reason Boeing’s new plane crashed twice.

Could Agile Foundations Have Prevented the Boeing 737 Max 8 Failures?

Absent Agile Foundation: Fail Early and Learn

The purpose of “fail early” is to discover successful outcomes in increments, steering to an ultimate completed product that works before full implementation. The core value is found in extensive testing and cutting losses quickly when something is not working. Had the 737 MAX 8’s larger, heavier and forward-mounted engines been mounted on a wing and floor tested, the discovery of the engine exhaust generating lift under the wing, causing it to “pitch up” with the application of power would have been known much sooner. If this result had been known and treated as unacceptable, then a different design, simply put, would have saved lives.

In the case of the 737 Max 8, the decision was to have software compensate for the extra lift generated by the engine exhaust. Boeing’s software, the Manoeuvring Characteristics Augmentation System (MCAS) was an attempt to use software to fix a flaw in the plane’s aerodynamic design. MCAS was needed because:

1. The plane’s “angle of attack” (pitch or tilt) changes caused by the larger, heavier, and forward-shifted engines would likely exceed regulatory criteria of the U.S. Federal Aviation Administration (FAA), thereby risking aircraft certification, and causing Boeing and customer Airlines to incur pilot re-training costs and delays;
2. A revised engine placement, novel wing design, or other hardware-based changes to fix the problem aerodynamically would be very costly;

In addition, because MCAS was “fixing” a design flaw, Boeing was not forthcoming about the MCAS operation with stakeholders:

1. The MCAS, its function, and overrides to its function were communicated transparently to pilots, perhaps given re-certification and re-training risks;
2. Key MCAS warning lights having been disabled or solely provided as an upgrade;
3. The two separate flight management systems for pilot and co-pilot each measure the angle of attack with different sensors on either side of the plane. Pilots check both systems and sensors to validate whether Angle of Attack is being measured correctly. However the MCAS only took input from one sensor on one side of the plane, something a Pilot would have recognized immediately as a flawed design. However Pilots were not informed on how MCAS worked.
4. Boeing has “standard procedures” for design, development, testing, documentation, training, and certification. Yet with MCAS it seems these procedures were not followed.

Absent Agile Foundation: Including Users in the Product Development Process

A critical quality element to a successful Agile implementation is connecting the development team to the end-users — directly and early-on.  The development team responsible for MCAS did not spend time in the cockpit with pilots to understand how they used their flight control systems and when they chose to ignore those systems. If the team had spent time with the Pilots, they would have understood how their users, the Pilots, do their work. That knowledge would have caused them to make different design decisions, more realistic tests, better documentation, and training informed by the pilot’s current practices.

A Culture of Profits Over People

Boeing’s cultural problems of prioritizing production and profits over quality don’t stop with the 737-Max. The New York Times recently published an expose on the production quality problems plaguing the 787 Dreamliner. The problems detailed in the article from many interviews and leaked documents indicate chronic problems, leading Qatar Airlines to refuse delivery on any 787s built in Charleston. The article paints a picture of management prioritizing production over quality and a desire to hide quality issues because they will slow down production.

Agile Adoption Now to Ensure Safety in the Future

So, how might service providers, IT organizations, and engineering firms introduce and sustain commitments to Agile so as not to fail their clients on quality and safety? Beyond Boeing, how might commitments to Agile by executives and managers, and alongside related quality paradigms, help align business priorities and development approaches to quality and safety? With software required to control an ever-increasing field of mission critical functions, it is time executives and organizations commit to Agile in the same way manufacturing companies increasingly commit to frameworks like the Toyota Production System or Lean.

Aligning to Customers Aligns with Success

Like Lean, Agile requires organizational change. Executives will need to step outside their comfort zone and work with collaborators to re-invent their organizations based on a deep commitment to their customers. By orienting to the customer, not the shareholder or the C-Suite, organizations align to value streams that are focused on improving their customers lives. In orienting to the customer, we quickly realize that the teams doing that work are essential to our success, and the better the teams the better the results. The closer the teams are to customers, the better they can meet their needs. Some of the world’s most valuable companies got there by being customer oriented and Agile. This reorientation is happening in many companies today, and yet many people in organizations today don’t see a reason to change. They don’t see their own personal interests reflected in this new paradigm.

Failures Beyond Boeing

The Boeing 737 Max 8 should be a warning to those who would put their personal interests ahead of their customer’s interests. So should the Volkswagen emissions scandal and The Wells Fargo account fraud scandal. These three are massive failures that have brought huge losses to shareholders, customers and employees. They have occurred because the organization is not oriented to the customer, it is oriented to shareholders and the C-Suite. According to the world’s best investor, that is exactly backwards:

“If a business does well, the stock eventually follows.” ~ Warren Buffet