Scaling Agile at Spotify. What True Organizational Agility Can Look Like.
Henrik Kniberg and Spotify released an illustrated video on how Spotify is Scaling Scrum and Agile from a 1 team startup to a 1200 person powerhouse of the online music industry. Spotify is a disruptive force in the music industry, forcing record companies, radio stations, and artists to rethink their business models. Spotify is also an Agile company from the ground up. Spotify got its start using Scrum with one team, so there has been no “Agile Transformation” for the company. Instead the company faced the problem of how to maintain the Agility they had with one team while growing quickly.
The video describes their team structure, the organizational structures that span multiple teams, and the key principles that guide their decision making. I think it is fantastic that Spotify was willing to open the Kimono and show the world how they work, and that Henrik, as their Organizational Coach was able to present it is such a compelling visual way.
This is the cutting edge of organizational agility. For most companies, this kind of organizational model is a fantasy, it is so far removed from their current organization’s structure. If I were a music company executive, I would be scared as hell. Instead, I look forward to continuing to help companies achieve their own improvements for Agility in their own business context.
Reviews of SAFe Program Consultant (SPC) training on the Scaled Agile Framework (SAFe), from Agile and Scrum experts
I have been following with interest the Scaled Agile Framework and its increasing popularity. There have been a number of reviews of the training to become an SPC or SAFe Process Consultant. I am also interested because SAFe came out of work done by Dean Leffingwell and other consultants at NAVTEQ in the US in 2010-2011. At the time I was providing Certified Scrum training to NAVTEQ subsidiaries in Germany who were implementing Scrum. I did not work with the US groups, however I am familiar with what was happening at NAVTEQ at the time. So I think any review of SAFe should include how it helped or did not help NAVTEQ be more successful as an organization. In the last few years since implementing SAFe, NAVTEQ has gone through a major downsizing, been acquired by Nokia, and then shrank some more. Speaking with a former Director from NAVTEQ now at another GIS client of mine, the automotive division is doing OK, but the rest of the company struggling and shrinking in the face of competition from Google and others. There are many reasons why companies fail, and we cannot blame SAFe for the shortcomings of NAVTEQ’s organizational culture and business model. However we can and should ask: how did SAFe help NAVTEQ?
In reviews the Scaled Agile Framework and SPC training, a number of experts Agile and Scrum have commented on their experience in the training, the training style, and the substance of the ideas in SAFe. Below are links to expert reviews of SAFe and SPC training.
Agile experts Ron Jeffries and Chet Hendrickson who are authors on extreme programming and Scrum, consultants and trainers recently took the SPC training in Washington DC. Ron’s review takes the stance that SAFe has some good things that Scrum or XP do not address, and he goes through those points in detail. Ron is an excellent writer and thinker, and I enjoyed his analysis of SAFe and the training. I think this is the best review so far. Scaled Agile Framework. SAFe: Good but not good enough.
Peter Saddington, a fellow Scrum trainer and consultant based in the DC area recently took the SPC course in Washington DC. Peter gives a balanced review of what he sees as the pros and cons of SAFe. While he likes Lean aspects SAFe has borrowed, he thinks the recommended “one week” roll out period carries significant risk. Peter strives to be open to the ideas that are contrary to some of the core principles and values of Agile. His highest praise is for SAFe’s marketing. Review of the Scaled Agile Framework SAFe SPC training and ideas
Scrum trainer Daniel Gullo recently took the Scaled Agile Framework and SPC training and reflected on the ideas and how they compared to his experiences working on Agile adoptions in large organizations. Daniel was also a agile consultant at NAVTEQ helping them adopt Scrum while the Scaled Agile Framework was first being implemented at NAVTEQ. “As long as I am able to work with a client implementing “SAFe” and we are allowed to tailor it to include the helpful parts and throw out the harmful parts, I don’t see anything really evil or wrong with SAFe here. What we are left with is doing precisely what I have already been doing for the last 8 years: looking for how to instill the values and principles Agile and Lean in the culture of an organization so that a paradigm shift happens.” Review of the Scaled Agile Framework SAFe SPC training and reflection on SAFe compared to Agile Coaching experiences
David Snowden, creator of the Cynefin (pronounced Kan av in) framework, a practical application of complexity theory to management science, recently weighed in on SAFe. In Dave’s post he weighs in on SAFe’s linear model and one size fits all approach. Snowden’s background is in complexity theory, and he knows that simple linear approaches fail to solve complex problems. Dave’s evaluation of SAFe is that it is not Agile. SAFe: the infantilism of management
Enjoy the reviews of the Scaled Agile Framework and SPC training. I will post more reviews from experienced Agilists on SAFe and the SPC training.
Moving to Agile development in hardware is more about mindset than it is about technological limitations. Let’s consider some of the limitations that are often used to resist Agile hardware development, and show how these limitations are more to do with thinking then with technology.
PCB board design and layout.
We can’t do Agile because we need to complete the full board design before sending the design for PCB fabrication and population of parts. Manufacturing is costly and takes a lot of time, so we want to minimize these costs.
Let’s examine each of these points:
Myth: Manufacturing a PCB takes a lot of time.
Facts: The last 30 years of continuous high speed innovation in PC motherboards and mobile phones has driven the PCB manufacturing business to be very quick and responsive. A five minute search for fast turnaround PCB manufacturers revealed many companies that can turn an 8 layer PCB in 24 hours. With overnight shipping, you can have your PCB in 2 days. If you plan ahead with the manufacturer to get parts ordered so it can be populated, this can be done at the same time.
Myth: Manufacturing a PCB is costly.
Facts: The costs for quick turn around PCBs are trivial when compared to the cost of an engineer waiting for hardware. You can calculate costs from a variety of quick turn manufacturers at http://www.ladyada.net/library/pcb/costcalc.html For a 3″x5″ PCB the highest cost for a 2 day turn around was $215, shipping included. For comparison, that average loaded cost of an engineer to their employer is (salary, benefits, office, etc.) is $80-200 per hour. An engineer who spends one day waiting for hardware is far more expensive then the fastest most expensive quick turn PCB manufacturer.
Myth: We need to complete the full design.
Facts: A well designed PCB is modular, with modules for CPU, memory, I/O, RF, etc. Each module is designed as a unit, with interfaces to other modules, just like software. Therefore we can design a module, or part of a module, and that completed design can be treated as a testable deliverable. For example, the CPU and the memory could be designed and shipped as a populated PCB ready for developers to start using. The developers can start to use that platform for coding while the hardware designers work on the next module. The hardware designers will need to implement some features, like a JTAG or USB port and terminate some pins so developers can use the PCB, but that is a small price to pay in exchange for getting hardware into the hands of the software developers.
Deploying iterative and Incremental hardware development.
If we change how we think of hardware from being a single physical device that has to be designed, to a set of functional modules that are deployed as hardware, then we can map the manufacturing process of PCBs to the deployment of code into production. If we can deploy functional modules into a physical device (PCB+parts) in a few days, then we can deliver hardware functionality using an iterative and incremental approach. For example we could deploy a PCB with a basic input output (I/O) circuit that meets our simplest design criteria. This circuit would be deployed into hardware and then could be used by developers for coding and testers. If it is determined from programming and testing that the simple I/O design needs changes, a revision of the I/O could be completed to make the desired improvements. This ensures that the designers are delivering a design that meets the needs of the developers and testers. Once the developers and testers were satisfied, the I/O could be evaluated by the customer to gain their feedback and ensure it meets their needs.
Iterative and Incremental firmware with the hardware.
If hardware is thought of as deployed functional modules, then firmware is the glue that enables these modules to provide customer value. Firmware engineers have numerous well understood options for software development when there is no hardware. Software simulators, emulators, evaluation boards, and FPGAs all provide different options. With every design there are business priorities and technical risks. Based on these risks and priorities the developers need to figure out how to slice the design into components of firmware and hardware that will deliver the most business value and risk reduction with each iteration. For example if I/O is risky then the hardware and firmware developers should figure out how to build, code and test the I/O module before the CPU module. For example this could be accomplished using an I/O PCB that interfaces to a CPU evaluation board.
Using business value to drive product design.
We are faced with the continual expansion of software into the hardware space. Shrinking device sizes, massive gate arrays, generic open source platforms like Arduino, and smart phones all are changing the hardware development game. However software will always need something to run on, and as more devices connect to the internet, the importance of real world interfaces will continue to expand. Great hardware developers like Hewlett and Packard always emphasized getting out of the lab and meeting your customers. As hardware is increasingly used to wire up the connected world, it is more important than ever that hardware designers find ways to quickly respond to the changing needs of their customers. Agile principles and values provide good ideas to help you get there… just replace the word software with firmware or hardware :).
Artwork “Post Apocalyptic Data Hunter: RONIX” by Steven Rodig at PCBCreations.com
With the massive failure of the healthcare.gov launch in the US, people are talking about why such an important software development initiative turned into such a failure. The US however is not alone in these kinds of failures. The UK government spent 10 years and 12 billion pounds on the failed NHS-IT electronic patient records program. Now 12 billion pounds is about 16 billion Euros or 20 billion US dollars. Ouch! I picture the Queen in earlier times shouting “off with their heads!” Instead the UK government started listening to people who knew about Agile and technology development. The UK Government has created the Government Digital Services, GDS, a 300 person department with the skills to deliver digital services online. In a recent interview with NPR, Mike Bracken, head of the UK Government’s GDS discussed the successes the GDS initiative has had in the two years since it was formed. Bracken mentions many Agile principles GDS are using to deliver a government that is digital at its core. Some of the highlights are:
Optimizing for business value: GDS is only going after the processes and services most used by citizens, such as passports, tax filing, benefits, etc. Within these areas they further optimize, so GDS is delivering digital versions of the highest value services first. Some processes and services may never be automated because the business value is not there, and that is OK.
Incremental delivery: GDS is delivering services in an iterative and incremental fashion. They are not trying to get it all right the first time. Instead they are getting the most important parts right, and then learning what works and what needs improvement.
Cross functional teams: GDS teams are cross functional, with developers, process and policy experts, UI designers, testers, etc.
Collaboration over contract negotiation: GDS has thrown out all of the traditional procurement processes and procedures. Instead of building contracts based on some wild ass guess wrapped in 300 pages of documentation, GDS admits they have no idea of where or what they need to do in 3, 5, or 10 years. Instead they focus on working collaboratively with users and stakeholders, quickly (1-3 weeks) delivering small amounts of working tested software to their users and getting their feedback.
Bracken was visiting Washington D.C. to talk to government officials and influencers to help them understand how the UK government is bringing digital services in a fast and cost effective manner to its Citizens. For everyone’s sake, I hope the bureaucrats in DC are listening. The $600 million spent could be a more cost effective lesson than the UK, or it could be a failure that is repeated over and over, just like in the movie Groundhog Day.
I am trying to find a specific kind of iOS application. I am using the App Store in iTunes, on a Macbook Pro with a 27″ external monitor, extended keyboard, and wireless mouse. But it doesn’t really matter. I could be on my ipad, and the experience would be just as frustrating. Apple says that they have 500,000 apps available for iOS. This is a big point of pride for Apple. Well let’s look at all the tools Apple gives iOS users to navigate 500,000 Apps to find what they want.
I would like to find a keyboard utility the can replace the iPad software keyboard with one that has arrow keys for editing and is more accurate. I don’t know if such a utility exists, however the keyboard and text editing capabilities of the iPad are a regular user complaint. I open iTunes and click on iTunes Store and then the app store. See the screen shot below. I am confronted with a giant rotating advertising banner that often features Apple products like ibooks 2 and iTunes U. Scanning the rest of the page I see headings New and Noteworthy, and Quicklinks, this section includes additional lists featuring Apps chosen by Apple.
The Apple App Store Front
On the right there is a “Top Charts” list featuring more Apps that are the top of some metric, also controlled by Apple. Other Navigation headings include the enlightening “What’s Hot” section, featuring another Apple controlled list of Apps. Scrolling down the page are 7 more lists defined by Apple staff.
What if you know generally what kind of App you want? Well out of the whole first page of the App Store, there is only one small button that might take you to where you want to go. Under Quicklinks is a dropdown list with broad categories. How long do you think it would take for most users to find this single button with its hidden categories? Well, they probably wouldn’t. I did, because I am writing this article and went over the App Store’s front page very carefully. Clicking on the drop down I am confronted with a long list in very small font, of all the categories. After a couple tries I click on the category Utilities.
iPad Utilities Landing page. No further categories, more Apple controlled lists. Click image to see UI detail.
The Utilities page loads more Apple lists, with New and What’s Hot again listing Apps that Apple has chosen to feature. This is the end of the category navigation. Only after scrolling down do we find the All Utilities iPad Apps section. The section has 3960 applications, of which we can see the first 24 with listed with “most recently released” at the top.
App Store Ipad Utilities page, 3960 Apps
To find a keyboard I re-order the list and click on “K-O” since I am looking for a keyboard utility. iTunes returns 180 applications from App iXML to LogVu. I find 2 Apps with the first word Keyboard. After viewing an App I click the Back button, however the list is different and I am disoriented. The 3960 Apps are reordered by release date again and I am back on the first page. After multiple tries I give up on browsing. However there are likely other keyboard utilities that use different names, so I will try another approach.
Searching for Something?
ITunes has a global search feature in the top right of the application. searching “keyboard” in this search utility returns results from movies, music and every other section of the iTunes store. Filtering by Apps, and then again by iPad Apps returns all iPad Apps for “keyboard” a whopping 581 entries, all conveniently sorted by well, actually I cannot determine how they are sorted, and I can’t change the sort. My search for a keyboard utility has returned Apps from almost every category, including Utilities. More specific search terms, such as “keyboard utility” or “keypad” or “arrow keys” returned more specific results, including everything from emoticon Apps to remote controls for PC and Mac applications. Apparently it is easier to turn the iPad into a keyboard for a PC application then it is to create a different keyboard for the iPad itself.
Searching returns long unordered lists with no descriptions
Power Search: Did you Miss it? I did.
In doing the search I stumbled upon the Power Search feature. Power Search allows you to select a more detailed search criteria. Where do you find it? Well that’s a good question as power search appears and disappears from the App Store UI, even while you use it. It also doesn’t seem work, missing some Apps, or returning Apps not in the category selected.
Too little information
As can been seen in the figure above, search results include a picture, the title of the application, the date it was updated and the price. There is literally no way to judge whether an App is worth your time unless you click on it and view the application’s page. Even on this page critical information about the App is hidden from view on the App’s own page, Users are forced to click again to read more than the first 2 lines of the App’s description.
Useful information is hidden from users on the App's own page
Your App is lost in the App Store
App developers should be very concerned about How Apple is controlling access to their applications. By not building in effective browsing and search features, Apple is preventing customers finding your application. This means all the time, effort, and money spent on your App is wasted, since unless the marketers running the App Store choose to promote your App in the Apple controlled lists, potential customers can’t find your product.
A lesson from Amazon and Musician’s Friend
Apple customers and iOS Developers both lose with Apple’s App Store User Interface. Navigating the App Store is painful, slow, and ineffective. It is clear that the App Store in iTunes was never designed to handle an inventory of 500,000 Apps. The App Store and the other storefronts in iTunes need a major investment to bring them up to the ecommerce standards of today. Instead of providing effective tools that allow users to find the Apps they need, Apple is pushing dozens of lists, in the hope that Apple taste makers know what Apps their customers want. This is bad for the App market, bad for App developers, and very bad for App Store customers. Apple needs to take a lesson from ecommerce sites such as Musician’s Friend and Amazon. Those sites handle as many or more products then the App Store and they provide the tools users need to find the products they want. App store users, whether on the PC, iPad or iPhone need to be able to browse and search effectively to find the Apps they need. Today the App Store is a little shop of horrors to use, devouring both customer’s and developer’s time and money. I still have not found that keyboard utility to replace Apple’s iPad keyboard, so I will continue to reach for the laptop and leave the emails for the Android powered Motorola Droid Pro World phone with its real keyboard.
This is a video of a talk I gave at Agile Tour 2011 in Vilnius Lithuania in October 2011.
Your New Customer has no clue what Agile is, however they have lots of assumptions about how they will “get the product done.” Do they know how to work effectively with you? Do they know all of the business and user issues that the product will need to solve and how to solve them? Have they built a product like this one before? Are the 100% committed to being the product owner or do they have other jobs too?
We’ll discuss how turn a customer into a Product Owner, from the first meeting to creating the first backlog, through to one year into development. We’ll go through key learning points that your new Product Owners and teams will have to transition through, and techniques you can use to make your life and theirs easier. Come prepared to learn tested hands on techniques you can apply in working with your customers.
I just taught a course and participated in a software conference in Vilnius Lithuania. Over the last 3 years I have taught Scrum to over 800 software development staff in the Ukraine, and hundreds more in the US and Brazil. The situation is much more complicated then you portray, and not as negative as the article implies for US/EU workers. The huge missing piece in this article is VALUE. The perspective presented is only focused on cost. It is about the same as saying if only there were lower taxes the US job market would improve. This is false thinking, and surprising from an investor. Why do we invest? Because of the potential opportunity. For example take the team of A players tweaking the Linux kernel yet again, and compare them to the B players who develop a hugely popular web site like gamespot. Which team generates more value? Another missed point is the huge demand for software. Unemployment in the US IT sector is 3.2%, compared to over 10% in other sectors. A more recent phenomena is companies who outsourced all development now rehiring developers because of communication, speed and quality problems. Capital One and Wells Fargo are examples.
By far the largest problem in software development is management. Managers who learn how to build and nuture lean and agile organizations will create organizations able to respond much more quickly to the market while growing the strength of their teams. This is where many companies fail miserably, and generate huge amounts of waste in failed projects, burned out staff and unmet expectations. There are no lean and agile organizations in our industry today.
Toyota became the largest and most profitable car company with staff from all over the world. They did it because the Toyota Production System is unmatched in terms of creating successful results. This is a management system and a way of viewing the world of work. Whether the cars are built in Manila or Kentucky, the driving force for success is not the cost of labor.
“I smile and start to count on my fingers: One, people are good. Two, every conflict can be removed. Three, every situation, no matter how complex it initially looks, is exceedingly simple. Four, every situation can be substantially improved; even the sky is not the limit. Five, every person can reach a full life. Six, there is always a win-win solution. Shall I continue to count?”
Dr. Eliyahu M. Goldratt 1947- 2011
When I first read Eli Goldratt’s book The Goal, it was in 2005. I had been using XP and Scrum for 3 years and understood the value of good process improvement ideas. I was stunned after reading it. Not only was the book fantastic, I was shocked: this book had been written in 1986?!?! I felt embarrassed that I had never known about these ideas. In 1986 I was still in engineering and these ideas were cutting edge at that time. Is Theory of Constraints taught in engineering schools 25 years years later? A search of goldrattschools.org reveals 9 affiliated universities, and google searches are not filled with education institutions teaching Theory of Constraints (TOC). It seems most people learn about Theory of Constraints through direct efforts of Goldratt’s various institutes, the Goal and subsequent books, and word of mouth. I have told many people about and given away copies of the Goal, creating numerous converts to Theory of Constraints thinking. Goldratt’s ideas provided many insights into systems thinking and process improvement. As a software development manager who had already transitioned from more traditional management to Scrum and Extreme programming, I found TOC provided another set of thinking tools to analyze a work system. These tools complimented Lean and Agile without reducing their value or conflicting with those ideas. For example Theory of Constraints provides a way to prioritize process changes found using Lean tools or impediments found by the team using Scrum. However the biggest change was that TOC made me think differently. It changed my perspective on systems, on software development and work in general. Like Lean thinking, TOC colors my thinking about any system and gives me greater insight into the natural properties of systems.
Ours and future generations are indebted to Goldratt’s ideas and his insights into people and systems. His ability to write about these ideas using non-technical stories was almost as important as the ideas themselves. His writing allowed many people to pickup his books and within a few hours understand the core concepts of Theory of Constraints. The appeal of TOC ideas caused them to look for ways to apply TOC to their work, while his easy and accessible style created powerful incentives for people to learn more.
I am truly sorry to hear that a genius such as Eli Goldratt has passed away at the relatively young age of 64. I am grateful to have been a student of his ideas and hope to carry on the work of spreading the ideas of TOC and its applications in software development and knowledge work.
Dr. Eliyahu M. Goldratt spent his entire adult life fighting to show that it is possible to make this world a better place. We must have the honesty to see reality as it is, we must have the courage to challenge assumptions, and above all, we must use the gift of thinking. Having applied these principles to various management fields, he created the Theory of Constraints. His concepts and teachings have expanded beyond management and are being used in healthcare, education, counseling, government, agriculture and personal growth – to name a few fields using TOC. His legacy is invaluable. On June 11th, 2011 at noon, Eli Goldratt passed away at his home in Israel in company of his family and close friends.
The strength and passion of Eli allowed him to spend his last days sharing and delivering his latest insights and breakthroughs to a group of people who have committed to transfer this knowledge to the TOC Community during the upcoming Theory of Constraints International Certification Organization Conference in New York. It was Eli’s last wish to take TOC to the next level – truly standing on the shoulders of the Giant he is.
A recent discussion on linkedin was started based on the question: Should teams talk directly with users or customers?
It provoked me to write the long response below.
“What do you mean by customer? Do you mean user? Sponsor? Another manager?
The Product Owner (PO) should be representing the various stakeholders for the system being built. The PO should also be creating alignment among the team and the stakeholders as to what needs to be accomplished in the release. This is a balancing act between the various needs, business and technical. The PO is not an information hub, everything doesn’t have to go through the PO. A good PO creates alignment around the vision and the solution, and steers based on their ongoing learning and subject matter expertise.
So I’d split my response into the stakeholder groups:
In general it is a good idea for a Scrum team members to talk to users. As Geir Amjso mentioned, the PO can become a bottleneck if they try to control all information. Also their maybe some nuance that a user can provide that will help a developer understand the problem more clearly. This conversation is framed in the context of a sprint and is about clarification and understanding to implement the solution.
Another benefit of having users and developers interact is that spark of innovation. Innovation happens when people with tools and techniques really start to understand a problem. Users sitting down with developers creates empathy and helps teams become motivated to provide pain relief or innovations that surprise or delight.
Not a good idea without the PO around. The sponsor is usually more concerned with the high level issues and the politics/expectation management for the product/project. The sponsors should engage the PO, and if they are going around them to the team, there is something else going on.
For other managers:
It depends. It may make sense if there are dependencies with other teams/vendors and the team is managing those issues within the sprint. It may also be an end run around the PO to get work done that is not planned in the sprint. In this case the team should refer them to the PO.
Scrum does specify meetings where interactions between teams and customers are more formal. However Scrum says nothing about restricting communications to the Scrum meetings. If communications are restricted to these meetings only the team will be slower and have less opportunities to learn. As a Scrummaster the goal is to get great software developed and do it with the highest level of engagement from all parties. This means there are lots of communications that should happen on an ongoing basis between all members of the project community. A Scrummaster needs to enable work to flow from request to working software, and this involves maintaining the flow of information. Will their be deviation? Sure, however the Scrummaster is there to detect when the team or the stakeholders are getting off track (i.e. adding scope, changing direction, etc.) and help them stay on track.
In September 2010 I gave a keynote talk at Agile Eastern Europe called “Dude, where’s my backlog?” The talk covers the common pitfalls seen with Product Owners and the Product Backlog. The talk covers the people that bring a product backlog into existence and the processes they can use to maintain it. This includes using ideas from Kanban for personal process and modifications to Scrum for more frequent involvement of the team in maintaining the backlog.
This 2 day course covers how to use Scrum to manage your Agile projects. Students gain an understanding of Agile principles, Scrum practices, and how to manage the Scrum process. Recommended for project managers, technical managers, and technical leads.
Current classes are listed on the Scrum Training site.
This 2 day immersive training is great for teams who are looking to adopt Agile for their project. Team members come away with a solid understanding of how to work day to day on an Agile project, how it changes their role and what they should expect. It covers Lean, Agile, and Scrum, with a focus on the Scrum project framework.