Agile Projects and Portfolios

Although the agile manifesto was defined and released in 2001, it is the result of many years of validating these practices in numerous real projects. I see the four statements defined in the manifesto as high-level direction to our industry. Recently I have to admit, I have an inner-conflict with one of the statements and the direction we take as a community. It is “Individuals and interaction over processes and tools”.

During the 1990s, agile development was seen as a beginning of a revolution in the IT industry. People referred to agilists as rebels. Please forgive me if this comparison is a bit strong but similar to the 1960s and 1970s generation of music, when new sounds and creativity erupted, agile development practices tackled many root causes of software engineering problems with really cool ideas. Have you seen the scene in the movie “Office Space” in which the actor decided to take down the walls of his cube? Agilists did do that! The fascinating aspect of this movement was the simplicity of the practices and that we focused on sometimes very basic ideas and tools. Among many others, index cards, pair programming and one co-located project team in one team room. Please also remember, that agile emerged during a time in which others explored with generating code directly from UML and heavy-weight process documentation. Today we know that we supported the right thing.

Important is however, that many of these early projects succeeded not only because we had a very basic tool of index cards, but because we worked in one team room. As a matter of fact, because we worked in one room, index cards were ideal and sufficient to manage our projects.Our project teams were also social networks which had due to their proximity a huge advantage, “individuals and frequent interaction over processes and tools”.Besides the daily scrum meeting, the team shared constantly small fragments of information.For example we didn’t write e-mails or call colleagues. We just asked “who wants t pair up and write some code?”, “Who broke the component xyz?” or “I am out for lunch for the next hour”. These were simple and good times.

Since the agile manifesto in 2001, many organizations experimented with distributed development, often motivated by accounting and financial departments.Reducing costs seemed to be directly linked to the wages of software engineers. On payroll paper, that appeared to be a great idea but caused in my opinion a major problem for agile projects. Let’s re-iterate.
The very first step was outsourcing “testing”, but we practiced test-driven design. What now? Well, with a little compromise, many projects fell back into old waterfall habits, and “just” did the QA at the end of the iteration? One team in one room was already history at that point, and our index cards didn’t work anymore. A few years later, with the increasing maturity level of off-shore organizations, we delegated programming as well. To do all the coding and testing, the team abroad needed specifications. Not a big deal, right? On one side of the planet a team which writes a specification, on the other side a team which digests it and tests at the end. That model has little to do with a team and the definition of agile assembled in the manifesto.

Instead of tackling the root cause, which is communication and collaboration of a geographically dispersed team, we transformed the tool instead.We digitized the index cards and card wall and created software products which we hoped will solve our problems.Psychologically, there is a huge difference between someone getting up a chair, walking to the card wall and move a card into the “done” column. Having the entire team in one room, we all witnessed the transition of the card, followed occasionally by applause. A card also carries a much stronger ownership because it is tangible. When a team member puts their initials down and moves the card into the “in progress” column, that team members really owned the card. In a distributed team these things are almost anonymous and may happen while the other part of the team sleeps (time-difference). Written words travel much slower than spoken and sometimes nothing needs to be said at all to share information. Neither the early steps of offshore development were agile, nor are many agile projects which were converted into agile distributed projects these days.Yes, many so-called agile distributed projects are workarounds. Industry leaders estimate at least a 50% productivity loss when distributing an agile team. But that said, even with a distributed agile solution, there are still benefits compared to waterfall projects. With a distributed model in place and the compromise in productivity, executives need to confirm that time-to-market is not as important as the development costs.That includes the potential revenue we sacrifice by deploying late instead of paying back the development costs earlier.Please don’t get me wrong, I am not suggesting that distributed agile does not work, I am just pointing out the productivity loss and that distributed projects live with a smaller profit margin.That might be perfectly suitable in many occasions.

Recently I have received a constructive comment about rock-star behavior of certain agile speakers. But perhaps that is exactly what our industry actually needs right now. Professionals with a wealth of experience who stand-up and take positions. Someone who may even exaggerate certain metaphors, entertains us with it and simultaneously encourages us to (re) think. Somebody who puts a stake in the ground, takes position and defends that side with healthy arguments.Agree with one side or not, it takes a difference in opinions to allow a good debate, otherwise everything is just blurry without substance. In a debate about distributed teams we will need to protect the term “agile” otherwise it will be blurry soon, too. For example, the agile conference’08 had a track dedicated to Distributed Agile and many more books are soon to be released around this topic. Some of the speakers and authors in our industry promote distributed agile frameworks which have in my opinion not been proven in practice, sometimes not even applied. It is ok that speakers admit that distributed teams are sub-optimal compared to co-located teams, but sub-optimal was not measure for success when agile originally emerged. Executives want to hear what is optimal and consultants advising executives with distributed frameworks need to be careful, because some of these immature solutions may come back like boomerangs.

When transitioning to scrum-style agile project management you have basically three different meetings; a sprint planning session, a retrospective and the daily scrum meeting. Although the concept of sprint planning and the retrospective are commonly well understood by executives, the daily scrum imposes some challenges. It should be kept short with a flat reporting hierarchy and it should be frequent.

When organizations transition to agile, they will need to revisit existing calendars and challenge them. Every periodic meeting on the calendar means less time dedicated to an increment. I am not proposing that all meetings beside the daily scrum should be eliminated, but they should be the result of a daily scrum meeting. The daily scrum could therefore be the only scheduled meeting. That is especially true for organizations that do not have the entire team co-located in one team-room. What I have learned from new agile teams is that issues are historically tackled with team meetings. The daily scrum is the place to expose the issues and make them part of the agenda, not to elaborate them. New meetings will however fill up calendars. If the team members are also part of different reporting hierarchies, the problem might get more complicated due to mixed agendas. Then, and that’s where things get worse, you will experience scheduling conflicts, excuses and slippage as a result of it. Let’s not even talk about distributed environments and very short iterations.

If additional meetings are scheduled as a result of the daily scrum meeting, the team can monitor the events and the effectiveness though their next daily scrum. If the meetings are already part of a regular schedule, their existence won’t be challenged. During the agile conference I learned about a “No Meeting Day” one particular company established. In this case, no meetings could be scheduled throughout the entire organization that weekday, including the daily scrum. That is not necessarily in-line with the scrum rule-book, but I like the idea and will experiment with it someday. Especially in very large organizations, this concept signals a great message from senior management to their project teams.

Executives have a different perspective on projects than team members.  That is related to the proximity of the work performed. Senior management sees parameters such as scope, schedule, costs, progress and eventually the implementation of of the project whereas team members are in the weeds of day-to-day activities.
 
Little has changed in this executive model since agile was introduced to waterfall companies.  One difference is however the observation of a project in-flight. While traditional methods produce paperwork, models and perhaps a prototype, agile projects present their results in an iterative fashion.  These results are shown in working software as early as after iteration 1.

From an executive perspective, and that is the primary audience of the book, defects appear consequently as early as the first iteration. Compared to traditional projects, which produce code much later, the agile project communicates problems and issues while the traditional project is “cruising”. That is true because there can’t be a defect without code, because the code will be produced much later.

It is ironic, although we know that the traditional project may hit a much bigger problems later and the impact could be severe, the problem is delayed. But for executive forecasting, traditional project look (wrongfully) better in early stages. 

Agile projects expose problems and impediments very early. Removing these impediments is part of the agile process and in the long-run an extremely positive thing. Executives who have not been exposed  to agile development practices may therefore mis-interpret the situation.

Therefore, agility naturally increases the number of defects in early iterations. If test-driven design or continuous integration practices are applied even more.  That these metrics represent a focus on quality and real progress will need to be communicated. In the same way, we will need to re-communicate the delayed response around progress, quality and technical debt for traditional projects.

The Agile’08 in Toronto is over. The conference had lots to offer, many different tracks and even more sessions. Although many attendees asked for more sessions which address the needs of agile developers a year earlier in Washington , I believe we overdid it this year.

Only a few attendees got lost in the open jam stage because attendees couldn’t carve out time for even another stage. Ironicially, there was little for executives and decision makers who will give the go/no-go for agile projects in organizations. The business value stage catered more to product owners and business analysts instead of managers.

I will submit my feedback to the conferene committee. I am thinking of offering more introductory sesions during the first two days targeting beginners, newbies and general broad topics. Towards the end of the week we could tackle more detailled ideas . Not everyone can spend five days at a conference.

A little disturbing was the absence of the Scrum Alliance. At least I could not find them. A year earlier they had a booth and several speakers who were missing this year entirely. Considering that the Agile conference should bring the agile community “together”, it was a poor showing in my opinion. I do think Robert C. Martin did an excellent job pointing it out in his keynote.

At the end, I think it was an impressive conference again. 1600+ attendees and the diversity of sessions and speakers are not typical in the IT community. See you in Chicago next year.

This book is primarily targeting project managers, portfolio managers, business analysts, members of the PMO, and executive managers who are interested in adopting agile practices and portfolio management. I hope it will give exactly this audience an easily digestible introduction to this topic with enough material to sponsor agility across the entire enterprise.

Although this book is not intended primarily for technical audiences, it may help them appreciate the motivations and involvement of the project external factors in their enterprise.

The book is a collection of thousands of words, but two very important ones were accidentially left out in the acknowledgement section. Saawan Pathange reviewed essential pieces of the book from a financial perspective and provided invaluable feedback. His name will be added to the next print-cycle but in the meantime I would like to acknowledge his contribution in this blog. Thank you Saawan!

The book became available this week at Amazon and other major retailers across the US.  In the future, I will use this blog to interact with the readers to discuss topics and questions which are related to the book. Please reach out to me with your feedback directly. The power of agile development practices is recognized by more and more enterprises. New ways of executing projects on an organizational level are therefore required. To provide answers to these enterprises and their social agile networks is the goal of this book. I know that new ideas will emerge and will turn into best practices over time. The book is the snapshot in-time and this blog the continuous evolution of the topic. And who knows, perhaps the content of this blog will drive a second edition of the book down the road…..

The current production schedule indicates that the book will be distributed to retailers at the16th of July’08 from the publisher’s warehouse. The books are then usually shipped within 1-2 weeks to their customers. Customers who had pre-ordered the book should therefore receive the book by the end of July’08.

In the past two weeks I have received many endorsements from industry professionals which will be printed as praises in the beginning of the book. This weekend is however the final deadline for the manuscript which will be final by Monday when the book will enter production. 

Nine out of twelve total chapters are reviewed, edited, paged, proofread, indexed etc. The remaining three chapters will be completed this week. That will trigger the production process. As of today, we are on track to make the book commercially available in July’08