Measuring Agile Success?

Last month our company announced its new Agility Awards – and the first set of awards were presented for five Agile projects that had been completed by customers and partners using  OutSystems’ Agile Platform and employing Agile methodology.

The question I want to pose is what are good criteria for assessing a successful Agile project? This question builds on Mike’s recent post about criteria for measuring an Agile project manager’s success – and we got lots of great responses and ideas in the comments.
agility award.jpg
The data points being used by the OutSystems team to evaluate whether projects are eligible for an award are:

1.    Size & scope of project: the project should be of medium to large scale (over 40,000 software units in size.)

2.    Project definition & objectives: the project should deliver significant and measurable business value.

3.    Project approach: the project should have been run based on Agile practices, following an iterative development approach with regular end user involvement.

4.    Project metrics: a baseline of project metrics must be submitted in order to measure the impact of using Agile to deliver the application.

The team then use these data points to assess whether an Agile approach was employed to deliver the project on time, on budget, with 100% user adoption and delivered true value to the business and IT.

Is this list a reasonable set of data points for measuring the success of an agile project?
BTW – Here are some examples of the results from the initial set of award winners (read more details here):

XDx – Analysis Request Management System

Time-to-market: 6 weeks + 1 for launch; Number of Agile sprints: 3

Customer quote: “This was our first Agile project and it achieved the two key business goals: avoiding tracking errors and improving real-time data consistency for our studies. Most importantly, we were able to deliver this value to the business in a record time, exceeding both developer’s and user’s expectations and establishing Agile as the preferred methodology for this type of development project.” – Jochen Scheel, Director at XDx.

RWE – Tiger, Implemented by Atos Origin
(BTW – nice blog from Atos Origin on the project & award here)

Time-to-market:14 weeks; Number of Agile sprints: 5 + 1 tuning

Customer quote: “The Gas Portfolio Management application was implemented over a period of three months which was only possible with the OutSystems’ Agile Platform and methodology. Their way of sharing information, processing activities and reviewing project deliverables with key users of RWE NL was instrumental to the success of this project.”- Perry van de Goorberg, Project Manager at RWE.

OK! teleseguros – Sales Platform and Home Insurance, Implemented by Keep It Simple

Time-to-market: 13 weeks; Number of Agile sprints: 3

Customer quote: “This project was a true success as it exceeded the business’s expectations in terms of objectives achieved and above all business benefits. The Agile Platform and methodology allowed the business to engage the development team and see the immediate impact and results of all project changes and decisions.” – Sérgio Carvalho, Marketing and Product Director at OK! teleseguros.


About the author

Robert Neri

Application development has always been Robert’s passion. From project management to customer interaction to hands-on coding and testing, Robert always looks at improving the process while delivering better applications faster with higher usability.


What about:
* From Development Team’s Point of View:
** Learning
** Satisfaction
* From Customer’s Point of View:
** Learning
** Delight using the Product
* Improving the state of Software Development? (both inside the company and in the community)
* Innovation and joy?
>> 1. Size & scope of project: the project should be of medium to large scale (over 40,000 software units in size.)
Its interesting to see the mapping of size and scope to software units (estimates). Who cares what the estimates? Esp. when your unit of estimate is very different from my unit of estimate. With this point you are almost encourage people to “create” (bloat) their requirements and estimates. When the opposite would be the right thing to do.
>> 2. Project definition & objectives: the project should deliver significant and measurable business value.
How do you measure business value? When do you decide you have achieved the business value is achieved?
IMHO the other 2 points are also very iffy.

Good input. I want to clarify the measurements Robert outlined in his post. They are part of OutSystem’s Agility Award program.
The scope and size of the project is specified in software units as this is a metric automatically generated by the Agile Platform. Software units are like function points – they estimate an application’s size and complexity. The minimum size of 40K was picked as we only want to recognize agile projects that delivered something more than a proof of concept.
Note, we are aligned in keeping applications as lean as possible while meeting the business needs. This is a core concept of OutSystems’ Agile Methodology.

Robert Neri

Thank you all for your input.
I guess it all boils down to how you define success after all. For OutSystems, we look at success as on-time, in-budget, and 100% user adoption. Seems fairly straight forward but not really. There are lots of things that factor into these, especially with 100% adoption.
Achieving an on-time and in-budget project requires a high-performing team that has the wherewithal to make things happen positively. Any team that can accomplish this always has some level of satisfaction; bragging rights at the very least; granted satisfaction and joy are very subjective. Frankly, measuring the team’s happiness does not really add value to the business. We’d rather keep this internal and recognize as such; but we also want to mention the team along with the award. Focus is really on the business itself.
What about value to the business? We let our customers measure this as this depends on the application itself; we recognize that each application has its own success criteria. For example, if you are deploying a call center application, wouldn’t you measure specifics about how the application has impacted the average call handle time or the first call resolution? We can’t very well outline criteria for all type of apps as part of the award so we figure, 100% user adoption would cover it. Here is why: the more an application is used, the more value it provides. When we deploy an application we want feedback from all users. This is very important to the adoption process. Once users see their suggestions implemented, the more satisfied they are, the more they feel empowered and the more they will use the application; the more value it provides.
Just as we let our customers measure how successful they are; we also let them decide when the application has indeed achieved the intended objectives. Besides, it’s their right to brag about their own successes; we merely recognize them for their achievements.
Regarding, quality and what happens after deploying an application… one of the tenets behind agile development is constant user involvement; to see their input evolve into a live application sprint after sprint is a great satisfaction for them. So why not extend that after the application goes live? We know the business evolves and we believe the application needs to evolve in-pace with the business. Our Agile Platform enables IT to change, test and deploy applications very quickly. To change or add new features very quickly means that quality must be built-in into the application; an easy feat with the Agile Platform even if you need to refactor at the architecture level. Once an application is deployed, it enters into an evolutionary maintenance where constant feedback from the users will keep the application evolving along with the business… agile maintenance.
Applications built with the Agile Platform are built for change. With evolutionary maintenance, success is not a one-time achievement; it is constant as the application continues to evolve with the business.

Leave Your Comment