What’s a good measure of an Agile project manager’s success?

In a recent conversation I was asked about PMBOK (Project Management Body of Knowledge) and its applicability to measuring an Agile project manager’s effectiveness.  While I did not know much about PMBOK and its measurement approaches I was curious to understand the problem.  As it was explained to me, the problem was along these lines… When following a PMBOK approach to measuring a project manager’s effectiveness, one of the things you would look at is on-time delivery.  If the project was delivered exactly on time the manager would get a rating of one.  If the project was delivered ahead of plan then the manager would get a rating of less than one (based on how much ahead) and vice versa if delivered late.  However, when following the OutSystems approach to Agile application delivery we define a project timebox and even if we deliver all features early, we will then fit any additional backlog items into the plan and keep working until the timebox is complete.  Thus, you never finish early – you always finish on time and in most cases you just exceed your customer’s expectations on how much you deliver!

sprints.jpgSo, off I went and did some research on PMBOK.  What I learned is that the OutSystems Agile Methodology, while based on SCRUM, incorporates lots of extra management concepts that align with PMBOK.  (BTW – If you are not familiar with the OutSystems Agile approach here’s a white paper.)  I will leave the convergence of Agile and PMBOK for a later discussion, but in my opinion, there are lots of PMBOK practices that are applicable to running Agile projects in Enterprise IT shops.

What I am interested in is the following – What is a good measure of an Agile project manager’s success?

In the conversations on Agile and PMBOK I’ve had over the last couple of weeks I have come to the conclusion that the best measure for an Agile project manager’s effectiveness is not based on on-time delivery, staying within budget, etc.  But rather a measure of a new application’s adoption by the business!  Everyone I have talked to on the topic agreed with this notion of adoption but none have really offered a concrete technique for measuring adoption.  In most of the discussions the notion of return on investment came up as a solution.  However, in drilling into ROI we always came to the conclusion that while ROI is important it is not necessarily a good measure of your project manager’s effectiveness.

So my quest for a good measure of an Agile project manger’s success continues.  Everyone agrees that application adoption could be it but I have yet to find anyone with a good definition on how to measure it. Your thoughts?
You should follow us on Twitter here.

 

About the author

Mike Jones

Mike is a professed 'hater' of complexity. He has been in the IT industry since the mid 80's where he learned his technical skills at EDS and Texas Instruments. He believes there is a simpler way for IT professionals to deliver business value that requires a pragmatic mix of agile methods and application development tools.

Comments

Maysoon

Just wanted to share an interesting link that Jim Highsmith from Cutter Consortium tweeted in response to this question – some ideas about agile measures of success. The Agile Triangle. http://bit.ly/3DxR0H.

David Updike, CSM

Mike, I understand your question but my initial gut reaction is fairly non-PMBOK. I teach Agile as a we all swim or we all sink. So everybody’s success is tied to a projects success. That said we still live in a world of “personal performance measurement” this is where the manager has to understand Agile AND be close enough to the team to know who is doing what but far enough away to not get in their way. This would apply to the PM as well. Are they correctly facilitating the meetings, are they inviting all necessary people, are they removing obstacles for the team, etc.

I think the using the measurement of delivering on time is a maybe not the best way to evaluate a PM. Given the statistic about 70-80% of IT projects failing, I always get scared of PMs who grade themselves by their ability to deliver on time or in budget. It would be hard to get up each day if I had to look at myself and say, “Today, 80% of the things I do will result in failure” (meaning late, over budget, etc.)
As a PM, the way I measure my own success, and the success of the PMs who work for me is based on ability to follow the PMBOK (assuming they are PMPs) in a way that works best within the org/culture, creates as little noise or process overhead as possible and keeps the client fully aware of status and making informed decisions.
For an Agile PM, I’d measure success on the person’s ability to apply Agile practices within their PM practice.
For me personally, I serve the project, then the client, then the company I work for. I manage the process very diligently, keep the client fully informed and I push the decisions back on them. If the client wants to move in a direction that is almost certainly going to result in failure, it is my job to make sure they know what they are doing, what the risks are, etc. If they still want to go that way, I am responsible for getting us to fail as efficiently and quickly as possible and then having a backup plan for recovery.

Hi Mike,
I echo what David has explained – it’s all for one and one for all when it comes to Agile.
It’s also worth noting that there is no such role as ‘Project Manager’ in an Agile environment. I don’t usually get caught up in semantics, but feel this is a pertinant point as the ‘Project Manager’ title implies that a single individual is responsible for the project outcome. This isn’t (shouldn’t) be the case.
In a Scrum environment you’ve got your Product Owner, Product Manager, Scrum Master and Scrum Team. All are responsible for different elements of the delivery, but all are jointly responsible for the combined outcome.
I’ve written a few posts about measuring the performance of an Agile Team… this is summarised in the Agile Balanced Scorecard series.
With that said, although these metrics are useful, the best measure of performance is stakeholder satisfaction.
Let me know your thoughts! :)
Thanks,
Tara

While I haven’t read it myself, Michelle Sliger and Stacia Broderick are both certified PMPs and wrote a whole book on this topic:
Software Project Manager’s Bridge to Agility
The Table of Contents and sample material can be found here: informit.com/title/0321502752

What is the purpose of AGILE/SCRUM? Is it about more than on-time and on-budget parameters as Dave mentioned?
Having managed Enterprise IT projects as well as product development for 10 years I see an aspect of Agile that I don’t see mentioned much. But it is the key to the project’s success.
Yes, it involves applying Agile practices — but it also involves the ability of a PM in enable a project team to become self-managing.
Many Agile PM’s can apply the practices, but enabling a team to truly be self-managing and owning the success of a project is where the magic lies.
When you look back on project success using Agile or PMBOK…….if the entire team owned the success, not just the PM and the core team members, then a project would shine and customer satisfaction would rate super high.
Thoughts?

My opinion is that the project’s success can be determined by the adoption of the business, and not the Project Manager’s. The Project Manager’s is always related to the on time on budget. If you’re way behind schedule, and you’re way over budget, regardless of the methodology, then there’s something that’s not very right. On the other hand, a project, in and for itself, can be a success even if you’re behind schedule and over budget.
I have published an article specifically on this subject: difference between project and project management success.

Mike/Tara,
Salient points from everyone. As a software consultancy, there is the purely financial success factor of actual profit versus forecast profit (we’re a subsidiary of an accounting firm – this is an obvious first consideration :-)). We cost all our projects based on resources and their base costs versus actual cost as the project progresses through to completion. Most projects expect to include an amount of bug-fixing but most clients clearly won’t pay for this aspect of ‘failing QA’ so this would affect the overall project efficiency. However, even with this in mind, our projects to date are still looking at >95% efficiency (this compares with 80-85% on previous fixed price projects using other approaches.
However, I agree with Tara’s last comment about customer satisfaction. The key aspect I impress upon everyone in my team is that if we make our project sponsor (for us, this is the external client) look good then we have a successful project – notwithstanding the financial success criterion above. Perhaps given that 80% of our business is based on repeat business within customer accounts this sort of success factor is continuously being self-tested – using an agile approach is enhancing our reputation within our clients and is a major contributor to us continuing to gain repeat business.
John

Tom Roussell

IMO, any measure of Agile success should pitted against the Agile Values …
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
I personally feel the delivery of working software is the number one goal. Since Schedule and Cost is fixed, and Scope is what varies, I would tend to want to measure success by looking at Scope; that is, did you meet or exceed your sprint goal? If so, you succeeded. If not, there is room for improvement somewhere. Look to your Retrospectives for that information.

Tara,
I am aligned with your comments! However, your conclusion still begs the question – how do you measure stakeholder satisfaction?

Amber Shah

- How closely the finished project ties to what the customer wants AT THAT MOMENT (not what was written in some spec doc 8 months ago)
- How little overhead the developers and testers and others need to spend conforming to your processes (ie. during this time they are not writing code, etc)
- How much turnover there is among the developers (the project manager plays a big role in setting the culture – he can be an enabler or a thorn in their side)
Schedule is irrelevant. Everyone wants things done faster than possible. Developers always agree to an overly optimistic estimate. There’ve been a million attempts to “fix” this but basically there is no fix so long as people are pushing to get things faster. The only way to fix this problem is to accept that the schedule is actually an irrelevant metric.

I often use as a measure or success the “Size of the Smile of the Customer”. Works quite well.

Robert Neri

Must there be metrics? Let’s see… this is a very subjective topic and very many factors involved. Adoption… hmmm, maybe a good metric. But for me adoption is more a measure of the application meeting the needs of the business (with the necessary usability factors applied of course.) But is that really a measure of the PM’s effectiveness? Indirectly perhaps; more bragging rights than anything else. In an agile project, the team should really be credited for that.
I think a PM’s effectiveness in any project, agile or not, is a combination of things – (1) managing the triple constraint; (2) managing the expectations across the board; (3) project delivery.
Things change over the course of a project; it’s inevitable. Typically what changes is one or more of the triple constraints. The PM needs to be on top of these so that expectations are set and reset accordingly. No surprises. Project delivery to expectations is key. It may not be delivered on the original date specified; could be before or after but if the PM properly manages the expectations and the impact to the triple constraint then I would say the PM was effective.
Of course, that’s just my opinion.

Jim Brisson

Lean/Poppendiecks say there’s no such thing as technical success, only business success. An architect can’t take credit for a beautiful design when the product doesn’t sell. Similarly, there’s no such thing as Project Management success, only business success. High productivity, high quality, using TDD, scrum etc. don’t count for much if your client is not happy or your product doesn’t sell.
All for one, one for all ….go Tara!

The authors of “Facilitating Organization Change” make an interesting point: whereas the success of a predictive system is measured by how close they are delivering to the original plan, the success of a complex adaptive system should be measured by how well the system adapts to changes in the environment.

Hi Mike (et al),
I would suggest that the true measure of the success of any project manager (‘PM’ from here on) is the final customer acceptance of an on-schedule, on-budget deliverable that meets the agreed-upon scope and quality. Whether you’ve obtained that acceptance thru multiple delivery cycles using methods or thru a Waterfall-equivalent single delivery cycle process, a PM’s reason for existence is to ensure customer satisfaction independent of the development methodology employed.
As Dave Prior points out, the project – hence the client – is first, then the company. Of course, in order to accomplish this goal, the PM will have necessarily ensured the success of the team and its members in terms of job satisfaction, career advancement, rewards, and overall performance realization, including himself/herself. This also pertains to the stakeholders as well – their satisfaction would also to have been met in order for the PM to meet the success criteria.
As far as ROI is concerned, that particular metric will have (must have?!) been determined prior to the initiation of any project, otherwise that project would not exist, and by meeting the success criteria for cost, schedule, scope, and quality would by definition indicate that the ROI for the project had been met, thereby contributing to the success of the company as well.

To be fair to PMBOK, it has not mentioned “on-time delivery” as the factor; it is only one of the factors.
I also agree with the view point that “there’s no such thing as technical success, only business success”. Solution which is not aligned with organization culture, does not enhance team synergy, fails to deliver on business value is not worth a dime for that content, no matter how technically superior that solution is.
However, I am concerned about attributing success or failure with a specific individual/role. It is conventional wisdom but goes against the core of what agile stands for, as I know it.
I believe, agile stands for demonstrated value, continuous and effective communication, honesty and transparency, and togetherness. I believe, just as quality, success/failure is a collective responsibility. Failure hits everyone hard, irrespective of the roles; not just customer, project manager, business analyst etc

Rewarding and measuring individual performance is biasing and potentially dysfuncional. (on-time, budget, etc.) Be careful! I really encourage to read Austin:”Measuring and Managing Performance in Organizations”. It is logically provable. You get the idea from my blog http://bit.ly/2ZQiKK
Exactly the same applies to projects. One project is a one-time endeavour. In long term product development, rewarding one project most probably leads to local optimization.
PMBOK gives advice how The Project Manager, takes care of The One-time Endeavour. However it may not be appropriate with software product development. In this blog I describe what I have seen happen. http://aritikka.wordpress.com/2009/08/18/focusing-on-projects-ruins-your-business/
As Poppendieck suggests, the business success is a good measure.
What I said concerns product development. From the customer perspective, the receiver, a one time-endeavour and the project metaphor may apply.
Best Wishes
Ari Tikka

Agile PMs know there are stakeholders, customers, project team members, outside vendors and others typically. To meet expectations and enable team ownership, an Agile PM must have those relationships in place to get the inside scoop. Inside scoop on what? The hidden expectations, hidden agendas, competing priorities your team members are battling with, etc. Successful Agile PM’s must know and deal with these issues and risks – and relationships are the key way to discover and manage them.

Things change over the course of a project; it’s inevitable. Typically what changes is one or more of the triple constraints. The PM needs to be on top of these so that expectations are set and reset accordingly.

RachelH

I think the record for success for a PM is simple. Customer satisfaction, and money earned. I’ve been a PM for almost a decade now, and while there are many different strategies to completing a project properly and on time, ( I prefer a midway status check) the fact of the matter is, if the client is satisfied, then the project is a success. I was looking for project manager jobs and salaries, and I was surprised at the varied salaries for a single job. I think it is keystone to know your value to a project and an internal knowledge of cost. If a project is asking too much for too little, then walk. Solid rules are hard to come by in this business, the only constant is the client will always want something by a deadline and we will have to adapt.

London Photographer

However, when following the OutSystems approach to Agile application delivery we define a project timebox and even if we deliver all features early, we will then fit any additional backlog items into the plan and keep working until the timebox is complete.

Leave Your Comment

contact pricing contact try