I was recently in a project that was based on the "Scrum Methodology" of working. I like Scrum and general Agile principles. I think when applied within a co-located team working to create a product it's a really good thing. However, I primarily work for an Indian IT Services Firm a.k.a- "Outsourcing". Things get very complicated in using such a methodology when it comes to outsourcing or working with teams which are in different companies.
A small history of software development models
In the early part of the new century CMMI was the "in-thing". The CMMI model primarily employed by offshoring companies was a way to assure customers who were wary of outsourcing their work that Indian companies who would be polar opposites culturally and time zone wise, could carry out the tasks given. It is a highly formal model very well-suited for Indian IT of that time which tended to hire low-wage and low-talent workers, put a process around them and get work done. This worked exceedingly well for repetitive boring work and with companies that had tons of money to spend, especially since Indian Outsourcing was cheap. It gave the statisticians and the QA guys great insight and software was reduced to a commodity. However, the key thing is that this only works in low value software tasks where everything is highly structured and certain.
The problems with the waterfall approach and CMMI are apparent to any company trying to build a new product or idea, it is just too structured and life can't follow that. Silicon Valley, and contemporary software went to the other end XP , Agile, Scrum and Kanban became buzz words for software development. I find these models much more suitable for working in situations where things are not so certain and structured like building a new product.
The outsourcing problem
Another change that has happened in the past 4-5 years is that offshoring and outsourcing to India has gone from just grunt work to a lot of high value work. People, are not just looking for vendors but partners. It was obvious that Indian companies would cash in on such buzzwords like XP, Agile, Scrum etc. to further show that they too can follow innovative processes and work in an unstructured manner.
Meanwhile, a lot of work went from Time and Materials to Fixed Bid. Outsourcing companies get typically higher margins on Fixed Bid work and customers get to control their costs.
The problem in my opinion with Agile Processes is two-folds:-
1.) There is no mention of the "Bid" or Proposal Stage. In this stage typically a vendor is supposed to bid for a project at a fixed price. Vendors, have to "estimate" the requirement and give a quote for the same. This very process is the root of all problems. To "Estimate" a requirement there needs to be clarity of the requirement, but the whole reason for the XP/Agile methodologies is because the requirements are complex and not clear.
2.) Consider the Product Manager role and the Product Backlog. If this is done by a client, he will ensure that even if it does not make sense the vendor delivers all the features mentioned in the proposal.
I have seen this play-out in funny ways. A project I was working with, we had defined a feature 'X' in the system during the requirements stage. However, 3-4 months into development, the team which consisted of the vendor product manager and us both concluded that feature 'X' did not make much sense. However, the 'Management' still forced us to do something to do feature 'X' even though it made absolutely no sense.
On the flip side the Product Manager's role can be given to the vendor. However, this is counter-productive as well. This first of all would need the Product Manager to sit in the client side and do all the business meetings required by "Stakeholders". This is generally hard to do especially in a vendor-client relationship.
Also a vendor PM in a fixed bid project has a different agenda- "deliver fast" than the business. Once again, he faces the problem that he cannot re-define scope on his own.
Thus both methods suffer in a fixed bid agile process based project. In a time and materials model this would work but then that is no different for a company to hire contractors.
The point I am trying to make is that a lot of the so called "Software Development" models do not take into account commercial considerations when multiple companies work together to make software. Their assumption of co-located teams working in synchronization and harmony breaks down in many such scenarios.
What I think is needed is that newer models that take into account - "Commercials" and "Organizational Boundaries" need to be developed to increase the success rates of IT Projects around the world.
