First, let me start by stating that it’s perfectly reasonable for a client—whether they are a business unit of a major corporation or an individual venture—to want to produce a plan that involves discrete costs, timelines, and outcomes. This is certainly a precedent that has been set by other industries—such as the construction and engineering of roads and buildings—where standardized processes, regulations, and historical factors allow for a relatively high level of certainty when it comes to outcomes and costs. Many of these considerations and constraints, however, are absent in the commissioning of software.
Why Does This Matter?
Initial estimates are likely very wrong (and often make everyone look bad).
Many clients expect to compare vendors using a variety of criteria with total project cost often weighted heavily. This results in a series of problems and threats to the likelihood of the success of the project due to the following:
- Apples vs oranges. Estimates from different vendors are heavily based on implicit assumptions they’ve made about the project, and the processes they’ll use to complete it. Since both the assumptions and processes can vary wildly vendor to vendor, it’s nearly impossible to compare.
- Introduction of risk. The costs of business strategies that require software development are inherently difficult to project at the outset and thus riskier than other strategies whose costs are more reliably projected.
- Bait and switch. Aggressive sales personnel and desperate vendors will push for the sale, representing an estimate as a true cost and hoping to successfully manage known problems as they occur down the road.
Why Does it Happen?
No one really understands a project in the beginning (but there’s pressure to ignore this fact).
- Technology, like, changes man. A priori estimation assumes the solutions and their level of effort are known. In reality, most non-trivial software development requires the developers to do or use something they’ve never done before.
- Requirements suck. For a lot of reasons. Let’s go through some:
- Nine-times-out-of-ten the client is not technically saavy or experienced in software development enough to produce specific requirements.
- Predetermined requirements put out for bid do not allow a vendor to research and propose creative solutions before producing an estimate.
- One-time-out-of-ten the client is technically savvy and overproduces requirements that are based in a different ontology and methodology than of the team that is expected to consume them. Moreover, detailed requirements are difficult and time-consuming to work through.
- No right answer. Developers know that, at best, the estimation process at such an early stage of a project is highly subjective. This leads them to base their estimates on factors like: the individual on the hook trying to cover themselves, the personality of the developer, the company needing business, the ‘coolness’ factor of the project, and a host of other factors that have little to do with the actual and unknown level of effort required.
- Estimation is expected too early. Expecting close estimates before an engagement is established is faulty. A great visualization of this concept is demonstrated by the ‘cone of uncertainty’ below.
The cone of uncertainty introduces a key insight, discussed in more detail in this article by Construx. The insight is this:
An important—and difficult—concept is that the Cone of Uncertainty represents the best case accuracy it’s possible to have in software estimates at different points in a project. The Cone represents the error in estimates created by skilled estimators. It’s easily possible to do worse. It isn’t possible to be more accurate; it’s only possible to be more lucky.
This brings us to the true problem with having to estimate total costs of any dev project in order to win a bid: to get it right, you have to be lucky—or prioritize gamesmanship over attempted accuracy in the production of estimates.
So, What Can You Do About It?
Multi-phase projects that involve re-evaluation and re-estimation (and allow you to test client developer chemistry).
The answer lies in one simple (and possibly tautological) truth:
The further along in the project you are, the more you know about the project.
The “Cone of Uncertainty” example above is a perfect illustration of this truth that the more a client and developer knows about a project and its roadmap to completion, the closer to a true estimate they can get.
It’s our recommendation, then, to engage the developer in a multi-phase project, one that allows for flexibility, redefinition, and re-estimation. This can be done by engaging the developer on scoping sessions and/or rapid prototyping workshops to define project requirements at the outset. We discuss rapid prototypes in this blog post, here I want to explain this answer further.
Evaluating the Answer
Multi-phase Projects for Software Development: Pros and Cons
Many if not most software projects do not incorporate discrete pre-production projects, I believe this is often due to misunderstanding the pros and cons of such an approach.
Perceived cons of paid pre-production planning projects:
- Without a ‘sticker price’ it is difficult to compare vendors. As noted above, comparing vendors on estimates generated up front will naturally vary to such a degree that it is a worthless measurement of the actual cost of a project. Compare vendors based on relevant experience, quality of their work, proposed process, and how you feel about the team that you’d work with.
- Without a fixed bid we can’t control costs. Costs aren’t something that get ‘controlled’ by a set-and-forget budget arbitrarily defined before the project, it’s something that needs to be part of the conversation and monitored regularly. Fixed bids are a whole other conversation, but a responsible vendor will understand that fixed bids need to account for risk and thus be marked up significantly. If I know that the initial estimate variance is high, then the subsequent markup to the estimate for the assumption of risk needs to be equally as high. Companies that don’t take this approach either go out of business, issue constant change orders, or figure that they can sacrifice quality to get out of the project as quickly as possible.
- The scope has already been defined. As noted above, scope documents created internally overlook the approach, process, and expertise of the potential vendor and often are underspecified. Furthermore, in most RFP processes the vendor is limited in the ways that they can further engage the issuer exacerbating the problem of estimate variance. This is why you often see agencies flat out refuse to respond to RFPs. If RFPs are required by your company or legally, I recommend putting out the strategic consulting/UX prototyping project out for bid first.
Pros of paid pre-production planning projects:
- Scope and budget are now variables. Projects are defined by their constraints. Often the vendor and client negotiate on scope/functionality, costs, and timeline—with the scope/cost relationship being primary. However, there is a fourth constraint, quality, which is often implicit. As the explicit constraints becomes fixed—costs are fixed, scope is non-negotiable, deadline is hard—it is usually the quality and often overall performance of the project that is sacrificed. When the vendor can work hand in hand with the client in defining the project we can better align the project scope with the desired budget and overall goals of the project.
- Low cost-of-entry for working with a vendor. According to a recent study, most application development project are not successful with 31.1% of projects being canceled (failed) and 52.7% of projects over budget, over time, and/or under delivered (challenged). 1 For any critical or large project it is important to select a vendor who is a fit and who can deliver. Just like the developer has a better sense of a project the further in they go, the client will better understand the developer to more they work together and have a better sense of project fit. Working with a vendor on an up-front strategy or planning project lets you see how they work once the sale is made. It might be painful to switch gears after a few weeks or months but doing so when the partnership is clearly not a fit will save time, money, and headaches down the road.
- Insights and documentation are often valuable even if you switch vendors. In the unfortunate circumstance that you decide to move on from a vendor after the initial project it is likely that significant portions of the work will be able to be reused.
Ultimately, by leveraging stand-alone pre-production projects clients are able to reduce risk while developing a collaborative working relationship with the developer. At S4 these projects usually take the form of rapid prototyping. This is where we work through the high-level strategy with the client before going through an intensive workshop where we create the user experience of the application while documenting the constituent tasks that will be required to develop it. With large tasks broken down into smaller ones and our increased understanding of the application, subsequent project cost and timeline estimates are more accurate.
Co-founder and Creative Partner at Station Four