Trend Watch: beta testing – a new software procurement strategy?

In brief 6 min read

Software company Palantir Technologies was recently announced as the US Army's choice to deploy a complex combat intelligence system valued at US$800 million. This followed Palantir's commercial off-the-shelf solution initially being rejected in favour of competitor Raytheon's developmental, customised approach, then a re-do during which the competing bidders conducted on-field beta testing with the solution's intended users. In light of this result, and the rise of COTS IT solutions in Australia, we examine the potential of beta testing as a viable new procurement strategy in software tendering processes.

Key trends and takeaways

The rise of COTS solutions

Governments and industry are increasingly preferring to procure COTS software over custom-built solutions. Research on these configurable and customisable solutions indicates that purchasers of COTS solutions are likely to enjoy greater functionality and flexibility, not to mention cost savings when introducing and maintaining COTS software, compared to custom-built alternatives.

Increased interest in beta testing as a software procurement strategy

Organisations may potentially minimise the risk of procuring unsatisfactory solutions by introducing beta testing into their procurement strategy. Adding a beta testing phase to the procurement process allows purchasers to test the merits of a solution in real world conditions by giving providers the opportunity to configure their solutions to the purchaser's specific needs or desired outcomes.

The rise of COTS solutions

Over the past decade, commercial off-the-shelf (COTS) solutions have increasingly been offered in complex government and commercial procurements. In March this year, the Australian Government's Digital Transformation Agency confirmed its commitment to deploying COTS models, by announcing a new COTS Software category on the Software Licensing and Services Panel for ICT procurement.¹

COTS refers to existing software solutions that are readily available to numerous customers, including ordinary consumers (eg the Microsoft Office suite). Customised solutions, on the other hand, are designed and built for the purpose of creating a unique solution, usually tailored to a customer's needs.

COTS versus customisation

In 2016, IBM commissioned Forrester Consulting to evaluate the benefits of using COTS, as opposed to custom software solutions.² Forrester found that while custom-developed software promises targeted, fit-for-purpose solutions, it often fails to deliver in terms of flexibility and functionality. Based on a survey of 100 government agencies across the UK, France and Germany, results indicated that custom-developed software:

• was often more expensive, yet less functional and flexible, than COTS software;
• frequently failed to perform as expected, but updating or altering the system proved difficult due to the complexity of the custom solution; and
• provided unfamiliar systems and code for in-house IT teams, with the upshot that third-party systems integrators are often required for maintenance and upkeep of the software. This is not only a costly exercise, it also impacts the predictability of budgeting for future maintenance.

But is it really the case that COTS software is more likely to be fit-for-purpose than customised software?

According to Forrester's findings, one important benefit of COTS software is that it is more easily tested for both functionality and useability. This is because beta testing (ie testing the software with the actual intended audience) before committing to a solution, may be an option that is accessible, cost-effective and (for the would-be customer) desirable in complex procurement processes. Prospective customers are given the opportunity to trial several solutions in specific, real-world conditions before contracting with a provider for the most suitable solution.

It is important to note that COTS solutions can still be configured for a specific customer – ie providers can use existing tools within the application to meet specific requirements, without needing to write new code. The concept of configuration is distinguishable from the concept of customisation – ie where new code is written into the software to meet particular requirements, culminating in the delivery of a bespoke, non-COTS solution.

A case study: Palantir's COTS solution for the US Army

In March 2019, US software company Palantir Technologies was announced as the US Army's choice over competitor Raytheon to supply combat intelligence hardware and software valued at US$800 million.

The tender process dates back to 2015, when the Army first solicited bids to develop its Distributed Common Ground System for Army – a system that lets users gather and analyse information about enemy movements, weather and terrain to create detailed maps and reports in real time. After narrowing its choices to Palantir's COTS solution and Raytheon's developmental, customised approach, in 2015 the US Army awarded a contract to Raytheon.

Palantir challenged the decision to award the contract to Raytheon, in the US Court of Federal Claims, and later the Federal Circuit Court, arguing that the US Army failed its statutory obligation under the Federal Acquisition Streamlining Act (the FASA) to consider whether its needs could be met by commercial off-the-shelf technology solutions before settling on a developmental approach.

Palantir's challenge was successful, and the Federal Circuit Court ordered the US Army to repeat the acquisition bid process, this time with greater attention to its obligations to give proper weight to commercially available solutions. Breathing renewed life into a much-neglected statute, the decision marked a watershed for the application of the FASA in tendering processes. More importantly for Palantir, though, the decision gave the company a second bite of the cherry.

In round two, Raytheon and Palantir both competed in a software capability demonstration, which gave each provider the opportunity to test its solution with its intended users (an audience of army soldiers) who then provided feedback, allowing Raytheon and Palantir to refine their solutions according to their users' needs. This test-fix-test process (ie beta testing + configuration) allowed the Army to test-drive both solutions, and gave Raytheon and Palantir the opportunity to showcase their models and learn from would-be users.

Following the repeat bid process and the beta testing component, the US Army awarded the US$800 million intelligence contract to Palantir.

What does this mean for beta testing as a software procurement strategy?

The Palantir example concerned a high-value contract, which made it worthwhile for the US Army to conduct a lengthy tendering process and spend money on the testing exercise. Understandably, not all purchasers will have the time or capital to commit to such extensive methods of vendor assessment.

Still, the benefits of beta testing are obvious: the purchaser has the opportunity to test-drive a solution in real-world conditions; and providers have the opportunity to configure their solutions to the purchaser's needs. Beta testing during tendering may also give start-ups and new entrants the opportunity to unseat legacy providers by getting their foot in what might have otherwise been a closed door.

As pragmatic commercial solutions that meet user needs become increasingly critical to government and industry, the benefits of beta testing in complex software procurement should not be overlooked.