“Is the Criticism of Rated Criteria in Public Procurement for Increasing the Risk of Overpaying for Public Contracts Justified?” (Based on the research paper “Money for Nothing: How Quality-Price Trade-offs in Bid Scoring Increase the Risk of Overpaying for Public Contracts” by Janet Izatt and Timothy Mullett)

Arguments Supporting the Criticism (Why Rated Criteria May Lead to Overpayment)
Subjective Judgement Increases Variability
The use of non-price criteria often relies on subjective evaluation, which introduces variability (noise) into scoring. Even small inconsistencies in quality assessment can disproportionately affect final bid outcomes.

Compensatory Scoring Creates Arbitrary Trade-offs
In rated criteria, higher prices can be offset by better quality scores. However, the paper demonstrates that these trade-offs are not anchored in real value differences-leading to situations where minor perceived quality advantages result in large price premiums.

Measurement Mismatch
Price is measured on a ratio scale (objective), while quality is often assessed on ordinal scales (subjective). Combining them without addressing their differing natures violates measurement theory and can distort procurement outcomes.

Rounding and Scoring Mechanics Can Skew Results
The arithmetic methods used in aggregating evaluator scores—such as averaging, rounding, and scoring cut-offs-can further amplify noise, leading to the selection of higher-priced bids with negligible or no real quality advantage.

Risk is Heightened in Tenders Without Negotiation
Without a negotiation phase, there is no opportunity to recalibrate price-quality trade-offs, making any scoring distortion final and potentially locking in overpayment.

Arguments Against the Criticism (In Defense of Rated Criteria)
Encourages Quality-Oriented Procurement
Rated criteria allow contracting authorities to emphasize long-term value and innovation over just cost minimization. This is particularly important in complex or service-based contracts where quality directly impacts outcomes.

Reduces Risk of Low-Quality Delivery
Pure price-based selection may incentivize underbidding and lead to poor performance, cost overruns, or contract failure. Rated criteria serve as a safeguard against such risks by rewarding bidders with credible, high-quality proposals.

Supports Policy Objectives and Social Value
Non-price scoring enables the incorporation of broader public policy goals-such as sustainability, local economic development, or SME participation-which may not be reflected in price alone.

Flexible Tool When Designed Well
The paper critiques the implementation of rated criteria, not the concept itself. When quality assessment is structured rigorously (e.g., using confidence-weighted scoring, clear rubrics, and robust moderation), the risks can be mitigated.

Useful in Competitive Markets with Comparable Price Ranges
In mature or commoditized markets, where price ranges are narrow, rated criteria can be a legitimate way to distinguish bids on value-added services, innovation, or responsiveness.

Conclusion: A Qualified Justification
The criticism of rated criteria as increasing the risk of overpaying for public contracts is justified in cases where the evaluation process is weak, subjective scoring is not well-controlled, and compensatory trade-offs are mechanically applied without scrutiny—as evidenced by the research paper.

However, when designed and executed with care, rated criteria can still be a powerful tool to achieve best value rather than lowest cost. The problem lies more in how rated criteria are implemented than in whether they should be used at all. The findings thus support a call for reform, not rejection-emphasizing the need for improved scoring frameworks, evaluator training, and safeguards against overpayment risks.

One of the most widespread phenoma in public procurement is the inclination to “reinvent the wheel”. Having just browsed over the paper (I promise I will be back with more detailed comments once I read it more carefully) I’d say that the authors are just 20 years late in tackling the thorny issue of how to apply rated criteria, that is, how to rank bids with financial and non-financial dimensions. Rajeev has already provides many avenues for a proper thinking so I will limit myself to state an almost obvious consideration: since there exist many ways of ranking quality and price pairs then bidders’ incentives to formulate their bids DO depend upon the specific evaluation algorithm so does the outcome. Even the concept of “overpaying” is a hollow one: overpaying with respect to what?? Say that a market price exists for a particular commodity, service or even civil work. Then the risk of “paying too much” can and should be kept at bay by using a “reserve price” (ceiling price) and does not have anything to do with bids evaluation.
More to come…

A similar blog is on the WB site.My comments on the blog are reproduced below-
—
On the use of rated criteria, at least in India I feel one has to proceed with caution where an element of subjectivity is introduced in bid evaluation.Even if not misused by the Purchaser,losing firms can create havoc with complaints. QCBS would be ideal for complicated contracts as in Nuclear Powr corporation etc. However it is not appropriate for routine works etc.There could be a case of giving marks to engineers/Key staff . Commitments made in bids may not be followed in actual deployment etc.Why the caution? Apart from the issues mentioned above, the role of AI which emerged later, changes the whole picture.
There is the possibility of AI overpowering/undermining the bid process by producing ideal english/data for bids (mostly not verifiable.).

Here are some more comments from WB experts-
VinaySharma –There has been a growing body of commentary—including a thoughtful blog post by Krish titled “Rethinking World Bank Operational Procurement”—raising concerns around the recent policy shifts in procurement practices. Having followed this debate closely, I am concerned that the rationale behind the decision to mandate broader use of rated criteria does not appear to be fully supported by robust analysis. In particular, the data used to justify the shift seems to warrant closer scrutiny.
While there has indeed been increased use of rated criteria within the EU, much of this has occurred in the context of Innovation Procurement or Framework Agreements, where the overall contract value may significantly exceed the actual drawdowns. My own analysis of the EU procurement database yielded results quite different from those cited, suggesting a need for greater clarity and nuance in interpreting these trends.
In my advisory work with several countries across Europe and Central Asia, I’ve encountered widespread concern about the implications of mandating rated criteria for all international tenders. Combined with the recent reduction in thresholds, this change effectively brings a much larger volume of Bank-financed procurement under the rated criteria regime—including contracts built around input-based specifications, where such an approach may not offer clear benefits. In practice, some countries have resorted to scoring all technically qualified bids within a very narrow band, thereby neutralizing the intended evaluative function of the criteria.
The trend toward overreach appears to be extending to consultancy contracts as well. I’ve observed some large consulting assignments—where QCBS would ordinarily be appropriate—being evaluated using weightings of 90% technical and 10% financial, alongside a minimum qualifying technical score of 90%. While presumably well-intentioned, this structure risks undermining the competitive nature of the process. In such cases, even modest differences in technical scoring can allow significantly more expensive bids to succeed, reducing the influence of price to a near-negligible level.This raises several issues of concern:
Erosion of Cost Consideration: When the technical score dominates the evaluation so completely, marginally superior technical proposals may secure contracts despite substantial cost differentials—particularly in assignments not materially different from standard contracts in terms of complexity.
Low Price Sensitivity: The evaluation model inherently deprioritizes cost, potentially leading to premium payments without a commensurate value gain.
Distortion of Competition: The intended competitive balance is skewed, and the process edges toward a form of direct selection under the guise of QCBS.

Vinay had referred to Krishna kumar’s piece in his mail.I am reproducing that with comments of Armando,Tes etc
.
“Krish Krishnakumar
Mentor | Trainer | International Procurement Consultant | ex-World Bank Practice Manager-Procurement | ex-General Manager, Export/Procurement Agency South Africa | ex-Head of Procurement, Large Public Sector, Zambia3mo Edited𝐑𝐞𝐭𝐡𝐢𝐧𝐤𝐢𝐧𝐠 𝐖𝐨𝐫𝐥𝐝 𝐁𝐚𝐧𝐤 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐏𝐫𝐨𝐜𝐮𝐫𝐞𝐦𝐞𝐧𝐭 In a rapidly evolving world, continuous improvements in procurement are essential for efficiency, transparency, and effectiveness. Recognizing this, the World Bank conducted extensive global consultations between 2013 and 2014, leading to a significant revision of its Procurement Framework in 2016. This reform empowered borrower agencies with greater autonomy while reducing the number of “prior reviews.” However, recent changes introduced by The World Bank, such as mandating a minimum quality weight of 50% in rated criteria for most international procurements, lowering thresholds for prior reviews, and increasing centralized oversight through the HQ-based Operational Procurement Review Committee (OPRC), seem to reverse progress. These measures contradict key objectives of the 2016 reforms: Strengthening and utilizing borrowers’ procurement systems. Reducing prior reviews by increasing thresholds. 𝐓𝐡𝐞 𝟓𝟎% 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐌𝐚𝐧𝐝𝐚𝐭𝐞 𝐫𝐞𝐩𝐫𝐞𝐬𝐞𝐧𝐭𝐬 𝐚 𝐬𝐡𝐢𝐟𝐭 𝐚𝐰𝐚𝐲 𝐟𝐫𝐨𝐦 𝐚 ‘𝐅𝐢𝐭 𝐟𝐨𝐫 𝐏𝐮𝐫𝐩𝐨𝐬𝐞’ 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡. Furthermore, a quality-based approach must be complemented by robust reforms in bidder prequalification. Even with early market engagement, true improvement starts with modernizing prequalification to ensure only capable firms participate in high-value, complex contracts. 𝐓𝐡𝐞 𝐍𝐞𝐞𝐝 𝐟𝐨𝐫 𝐒𝐦𝐚𝐫𝐭𝐞𝐫 𝐁𝐢𝐝𝐝𝐞𝐫 𝐐𝐮𝐚𝐥𝐢𝐟𝐢𝐜𝐚𝐭𝐢𝐨𝐧 Bidder qualification is critical in public procurement, particularly for large, complex projects. Despite rigorous numerical evaluations, many prequalified bidders still fail to deliver. The current framework remains overly focused on compliance rather than actual capability, leading to inefficiencies and increased risks. Given the large number of high-value contracts financed by the World Bank, it must lead in modernizing bidder verification. A key step could be a blockchain-based platform for digital submission and verification of qualification requirements, which would: Enhance data transparency and integrity. Enable seamless verification of bidder credentials. Reduce reliance on outdated, paper-based methods. By pioneering such an initiative, the World Bank could set a precedent for other Multilateral Development Banks (MDBs), potentially leading to a global blockchain platform for bidder qualifications. 𝐀 𝐂𝐚𝐥𝐥 𝐭𝐨 𝐀𝐜𝐭𝐢𝐨𝐧: 𝐂𝐨𝐥𝐥𝐚𝐛𝐨𝐫𝐚𝐭𝐢𝐨𝐧 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐅𝐮𝐭𝐮𝐫𝐞 Experts in AI and blockchain across both the public and private sectors can offer innovative solutions and strategic guidance for this transformation. As an initial step, the World Bank could take a proactive approach by organizing a workshop that convenes key experts to brainstorm and develop a roadmap for integrating AI and blockchain into various aspects of the operational procurement process. What are your thoughts?The content covered in training sessions for Borrowing/Implementing agencies is reflected in the slide below. The 50% quality mandate in most international tenders implies that nearly all such tenders under World Bank-financed investment projects globally will be high-cost and carry a high risk of vulnerability, regardless of procurement type, country, project context, or specific needs. 𝐓𝐡𝐞 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧 𝐢𝐬: 𝐒𝐡𝐨𝐮𝐥𝐝 𝐰𝐞 𝐭𝐫𝐞𝐚𝐭 𝐧𝐞𝐚𝐫𝐥𝐲 𝐚𝐥𝐥 𝐢𝐧𝐭𝐞𝐫𝐧𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐭𝐞𝐧𝐝𝐞𝐫𝐬 𝐚𝐜𝐫𝐨𝐬𝐬 𝐖𝐨𝐫𝐥𝐝 𝐁𝐚𝐧𝐤-𝐟𝐢𝐧𝐚𝐧𝐜𝐞𝐝 𝐩𝐫𝐨𝐣𝐞𝐜𝐭𝐬 𝐠𝐥𝐨𝐛𝐚𝐥𝐥𝐲 𝐚𝐬 𝐡𝐢𝐠𝐡-𝐜𝐨𝐬𝐭 𝐚𝐧𝐝 𝐡𝐢𝐠𝐡-𝐫𝐢𝐬𝐤 𝐛𝐲 𝐝𝐞𝐟𝐚𝐮𝐥𝐭, 𝐚𝐩𝐩𝐥𝐲𝐢𝐧𝐠 “𝐨𝐧𝐞-𝐬𝐢𝐳𝐞-𝐟𝐢𝐭𝐬-𝐚𝐥𝐥 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡”? Ironically, the World Bank moved away from a “one-size-fits-all” approach in 2016 as part of its Procurement Reform. 𝐀𝐧𝐝 𝐰𝐡𝐚𝐭 𝐢𝐬 𝐝𝐫𝐢𝐯𝐢𝐧𝐠 𝐭𝐡𝐢𝐬 𝐜𝐡𝐚𝐧𝐠𝐞, 𝐰𝐡𝐢𝐜𝐡 𝐬𝐞𝐞𝐦𝐬 𝐭𝐨 𝐜𝐨𝐧𝐭𝐫𝐚𝐝𝐢𝐜𝐭 𝐭𝐡𝐞 𝐟𝐮𝐧𝐝𝐚𝐦𝐞𝐧𝐭𝐚𝐥 𝐩𝐫𝐢𝐧𝐜𝐢𝐩𝐥𝐞 𝐨𝐟 ‘𝐅𝐢𝐭 𝐟𝐨𝐫 𝐏𝐮𝐫𝐩𝐨𝐬𝐞’?No alternative text description for this imageArmando Araujo
Chief Technical Officer of Belo Monte Transmissora de Energia3moKrish The use of 50% for technical evaluation is a big challenge for the great majority of borrowers and probably will create much more complaints from losing biddersLike
Reply
2 Reactions
Tesfaye Ayele, graphic
Tesfaye Ayele
—3moThank you Krish. The recent changes to the 𝐖𝐨𝐫𝐥𝐝 𝐁𝐚𝐧𝐤 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐏𝐫𝐨𝐜𝐮𝐫𝐞𝐦𝐞𝐧𝐭 approaches are mostly driven by IEG’s findings and recommendations for improvement. I fully agree with your insights & concerns from principles of ‘𝐅𝐢𝐭 𝐟𝐨𝐫 𝐏𝐮𝐫𝐩𝐨𝐬𝐞’, leveraging technology & shifting Bank’s support to more value adding roles than transaction-based approach.”

Thank you, Mr. Krishnan, for drawing attention to my LinkedIn post and the insightful comments it received. Your remarks are especially pertinent in light of the UK paper currently under discussion. We have observed numerous unintended consequences arising from the use of rated criteria in World Bank–funded projects.

To be clear, I am not suggesting that introducing rated criteria in proposal evaluation is inherently a bad idea. However, moving from a “fit-for-purpose” approach to a “one-size-fits-all” model is, in my view, a step in the wrong direction.

Vinay Sharma’s response effectively highlights several concerns we encounter in consultant selection – QCBS as an example. About two decades ago, the World Bank conducted an internal (non-public) study, which – based on detailed data analysis — concluded that Quality-Based Selection (QBS) tended to yield better results than Quality and Cost-Based Selection (QCBS). Many firms have learned to game the system, securing inflated technical scores that enable them to win contracts at significantly higher prices. A major issue is the lack of any systematic benchmarking or measurement to validate whether the high technical scores—and consequently the higher costs—are reflected in better quality outputs.

This problem is not limited to consulting services. It extends to the use of rated criteria in the procurement of goods and works. We often refer to key performance indicators (KPIs) tied to promised features, sustainability elements, efficiency gains, or life-cycle cost benefits—factors that justify awarding higher technical marks. These higher scores can tip the scale in favor of more expensive proposals. Yet, I have not come across any robust cost-benefit analysis that verifies whether these higher-priced contracts actually delivered value over the life of the procured goods or equipment. In practice, once the contract is awarded, attention shifts to implementation, and the verification of promised quality features becomes partial at best. A full cost-benefit evaluation rarely occurs—perhaps because such assessments are inherently complex and resource-intensive.

I recall a conversation I had a few years ago with a Minister of Finance from an African country. I tried to make the case that paying a higher upfront cost for better quality products could lead to long-term savings and durability. His response was candid and poignant: “When people are dying of hunger and the poverty rate is high, there is no justification for spending millions more today in the hope of possible benefits sometime in the distant future. I would rather use that money to feed the poor and alleviate poverty now.” I would like to add that this example and topic bears no relevance to the ongoing discussions and efforts to address global climate change, which in my view needs action now to avert a disaster in the future.

Thanks to Vinay Sharma for sharing this article: https://www.uu.nl/en/news/government-can-save-almost-a-billion-euros-a-year-through-smarter-procurement

Government can save almost a billion euros a year through smarter procurement:

“In addition to the savings we quantified, there are probably other savings opportunities that we have not been able to calculate in these studies. Think of purchasing less often and critically reconsidering high quality standards where they do not provide high demonstrable added value.

This concerns, for example, situations in which existing solutions can be used for a longer period of time or situations in which very strict quality requirements are imposed in tenders, which, although guaranteeing quality, provide little added value compared to lower, cheaper alternatives. Especially in times of budget cuts, it is important to also take a closer look at these kinds of broader strategic choices.”