Can Governments Pick Quality Suppliers?

Yes, they can. But in practice they mostly don't - 66 % of suppliers get chosen, just because they are the cheapest. This EU-wide analysis provides a novel insight into real supplier selection processes.

If you shop for a yogurt, you always buy the cheapest one, right? You don't? Maybe because you know, that the cheap white thing hardly is a yogurt. Probably it is impossible to make a decent one for such a low price. That is a common instinct we employ while shopping. Unfortunately, our governments don't have such an instinct. They need to rely on objective & transparent criteria - which however often boils down to comparing prices. As a result, we save some public money. But we also risk getting a strange white thing instead of yogurt: poor roads, confusing IT systems or bad quality of public transport.

The ability to select decent contractors is crucial for most of the government agenda. Whether it is building highways, drafting bills or supplying medicines to hospitals - all that gets often outsourced to private companies. How are these suppliers typically chosen? That is the main question of this study.

Note: besides our topic of award criteria (how we choose the best yogurt supplier), two other relevant filters are typically applied to ensure a reasonable quality. The product specification is the most straightforward (buyer defines demanded qualities of a yogurt). The qualification then sets requirements on a supplier (buyer requires proven ability to make a yogurt, certifications, etc.). Those two filters, however, do only set out necessary minimum requirements. They do not allow for any flexibility to prioritize a better product or service offered. If a buyer relies on those (which is frequently the case), he typically ends up with exactly the minimal quality of product and minimal qualifications of supplier he defined. That is not always optimal.  

What criteria are used in practice?

Let's take a look at one of the biggest procurement market out there - EU. We cleaned and analyzed a huge chunk of tenders from its member countries (more on the data processing later), to get the following picture:

Is it ok to use only price? This, of course, depends on the type of goods procured. If you buy electricity, it's ok to get it cheaply, there are not too many quality issues to discuss. Cars? Maybe. Yogurts? No! :). Lawyers .. seriously? For what type of job would you be content with the cheapest lawyer you can get? Though in practice such tenders are not uncommon. It is hard to generally judge here, what we can do is to compare. If some countries pick lawyers using price only and others don't, then someone might be doing it wrong. Does that happen?

International comparison - East vs. West

The practice within EU is anything, but unified. Despite the common legislation, we see deep differences between member states. Following map demonstrates, how often the qualitative criteria are used (either quality, qualification, life-cycle costs, environment or social). Apparently, there is quite remarkable east-west division, where "old" EU-15 member states led by UK and France routinely use qualitative criteria, whereas in former eastern block, price is used as a single criterion in over 90 % of cases (except Poland).

Why is that? Three explanations are the most prominent:

1. EU funds. New member countries get much more structural funds, together with various conditions and additional audit layer, leading to stress on formal correctness. Our other results, however, indicate that this is not the case: EU-funded tenders use more qualitative criteria than their counterparts in the same country (typically life-cycle costs) in 2016+.
2. Institutional differences. Anti-corruption and pro-transparency efforts are more severe in new member states, as they are responding to a more powerful public distrust in institutions. Transparency of the selection process is often a replacement for an underdeveloped audit, which should prevent corruption and inefficiency.
3. Expertise. Choosing the cheapest bid is simple. For evaluating more complex criteria, you need more time, but also more experienced staff - ideally experts in the given field, who have done a couple of similar tenders before. Such learning takes time and personal stability. For additional argumentation see for example EC Guidance for Procurement Practitioners point 4.2)

It needs to be said, that "overusing" of price criteria is not a priori a bad thing. As options 2 and 3 suggest, its rather an adaptation to public demand, rushing of projects and/or experience of civic servants in procurement or audit. Nonetheless, such adaptation still does take its toll on quality - which is sacrificed to achieve a higher level of transparency and accountability. (Of course, transparency can be achieved jointly with quality, though this might again require better personnel, institutions, etc...).

Ok, how to improve the situation?

It's not good enough to tell buyers: please use qualitative criteria more. Many have done that, many have failed. To name one example: the new EU procurement directive gave various reinforcements to multi-criteria evaluation, it also explicitly stressed the possibility to use various qualitative criteria ( namely social, ethical and environmental criteria ). It became effective in 2016. Following graph depicts its weak impact on the use of qualitative criteria: we saw a mild short-term increase, but the use of quality criteria is still below rates achieved a few years ago.

The point here is, that legislation is not the main tool here. Law-makers are frequently blamed for something, which is, in fact, a deeper problem of insufficient experience and learning capacities of public buyers - as noted above.

So, can we somehow speed up the learning process? This study brings a modest contribution to such goal: we provide an extensive list of examples, how various criteria types are used in different fields (Technical Annex - examples). This can be a starting point for a buyer or policymaker (concrete specification behind the criteria needs to be requested from the given buyer though).

This also brings a solid argument against "this can't be done" skepticism, present in many countries (including our own - Czechia). Ecological criteria to buy cars? Sure. Comparing personnel track records for legal services? No problem. Fix the price, and choose a piece of medical equipment entirely based on technical features? Yes! For all that we can provide many examples from other countries - Technical Annex provides an extensive list. That is especially relevant in EU, where countries are sharing most of the legislation for the large tenders - however, we believe that applicability goes beyond the EU.

Think this is relevant? Thanks for sharing!

P.S. - Cool numbers, where did you get them?

We are a team of economic and IT specialists, who focus on public procurement. Our unique database allowed us to do this within a relatively small project (except for step 1):

1. Get data on 18 million public tenders in EU since 2006 (we did that through digiwhist.eu project jointly with the University of Cambridge and other partners).
2. Translate the 100.000 most frequently used criteria into English (covering about 85 % of all tenders) using machine translation.
3. Manually categorize most of them into the following categories: price, quality, life-cycle costs, qualification, deadlines, contract details, environment, social (or conclude that no relevant information is provided). More concrete categories were prefered to more general. Detailed definitions in Technical Annex.
4. Prepare this nice analysis (a more detailed study was prepared for the Czech Ministry of Social Affairs, which kindly allowed us to publish these results).