Tallinn, 18 February 2015 – Stage agencies commissioned over 1300 studies to a value of 41 million euros from 2010–2013 Studies are primarily used as background information in the development of policies, but the National Audit Office is of the opinion that they could be used more in knowledge-based policy-making. The audit indicated that the capacity to commission studies is uneven in state agencies, the manner in which they are commissioned is not properly coordinated and the accessibility of the commissioned studies should be improved via the creation of a national study database.

The audit of the National Audit Office revealed that studies are scarcely used when decisions are made. The majority (63%) of the 56 studies reviewed in depth in the course of the audit had been commissioned as background information in order to obtain an overview of problems or sectors. In the case of many studies, however, it is unclear whether and to what extent they were used. The use of a study in the development of policies was clearly identifiable in the case of 46% of all studies reviewed (the study being referred to in the policy document or development plan). There were signs that the remaining studies had been used to a certain extent, but specific connections were indirect and difficult to identify.

The accessibility of the studies commissioned by the state should be improved. Irrespective of the similar website structure used by ministries, finding studies is difficult, because the ministries upload them to different parts of their websites. Obtaining a comprehensive overview of the studies commissioned by the state would require a lot of manual work. A central national database of studies which also contains abstracts of and keywords related to the studies should be created in order to solve the problem. A joint database would allow the general public to obtain a full picture of all of the studies commissioned by the state and guarantee that the studies are accessible over time. For example, the interviews carried out during the audit revealed that many studies have been prepared within the scope of programmes financed from the European structural funds, but that once the programmes end, the relevant websites and analyses also disappear.

The audit further revealed that not all of the studies have been published on websites. Sometimes this was simply forgotten; sometimes the studies were regarded as being meant for internal use. A few of the people interviewed by the National Audit Office said that in some cases the studies were left unpublished for political reasons. For example, there was a case where the plan was to publish the study almost five months later because the minister wished to present it at a politically suitable moment.

The terms of reference of commissioning studies are not always well thought through. An analysis of the terms of reference of the studies in the area of administration of five ministries carried out in the course of the audit revealed significant differences in the capability of state agencies to commission high-quality studies. In many cases, the objective or questions of the study had not been clearly set in the terms of reference. Background descriptions of the studied problem were often scarce and there was no information about how the results of the study were to be implemented.

Reasonable use of public procurements causes problems when studies are commissioned. Many state agencies have gone down the path of least resistance and used the cost of the study as the only criterion for selecting the best tender. Preferring the lowest cost to the assessment criteria that allow for the submission of the tender that is the best economically casts doubt over the possibility of achieving a high-quality result. When the goal is to obtain a simple survey service, a highly detailed description of methodology (provided that its use is monitored) makes it possible to organise a price-based competition and, as a result, to commission the study, but price has also been used as the only criterion in cases of substantive procurements of studies, such as the identification of the competency of and need for workforce.

As they are afraid of substantive assessment of procurements and possible disputes, many state agencies do not commission answers to their questions with the studies, but try to offer a specific methodology themselves. This is why research companies cannot use their know-how to carry out the study in the most efficient manner, but have to follow the methodology prescribed by the contracting authority. However, state agencies often do not have enough experience in research methodology, which is why the methods they suggest in the terms of references of various studies are questionable. Unprofessional procurements are characterised by excessive precision of methodology. For example, exactly 108 interviews had to be carried out within the scope of one study. The representatives of research companies also mentioned unrealistic expectations regarding population coverage (e.g. 50% of the population having to be interviewed).

Although the level was uneven across ministries, it can be said that the terms of reference were the best in the Ministry of Social Affairs, as they almost always contained a detailed description of the problem and clear questions to be asked in the study. The procurements made by the Ministry of Social Affairs can also be highlighted, as the price typically comprised 30% of the score when tenders were evaluated and the substantive solution offered by the tenderer was considered more important. In the case of the latter, the tenderer’s understanding of the objective of the analysis, the description and justification of the process of analysis and the project plan were also evaluated.

Activities to improve the planning and commissioning of the studies have not been carried out. The audit revealed that although the Government Office had assumed the task of developing a system for the distribution of study results that support policy-making and implementation and to make the planning of studies more transparent, most of the promised activities have still not been carried out. There are still no uniform agreed principles in the state which would guarantee that the planning and commissioning of studies is more transparent and better coordinated.

The National Audit Office made several recommendations on the basis of the audit results about guaranteeing that the planning, commissioning and use of studies become more transparent. The ministries and the Government Office largely agreed with the recommendations made by the National Audit Office and they have already started implementing various activities to make the commissioning and publication of studies more efficient.

***

Background

The National Audit Office assessed whether and how studies are used in the decision-making process, whether the studies are accessible to the general public and what the capability of ministries is in commission high-quality studies.

The National Audit Office presumed that the money spent on commissioning studies was expediently used if a study yielded benefits. A study must produce information that helps to plan activities and make decisions. Also, it must be possible for the general public to understand whether and how the results of studies commissioned using public funds have been used.

Since 2012 Estonia has been one of the 64 countries that has joined the Open Government Partnership – a global effort to make governments better. Open government means acting honestly, transparently and in dialogue with citizens. It is important for the general public to see and understand what governments rely on when they make decisions and whether spending money on commissioning studies has justified itself.

A ministry or an agency in its area of administration is responsible for commissioning and financing studies. The Government Office has assumed the obligation of improving the planning and distribution of studies in the state as a whole and it is carrying out the programme “Development and Streamlining of the System of Long-term Planning of the Government’s Work”.

The audit covered the studies commissioned by ministries and state agencies (incl. foundations) from 2010-2013. Works in which a certain area or problem was analysed in depth and which were used to plan and/or implement a policy or activity were regarded as studies in the audit.

An overview of the number and cost of the commissioned studies is given in Table 1.

Table 1. Number and cost of studies commissioned from 2010-2013 by area of administration

* In addition to ministries, the information request also covered the following agencies (with the agencies given in italics not having commissioned any studies): Area of administration of the Ministry of Education and Research - Language Inspectorate, National Archives, Estonian Youth Work Centre, Estonian Research Council, Innove Foundation, Archimedes Foundation, Estonian Qualifications Authority, Sports Training and Information Foundation. Area of Administration of the Ministry of Justice - Data Protection Inspectorate, Patent Office. Area of administration of the Ministry of Defence - Defence Forces, Defence Resources Agency. Area of administration of the Ministry of the Environment - Estonian Land Board, Environmental Board, Environmental Inspectorate. Area of administration of the Ministry of Culture - National Heritage Board, ‘Our People’ Integration and Migration Foundation. Area of administration of the Ministry of Economic Affairs and Communications - Competition Board, Civil Aviation Administration, Road Administration, Consumer Protection Board, Maritime Administration, Technical Surveillance Authority, Republic of Estonian Information System Authority, Enterprise Estonia, KredEx. Area of administration of the Ministry of Agriculture - Agricultural Board, Estonian Agricultural Registers and Information Board, Veterinary and Food Board, Rural Development Foundation. Area of administration of the Ministry of Finance - Tax and Customs Board, Statistics Estonia. Area of administration of the Ministry of the Interior - Police and Border Guard Board, Rescue Board, Estonian Rescue Service, 15 counties. Area of administration of the Ministry of Social Affairs - State Agency of Medicines, Social Insurance Board, Health Board, Labour Inspectorate. Also the Government Office, Estonian Unemployment Insurance Fund and Estonian Health Insurance Fund.
Source: responses of agencies to the information request made by the National Audit Office

Analyses and conclusions were made on the basis of 56 studies commissioned by five ministries in the course of the audit (Ministry of Education and Research, Ministry of Economic Affairs and Communications, Ministry of Agriculture, Ministry of the Interior and Ministry of Social Affairs) and their agencies.

A list of all studies commissioned from 2010-2013 has been published as an electronic annex to the audit report in support of open government. The list is based on a survey of all ministries and state agencies carried out in the course of the audit.

Toomas Mattson
Communication Manager, National Audit Office
+372 640 0777
+372 51 34 900
[email protected]
[email protected]
www.riigikontroll.ee
www.facebook.com/riigikontroll
www.twitter.com/riigikontroll

• Posted: 2/18/2015 10:00 AM
• Last Update: 11/10/2015 5:22 PM
• Last Review: 11/10/2015 5:22 PM