These days, government agencies rely, to a great extent, on information and communication technology (ICT) solutions in their daily operations; they store data in digital form and provide services via electronic channels. Therefore the number of software development projects required to develop and maintain e-services has increased considerably over the last decade, and access to EU funding has largely contributed to the increasing number of software developments.

What did we audit?

The auditors analysed whether and why the software development projects of government agencies fail. At the same time, they tried to highlight the best software development practices of various agencies and major success factors in implementing development projects.

In its audit the NAO analysed the development of nine information systems across the government areas of eight ministries: the Building Register (EHR) and the Management System of National Information Systems (RIHA) (both under the Ministry of Economic Affairs and Communications), the Estonian Research Portal (ETIS) (Ministry of Education and Research), the Enforcement Proceedings Information System for Bailiffs (e-Täitur) (Ministry of Justice), the Environmental Permits Information System (KOTKAS) (Ministry of the Environment), the Rural Development Funding System (MATS) (Ministry of Rural Affairs), the Public Procurement Register (RHR) (Ministry of Finance), the Social Protection Information System (SKAIS) (Ministry of Social Affairs), and the Identification and Proceedings Information System (UUSIS) of the Police and Border Guard Administration (Ministry of the Interior). In addition, the management of software development at the Ministry of Culture and the Ministry of Foreign Affairs was reviewed.

Why is this important for taxpayers?

The main incentive for the audit was the fact that considerable funds are spent on software development, and the developed software and information systems play an important role in public governance and service provision.

232 ICT projects that relied on EU aid and totalled 53.4 million euros were completed during the programming period 2007-2013. According to the latest budget forecast of the Digital Agenda for Estonia, a total of 223 million euros should be spent on ICT solutions in 2014-2020. For example, 46 million euros of EU funding is available for the development of smart infrastructure for services, and 99 million euros can be used for the development of public services during the current EU programming period.

Since the administrative procedures of government agencies and the provision of services to the general public are increasingly dependent on e-services, the damage arising from software development failures is far greater than just the money spent on such developments.

What did we find and conclude as a result of the audit?

The audit demonstrated that five out of nine development projects (EHR, KOTKAS I, MATS, RHR and RIHA) had been implemented efficiently in all material respects, or in other words, in line with the specified budget and time limits, and the developed software featured the required functionality.

The NAO finds that four projects failed (ETIS2, e-Täitur, SKAIS2, UUSIS) for the following reasons:

• ETIS2 - requirements for the information system were modified due to changes in the research domain, and these modifications had to be implemented during the development process; also, the development schedule was too optimistic.
• E-Täitur - the head of the development project ignored the users’ needs, the roles and tasks of project participants were not clearly defined, and cooperation between various parties was complicated.
• SKAIS2 - core processes were not outlined nor optimised, and there were frequent legislative amendments, which necessitated adjustments to the ongoing development work; the roles and tasks of project participants were not clearly defined, and cooperation between various parties was complicated; there were also problems with funding.
• UUSIS - legislative amendments were frequent, there was a shortage of software developers and funding, quality control was inadequate, plus there were difficulties in renewing the outdated information system.

Further to the aforementioned issues, the NAO made the following observations:

• Efficient implementation of development projects is impeded by legislative amendments made during the development process. Said amendments notably slowed down the development work or gave rise to extra costs (for example, EHR, ETIS2, SKAIS2).
• Future sustainability of information systems is not properly considered in the initiation phase of software developments. The planning phase of four audited projects (Kotkas I, SKAIS2, e-Täitur and UUSIS) did not include an analysis of future maintenance of developed software or the provision of support for the agencies’ core processes, or an estimate of the annual maintenance and improvement costs.
• The developed software often fails to meet the needs of all user groups. For six of the audited projects, it was hard to determine whether the developed software met the user’s needs. In some cases, regular user satisfaction or usability (incl. user convenience) surveys were not conducted after the completion of development work.
• In some cases, there were no rules put in place to govern software development. Several agencies and government areas had failed to put in place rules for managing the implementation of software development projects in their initial phase. By the time of the audit, most audited entities had developed the relevant rules and procedures but some of these need further improvement.
• Government agencies do not have a clear idea whether the rules for using EU aid allow implementing agile methods in software development. Breaking down the development work into stages might often be practicable but the agencies have abandoned this practice believing that EU rules do not allow using such development methods. The Ministry of Economic Affairs and Communications has not identified the possibilities of implementing agile development methods in the case of EU funding nor provided advice to support applicants related to the use of these methods.

What were the major success factors?

Having audited nine projects, the following factors contributing to the success of software development projects can be highlighted:

• The core processes should be outlined and optimised before the start of software development to ensure that the developed software helps streamline the agency’s core functions and provide e services efficiently.
• The outlined and optimised core processes must be coordinated with the representative of the relevant domain, i.e. the main user of the information system. The system’s end users, too, should be involved in the process.
• The staffing of software development teams should be prioritised. It is essential to find competent project managers and key personnel as the success of the development work depends on them.
• There are no best or generally applicable software development methods that could be recommended universally. However, it would be advisable to implement the development work in smaller stages which would allow for early identification of potential problems and the implementation of timely adjustments.
• People involved in software development projects should be trained in advance (for example, the nature of software development and the implementation process should be explained, and the parties should be identified). Also, the roles, tasks and responsibilities of project participants should be defined at the outset.
• The users of information systems should be asked to provide regular feedback (preferably once a year) on the usability of information systems and user satisfaction, and any new developments should rely on this information.
• Legislative drafting should take into account the need for the development or modification of information systems to ensure that the development work can be completed in due time and legislative instruments are enforceable.