Wereldwijd staan overheden voor grote opgaven, zoals klimaatverandering en maatschappelijke ongelijkheid. Hoe kunnen zij zich beter toerusten voor deze opgaven? En welke rol kan evaluatieonderzoek hierbij spelen? Deze vragen staan centraal in het toegankelijk geschreven boek Changing bureaucracies: Adapting to uncertainty, and how evaluation can help (Abingdon/New York: Routledge 2021), onder eindredactie van Burt Perrin en Tony Tyrrell. Jedid-Jah Jonker bespreekt deze bundel en staat hierbij onder meer stil bij de relevantie van de inzichten uit het boek voor de Nederlandse overheid.

Dutch social policies are aimed at the integration and participation of all people. This creates challenges for persons with a mental vulnerability and their neighbourhoods. Five municipalities in the province of Flevoland asked us to help them improve the move from protected living to living in the neighbourhood. We used an arts-based participatory action research design and we followed ‘hot topics’, topics that sparked people’s energy and emotion, and which led to empowerment and participation. Focusing on these topics, which initially might not seem to be closely connected to the main research topic, might produce more information, and energy to take action on it, rather than rigidly sticking with the initial research topics. In this article, we focus on the first phases of the project. The emotions that people with mental vulnerabilities expressed when talking about their dogs, led us to the core of what really mattered to them in terms of inclusive living and participation. The dog functioned as an unexpected ‘hot topic’. In a symbolic sense, ‘the dog’ stands for a diverse range of lifeworld topics that can act as a creative catalyst for social change.

Vooraf

Participatief actieonderzoek en responsieve evaluatie staan volop in de belangstelling bij beleidsmakers en onderzoekers. Dit type beleidsonderzoek en -evaluatie beoogt democratisch, inclusief én impactvol te zijn. Het gaat om onderzoek mét in plaats van óver mensen. En het is actiegericht: onderzoek wil bijdragen aan concrete oplossingen door met betrokkenen gezamenlijke (verbeter)acties te ontwikkelen in de praktijk, en daarop te reflecteren en van te leren. Dit alles met het oog op sociale inclusie. Het zijn mooie idealen, maar wat betekent dit in de alledaagse, vaak weerbarstige onderzoekspraktijk?

Op 20 januari 2020 organiseerde prof. Abma daarover een symposium, getiteld ‘Responsive, Participatory Research: Past, Present and Future Perspectives’ (Vrije Universiteit, Amsterdam). De rode draad op het symposium was de vraag wat goed en ethisch verantwoord participatief onderzoek is, en wat dit vraagt van onderzoekers en beleidsmakers. Drie lezingen op deze conferentie zijn nadien omgewerkt tot essays om lezers van Beleidsonderzoek Online vanuit verschillende perspectieven beter kennis te laten maken met deze vorm van onderzoek:

Prof. Weerman en haar team focussen in hun bijdrage op het zich in de praktijk ontwikkelende onderzoeksdesign en het inzetten van creatieve methoden om participatie te bevorderen. Ze gaan na welke kwaliteitscriteria aan participatief actieonderzoek worden gesteld en hechten daarbij met name aan eisen ten aanzien van participatie, samen leren en verschil maken (zie BoO juli 2021). Ze benadrukken het belang van creativiteit en flexibiliteit.

Prof. Abma bespreekt in haar artikel de normatieve dimensies en de ethiek van participatief actieonderzoek (zie BoO september 2020). Ze illustreert met een voorbeeld uit de crisishulpverlening aan GGZ-cliënten dat participatief actieonderzoek niet slechts een methodisch-technische exercitie is, maar een sociaal-politiek proces waarbij bestaande machtsverhoudingen verschuiven om ruimte te geven aan nieuwe stemmen en kennis. Dit omvat het zien van en stilstaan bij ethisch saillante dilemma’s en morele reflectie.

De bijdrage van prof. Cook (zie BoO februari 2021) gaat over de weerbarstige praktijk van participatief actieonderzoek. Het doel is samen leren en voorbij geijkte oplossingen komen. Zij laat zien dat dit uitdagend is voor professionals die geconfronteerd worden met burgers die feedback geven en vragen om het (deels) loslaten van vaststaande professionele kaders. Er ontstaat dan ongemak en onzekerheid, maar zo beoogt en laat Cook overtuigend zien, deze ‘mess’ (niet meer goed weten wat goed en nodig is) is productief om te komen tot hernieuwde inzichten en innovaties.

(Introductietekst opgesteld door prof. T. Abma)

Since 2003, decentralized audit offices in the Netherlands have been authorized to investigate the regularity of the administration conducted. The definition of ‘regularity’ and the scope of the regularity investigation is not described in the law or in the literature. In this article, a regularity investigation is defined as ‘testing whether the administration has complied with applicable law’. That applicable law consists of written and unwritten rules of law, and case law. Audit offices examine regularity less often than efficiency and effectiveness. However, they have started researching it more often than in the past, according to this article. Of the administrative audit office reports in 2019, 42% contained a regularity finding, conclusion or recommendation. Accountants also investigate the regularity. They do this in the context of the annual audit and limit themselves to financial regularity. The regularity audit carried out by decentralized audit offices is broader. In addition to written legal rules, it also focuses on unwritten legal rules and case law, it is not limited to financial subjects and the investigation period can be longer than one reporting year. The findings and conclusions of the audit offices regarding the lawful unlawful actions of the administration concern the consequences for citizens and companies and the consequences for the efficiency and effectiveness of the administration’s actions.

In recent years there has been a lot of discussion about and movement in the position of local audit offices in the Netherlands, For example, there is currently a bill to strengthen the audit offices in the Dutch House of Representatives (‘Tweede Kamer’), and in recent years there have been drastic cuts in the budgets of many local audit offices. The development of local audit offices nationally is therefore strongly determined by politics. Partly for this reason, it is wise to look beyond national borders and gain insight into the development and position of the local audit institutions in other European countries. It is striking that the majority of the regional European audit offices are not only considerably larger and perform more tasks – in addition to performance audits, they also audit the annual accounts – but also that there are considerably fewer decentralized audit offices per country. In addition, there are major differences in powers. For example, regional audit offices in France can in some cases impose fines and take certain cases to court. Despite the large differences, however, there are also many similarities between the various European decentralized audit institutions. For example, all audit offices conduct performance audits and independence is almost always properly guaranteed by law. The article concludes with a plea for the Dutch situation to be somewhat more in line with the development of the European decentralized audit institutions.

Dutch municipalities and provinces have been obliged to have an audit office or audit office function for about 20 years. How does the audit office work nowadays and what contribution does it make to decentralized administration? That is the question at the center of this article. To this end, the authors list the available knowledge about audit offices or committees and present the results of their own analysis of 982 audit reports from 234 audit offices or committees from 308 Dutch municipalities. The audit office or committee has been institutionalized in the vast majority of municipalities and in all provinces. Council members are increasingly less likely to (also) be members of this board and the output has increased slightly from approximately one to an average of one and a half surveys per year. Where initially mainly business management-oriented subjects were examined, some broadening to more policy-related themes has taken place. Municipal councilors are quite satisfied with their audit office or committee. At the same time, the actual social effects of policy are rarely measured in audit institutions. Moreover, council members make little use of audit reports in controlling the municipal board, and audit offices or committees also add little to their framework-setting role. Little is known about the extent to which the research of the audit offices makes a more objective contribution to the administration and strengthening of the functioning of the municipal council, which is also a theme for future research.

In the more than 15 years that decentralized audit offices have existed in the Netherlands, little attention has been paid to the research methods they use. This article focuses on how the research methods used by decentralized audit offices have developed and to what extent they use new technology. New technology has changed a lot in 15 years, which offers new possibilities for research, but also raises new questions. Based on an empirical analysis of audit reports, it can be concluded that decentralized audit offices adopt a standard approach to document and file analysis and interviews, with only limited application of innovative technology. On the basis of a theoretical exploration of the relevant literature and a simple qualitative analysis of research by the Netherlands Court of Audit and the Rathenau Institute, a framework has been developed in which the opportunities and risks of the application of new technology in decentralized audit office research are described. This can provide a handle for future application. Decentralized audit offices can use this for (more) reflection on their research methods and innovation, in order to develop to maturity while remaining young.

The impact of audit office reports has received little attention in the scientific literature. In this article, various forms of impact have been distinguished with the help of Public Administration literature and factors that promote the use of evaluations have been distinguished. This theoretical framework was subsequently used for empirical research into the effect of audit office research. The extent to which the recommendations have had an impact was investigated in 20 Dutch municipalities with the aid of impact reports from audit institutions. Out of 176 publications, 94% of the 1216 recommendations were adopted by the city council. This means that the procedural impact is high. Of the 731 recommendations that could be checked at 17 municipalities, the local audit offices report that 58% had been fully implemented, 19% partially and 15% not or not tackled differently. The three categories of success factors from the scientific literature were visible in the practice of the audit offices. This applies most strongly to impact factors related to evaluation quality, in particular the factors related to communications standards, clear recommendations, timeliness and relevance to the decision maker. As far as research and decision-making factors are concerned, the commitment of the organization and the political climate are the most important factors for audit institutions. Finally, the involvement of stakeholders promotes the impact as a catalyst. The article concludes with practical lessons for promoting the processing of audit reports.

This article focuses on the regularity audit carried out by decentralized audit offices in the Netherlands. Decentralized audit institutions have been given a limited task assignment from the legislator for this type of research. It was expressly not the intention that the audit office repeats the (financial) regularity audit done by the accountant, nor was the decentralized audit office given a role in the so-called indemnity procedure. The decentralized audit office’s role is primarily to carry out a system test of regularity. A positive side effect of this limited task assignment has been that decentralized audit institutions have not started to practice regularity audits as a separate activity. Monitoring the relationship between regularity, efficiency and effectiveness can protect an audit office from pitfalls. While this working method is maintained, new opportunities will arise for decentralized audit institutions. The accountant will soon no longer have primacy in assessing (financial) regularity, but the municipal and provincial Executive will instead report directly to the municipal and provincial council in an annex to the annual accounts. The accountant will continue to monitor whether what is stated in this annex about regularity is correct and complete. This offers new possibilities for the decentralized court of auditors to assist the council in its monitoring and contextualizing role, and in forming an opinion on regularity.