GOVERNANCE GOVERNANCE EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions December 2020 © 2020 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO), http:// creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: 2020. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions. EFI Insight-Governance. Washington, DC: World Bank. Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third- party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to reuse a component of the work, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; e-mail: pubrights@worldbank.org. Cover design and layout: Diego Catto / www.diegocatto.com >>> Contents Responsibilities 5 Acknowledgments 5 Abbreviations 7 Executive Summary 9 1. Introduction 13 Background 13 Objective 14 Scope and Methodology 14 Rating System 15 2. Review Findings of Compliance with ISSAIs 17 Overall Review of the Implementation of ISSAIs 19 SAI’s Independence 19 Compliance with the ISSAI 20 System of Quality Control Compliance with the ISSAI Audit Planning Phase 21 Audit Execution Phase 22 Audit Reporting and Follow-Up 23 3. Findings, Conclusion, and Suggested Next Steps 25 Findings 25 Conclusion 28 Suggested Next Steps 29 Appendix A. INTOSAI Framework 31 of Professional Pronouncements Appendix B. Self-Assessment Questionnaire 33 >>> Responsibilities Directors Bob Saum; Ed Olowo-Okere Managers Roberto Tarallo; Manuel Vargas Task Team Leaders Mona El-Chami; Zohra Farooq Nooryar This review paper is a joint effort by the Governance Global Practice (GGP) and the Standards, Procurement, and Financial Management (OPSPF) Department of the Operations Policy and Country Services (OPCS) Vice Presidency, in collaboration with the Accountability and Over- sight Institutions Community of Practice. >>> Acknowledgments The team would also like to thank the World Bank staff and in particular financial management specialists for the time and support provided to collect the information from the SAIs, including Susana Amaral, Novira Asra, Irina Babich, Natalia Konovalenko, Lingson Chikoti, Baison, Fran- cis Zulu, Yi Dong, Oxana Druta, Hasib Ehsan, Christopher Fabling, Saidu Goje, Trust Chimaliro, Hawazin Hameed, Riham Hussein, Patrick Kabuya, Waseem Kazmi, Ahmed Zai, Arun Manuja, Jad Mazahreh, Vida Nkya, Joao Tinga, Siriphone Vanitsaveth, Maxwell Dapaah, and Viorela Voinea as well as the peer reviewers of the review paper, including Srinivas Gurazada, Patricia Hoyes, and Asha Narayan, in addition to Lucy Mukuka and Phuong Thu Nguyen for their logis- tical support. The team would like to recognize and thank Serge Nkuindja, consultant, for his diligent work on this review. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 5 >>> Abbreviations IDI INTOSAI Development Initiative IFAC International Federation of Accountants IFPP INTOSAI Framework of Professional Pronouncements INCOSAI International Congress of Supreme Audit Institutions INTOSAI International Organization of Supreme Audit Institutions ISQC International Standards on Quality Control ISSAI International Standards of Supreme Audit Institutions FM Financial Management SAI Supreme Audit Institution SAI PMF Supreme Audit Institution Performance Measurement Framework The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 7 >>> Executive Summary Public sector auditing provides unbiased, objective assessments of public sector programs, policies, operations, and results to detect whether public resources are managed responsibly and effectively to achieve intended results and to instill confidence among citizens and stakeholders. Supreme Audit Institutions (SAIs)1 perform a vital role in the functioning of governments as they inform legislatures and other stakeholders through their independent audit reports. They help promote good governance, accountability, and transparency. The work of SAIs in reducing waste and abuse of public resources has the indirect effect of making more money available for programs that fight poverty, which lie at the core of the World Bank’s work to end extreme poverty. 1. SAIs include all forms public external audit bodies, including the Court of Accounts and audit general offices. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 9 The Bank’s fiduciary assurance mandate requires making by the International Organization of Supreme Audit Institutions arrangements to ensure that the proceeds of any financing (INTOSAI). It aims to inform, among other factors, the Bank are used only for the purposes for which it was granted, with strategy on reliance of SAIs for auditing Bank-financed due attention to considerations of economy and efficiency. operations, as part of the use of a country systems strategy. This mandate includes requiring borrowers and recipients of Bank financing to have financial statements of Bank-financed The review was carried out through a survey3 of 18 SAIs that operations audited by auditors acceptable to the Bank in comprise 33 percent of SAIs that fit the first selection criteria, accordance with auditing standards acceptable to the Bank. including SAIs which had produced and published at least one audit report during 2012–2017.4 The aim is the assessment This review assesses the quality of audits performed by of the institutional and organizational capacity of these SAIs SAIs of Bank-financed projects through the lens of audit based on these evaluation domains5: compliance with International Standards of Supreme Audit • SAI independence Institutions (ISSAI), using the methodology prescribed • Quality control by the ISSAI Compliance Assessment Tools developed • Audit planning phase by INTOSAI Development Initiative (IDI) and the SAI • Audit execution phase Performance Measurement Framework2 (SAI PMF) adopted • Audit reporting and follow-up phase. 2. The SAI PMF is a methodological tool elaborated by the IDI that has been used as an inspiration in this peer review exercise, in accordance with the provisions of the INTOSAI GUID 1900 Peer Review Guidelines. 3. The selection of SAIs was based on their capacity to produce an audit report of a Bank-financed project. Twenty-two SAIs were selected across the six World Bank Regions, and a questionnaire was submitted to them by email. 4. They number 55 SAIs, which includes one SAI that produced its first audit report after the selection process. 5. The assessment domains were inspired by the SAI PMF assessment model of the SAI’s environment, capability, and performance. 10 >>> Equitable Growth, Finance & Institutions Insight The review found that only two SAIs, or 11 percent of the With most of the SAIs partially complying with ISSAIs, the selected SAIs, complied with the requirements related to all review suggests that while SAIs operate at various capacity evaluation domains. With regard to each individual evaluation levels based on the available resources and constitutional domain, the review found that: framework, targeted capacity building programs will help in • Four out of 18 SAIs, or 22 percent, complied with the improving the quality of audits performed. Political will and implementation of the ISSAI independence requirements. commitment from senior management coupled with technical • Nine out of 18 SAIs, or 50 percent, complied with the assistance are key for driving reforms in SAIs. implementation of the ISSAI quality control requirements. • Eleven out of 18 SAIs, or 61 percent, complied with the The World Bank remains committed to support SAIs in implementation of the ISSAI audit planning phase, audit building their capacity. In collaboration with INTOSAI and execution phase, and audit reporting and follow-up phase other multilateral and bilateral agencies, the Bank has created requirements. various platforms to stay engaged with SAIs and facilitate knowledge exchange and learning events. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 11 1. >>> Introduction Public sector auditing provides unbiased, objective assessments of public sector programs, poli- cies, operations, and results to detect whether public resources are managed responsibly and effectively to achieve intended results and instill confidence among citizens and stakeholders. Supreme Audit Institutions (SAIs)6 perform a vital role in the functioning of governments as they inform legislative bodies and other stakeholders through their independent audit reports. They help promote good governance, accountability, and transparency. The work of SAIs in reducing waste and abuse of public resources has the indirect effect of making more money available for programs that fight poverty. The Crossroads The International Standards of SAIs (ISSAIs), updated as the INTOSAI Framework of Professional Pronouncements7 (IFPP), are the authoritative international standards on public sector auditing that aim to ensure the quality of the audits conducted. To comply or meet with ISSAIs, SAIs establish policies and procedures for their engagements. Required steps are outlined in policies, audit manuals and guidelines, systems of quality control, and various other audit tools. They guide auditors to ensure audit engagements are conducted according to professional standards and SAI policies. The World Bank’s fiduciary assurance mandate requires making arrangements to ensure that the proceeds of any financing are used only for the purposes for which the financing was granted, with due attention to considerations of economy and efficiency. This mandate includes requir- ing borrowers and recipients to have financial statements of Bank-financed operations audited by auditors acceptable to the Bank and in accordance with acceptable auditing standards. The audit of Bank-financed projects, as required by the Articles of Agreement, is the financial state- ments audit, and can be complemented by other types of audit as compliance and performance audits. International Standards on Auditing and ISSAIs are deemed acceptable. The Bank regards SAIs as key players in the development processes of countries through their important role in promoting efficient, accountable, effective, and transparent public administra- tion. Thus, the Bank sought to rely on SAIs as part of implementing its strategy on the use of country systems.8 Under the Bank’s Access to Information Policy, introduced in 2010, borrowers and recipients are expected to publish the audited financial statements of Bank-financed opera- tions, while the Bank makes audits reports more accessible and transparent. Based on the last stock take of published9 audit reports of Bank-financed projects, SAIs produce 40 percent of these reports (table 1), while private-sector auditors produce the rest (with a minor percentage produced by private-sector auditors hired by SAIs). 6. SAIs include all forms public external audit bodies, including court of accounts and audit general offices. 7. Details about IFPP are included in appendix A. 8. Commencing July 1, 2003, the World Bank adopted a strategy to commit to greater use of country systems in auditing of Bank-financed operations through: (i) providing technical assistance funding and leveraging Bank operations to strengthen in-country SAI capacity and impact; (ii) promoting global, regional, and bilateral SAI partnerships; and (iii) providing learning opportunities and support to Bank staff to enable implementation of the strategy. 9; Aid effectiveness forums, starting with the 2005 Paris High-Level Forum on Aid Effectiveness, have progressively called for further strengthening of country fiduciary systems including SAIs. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 13 > > > T A B L E 1 - Audit Reports of World Bank-Financed Projects per Region during 2012–2017 Countries with Representation of Countries with an Audit World Bank Region SAI Audit Report Countries with SAI Audit Report (number) (number) Report (percent) Africa 47 21 45% East Asia and Pacific 20 13 65% Europe and Central Asia 28 5 18% Middle East and North Africa 8 1 13% Latin America and the Caribbean 25 7 28% South Asia 8 8 100% Total 136 55 40 Source: World Bank. Objective Scope and Methodology The objective of the review was to assess the quality of the The review’s original scope covered SAIs that produced and audits performed by SAIs of Bank-financed projects through published at least one audit report during 2012–2017, which reviewing and validating their level of compliance with ISSAIs. came to 55 SAIs (table 2). The scope was further reduced to The process involved gathering data and evidence to support 18 SAIs, from the 6 regions of the world in agreement between the assessment. While there are multiple factors for consider- Operations, Policy, and Country Services and Governance ation in selecting SAIs for audit of Bank-financed projects, the Global Practice, Financial Management (FM) staff. The review review’s findings provide more insight on the level of quality of team selected the final sample SAIs based on interest in col- the audits carried out by SAIs. The findings also inform Bank laboration in order to foster buy-in for the review’s potential capacity building programs with SAIs by highlighting areas for policy or operational suggested next steps. improvement. 14 >>> Equitable Growth, Finance & Institutions Insight > > > T A B L E 2 - Selected SAIs for Review Countries with Representation of Countries with SAI Audit World Bank Region SAI Audit Report Countries with SAI Audit Report (number) (number) Report (percent) Africa 21 6 29 East Asia and Pacific 13 4 31 Europe and Central Asia 5 3 60 Middle East and North Africa 1 2 100 Latin America and the Caribbean 7 2 29 South Asia 8 2 25 Total 55 18 33 Source: World Bank. Note: One SAI did not meet selection criteria at the selection stage but did during the life of the review and was added. The review focused on five components of ISSAIs (evaluation Rating System domains): • SAI independence • Quality control The review team applied a simplified rating system to rate the • Audit planning phase degree of compliance with ISSAIs (table 3). • Audit execution phase • Audit reporting and follow-up > > > T A B L E 3 - Simplified Rating System to The methodology included: Rate the Degree of Compliance with ISSAIs • Self-assessment survey. Develop and circulate a ques- Compliant The ISSAI requirements and applicable tionnaire (see appendix B ) to have the selected SAIs self- (SAI met auditing standards were implemented assess their level of compliance with quality standards to requirement) in SAI’s audit practice be considered when conducting an audit; independence Partially compliant Improvements are necessary in standards; International Standards on Quality Control (requirement is some areas to fully comply with (ISQC); and standards related to the planning, execution, partially met) the ISSAI requirements reporting, and follow-up. Noncompliant Major deficiencies exist; there • Validation through relevant documentation and audit (requirement is is noncompliance with the not met) ISSAI requirement file review. Have FM staff visit the selected SAIs to con- duct an audit file review to obtain evidence and validate the Source: World Bank. Note: ISSAI = International Standards of Supreme Audit Institutions; results of the completed questionnaires. SAI = Supreme Audit Institution. The overall assessment of the compliance is an average of the rating of the five evaluation domains. However, if the SAI was rated as noncompliant or partially compliant on the inde- pendence domain, the overall rating will be partially compliant even if it was rated as compliant in all the other domains. This is due to the importance and more weight of independence in SAI effectiveness. If the SAI self-assessed itself as com- pliant but did not facilitate the validation exercise, the over- all rating was downgraded to the next lower rating. If the SAI self-assessed itself as partially compliant and results were not validated, the overall rating will be partially compliant. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 15 2. >>> Review Findings of Compliance with ISSAIs The 18 selected SAIs completed the self-assessment questionnaire. Table 4 presents the summary of the analysis of the self-assessment followed by a detailed analysis. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 17 > > > T A B L E 4 - Summary of the Survey Analysis by Evaluation Domain and Ratings Audit Audit Audit Reporting Quality Overall SAI Region Independence Planning Execution and Comment Control Assessment Phase Phase Follow-Up Phase Partially Partially Partially Partially SAI 1 SAR Noncompliant Compliant compliant compliant compliant compliant Partially Partially Partially SAI 2 SAR Compliant Compliant Compliant compliant compliant compliant Partially Partially Partially Partially Partially SAI 3 LAC Compliant compliant compliant compliant compliant compliant Partially Partially Partially Partially SAI 4 EAP Compliant Compliant compliant compliant compliant compliant Partially SAI 5 EAP Compliant Compliant Compliant Compliant Compliant compliant Partially Partially SAI 6 MNA Compliant Compliant Compliant Compliant compliant compliant Partially Partially SAI 7 EAP Noncompliant Noncompliant Noncompliant Noncompliant compliant compliant Partially Partially Partially SAI 8 AFR Compliant Compliant Compliant compliant compliant compliant Validation Partially Partially Partially Partially Partially Partially SAI 9 AFR phase not compliant compliant compliant compliant compliant compliant completed Partially Partially SAI 10 ECA Compliant Compliant Compliant Compliant compliant compliant Partially Partially Partially Partially Partially Partially SAI 11 AFR compliant compliant compliant compliant compliant compliant Validation Partially Partially Partially Partially Partially Partially SAI 12 EAP phase not compliant compliant compliant compliant compliant compliant completed Partially Partially Partially SAI 13 MNA Compliant Compliant Compliant compliant compliant compliant Validation Partially SAI 14 ECA Compliant Compliant Compliant Compliant Compliant phase not Compliant completed Validation Partially Partially SAI 15 LAC Compliant Compliant Compliant Compliant phase not compliant compliant completed SAI 16 AFR Compliant Compliant Compliant Compliant Compliant Compliant Partially Partially Partially SAI 17 ECA Compliant Compliant Compliant compliant compliant compliant Validation Partially Partially SAI 18 AFR Compliant Compliant Compliant Compliant phase not compliant compliant completed 1 0 1 1 1 1 Noncompliant n.a. 5.6% 0.0% 5.6% 5.6% 5.6% 5.56% Partially 13 9 6 6 6 15 n.a. compliant 72.2% 50.0% 33.3% 33.3% 33% 83.33% 4 9 11 11 11 2 Compliant n.a. 22.2% 50.0% 61.1% 61.1% 61.1% 11.11% Source: World Bank. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; n.a. = not applicable. SAI = Supreme Audit Institution; SAR = South Asia. 18 >>> Equitable Growth, Finance & Institutions Insight Overall Review of the The SAIs mainly faced operational, rather than constitutional independence, challenges from the Executive branch. To fulfill Implementation of ISSAIs their mandate, SAIs indicated they could carry out their work largely in an independent manner and did not experience severe interference by the Executive branch, including the Ministry of The review found that overall, only 2 out of 18 SAIs, or 11 Finance. However, the operational independence of managing percent, successfully complied with ISSAI requirements for all their finances and personnel is not guaranteed and, as a result, the evaluation domains in the audit practice. The review also the SAI’s financial and operational independence is restricted. found that the audit manual framework, consisting of rules, regulations, manuals, and guidelines, complies with interna- The domain’s purpose is to consider the institutional basis for tional standards for these SAIs. the SAI’s operations. According to INTOSAI–P 1, an objective SAI operating in an effective manner can only be achieved if the The review concluded that the majority (15 out of 18, or 83 SAI is independent from the executive branch, and the indepen- percent) of the selected SAIs were partially compliant with the dence of the SAI’s head is protected against outside influence. ISSAIs. For these SAIs, the review observed that general au- Therefore, the domain measures the degree of independence dit manuals and guidelines explain the different types of audits enjoyed by the SAI by assessing the key aspects of indepen- (i.e., financial, compliance, and performance). While some dence as identified by ISSAIs through the Lima Declaration SAIs continue to develop the audit manuals and guidelines (INTOSAI–P 1) and the Mexico Declaration (INTOSAI–P 10). which would be quite useful to the auditors. The survey included four questions related to these sections: The SAIs have put in place a considerable amount of work • Constitutional and legal framework supported by the documentation provided by these institu- • Financial independence and autonomy tions. They have laid strong foundations to function in compli- • Organizational independence autonomy ance with ISSAIs. However, they still need to make significant • Independence of the head of the SAI and its officials. improvements to implement the audit methodology during the audit process phases (i.e., planning, execution, reporting, and Four SAIs indicated compliance with the ISSAI requirements follow-up) and to attain the practice level set by ISSAIs in key related to SAI independence. For these SAIs, the review found areas: sufficient provisions in the constitution, the budget law, and • Independence and mandate the audit law to guarantee independence. On the other hand, • Quality control 13 SAIs partially complied with the requirements. The reasons • Audit standards and guidelines. cited by them included: • Lack of sufficient guarantees in the constitution against Finally, compared to others, one out of the 18 SAIs, or 6 percent, interference in the SAI’s affairs. provided little supporting information related to its implementa- • Control of the SAI’s access to financial resources by the tion of the ISSAI requirements in all evaluation domains in its Ministry of Finance, just as it does for other government audit practice, which led to an overall rating of noncompliance. budget entities. • None had the right to independently hire their own per- SAI’s Independence sonnel or decide on their terms and conditions. • Lack of a framework of incentives to retain, motivate, pro- mote, and build the capacity of SAI staff. The review found that overall, 4 SAIs, or 22 percent, complied with ISSAI requirements related to SAI independence. Thir- Finally, one SAI, or 6 percent, had not complied with the re- teen SAIs, or 72 percent, had partially complied with the inde- quirements for SAI independence under the ISSAIs through pendence requirements even though the SAI independence INTOSAI–P 1 and INTOSAI–P 10. requirements were identified by INTOSAI members them- selves through the Lima Declaration ISSAI 1 (now INTOSAI–P 1) and the Mexico Declaration ISSAI 10 (now INTOSAI–P 10). The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 19 Compliance with the ISSAI System of Quality Control The review found that only 9 out of 18 SAIs, or 50 percent, • Customized audit methodologies are designed to suit IS- complied with the quality control requirements of the interna- SAI requirements by developing tools such as the Regu- tional standards. The other 9 SAIs partially complied with and larity Audit Manual, Performance Audit Manual, and the implemented them in their audit practices. recently introduced Financial Audit Manual and Compli- ance Audit Manuals. The SAIs surveyed have confirmed the implementation of a • Supervision and review procedures at the engagement number of manuals and policies aimed at ensuring uniform level are stated in the Regularity Audit Manual, Perfor- practices and adoption of the ISSAIs. This demonstrates they mance Audit Manual, and the recently introduced Finan- have ownership of the system of quality control. Therefore, the cial Audit Manual and Compliance Audit Manuals. overall trend is the gradual movement of the SAIs’ audit work • The Quality Assurance Manual and Quality Assurance toward ISSAI compliance. Checklists (tools) are in line with the ISSAIs that guide the quality assurance team during review. The SAI’s system for quality control of audit processes is the • The in-house post-issuance team (reviewers) are inde- sum of all measures taken to ensure high quality of each au- pendent from all other institution functions and engage- dit product. It is carried out as an integrated part of the audit ments and have the required ability as stated in Quality process and needs to be part of each SAI’s strategy, culture, Assurance Policy. policies, and procedures to be effective. As stated in ISSAI 140 — Quality Control, the SAIs should consider their work The review also found that nine SAIs had only partially com- program and whether they have resources to deliver the range plied with the ISSAI requirements related to quality control. of work to the desired level of quality; and that to achieve this, The key reasons included the absence of a quality assurance SAIs should have a system to prioritize their work in a way that unit to monitor quality and the lack of policies and procedures takes into account the need to maintain quality. for the monitoring of compliance with quality control proce- dures. In line with ISSAI 140, quality control and quality as- The review only looked into the approach to quality and fo- surance are preconditions for continuous improvement and cused on quality control procedures, which are mainly imple- for SAIs to be effective; however, the role and function of all mented after an audit is completed, to see whether the audit actors and instruments must be clear. Maintaining a system quality control effectively operated and how policies and pro- of quality control for a SAI requires ongoing monitoring and a cedures could be improved. Quality control, however, cannot commitment to continuous improvement — Element 6 of the replace the important function of quality assurance, which ISSAI 140 (monitoring), in other words, the quality assurance should be provided on a continuous basis — as an ongoing review process. element of the audit reporting process, rather than as an ex post assessment of the quality of audit files. Overall, the assessment indicates that the implementation ef- fort related to team competency, supervision, monitoring, au- The review found that nine SAIs provided indications that they dit management, and quality control review responsibilities is implemented quality control requirements in their audit prac- not sufficient. Therefore, SAIs need to strengthen reasonable tices. Examples from their audit methodology include: assurance that the quality control requirements are carried out • The official code of ethics is communicated to all staff. In following relevant standards and applicable legal and regula- addition, the office has an ethics committee which adds tory requirements, such as ISSAI 130, the INTOSAI Code of value to the process of ensuring compliance with the code Ethics and ISSAI 140 on Quality Control, or the International of ethics. Federation of Accountants’ (IFAC) guidance ISQC 1. • A training program is prepared on the basis of a needs assessment and is updated every three years. 20 >>> Equitable Growth, Finance & Institutions Insight Compliance with the ISSAI Audit Planning Phase The review found that 11 out of 18 SAIs, or 61 percent, com- Meanwhile, 6 SAIs had only partially complied with the re- plied with the ISSAI requirements related to the audit planning quirements related to the audit planning phase. These SAIs phase in their audit practices. Six SAIs, or 33 percent, only have implemented a rich set of policies, procedures, and audit partially complied. manuals. All are aimed at ensuring that audit engagements are of high quality and follow the ISSAIs. However, the review The challenge for most SAIs is to implement the audit plan- found that the SAIs did not give sufficient indications related to ning process using a risk-based methodology (Element 3 of the key audit planning activities: ISSAI 140). Risk assessment is the main element in the de- • Risk assessment considerations, understanding of inter- velopment of the annual risk-based audit plan. Identification nal control relevant to the audit objectives, and test con- and prioritization of key risks is crucial to ensuring that audit trols on which they expect to rely resources are allocated to the significant areas of the state’s • Fraud risk assessment to enable auditors to plan and activity. gather sufficient appropriate evidence related to identified fraud risks through the performance of suitable audit pro- Principle 5 of INTOSAI–P 12 maintains that the SAI should cedures ensure that stakeholders’ expectations and emerging risks are • Discussion and documentation of significant matters with factored into audit plans as appropriate. Therefore, it would management, those charged with governance, and oth- be advisable that SAIs apply the risk-based approach in the ers, including the nature of the significant matters dis- planning process, using the top-down and bottom-up risk as- cussed and when and with whom the discussions took sessment element. However, further steps should be taken to place make it more systematic. • Cost estimates per audit, thus impeding the comparison between costs and benefits per audit engagement. The questionnaire review focused on the steps of the audits (compliance, financial, and performance) the SAIs plan to On the other hand, the quality of audit planning varies between conduct in a set period. The overall audit plan supports the SAIs, especially regarding the ability to prioritize a systematic SAI in reaching its objectives efficiently and effectively. It is, and documented risk-based approach for the selection of enti- therefore, important that the overall audit plan is feasible, re- ties to be audited against identified risks. flecting SAI budget and workforce. This gap can be filled by following an approach laid The review found that for 11 SAIs, the audit manual frame- down by INTOSAI–P 100, Fundamental Principles of Public- work, consisting of rules, regulations, manuals, and guide- Sector Auditing: lines, complies with international standards. In the responses • Establish the terms of the audit received, the review noted the implementation of important • Obtain understanding planning elements, such as background and information • Conduct risk assessment of problem analysis about the audited entities, audit objective, scope, questions or • Identify risks of fraud hypotheses, criteria, and period to be covered by the audit as • Develop an audit plan. well as key project timeframes and milestones and the main points for control. Finally, one out of 18 SAIs provided very little supporting infor- mation related to its implementation of the ISSAI requirements for all evaluation domains in its audit practice. This led to the rating of noncompliance. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 21 Audit Execution Phase The review found that 11 out of 18 SAIs, or 61 percent, com- The review of the selected performance audit files found that plied with the ISSAI requirements related to the audit execu- the auditors used financial audit methodology, and the audit tion phase in their audit practices. Six SAIs, or 33 percent, had design and execution focused on compliance. Nevertheless, only partially complied with them. For the 11SAIs, the manuals designing and conducting the audit execution for a good per- were appropriate and explained the different types of audit. formance audit takes time. They defined general principles and set more specific criteria and approaches for the audit execution phase in the audit pro- On the positive side, the review noted indications from the SAI cess. The selected audit files included audit procedures, test responses received that audit engagements are implemented of control worksheet, and substantive procedures worksheets. according to the audit plans and following the guidelines. The Audit queries raised in the audit tests were escalated to the SAIs surveyed indicated they gather: reporting phase. Supervision was conducted by three levels • audit evidence using appropriate audit procedures of reviewers. At the end of each audit phase, a quality control • sufficient, appropriate (valid, reliable, and relevant) audit checklist is used to review and determine adherence to rel- evidence to establish findings, reach conclusions in re- evant procedures aligned with ISSAI requirements. sponse to the audit objectives and questions, and issue recommendations. The review found that for the 6 SAIs that only partially com- plied, audits were implemented according to the audit plans The review found that SAIs need to strengthen audit practice and following the guidelines. A number of SAIs provided ge- to be fully equipped with tools to enable audit staff to collect neric responses without references or supporting documenta- and analyze sufficient and appropriate evidence to support tion that were not considered to be valid inputs in the course sound conclusions: of the present assessment. • SAIs need to ensure more consistent application of IS- SAIs across all of their audit work. In this category, the challenge for SAIs is to move away from • Audit manuals and guidance need to be developed for a compliance audit and toward a performance audit, which each type of audit (i.e., financial, compliance, and perfor- remains at an early stage for most of the SAIs reviewed. How- mance) to ensure full compliance with ISSAIs. ever, important foundations for financial and compliance au- • Adopt more intensive quality assurance and consistent dits are in place for many SAIs. use of audit management software. 22 >>> Equitable Growth, Finance & Institutions Insight Audit Reporting and Follow-Up The review found that 11 out of 18 SAIs, or 61 percent, com- sight bodies, those charged with governance, and the public. plied with the ISSAI requirements related to audit reporting These reports include the following relevant information: and follow-up in their audit practices. Likewise, another 6 • Audit objectives SAIs, or 33 percent, had only partially complied. • Audit questions and answers to those questions • Subject matter SAIs face an important challenge to comply with the Mexico • Criteria Declaration on SAI Independence (INTOSAI–P 10), which sup- • Methodology ports using audit reports more often and promoting transpar- • Sources of data ency. The following three principles are especially important: • Limitations on the data used • Principle 5: The right and obligation of SAIs to report on • Audit findings, conclusions, and recommendations. their work • Principle 6: The freedom to decide the content and timing The follow-up of the recommendations is in place and seems of audit reports and to publish and disseminate them effective. However, the review found that 1 SAI did not com- • Principle 7: The existence of effective follow-up mecha- ply with the ISSAI requirements related to audit reporting and nisms on SAI recommendations. follow-up. The review also found that 11 SAIs complied with the requirements related to the audit execution phase. On the other The questionnaire included 10 questions related to the audit hand, the review found that 6 SAIs had only partially complied reporting phase and report follow-up and key points, such as with the requirements related to the audit follow-up phase. connecting conclusions with the audit objectives, criteria, and evidence. Wrap-up procedures are also included to make sure The review found that SAIs need to strengthen the following auditees are given an opportunity to comment on the draft, audit practices to be fully equipped with tools to enable them to and their points are considered before issuing the final report meet the requirements related to audit reporting and follow-up: of the execution phase to ascertain how SAIs collect and ana- • Publication and dissemination of audit reports to auditees lyze evidence concerning financial, compliance, and perfor- as well as other stakeholders mance audits. • Development of an external communication strategy to maintain constructive relationships with the executive, the The majority of the respondent SAIs indicated they produce media, and civil society audit reports in compliance with ISSAI requirements that pro- • Systematic follow-up of the implementation of recommen- vide independent and reliable information to legislatures, over- dations by the audited entities. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 23 3. >>> Findings, Conclusion, and Suggested Next Steps Findings The review, which involved self-assessment followed by validation through Bank staff, provided insights with respect to SAI’s compliance with ISSAIs. Specific findings under each domain and its implications are presented. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 25 Findings per Degree of Compliance tion of ISSAIs, including a code of ethics, audit methodologies, Two out of 18 SAIs, or 11 percent, successfully complied with and quality assurance manuals. This demonstrates that they the ISSAI requirements of all the evaluation domains in the have ownership of the system of quality control. The overall audit practice. Their audit manual frameworks, consisting trend is that the SAIs’ audit work is gradually moving closer to of rules, regulations, manuals, and guidelines, complied ISSAI compliance. Key reasons for partial compliance includ- with ISSAIs. ed the absence of a quality assurance unit to monitor quality; the lack of policies and procedures for the monitoring of com- The majority of selected SAIs, 15 out of 18, or 83 percent, pliance with quality control procedures; and the insufficiency partially complied with ISSAIs. They laid strong foundations of the implementation effort related to team competency, su- to function in compliance with ISSAIs as demonstrated by the pervision, monitoring, audit management, and quality control documentation they had shared. Their general audit manuals review responsibilities. and guidelines explained the different types of audits (i.e., fi- nancial, compliance, and performance). However, SAIs need FOR AUDIT PLANNING. Eleven out of 18 SAIs, or 61 per- to make significant improvements to implement the audit cent, complied with the ISSAI requirements related to the methodology during the audit process phases (i.e., planning, audit planning in their audit practice. Their audit manual execution, reporting, and follow-up) and to attain the practice frameworks, consisting of rules, regulations, manuals, and level set by ISSAIs in regard to independence and mandate; guidelines, comply ISSAIs. The implementation of important quality control; and audit standards and guidelines. One of the planning elements such as background and information about 15 SAIs complied with all requirements of ISSAIs, but the re- the audited entities, audit objective, scope, questions or hy- view team could not validate the results, and its rating was potheses, criteria and period to be covered by the audit, and downgraded to partially compliant. key project timeframes and milestones and the main points for control, is in compliance with ISSAII requirements for the audit Compared to others, 1 out of 18 SAIs, or 6 percent, provided planning phase. little supporting information related to its implementation of the ISSAI requirements for all evaluation domains in its audit prac- Six SAIs, or 33 percent, only partially complied. The challenge tice, which led to its noncompliance rating. for most SAIs was to implement the audit planning process following a risk-based methodology (Element 3 of ISSAI 140). The SAIs implemented a rich set of policies, procedures, and Findings per Evaluation Domain audit manuals aimed at ensuring that audit engagements are FOR INDEPENDENCE. Four out of 18 SAIs, or 22 percent, of high quality and follow the ISSAIs. However, the responses complied with the independence requirements of ISSAIs. The they provided did not include sufficient indications related to constitution, the budget law, and the audit law had sufficient key audit planning activities: risk assessment considerations, provisions to guarantee their independence. Thirteen SAIs, or understanding of internal control relevant to the audit objec- 72 percent, had complied partially. The main challenge faced tives and test controls on which they expect to rely; fraud risk by these SAIs is operational rather than constitutional inde- assessment to enable auditors to plan and gather sufficient pendence from the Executive branch. They can carry out audit appropriate evidence related to identified fraud risks through work largely in an independent manner and do not experience the performance of suitable audit procedures; discussion severe interference by the Executive branch, including Min- and documentation of significant matters with management; istry of Finance. However, the operational independence of and cost estimates per audit, thus impeding the comparison managing their finances and personnel is not guaranteed and, between costs and benefits per audit engagement. Finally, therefore, the SAI’s financial and operational independence is one out of 18 SAIs provided very little supporting information restricted. One SAI had not complied with the ISSAI require- related to its implementation of the ISSAI requirements for ment for SAIs independence. Its constitutional and legisla- audit planning in its audit practice, which led to a rating of tive framework did not provide for its financial independence, noncompliance. and its audit law did not guarantee its independence from the executive. FOR AUDIT EXECUTION. Eleven out of 18 SAIs, or 61 per- cent, complied with the ISSAI requirements related to the audit FOR QUALITY CONTROL. Nine out of 18 SAIs, or 50 per- execution in their audit practices. Their manuals are appropri- cent, complied with quality control requirements of ISSAIs, ate and explain the different types of audit. They define gen- whereas the other 9 partially complied. SAIs have confirmed eral principles and set more specific criteria and approaches the development and implementation of a number of manuals for the audit execution phase in the audit process. The select- and policies aimed at ensuring uniform practices and adop- 26 >>> Equitable Growth, Finance & Institutions Insight ed audit files included audit procedures, test of control work- Suggestions per Evaluation Domain sheet, and substantive procedures worksheets. Audit queries FOR INDEPENDENCE. As country authorities, SAIs (where raised in the audit tests were escalated to the reporting phase. possible with international development partners support) can Supervision is conducted by three levels of reviewers. At the fill this gap by ensuring the minimum conditions provided by end of each audit phase there is quality control checklist which the Mexico Declaration (INTOSAI–P 10) apply to them: reviews and concludes on adherence to relevant procedures • Appropriate and effective constitutional framework aligned with ISSAI requirements. • Financial independence autonomy • Organizational independence and autonomy Six SAIs, or 33 percent, only partially complied. They imple- • Independence of the head of SAI and its officials. mented audits according to audit plans and guidelines. They also had sufficient, appropriate (valid, reliable, and relevant) FOR QUALITY CONTROL. SAIs need to strengthen reason- audit evidence to establish findings, reach conclusions in re- able assurance that the quality control requirements are car- sponse to the audit objectives and questions, and issue rec- ried out following relevant standards and applicable legal and ommendations. A number of SAIs provided generic responses regulatory requirements, such as ISSAI 130, the INTOSAI without references or supporting documentations, which were Code of Ethics and ISSAI 140 on Quality Control, or the In- not considered to be valid inputs in the course of the pres- ternational Federation of Accountants IFAC guidance ISQC 1. ent assessment. In this category, the challenge for SAIs is to move away from a compliance audit. The assessment found For Audit Planning. The quality of audit planning varies be- that performance audits remain at an early stage for most tween SAIs, especially regarding the ability to prioritize a sys- of the SAIs reviewed. On the other hand, important founda- tematic and documented risk-based approach for selection of tions for financial and compliance audits are in place for many entities to be audited against identified risks. INTOSAI–P 100, of them. Fundamental Principles of Public-Sector Auditing, can fill this gap as follows: FOR AUDIT REPORTING AND FOLLOW-UP. Eleven out of • Establish the terms of the audit 18 SAIs, or 61 percent, complied with the ISSAI requirements • Obtain understanding related to audit reporting and follow-up in their audit practices, • Conduct risk assessment of problem analysis whereas 6 SAIs, or 33 percent, had only partially complied • Identify risks of fraud SAIs face an important challenge to comply with the Mexico • Develop an audit plan. Declaration on SAI, which supports using audit reports more often and promoting transparency. Three principles are espe- FOR AUDIT EXECUTION. SAIs need to strengthen the fol- cially important: lowing audit practice to be fully equipped with tools that will • Principle 5: The right and obligation of SAIs to report on enable audit staff to collect and analyze sufficient and appro- their work priate audit evidence to support sound audit conclusions: • Principle 6: The freedom to decide the content and timing • Consistent application of ISSAIs across all of the audit work of audit reports and to publish and disseminate them • Development of audit manuals and guidance for each type • Principle 7: The existence of effective follow-up mecha- of audit (i.e., financial, compliance, and performance) to nisms on SAI recommendations. ensure full compliance with ISSAIs • More intensive quality assurance and consistent use of The follow-up of the recommendations is in place and seems the audit management software. effective. The majority of the respondent SAIs indicated they produce audit reports in compliance with ISSAI requirements FOR AUDIT REPORTING AND FOLLOW-UP. SAIs need to that provide independent and reliable information to legisla- strengthen the following audit practices to be fully equipped tures, oversight bodies, those charged with governance, and with tools that will enable then to meet the requirements re- the public. These reports include the following relevant infor- lated to audit reporting and follow-up: mation: audit objectives; audit questions and answers to those • Publication and dissemination of audit reports to auditees questions; subject matter; criteria; methodology; sources of as well as other stakeholders data; limitations on the data used; audit findings; and conclu- • Development of an external communication strategy to sions and recommendations. One out of 18 SAIs did not com- maintain constructive relationship with the executive, the ply with the ISSAI requirements related to audit reporting and media, and civil society follow-up. • Systematic follow-up of the implementation of recommen- dations by the audited entities. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 27 Conclusion The review sets out observations for a select number of SAIs With regard to compliance and financial audits, the review on their compliance with ISSAIs. It was conducted through a found that the general audit approach overall complies with survey using self-assessed responses provided by SAIs and ISSAIs. However, SAIs need to invest further in understanding the validation of supporting documentation by the World Bank the concept of performance audit, especially the concepts of review team. efficiency and effectiveness. The review noted that SAIs have successfully complied with The quality of SAI interventions should be continuously as- ISSAIs at the foundational level but remain partially compliant sessed in terms of their impact on the SAIs’ environment (rel- with the standards. The partial compliance rating represents evant stakeholders such as auditees and related institutions). a spectrum where each SAI scores differently and demon- Progress on improvement in relation to ISSAI compliance strates a capability to operate at different capacity levels. As necessitate a longer period to complete, and it is crucial to most SAIs have dual capacity and channel most of their re- invest time and resources in building success. SAIs benefit sources into audits of donor-funded projects, the findings of from Bank projects, peer networks, INTOSAI, and other mul- this review cannot be applied to the full range of audits carried tilateral and bilateral agencies. Although the review was com- out by SAIs. They indeed have the potential and the capacity pleted after the COVID-19 pandemic began, the review team to perform at higher levels provided they have the resources notes that support to SAIs continues in order to strengthen and commitment. their ability to audit Bank-financed projects and conduct high- priority work. During the post-emergency phase of COVID-19, While further capacity building projects and technical assis- a broader coalition of partners could be built to support SAIs tance aimed at strengthening SAIs capacity to perform audits in further strengthening donor-funded projects will improve compliance with ISSAIs, the SAIs’ independence remains a concern. Revisions to the constitutional framework to allow for financial and organizational independence as well as the independence of the head of SAI and its officials will be critical to the successful implementation of SAI reforms. 28 >>> Equitable Growth, Finance & Institutions Insight Suggested Next Steps Suggested next steps for the World Bank to consider include: • Draw on the review report and explore a next phase with • Capitalize on the review report to strengthen the World broader scope including assessment of SAIs’ capacity to Bank’s engagement with SAIs and support the use of a review ex post procurement of Bank-financed projects; country systems strategy. conducting audits during the COVID-19 crisis. Deeper • Organize a knowledge exchange session with other mul- analysis based on Public Expenditure and Financial Ac- tilateral development banks, which also rely on SAIs for countability and SAI PMF reports could capitalize on the project audits, to discuss the dynamics and explore fur- findings of this report. ther collaboration with SAIs. • Consider developing guidelines to assist Bank staff in pur- • Continue collaboration with IDI on taking this agenda for- suing further reliance on SAIs for auditing Bank-financed ward and strengthening SAI’s compliance with ISSAIs. projects. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 29 >>> Appendix A. INTOSAI Framework of Professional Pronouncements The International Organization of Supreme Audit Institutions (INTOSAI) enables Supreme Audit Institutions (SAIs) to help their respective governments improve performance by producing robust auditing standards through the International Standards of Supreme Audit Institutions (ISSAI). The INTOSAI Framework of Professional Pronouncements, endorsed by the XXIII International Congress of Supreme Audit Institutions held September 25–27, 2019, in Moscow, contain three categories of professional pronouncements: the ISSAI, INTOSAI Principles (INTOSAI–P), and INTOSAI Guidance (GUID). INTOSAI–P consists of founding and core principles. The founding principles have historical significance and specify the role and functions that SAIs should aspire to. These principles may be informative to governments and legislatures as well as SAIs and the wider public and may be used as a reference in establishing national mandates for SAIs. The core principles support the founding principles for the SAI, clarifying its role in society as well as high- level prerequisites for its proper functioning and professional conduct. The ISSAIs are the authoritative international standards on public sector auditing. Their purpose is to: • Ensure the quality of the audits conducted • Strengthen the credibility of the audit reports for users • Enhance transparency of the audit process • Specify the auditor’s responsibility in relation to the other parties involved • Define the different types of audit engagements and the related set of concepts that provides a common language for public sector auditing. GUID is developed by INTOSAI to support the SAI and individual auditors in: • How to apply the ISSAIs in practice in the financial, performance, or compliance audit processes • How to apply the ISSAIs in practice in other engagements • Understanding a specific subject matter and the application of the relevant ISSAIs. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 31 INTOSAI-P INTOSAI Principles INTOSAI Founding Principles INTOSAI-P 1-9 INTOSAI Core Principles INTOSAI-P 10-99 INTOSAI Standards Fundamental Principles of Public Sector Auditing ISSAI 100-129 Competency Standards SAI Organisational Requirements ISSAI 130-199 (COMP) Financial Performance Compliance Other COMP Audit Audit Audit Engagements ISSAI 700-799 FA Principles PA Principles CA Principles ISSAI ISSAI 200-299 ISSAI 300-399 ISSAI 400-499 600-699 COMP 7000-7499 FA Standards PA Standards CA Standards ISSAI ISSAI 2000-2899 ISSAI 3000-3899 ISSAI 4000-4899 6000-6499 INTOSAI Guidance SAI Organisational Guidance GUID 1900-1999 Competency Standards Supplementary Supplementary Supplementary Other (COMP) Financial Performance Compliance Engagements Supplementary Audit Audit Audit GUID 2900-2999 GUID Competency Guidance Guidance Guidance Guidance GUID 2900-2999 GUID 3900-3999 GUID 4900-4999 COMP 7500-7999 Subject Matter Specific Guidance GUID 5000-5999 Other Guidance GUID 9000-9999 Source: Reprinted from the INTOSAI website (https://www.intosai.org/focus-areas/audit-standards). 32 >>> Equitable Growth, Finance & Institutions Insight >>> Appendix B. Self-Assessment Questionnaire The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 33 > > > T A B L E B . 1 . - Self-Assessment Questionnaire for Supreme Audit Institutions to Determine Their Level of Compliance with Quality Standards to be Considered When Conducting an Audit SAI Independence Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents DIMENSION (I) APPROPRIATE AND EFFECTIVE CONSTITUTIONAL AND LEGAL FRAMEWORK (ISSAI 1) a) The establishment, role, powers and duties of the SAI are laid down in the constitutional or ISSAI 1.5 comparable legal framework. b) The independence of SAI from the executive is embedded in the constitutional or ISSAI 1.5 comparable legal framework. c) The appointment, term, removal and 1 dismissal of the SAI’s head (and members, in the case of collegiate bodies) and the ISSAI 1.6 independence of their decision-making powers are guaranteed in the constitutional or comparable legal framework. d) There is adequate legal protection against ISSAI 1.5 any interference with the SAI’s independence. DIMENSION (II) FINANCIAL INDEPENDENCE/AUTONOMY (ISSAI 1 AND ISSAI 10) a) The legal framework explicitly or implicitly provides for the SAI’s financial independence ISSAI 1:7 from the executive. b) The SAI’s budget is not part of the ISSAI 10:8 executive’s budget (e.g., Ministry of Finance). c) The SAI has the right to prepare its own ISSAI 10:8 budget and submit this to Parliament. d) The SAI’s budget is approved by parliament. ISSAI 1:7 e) The SAI has flexibility to use its budget 2 according to its own priorities in accordance ISSAI 1:7 with the national budgetary laws. f) “If required, SAIs shall be entitled to apply directly for the necessary financial means ISSAI 1:7 to the public body deciding on the national budget.” g) During the past 3 years there have been no cases of interference from the executive ISSAI 10:8 regarding the SAI’s use of its budget. 34 >>> Equitable Growth, Finance & Institutions Insight Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents DIMENSION (III) ORGANIZATIONAL INDEPENDENCE/AUTONOMY (ISSAI 1 AND ISSAI 10) a) The legal framework contains specific provisions which guarantee the organizational ISSAI 10:3 independence of the SAI. b) The relationship between the SAI and parlia- ISSAI 1:8,9 ment and also government is clearly defined. c) The SAI has the power to determine its own rules and procedures for managing business, ISSAI 1:5 consistent with prevalent rules affecting other public bodies. d) The SAI has clear governance structures 3 ISSAI 1:5 documenting how decisions are made. e) The SAI decides on all human resource matters linked to its budget in accordance with ISSAI 10:8 the constitution/legislation. f) The SAI is entitled to call on and pay for ISSAI 1:14 external expertise as necessary. g) The head of the SAI is free to appoint staff and establish their terms and conditions, constrained ISSAI 10:8 only by staffing and/or budgetary frameworks approved by parliament or its committees. DIMENSION (IV) INDEPENDENCE OF THE HEAD OF THE SAI AND ITS OFFICIALS (ISSAI 1, ISSAI 19, AND ISSAI 11) a) “The applicable legislation specifies the conditions for appointments, reappointments… removal… of the Head of the SAI and [where relevant] members of collegial institutions… by a process that ensures their independence ISSAI 11:2 from the Executive” ISSAI 10:2 (e.g., with the approval of the legislature, and where relevant, the head of state; removal only for just cause/ impeachment, similar protections to those that apply to a high court judge). b) “the Head of the SAI, and [where relevant] members of collegial institutions… [are] given appointments with sufficiently long and ISSAI 10:2 4 fixed terms, to allow them to carry out their mandates without fear of retaliation.” c) There have been no periods longer than 3 months during which there has been no official head of the SAI within the past three years. d) The last appointment of the head of the SAI was done according to the legal framework and through a transparent process that ensures their independence from the executive. e) During the last 3 years, there have been no attempts to remove the head of the SAI for reasons not covered in the legal framework and without following due process of law. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 35 International Standard on Quality Control (ISQC) 1 Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents KEY ETHICAL REQUIREMENTS (ISQC 1 PARAGRAPHS 20 TO 25) a) The SAI has the policies and procedures for establishing, promoting, and monitoring ethical ISQC 1.20 conduct among all personnel. b) The SAI has the policies and procedures for its personnel, experts contracted by the firm (SAI), and network firm personnel relating to the following: i. Integrity; ISSAI 1.5 5 ii. Objectivity; iii. Professional competence and due care; iv. Confidentiality; and v. Professional behavior. c) The SAI has the policies and procedures for identifying, communicating, and monitoring compliance with specific independence ISQC 1.21 requirements, including maintenance of adequate records. KEY HUMAN RESOURCES REQUIREMENTS (ISQC 1 PARAGRAPHS 29 TO 31) a) The SAI has the policies and procedures in place for the following: i. Recruitment; ii. Performance evaluation; iii. Capabilities; ISSAI 1:7 iv. Competence; v. Career development; vi. Promotion; and vii. Disciplinary actions. 6 b) The SAI has professional development policies and procedures, and they are communicated to staff, how these are ISQC1. 31 documented and how they are reinforced and monitored. c) The SAI has policies and procedures for assignation of responsibility for each ISQC1. 31 engagement to an engagement partner and appropriate engagement team. 36 >>> Equitable Growth, Finance & Institutions Insight Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents KEY ENGAGEMENT PERFORMANCE REQUIREMENTS (ISQC 1 PARAGRAPHS 32 TO 47) a) The SAI has policies and procedures that provide the firm (SAI) with reasonable assurance that engagements are performed in accordance with professional standards — International Standards of Auditing — and applicable legal and regulatory requirements and that appropriate reports are issued in the ISQC1. 32 circumstances. 7 Remark: Provide details of the audit methodol- ogy, whether the methodology is based on ISA or acceptable national standards, manuals, and software tools used (mapping). How regularly are the manuals and software updated? b) The SAI has supervision and review policies ISQC1. 32 and procedures at the engagement level. c) The SAI has policies and procedures regarding ISQC1. 34 required and voluntary consultations? KEY MONITORING REQUIREMENTS (ISQC–1 PARAGRAPHS 48 TO 59) a) The SAI has policies and procedures for the monitoring of compliance with quality control policies and procedures. Is your monitoring ISQC1. 48 (post issuance) in house or outsourced? How do you ensure the independence and evaluate the ability of the monitoring reviewers? b) The SAI has policies and procedures for 8 the inspection of a sample of completed ISQC1. 48 engagements. c) The SAI has policies and procedures that provide how the effects of monitoring deficiencies found are evaluated and how the ISQC1. 49 results of the monitoring is documented and communicated to engagement partners and other appropriate personnel. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 37 Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents AUDIT PLANNING REQUIREMENTS The SAI has policies and procedures to: 1) identify its stakeholders 2) develop procedures for communicating ISSAI 3100/ 9 with the media, the auditee and other key Appendix stakeholders 3.3 3) establish effective two-way communications with them. ISSAI The SAI has policies and procedures that 300/34 provide to document the audit in accordance ISSAI with the particular circumstances thereof. The 3100/23, working papers shall contain: 24 10 ISSAI a) details of audit planning 3000/4.2 b) results of the audit execution ISSAI c) audit evidence to support all audit findings, 3000/ Appendix 3, conclusions, and recommendations. 3.9, 4 The SAI plans and performs the audit to determine whether the subject matter, in all ISSAI 11 4100/69 material respects, is in compliance with the stated criteria. The SAI obtains an understanding of internal ISSAI 12 control relevant to the audit objective and test 4100/66 controls on which they expect to rely. The SAI identifies and assesses fraud risk and auditor plan and gathers sufficient appropriate ISSAI 13 evidence related to identified fraud risks through 4100/82 the performance of suitable audit procedures. The SAI has policies and procedures that require the auditor, before starting the main study, to define the audit objectives, the scope, and the methodology to achieve the objectives. This could be done in the form of a pre-study. Where a pre-study is conducted it: 14 a) establishes whether the conditions for a main study exist b) provides background knowledge and information needed to understand the entity, program, or function c) is carried out in a fairly short period. 38 >>> Equitable Growth, Finance & Institutions Insight Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents AUDIT PLANNING REQUIREMENTS The SAI includes the following elements in the planning documents: a) the background and information about the audited entities, so as to allow an assessment of the problem and risk, possible sources of evidence, auditability, and the significance of the area considered for audit; b) the audit objective, scope, questions or hypotheses, criteria, and period to be covered by ISSAI the audit, methodology (framework, perspective, 300/34, 37 ISSAI 15 and analytical structure that were adopted and 3100/12 the process that was followed to arrive at the ISSAI conclusions, including techniques for gathering 3000/3.3 evidence and conducting the audit analysis); c) the necessary activities, staffing, and skills requirements (including the independence of the audit team, human resources, and possible external expertise); d) the estimated cost of the audit, the key project timeframes and milestones, and the main points for control. The SAI documents discussions of significant matters with management, those charged with ISSAI 16 governance, and others, including the nature of 1230.10 the significant matters discussed and when and with whom the discussions took place. The SAI makes inquiries of management regarding: a) Management’s assessment of the risk that the financial statements may be materially misstated due to fraud, including the nature, extent, and frequency of such assessments; b) Management’s process for identifying and responding to the risks of fraud in the entity, including any specific risks of fraud that management has identified or that have ISSAI 17 1240.17 been brought to its attention, or classes of transactions, account balances, or disclosures for which a risk of fraud is likely to exist; c) Management’s communication, if any, to those charged with governance regarding its processes for identifying and responding to the risks of fraud in the entity; and d) Management’s communication, if any, to employees regarding its views on business practices and ethical behavior. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 39 AUDIT EXECUTION — Collecting and analyzing evidence Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents AUDIT EXECUTION REQUIREMENTS The SAI has a systematic and iterative evidence gathering process that involves a) Gathering the evidence through the use of appropriate audit procedures ISSAI 18 4100/94 b) Evaluating the evidence obtained as to its sufficiency and appropriateness c) Re-assessing risk and gathering further evidence as necessary. Audit evidence is gathered using a variety of techniques such as: a) Observation b) Inspection ISSAI 19 4100/97 c) Inquiry d) Re-performance e) Confirmation f) Analytical procedures. ISSAI 300/38 ISSAI 3000/2.3 The SAI arranges that the work delegated to ISSAI 20 the auditor is carefully directed, supervised and 3100/19 reviewed. ISSAI 3100/2.5, 38 ISSAI 3000/ Appendix 4 ISSAI 300/38 ISSAI 3100/21 ISSAI 3000/4.2, 4.3 ISSAI 3000/ Appendix The SAI obtains sufficient, appropriate 3, 2 ISSAI 3000/ (valid, reliable, and relevant) audit evidence Appendix 21 to establish findings, reach conclusions in 3, 1.2 response to the audit objectives and questions, ISSAI 3000/ Appendix 3, and issue recommendations. 2.2 ISSAI 3100/20 ISSAI 3000/5.2 ISSAI 3000/ Appendix 3, 1 ISSAI 3000/5.3 40 >>> Equitable Growth, Finance & Institutions Insight Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents AUDIT PLANNING REQUIREMENTS The SAI communicates to management at an appropriate level of responsibility on a timely basis: (a) in writing, significant deficiencies in internal control that the auditor has communicated or intends to communicate to those charged with governance, unless it would ISSAI 22 be inappropriate to communicate directly to 1265:10 management in the circumstances; and (b) other deficiencies in internal control identified during the audit that have not been communicated to management by other parties and that, in the auditor’s professional judgment, are of sufficient importance to merit management’s attention. On or before the date of the SAI’s report, the engagement partner, through a review of the audit documentation and discussion with the ISSAI 23 engagement team, is satisfied that sufficient 1220.17 appropriate audit evidence has been obtained to support the conclusions reached and for the auditor’s report to be issued. SAI takes care to ensure that reports presented are factually correct, and that findings and ISSAI 24 4100/137 conclusions are presented in the proper perspective and in a balanced manner. SAI reports are constructive and include, as ISSAI 25 appropriate, recommendations designed to 4100/153 result in improvements. The SAI includes in the audit report information about the: • Audit objectives • Audit questions and answers to those ISSAI questions 300/39 • Subject matter ISSAI 3100/ 26 sec. 28,30 • Criteria ISSAI • Methodology 3000/sec. • Sources of data 5.2 • Limitations on the data used • Audit findings, conclusions, and recommendations. The SAI ensures that the findings clearly ISSAI 3100/ 27 conclude against the audit question or explain sec. 30 why this was not possible. The Quality of Audits by Supreme Audit Institutions: A Review of Compliance with International Standards of Supreme Audit Institutions <<< 41 Provide Reasons Evidence to Compliance SAI Validation for Noncom- Confirm Com- No. Requirements Reference Assessment of Assessment pliance and pliance Status (Yes or No) (Yes or No) Partial Com- and Reference pliance Documents AUDIT EXECUTION REQUIREMENTS The SAI explains in the report why and how problems noted in the findings hamper ISSAI 28 performance in order to encourage the audited 1220.20 entity or report user to initiate corrective action. The SAI’s engagement quality control reviewer performs an objective evaluation of the significant judgments made by the engagement team and the conclusions reached in formulating the auditor’s report. This evaluation shall involve: (a) Discussion of significant matters with the engagement partner; (b) Review of the financial statements and the ISSAI 29 1230.14 proposed auditor’s report; (c) Review of selected audit documentation relating to the significant judgments the engagement team made and the conclusions it reached; and (d) Evaluation of the conclusions reached in formulating the auditor’s report and in consideration of whether the proposed auditor’s report is appropriate. The SAI assembles the audit documentation in an audit file and completes the administrative ISSAI 30 process of assembling the final audit file on 1230.14 a timely basis after the date of the auditor’s report. Source: World Bank. Note: ISA = International Standards on Auditing; ISQC = International Standards on Quality Control; ISSAI = International Standards of Su- preme Audit Institutions; SAI = Supreme Audit Institution. 42 >>> Equitable Growth, Finance & Institutions Insight