GOVERNANCE GOVERNANCE EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT Supreme Audit Institutions’ Use of Information Technology Globally for More Efficient and Effective Audits © 2021 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO), http:// creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: 2021. Supreme Audit Institutions’ Use of Information Technology Globally for More Efficient and Effective Audits. EFI Insight-Governance. Washington, DC: World Bank. Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an official World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third- party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to reuse a component of the work, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; e-mail: pubrights@worldbank.org. Graphic Designer: Maria Lopez / lopez.ten@gmail.com >>> Contents Acknowledgments 4 Abbreviations 5 Executive Summary 6 Introduction 8 Snapshot of the Use of Information Technology-Based Audit Methods and Procedures by Supreme Audit Institutions 10 Private Sector Audit Companies More Advanced in Use of IT 11 Increased Use of Advanced IT by SAIs 12 Challenges Faced by SAIs in Making More Extensive Use of IT 16 Auditor-General of South Africa Capitalizes on the Opportunities Created by Technology for Audits 17 INTOSAI’s Advocacy of the Use of IT for Audit 19 SAIs’ Response to COVID-19 21 International Audit Standards and IT-Based Methods and Procedures 22 Conclusions and Recommendations for the Way Forward 24 Appendix A: How SAIs Are Responding to the Challenges of Implementing Advanced IT Audit Methods and Procedures 26 Appendix B: Challenges Faced by SAIs in Developing Countries and the Experience of the Auditor-General of South Africa 37 Appendix C: INTOSAI Strategy and Initiatives to Promote SAIs’ Use of IT 41 Appendix D: Private Sector’s Experience Using Audit Data Analytics and Other Advanced IT-Based Audit Methods and Procedures 46 The Experience of Private Sector External Audit Companies 47 Internal Audit Examples 51 Coca Cola Case Study 51 Credit Suisse Case Study 52 Dublin Airports Authority Case Study 52 Appendix E: International Audit Standards and IT-Based Audit Methods and Procedures 54 Appendix F: Reference Websites 58 >>> Acknowledgments This paper was developed by Francis Grogan and by Mona El-Chami, Zohra Farooq Nooryar, Srinivas Gurazada, Arun Manuja, Moses Wasike, and Seongjun Kim from the World Bank Accountability and Oversight Institutions Community of Practice, led by Manuel Vargas. It was reviewed by Patrick Kabuya, Joao Vicente, Love Ghunney, Khuram Farooq, and Suraiya Zannath. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 4 >>> Abbreviations ACL Audit Command Language AI Artificial Intelligence AGSA Auditor-General of South Africa ANAO Australian National Audit Office CNAO National Audit Office of China CAATs Computer-Assisted Audit Techniques DLT Distributed Ledger Technology ECA European Court of Auditors FRC Financial Reporting Council GAO Government Accountability Office IAASB International Auditing and Assurance Standards Board ICT Information and Communication Technology IDEA Interactive Data Extraction and Analysis IFAC International Federation of Accountants INCOSAI International Congress of Supreme Audit Institutions INTOSAI International Organization of Supreme Audit Institutions ISAs International Standards on Auditing ISSAIs International Standards of Supreme Audit Institutions IT Information Technology ML Machine Learning NAO National Audit Office NLP Natural Language Processing OECD Organisation for Economic Co-operation and Development SAI Supreme Audit Institution SCEI Supervisory Committee on Emerging Issues SDGs Sustainable Development Goals STAA Science, Technology Assessment, and Analytics SQL Structured Query Language TCU Tribunal de Contas da União WGISTA Working Group on Impacts of Science and Technology on Auditing EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 5 >>> Executive Summary Supreme audit institutions (SAIs) recognize the benefits To help SAIs address these challenges and enable them to of using technology to improve the quality and impact of capitalize on the benefits and opportunities created by rapid their audits. This benefit has further intensified during the advances in the technology, this paper suggests initiatives that COVID-19 pandemic; SAIs with existing technology capacity fall into two broad categories: quick, short-term actions and have continued to perform their role effectively and efficiently. medium- to long-term actions. SAIs must be the main driver of As governments are advancing in utilizing technology, by these initiatives, with help from the International Organization developing sophisticated systems and exploring innovative of Supreme Audit Institutions (INTOSAI), regional bodies, tools and solutions including artificial intelligence and data donors, and development organizations. analytics, among others, SAIs must strive to be equally equipped to provide the needed oversight. In the short-term, SAIs can: The available evidence suggests a wide range in SAIs’ use • Conduct a survey of INTOSAI members to identify the level of information technology–based tools and procedures. At one of maturity of each SAI in the use of audit data analytics extreme, SAIs in some developing countries have little or no and IT-based audit tools. The results of such a survey information technology (IT) facilities or infrastructure. At the could be used to support SAIs’ use of IT and to facilitate other extreme, a small number of SAIs have the resources the exchange of experiences and good practices among to invest in the development, implementation, and use of SAIs. The survey results would be a valuable source of highly sophisticated IT-based audit tools and procedures. The information for other development partners in designing majority of SAIs are somewhere between these two positions their support to SAIs at the national or regional level. and use software packages that are available commercially or under license, such as desktop analytics, electronic document • Design a set of diagnostic/case studies that would management systems, and specialist audit tools. encompass lessons learned, the potential pitfalls to be avoided, quick fixes to common problems, and the key SAIs face three challenges in responding to the evolution components of successful longer-term IT strategies and growth of the sophisticated IT tools and applications that for SAIs. are available to them for the audit of governments and other public bodies and entities. The first challenge is the need to • Organize targeted events and seminars encouraging develop techniques and procedures for the audit of the more the use of IT by SAIs to more effectively use audit data sophisticated systems and applications used by auditees. analytics and IT-based audit tools and to identify ways to The second challenge is harnessing the opportunities created facilitate that process. These include, for example, the use by IT to conduct audits more efficiently and to develop audit of smartphones and drones for audit purposes; the audit procedures that will enhance SAIs’ effectiveness and that will of blockchain transactions; the implications for audit of the contribute to more impactful audit outcomes. The third, and Internet of Things; the use of augmented intelligence or arguably most difficult challenge, is to find the resources, artificial intelligence (AI) for audit purposes (for example, experience, and expertise needed to develop, implement, to contribute to risk assessments); and the lessons learned and use audit data analytics and IT-based audit tools and from SAIs’ responses to the challenges of COVID-19. procedures. There is also scope for sponsoring events that consider EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 6 how SAIs can use IT to support new or evolving types of In the long-term, SAIs can: audit, such as work focusing on social audit mechanisms. • Establish a fund that international partners can contribute In the medium-term, SAIs can: to, to support professional training for SAI auditors in the use and application of audit data analytics and other IT- • Support the development of SAIs’ use of IT and IT-based based audit tools. If sufficient resources are made available, audit processes and procedures customized to low- the proposed fund could also support the provision of the capacity settings; the development of appropriate case relevant audit software (such as licenses) as well as the studies to encourage SAIs to make more extensive use required hardware. of, for example, audit data analytics; and encourage more advanced SAIs to establish a mentoring role with individual • Promote complementary technical assistance to SAIs SAIs in developing countries with a view to providing alongside the financing of public sector IT capacity continuing support and expertise. strengthening. For example, if a donor-funded project includes financing of automation of public procurement, • Develop detailed technical guidance for all SAIs (but revenue administration, integrated financial management specifically for SAIs in developing countries) that focuses information systems, etc., the project design is encouraged on best practice in developing, implementing, and using to include a component for SAI capacity strengthening to audit data analytics and IT-based audit methods and tools. effectively audit those systems. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 7 >>> Introduction The World Bank prepared this paper on the basis of a desk of IT to enhance the quality and impact of their audits. Though review of recent relevant literature to inform the work of a range of factors inhibit the more extensive use of IT by SAIs staff concerned with developing policies and approaches to in general, and by SAIs in developing countries in particular, support and enhance the capacity development of supreme the key factors revolve around a lack of resources to facilitate audit institutions (SAIs). The paper explores how at a global investment in the necessary hardware and software and a lack level SAIs are using technology to perform more efficient and of expertise to maintain and operate IT systems once they are more effective audits. It provides a brief overview of how some in place. Private sector experience illustrates that the potential SAIs are harnessing the possibilities created by advances benefits to SAIs that are making more extensive use of IT are in technology to develop new, innovative audit methods and more efficient, high quality audits. procedures. It also seeks to identify the factors inhibiting other SAIs—in particular SAIs in developing countries—from The potential benefits are also borne out by SAIs’ early implementing and using audit methods based on information experience of responding to COVID-19. The initial evidence technology (IT). Against this background, the paper suggests suggests that those SAIs that had already invested in their ways in which the World Bank, working with other stakeholders, IT infrastructure, and in particular in remote working, had can facilitate the more extensive and more effective use of IT- the necessary building blocks in place and, so, have been based tools and methods by SAIs. able to respond with greater agility and resilience to meet the challenges the pandemic created. Also, the experience Initially, this paper was prepared prior to the COVID-19 of working through the pandemic has accelerated the trend outbreak with a view to assessing how SAIs can respond to toward remote working among SAIs, and a number of SAIs are the challenges posed by advances in IT and how they can looking again at their traditional audit approach to see how they use more sophisticated IT-based techniques to deliver better, can capitalize on the innovative IT-based approaches they more effective audits across the full range of their remits and have developed in response to the challenges of the pandemic. mandates. The impact of COVID-19 has introduced a new important consideration: namely, how IT has helped some The paper is based on the extensive review of material that SAIs respond with agility and resilience to the unprecedented is available in the public domain. Because of the nature and and completely unforeseen circumstances created by range of the issues identified in the process of preparing the the pandemic. paper, and their interconnections, the paper is structured around the following key themes: In brief, the paper illustrates wide differences between SAIs in the use of IT. A number of SAIs in several Organization for • SAIs use of IT-based audit methods and procedures Economic Co-operation and Development (OECD) countries, • Private sector audit companies more advanced use of IT and in Brazil, China, India, and others, operate at a very • SAIs increased use of advanced IT sophisticated level and have developed the capacity to use • Challenges faced by SAIs in making a more extensive advanced IT-based technology and methods. But this is not the use of IT case for the majority of SAIs. For example, research carried • Example of the Auditor-General of South Africa (AGSA) out by AFROSAI-E shows that SAIs in developing countries in capitalizing on the opportunities created by technology are at a very basic level in their use of IT and the availability for audit EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 8 • Advocacy of the International Organization of Supreme research undertaken by other organizations. The following Audit Institutions (INTOSAI) of the use of IT for audit topics are covered in the appendixes: • SAIs’ response to COVID-19 • International audit standards and IT-based methods • Appendix A: How SAIs are responding to the challenges of and procedures implementing advanced IT audit methods and procedures. • Appendix B: Challenges faced by SAIs in developing The paper ends with conclusions and suggestions for the way countries and the experience of AGSA. forward for the World Bank, SAIs, and stakeholders as they • Appendix C: INTOSAI’s strategy and initiatives to promote plan, support, and implement initiatives to encourage SAIs to the use of IT by SAIs make more extensive and ambitious use of IT and IT-based • Appendix D: Private sector experience of using audit data audit methods to increase the efficiency, effectiveness, and analytics and other advanced IT-based audit methods impact of their audits. and procedures • Appendix E: International audit standards and IT-based The conclusions and suggestions are followed by a series of audit methods and procedures supporting appendixes that capture relevant information and • Appendix F: Reference websites EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 9 >>> Snapshot of the Use of Information Technology-Based Audit Methods and Procedures by Supreme Audit Institutions Although there has not as yet been a comprehensive, definitive financial audits lend themselves to the use of a common set survey of supreme audit institutions (SAIs) that use information of IT-based audit procedures and of IT-based audit software technology-based (IT-based) audit methods and procedures, it and packages that can be bought from a specialist commercial is possible to give a reasonably accurate—though broad-brush supplier or used under license. snapshot—of SAIs’ experience using IT-based tools. Third, the range of basic IT tools and applications that most First, a wide range of relative SAI experiences in general IT SAIs use encompasses the following: capacity and expertise exists. At one extreme, SAIs in the poorest developing countries have access to either very limited • Basic or ad hoc and desktop analytics such as Excel and or no IT resources and applications. At the other extreme, a the use of spreadsheets small number of SAIs are developing and implementing • Specialist audit document management systems available increasingly sophisticated and powerful IT tools and IT-based under license such as TeamMate and CaseWare analytical methods. • Specialist audit tools such as IDEA (interactive data extraction and analysis) and ACL (audit command Second, SAIs use IT tools and applications predominantly in language) the course of their financial audits. Some IT applications are • SQL (structured query language) and SQL-based tools that used for performance audits. But because performance audits, manipulate data stored in management systems operated overall, account for a relatively small proportion of SAI activities on Oracle and Microsoft platforms when taken as a whole, the IT applications used tend to be • Statistical analysis tools such as IBM’s SPSS software those needed for a piece of analysis specific to the individual and SAS (statistical analysis software) developed by the performance audit. By contrast, because each SAI applies a SAS Institute standard methodology and approach to all of its financial audits, EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 10 >>> Private Sector Audit Companies More Advanced in Use of IT By contrast, in general, private sector audit firms are more increasing volumes of data. These technologies may enable advanced in their use of sophisticated information technology- auditors to assess and potentially interpret or extrapolate based (IT-based) audit methods and procedures. When private entire populations of data. They include technologies linked to sector audits are regularly rotated, the available evidence artificial intelligence (AI) such as machine learning (ML) and indicates that competition between audit firms has encouraged natural language processing (NLP). Emerging technologies them to develop and implement better IT-based audit methods that contribute to more efficient audit processes include the and procedures. To date, the use of IT-based advanced use of drone, Internet of Things, and sensor technologies in, for analytics has been most prevalent for risk assessment, audit of example, inventory inspection in which scale and distribution revenue, and in testing technology-based internal controls and is an issue. Firms also are developing their own smartphone- automated processing systems’ integrity. Although firms have based “apps” for recording the results of specific audit tasks established different priorities in their adoption and rollout of and procedures. automated tools, it is evident that all firms continue to develop and enhance their use of technological resources by making Looking further into the future, private sector audit firms are substantial investment in the infrastructure needed for the assessing the implications for their work of other emerging capture, collation, and analysis of the relevant data. technologies. These other technologies include cloud technologies, particularly with regard to security and providing In addition, private sector audit firms are exploring the assurance around cloud-based systems, and also distributed potential benefits of using emerging technologies for audit ledger technology (DLT), a family of technologies that includes purposes. The emerging technologies fall into two broad blockchain. DLT is of great interest to both auditors and categories: those that have the potential to improve audit businesses because it has the potential to become a sort quality and those that drive efficiencies in the audit process. of universal bookkeeping service that removes the need to The emerging technologies that may contribute to improving reconcile multiple databases of records and, so, potentially audit quality potentially enable auditors to deal with complex provides a clear audit trail. corporate entities, complex electronic transaction flows, and EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 11 >>> Increased Use of Advanced IT by SAIs Among supreme audit institutions (SAIs), there is a shared that will expand GAO’s support to lawmakers on cutting-edge recognition that they need to respond to the challenge posed topics, such as artificial intelligence, regenerative medicine, by governments increasing their use of sophisticated IT to 5G wireless communication, and quantum computing.”1 This manage and deliver their policies and programs. At the same new unit also encompasses a specialist audit innovation lab “to time, SAIs also want to capitalize on the opportunities created explore, pilot, and deploy new advanced analytic capabilities, for better, more effective audits through the use of IT audit conduct research in information assurance, and explore tools and procedures. This is being driven by a number of emerging technologies that will impact future audit practices.”2 factors. First, SAIs recognize the need to address the audit risks posed by the use of IT financial management systems. In turn, this development flowed from the GAO’s concern Second and more broadly, SAIs must respond to stakeholder “to increase [its] capabilities to review the opportunities and needs in relation to the assurance, advice, and analysis that challenges associated with evolving science and technology they are expected to provide regarding governments use of issues; the risks and management needs to address complex and increasing dependence on IT systems. A third key factor and growing cyber security developments; increased is the need for SAIs, in particular those in countries with investments in the Department of Defense; and rising more advanced economies, to keep pace with the capability health care costs.” In establishing its Science, Technology of audit firms in the private sector and the quality of service Assessment, and Analytics (STAA) team, the GAO’s aim they provide. was to expand the support and advice it provides to the US Congress by The experiences of the US Government Accountability Office (GAO), the Australian National Audit Office (ANAO), the • Conducting technology assessments and providing European Court of Auditors (ECA), and the National Audit technical services; Office of the United Kingdom (NAO) all serve to illustrate • Reviewing science and technology programs and initiatives this need. to assist in oversight of federal investments in research, development, and advanced manufacturing; In 2018, the US Congress passed spending legislation that • Compiling and utilizing best practices in engineering directed the GAO to increase its capacity in science and sciences, including cost, schedule, and technology technology analysis. And in January 2019, the GAO established readiness assessments; and a new science, technology assessment, and analytics team to • Establishing an audit innovation lab to explore, pilot, better meet Congress’s growing need for information on science and deploy new advanced analytic capabilities, conduct and technology issues. In doing this, the GAO combined and research in information assurance, and explore emerging enhanced “its technology assessment functions and its science technologies that will impact future audit practices. and technology evaluation into a single, more prominent office 1. Government Accountability Office, “GAO Deepens Science and Technology Capabilities,” Press release, January 29, 2019. 2. American Institute of Physics, “Congress Bolstering Its Access to S&T Expertise,” May 9, 2019, https://www.aip.org/fyi/2019/congress-bolstering-its-access-st-expertise. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 12 The GAO’s budget request for fiscal 2021 built on these For many years, the private sector audit firms have been themes.3 An important part of the request focused on the exploring how to make best use of data-driven audits. resources required for its STAA team and to support “priority The Big Four audit firms have embraced the opportunity investments” in information technology. According to the to explore areas such as artificial intelligence (AI) and budget request, in 2021 the STAA will focus on the following: blockchain, so why should supreme audit institutions not follow the same path? In 2017, the ECA responded to this • Equipping a state-of-the-art collaborative space to conduct question by launching the ECALab, an in-house center data science experiments and to define, design, and for research and innovation. This space for sharing ideas, execute advanced analytics pilot projects. exploring, testing, and implementing technologies in the • Building prototypes of emerging technologies such as audit process is part of the ECA’s digital transformation blockchain/digital ledgers as well as machine learning/ initiative. The ECALab is comprised of data science artificial intelligence systems. enthusiasts and expert auditors who cooperate on finding • Developing a framework for algorithmic accountability in tailored solutions to audit tasks and audit-related projects. machine learning/artificial intelligence systems. The work of the ECALab is still at an early stage and its initial The GAO believes that the lab will “provide Congress with activities have focused on data visualization. In ECA’s view, critical foresight, oversight, and insight of science and this allows for innovative ways of presenting data graphically technology issues to help ensure continued American and greatly facilitates the task of data analysis. The ECALab innovation, competitiveness, and security.” is also exploring the potential of text mining and developing a new tool, Document Navigator, to help ECA’s auditors in The ANAO created its Systems Assurance and Data Analytics this regard. Group in 2017. ANAO has stated that the focus of the work of this group will be on increasing the quality and productivity of For the NAO, a central element of its strategic planning is its audit work “in an environment of increased data collection investment in its in-house IT capability to make greater use and generation in the public sector.”4 of technology in its audit work.6 The NAO sees three benefits flowing from this approach: In July 2020, the ANAO issued its Corporate Plan for 2020– 21.5 This plan confirmed that with regard to technology, the • Improvement in the quality of NAO’s work through the use ANAO continues to focus on “keeping pace with advances in of automated tools that would help NAO auditors focus technology” by their efforts on the important risks to their audits • A better understanding of business processes and • Improving ANAO’s connective technology; supporting technology • Investing in “modern consumption-based cloud computing”; • Enhanced technology that would help the NAO • Building high-capacity data storage and computer keep pace with other audit institutions and UK platforms to support enhanced data analytics, modeling, government departments and automation; and • Enhancing ANAO’s security environment with a focus The NAO sees technology as a central part of its financial audit on maintaining organizational cyber resilience and methodology. The office notes that to advance its testing of streamlining secure data governance collection and financial systems and its understanding of audited bodies, it classification methods. has been trialing new analytical techniques in its audits. This work has identified opportunities to analyze transactions The ECA is another audit institution that is developing its IT more completely, supporting additional insight and value- capability. The ECA has established the ECALab to serve “as adding activities. an in-house incubator for exploring new ideas in audit and offers auditors a variety of techniques that can assist in achieving In this context, the NAO has commented that “in line with other individual audit goals.” The ECA explains its rationale for this firms in the industry” it will continue to develop this work stream initiative as follows: as a priority. It plans to maintain its annual investment in the 3. Government Accountability Office, “Fiscal Year 2021 Budget Request,” GAO-20-429T, 15–16. 4. Australian National Audit Office, “Corporate Plan 2018–19,” July 5, 2018, https://www.anao.gov.au/work/corporate/anao-corporate-plan-2018-19. 5. Australian National Audit Office, “Corporate Plan 2020–21,” July 1, 2020, https://www.anao.gov.au/work/corporate/anao-corporate-plan-2020-21. 6. National Audit Office, “NAO Strategy 2019–20 to 2021–22,” December 2018, 15–34. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 13 use of technology to augment its financial audit approach. The The Tribunal de Contas da União (TCU, the supreme audit NAO sees this investment as essential to it being able to audit institution of Brazil) is one of the more advanced supreme audit new technologies (such as cloud-based computing) that are institutions with regard to its thinking in relation to exploiting the used increasingly by the entities that it audits. In addition, NAO possibilities created by information technology, specifically the sees this investment as necessary to ensure that its use of use of data analytics. The use of information technology has technology stays in line with developments in the wider audit enabled the TCU to adopt a new approach that it describes profession. The NAO concludes that “in this way, we can as “continuous monitoring” in auditing welfare programs that make sure that Parliament receives audit services that are account for expenditure of about R$500 billion a year. The comparable in quality and capability to those offered by the top process enables auditors to specify and set the parameters commercial audit firms.” for their audit tests and audit procedures. In this way, the TCU believes its testing provides for a more selective way of In addition to embedding IT investment as a core element of defining control actions and also provides a more risk-based the SAI strategy, SAIs are increasingly using sophisticated IT- audit approach.9 Also, the TCU has experimented with other based methods and analytics for specific audit purposes. forms of new technology to enhance the quality and impact of its audits. One example is the work that the SAI is doing to The National Audit Office of China (CNAO) uses sophisticated assess the contribution that georeferenced data could make to IT tools for its audits.7 For example, in undertaking audit of land its audits in which environmental issues are a key aspect (such transfer payments and the protection of arable land, the CNAO as audits involving the assessment of schemes affecting the analyzed and correlated structured and unstructured data from environment in the Amazon basin or the construction of roads multiple sources. These sources included remote sensing in remote areas). This approach involves using and analyzing image data from the Department of Land, planning data, data data available from government and other sources based on on and mapping of forests, data from the agricultural sector, GPS systems. Another striking innovation is the TCU’s initiative data from the Ministry for Development and Reform, as well as to encourage greater citizen awareness of and participation in relevant industrial, commercial, and tax data. The CNAO has its work through the release of mobile applications that can be also used data matching for an audit of an affordable housing downloaded to Android and IOS devices.10 scheme. The audit examined a range of issues including the construction of houses and their distribution to beneficiaries. The Office of the Comptroller and Auditor General of India (SAI In planning the audit, the CNAO concluded that reviewing India) provides another example of an SAI that is developing each beneficiary’s application would be too time consuming a sophisticated approach to the use of data analytics.11 It has and that focusing on a small, random sample of applications carried out two pilot studies to explore and test the potential would not provide the insights it wanted to obtain. Instead, contribution of “big data” and the use of data analytics. the CNAO opted for an approach that involved matching the data from different sources. It concentrated its examination The first pilot study was a performance audit that focused on one key qualification for access to houses constructed on the administration in one Indian state of three social under the scheme, namely that beneficiaries did in fact have assistance programs—old-age pension, widows’ pension, low incomes and did not own other assets such as other and disability pension. Although the structure of the audit houses or expensive motor vehicles. Accordingly, the auditors followed a traditional, established approach to undertaking crosschecked details of the scheme’s beneficiaries with data performance audits, in the design and implementation of the held by other state institutions on car ownership and property audit the SAI decided to make increased use of available data ownership. This enabled the CNAO to identify individuals who and research evidence. Specifically, in addition to analyzing had obtained housing under the scheme but who were not in and testing databases of beneficiaries, the SAI made use of fact entitled to that housing. The SAI found that a number of external databases such as census data and data on people these individuals were friends or relatives of the individual in with incomes below the poverty line. This enabled the SAI to charge of the affordable housing scheme go beyond a traditional outcome of this type of audit—with its 7. National Office of China, “National Audit in the Big Data Environment” (Paper presented at meeting of INTOSAI Working Group on IT audit, 2017). 8. Alexander Juliano, “The 10th ASOSAI Research Project Report: Audit to Detect Fraud and Corruption - Evaluation of the Fight against Corruption and Money Laundering,” Fraud Audit Office, Philippines Commission on Audit, Asian Organization of Supreme Audit Institutions, 2015, 197 and 200. 9. Revista do TCU (Federal Court of Accounts Journal, Brazil) 47, no. 133 (May/August 2015): 13. 10. Revista do TCU (English version) 47, no. 133 (May/August 2015): 36–37. 11. The source for the material about SAI India is a document entitled “A Paper on Big data Analytics in SAI India.” The document is not dated but its contents indicate that it was prepared after February 2016. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 14 focus on the payment of benefits to individuals who do not about rates of infant mortality in different regions) and linked meet the appropriate eligibility criteria—and identify potential this data with the variables identified from the word cloud using beneficiaries who, for whatever reason, had been excluded linear correlation analysis and multivariate regression analysis. from the relevant scheme. The results served to confirm which variables appeared to have the most impact on infant mortality rates in the districts The second pilot study was concerned with an analysis of infant being examined. In turn, this enabled the SAI audit team to mortality rates. A conventional audit on this type of subject concentrate its examination on those government interventions would have focused on the implementation of a scheme or that had potentially the most effect and, conversely, to identify schemes intended to reduce infant mortality rates and would those interventions likely to be less effective. probably have refrained from commenting on whether the schemes examined had factored into their design all significant SAIs that are at the leading edge in the development and factors that affect the level of infant mortality rates. The application of a sophisticated and advanced IT capability study used a number of innovative techniques as far as SAI share three characteristics. First, they are able to build on a performance audits are concerned. The first step was a text strong existing IT infrastructure. Second, they have access analysis of the relevant research literature using an IT tool that to the resources, experience, and expertise necessary to enabled the study team to produce a word cloud that identified implement and support the necessary investment in new IT the key variables that appeared from the research to have the systems and methods. Third, using IT to improve the efficiency most significant impact on infant mortality. The study team next and quality of the SAI’s audit is a core component of the SAI’s used data from official sources (such as census databases business strategy. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 15 >>> Challenges Faced by SAIs in Making More Extensive Use of IT The contrast with SAIs in developing countries is striking. in the region and specifically in relation to the audit of IT and Research and analysis conducted by AFROSAI-E, the regional information systems. According to AFROSAI-E, this is illustrated group of predominantly (but not exclusively) English-speaking specifically by the small number of specialist IT auditors African SAIs, provide the most comprehensive snapshot of the employed by SAIs in the region, by limited IT infrastructure and nature and extent of the use of IT by African SAIs. In summary, equipment, and by the shortage of technical IT audit skills in these SAIs face the following challenges: the region. • A chronic lack of information and communications Taking a wider view, in addition to the challenges around technology (ICT) staff across the region resources and skills identified in the AFROSAI-E research, it • A lack of professional capacity to audit IT-based information is evident that other factors constrain the capability of SAIs in systems developing countries to capitalize on the opportunities created • A lack of technical IT capacity and resource constraints by advances in IT. These factors revolve around structural means that for 2019, 14 SAIs of the 23 countries that have issues such as the absence of uniformity between a country’s integrated financial management expenditure and revenue financial management information system and its public systems do not audit those systems financial management and external audit systems. These • According to the report for 2017, just seven of the SAIs issues are then exacerbated by poor ICT infrastructure; a lack that completed the relevant sections of the AFROSAI-E of relevant data and poor quality and reliability even when research document had developed and implemented data is available; and capacity issues around the collection documented IT/information system audit strategies. of data and poor legal and regulatory requirements about the handling and security of data. In general, there has been a lack Overall, AFROSAI-E believes that these results show that of coordinated, concerted action on the part of donors (and funding constraints and inadequate technical skills are limiting arguably key stakeholders such as INTOSAI) to recognize the factors when it comes to the development of audit knowledge links between these issues and to address them accordingly. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 16 >>> Auditor-General of South Africa Capitalizes on the Opportunities Created by Technology for Audits Despite the challenges that many countries face, some systems auditing.14 It defines this as the examination of supreme audit institutions (SAIs) are putting technology at controls within an entity’s IT environment that focuses on the heart of what they do and they are capitalizing on the “determining whether adequate controls that are operating opportunities that technology creates. The Auditor-General of effectively to mitigate risks to ensure that data is protected, South Africa (AGSA) is one such example. reliable, and available.” The AGSA began its information technology audits in the early Among the initiatives that the AGSA is taking to implement its 1990s and subsequently established a specialist unit to support strategic plan for the period 2017 to 2020, it is implementing a its work in this area.12 The AGSA Information Systems Auditing project to enhance its audits through the use of data analytics. business unit provides support for all of the organization’s In the AGSA, the main purpose of the use of data analytics other business units. has been to support auditors during the planning or execution of an audit. To date, the AGSA has typically performed this The Information Systems Auditing business unit is an integral work under the banner of computer-assisted audit techniques part of the AGSA’s audit processes and supports the office’s (CAATs). In addition, the AGSA has used trend analysis tools units that are responsible for financial audit by, for example, and data visualization techniques “to identify areas of high risk audited entities’ IT internal controls. It also supports the AGSA’s and anomalies to improve audit planning” and to enhance the Investigations unit by using data analytics to monitor public AGSA’s ability to focus its audit effort “on what matters.”15 entities’ payment systems to identify questionable transactions. Building on this work, the AGSA is developing enhanced CAATs In addition, the Information Systems Auditing business unit (that is, enhanced data analytics) to provide an automated carries out financial audits in relation to entities at the national, “solution” for two key areas of its audit activities: the audit of provincial, and municipal levels.13 human resources and payroll and the audit of supply chain management.16 In relation to the use of data analytics for the The AGSA describes the examinations that the Information audit of human resources and payroll, the AGSA believes Systems Auditing business unit carries out as information that “fewer hours are spent on the audit . . . and efficiencies 12. Auditor-General of South Africa, “The Supreme Audit Institution (SAI) of South Africa’s Approach to Supporting Financial Audits on Human Resources, Payroll and Supply Chain Management through the Use of Data Analytics” (country paper, May 23, 2017). 13. Auditor-General of South Africa, 3. 14. Auditor-General of South Africa, 2. 15. Auditor-General of South Africa, 4–9. 16. Auditor-General of South Africa, 10–12. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 17 are gained. The quality of the audit procedures is centrally accurate and up-to-date. The AGSA’s audit of procurement monitored through the development of CAATs and associated has had a significant impact. In the period between 2012/13 working papers. Reporting . . . will be more consistent. The and 2015/16, it identified questionable payments to the value approach eliminates the need for ad hoc audit procedure of R$2.636 billion. This discovery led to the introduction of new design and different approaches having to be followed for the government procurement regulations intended to strengthen same system and similar business processes.”17 procurement and contract management in the public sector of South Africa. The audit of supply chain management represents a major area of concern for the AGSA. It currently uses procurement The approach adopted by the AGSA in strengthening its IT data analytics as a key element of its audit approach. Its capability illustrates again the key theme emerging from our planning in this regard is predicated on the assumption that assessment of the use of technology by SAIs and private there is a significant risk of fraud in the procurement practices sector audit firms—namely, that the impact of technology lies of the entities it audits; and, in its audit risk assessments, the in its capacity to improve the efficiency of the audit process AGSA has identified a significant number of audited entities and the quality and effectiveness of the audit itself. displaying fraud indicators. To respond to the challenge this creates for the audit of such a large number of entities, the However, an important difference exists between the AGSA has standardized its audit approach across the board in experience of private sector audit firms and that of SAIs. this area and relies on automated data analytics tests. Broadly, competition between firms has acted as a spur to encourage experimentation with, and the development However, the AGSA notes that although its data analytics of, new IT-based audit methods and procedures. Although tests are automated, its data gathering from audited entities competition has played a part in encouraging some initiatives is not automated and requires extensive manual intervention. among SAIs (in particular, as noted previously, the goal of In summary, the AGSA’s response to this situation is to some of the more technologically advanced SAIs to provide specify the data it requires from each of the entities it audits. an audit service on a par with that delivered by private sector The office’s Information Systems Auditing business unit then audit firms), the predisposition of SAIs to share knowledge ensures the data provided is properly formatted and usable. and experience and in this way encourage more extensive This business unit also is responsible for ensuring that the and ambitious use of technology is striking. It is something comparative information that the AGSA uses for procedures that the ethos and structure of the International Organization for the audit of procurement and contract management is of Supreme Audit Institutions (INTOSAI) has encouraged. 17. Auditor-General of South Africa, 12–17. 18. Auditor-General of South Africa, 18–26. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 18 >>> INTOSAI’s Advocacy of the Use of IT for Audit The current support of the International Organization of • Theme 2: The role of SAIs in the achievement of the Supreme Audit Institutions (INTOSAI) of the use of IT for audit national priorities and goals including (a) core strategic purposes reflects its priority of encouraging SAIs to assess activities within audit and advisory functions, and (b) and report on the progress within their individual jurisdictions foresight and analytics in SAIs/advanced communication toward achieving the Sustainable Development Goals (SDGs). strategy. Against this background, since the 2016 INTOSAI International Congress of Supreme Audit Institutions (INCOSAI), INTOSAI’s For INTOSAI, the key output of the Congress was the interest in data analytics has grown significantly. Elaborating summary of the proceedings and decisions as encapsulated on the priority given to this in its strategic plan for the period in the Moscow Declaration.20 The declaration is intended to 2017 to 2022, INTOSAI emphasized the scope for SAIs to help guide INTOSAI’s future. It comprises the results of the strengthen their work in this area through the use of data discussions of the Congress’s two themes. Also, the declaration analytics and other audit techniques that exploit the power informs the future activities and strategies of INTOSAI and of information technology.19 In parallel, and agreeing with its SAIs while taking into account the Sustainable Development strategic plan at the 2016 Congress, INTOSAI also established Goals, as well as “fundamental changes in public auditing a working group to consider the issue of “big data” and data and public policy worldwide.” The declaration states that SAIs analytics. The establishment of this working group reflected and INTOSAI “commit themselves to provide independent a growing interest in this subject among INTOSAI members. external oversight on achieving nationally agreed on targets (including those linked to the SDGs); to effectively respond to The concern and interest among SAIs in these issues opportunities brought by technological advancement; and to subsequently influenced the development and definition of the enhance the impact of SAIs.” The declaration also included themes for INTOSAI’s XXIII Congress held in September 2019 the following commitments: in Russia. The following two themes were identified for this Congress: • Contribute to more effective, transparent, and informative accountability for outcomes. • Theme 1: Information technology for the development of • Develop a strategic approach to public auditing to support public administration including (a) data application in public achieving national priorities and the SDGs. administration, and (b) the role of big data analytics in the • Enhance public auditing value by extending the provision activities of SAIs. of audit-based advice on important and strategic issues to parliament, government, and public administration. 19. In agreeing on the 2017–2022 Strategic Plan, discussion among INTOSAI members focused on how INTOSAI could contribute to the 2030 Agenda for Sustainable Development, including good governance, to strengthen the fight against corruption. The plan as agreed on at the 2016 Congress contains four strategic goals and five cross-cutting priorities, including one cross-cutting priority to contribute to the follow-up and review of the SDGs within the context of each nation’s specific sustainable development efforts and SAIs’ individual mandates. 20. INTOSAI, “Summary of the Moscow Declaration,” International Journal of Government Auditing 46, no. 4 (Autumn 2019). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 19 • Promote the principle of availability and openness of data, technology, cybersecurity, data analytics, 5G cellular network source code, and algorithms. technology, machine learning, and quantum computing.21 • Make better use of data analytics in audits, including developing adaptation strategies and experienced data The WGISTA concept evolved from the work of the INTOSAI analytics teams and introducing new techniques in public Supervisory Committee on Emerging Issues (SCEI). The SCEI audit practices. is responsible for INTOSAI’s enterprise risk management and • Foster an experimental mindset. provides recommendations on important issues and emerging • Extend focus on identifying risk areas, raising awareness challenges faced by INTOSAI and individual SAIs. It aims to of risks and managing systemic ones. help coordinate and support the sharing of knowledge in this • Nurture “auditors of the future” and act as strategic players, regard. In this context, the SCEI had identified science and knowledge exchangers, and foresight producers. technology trends as key issues that would increasingly affect • Address inclusiveness when conducting audits and follow governments and SAIs in the future. An interim task force was the principle of leaving no one behind. established at the 2018 INTOSAI Governing Board meeting in • Establish productive interactions with auditees and enhance Moscow, Russia, to develop Terms of Reference for the new cooperation and communication with stakeholders. working group.22 INTOSAI believes that with the establishment of the WGISTA, the international SAI community now has A specific, practical outcome of the Congress was the decision a forward-looking working group that will monitor potential to create the INTOSAI Working Group on Impacts of Science impacts that science and technology may have on the audit and Technology on Auditing (WGISTA), which will focus on profession; share this specialized knowledge and experiences; key trends in areas such as artificial intelligence, blockchain and ensure INTOSAI’s work remains relevant and impactful.23 21. The INTOSAI Working Group on Impacts of Science and Technology on Auditing (WGISTA) is chaired by the State Audit Institution of the United Arab Emirates, with the US Government Accountability Office acting as vice chair. The WGISTA’s primary strategic objectives include conducting environmental scanning, sharing best practices, maintaining expertise within SAIs, applying science and technology in auditing, and developing competencies required by SAIs and auditors. 22. The task force, led by the State Audit Institution of the United Arab Emirates and the US Government Accountability Office, included the American Institute of Certified Public Accountants; Audit Board of the Republic of Indonesia; China National Audit Office; Institute of Internal Auditors; INTOSAI Development Initiative; Office of the Comptroller and Auditor General of India; Organisation for Economic Co-operation and Development; Pacific Association of Supreme Audit Institutions; United Nations Department of Economic and Social Affairs; and the World Bank. 23. For more information, see the INTOSAI International Journal of Government Auditing website at http://intosaijournal.org/wgista/. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 20 >>> SAIs’ Response to COVID-19 The Supervisory Committee on Emerging Issues (SCEI) able to respond with greater agility and resilience to meet the structure that was put in place by the International Organization challenges the pandemic created. Nevertheless, in having to of Supreme Audit Institutions (INTOSAI) has been central to work remotely, SAIs have had to deal with issues common to developing and coordinating its response to the COVID-19 all organizations pursuing this aim, namely ensuring staff have pandemic, in particular by providing a source of advice and the necessary “kit” to work from home; improving connectivity encouragement for SAIs.24 INTOSAI has established a and capacity to facilitate remote working; and liaising with audit COVID-19 expert group under the auspices of the SCEI. The entities and government to find solutions to issues such as group has held two online events (in June and July 2020) to security and the implementation of innovative approaches to share emerging experiences and strategies. The group also remote working. has surveyed SAIs’ responses to the pandemic with the aim of presenting the results to the INTOSAI Governing Board at its For those SAIs that already had a strong, well-developed IT November 2020 meeting. In addition, the group has established infrastructure, the lessons coming out of their response to the a website to make relevant material as widely available and challenge of the COVID-19 pandemic seem to be reasonably accessible as possible. positive. Those SAIs contributing to the SCEI online events highlighted improvements in productivity. They also It is clear that the pandemic has created three broad challenges acknowledged the ingenuity of their auditors in adapting their for SAIs: audit methods to overcome perceived obstacles. For example, one initiative introduced by the UK National Audit Office was • Dealing with the practical issues of SAI staff working the use of webcams to check stocks as part of an audit. The remotely with minimal physical contact between colleagues consensus emerging from the early work of INTOSAI’s SCEI and minimal physical contact with audited entities and COVID-19 Expert Group is that the experience of working their staff through the pandemic has accelerated the trend toward remote • Maintaining the normal cycle of audits and audit-related working among SAIs, and that a number of SAIs are looking work and activities including the day-to-day management again at their traditional audit approach to see how they can of the SAI itself capitalize on the innovative approaches they have developed • Auditing the government’s response to the pandemic in response to the challenges of the pandemic. Although SAIs’ responses to the pandemic continue to develop For other less-developed SAIs, the available evidence of the and evolve, it is possible to make some initial observations on impact of the COVID-19 pandemic and their response to it is the basis of information currently available. It is clear that those not available at the moment. A clearer picture may emerge as SAIs that had already invested in their information technology the INTOSAI COVID-19 Expert Group completes its survey (IT) infrastructure, and in particular in remote working, had the of SAIs. necessary building blocks in place and, therefore, have been 24. Gene L. Dodaro, “Addressing COVID-19 Implications Nationally and Globally,” International Journal of Government Auditing 47, no. 3 (Summer 2020). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 21 >>> International Audit Standards and IT-Based Methods and Procedures It is important to reflect on the implications of the use of Over time, the risk-based audit approach has evolved due increasingly sophisticated IT-based audit methods and to a range of factors including higher transaction volumes procedures for the way in which the audit profession is overseen such that auditors were not able to test all transactions and regulated. This understanding is important because any underlying the financial statements; increased complexity; issues around IT and international audit standards will have regulation stimulated by highly public failures of companies; direct implications for supreme audit institutions (SAIs) and the and technological limitations. A risk-based audit focuses on audits they undertake. the nature and extent of risks of material misstatement for the particular engagement, with greater emphasis on obtaining an The analysis and research underpinning reports issued by the understanding of internal control established by an entity and, International Auditing and Assurance Standards Board (IAASB) when appropriate, obtaining audit evidence from the auditor’s and by the Financial Reporting Council of the United Kingdom testing of the effectiveness of such internal control.25 provide a useful starting point for an accurate summary of the audit profession’s growing use of data analytics and the Against this background, the key concern for the IAASB is how issues that this raises for the regulation and oversight of to accommodate audit data analytics and its potential benefits the profession. within the structure of the international standards on auditing (ISAs) and facilitate the regulation of the audit profession in In relation to the impact of technology on the audit of private this rapidly changing environment. As they currently stand, the sector entities, the IAASB has commented that in the history of ISAs “do not prohibit, nor stimulate, the use of data analytics. . the audit profession, there have been shifts in how the audit is . . The ISAs acknowledge the use of technology by the auditor executed. These shifts have been a result of transformations in executing the audit, through use of computer-assisted audit in the environment in which companies operate, and in techniques (CAATs). However, the reference to CAATs in the which audits are performed. Prior to the current risk-based ISAs was created in a completely different technological era audit approach, companies operated in a far less complex and CAATs have evolved significantly into what is now being environment. As a result, the audit was carried out in a largely referred to as data analytics.”26 The IAASB acknowledges that manual way with a relatively high proportion of the financial this absence of reference to data analytics “beyond mention information underlying the financial statements being tested of traditional CAATs in the ISAs may be viewed as a barrier without any significant emphasis on the nature and extent of to their adoption more broadly.” And the IAASB recognizes the risks of material misstatement. that without a firm framework of standards, there is a risk that 25. IAASB Data Analytics Working Group, “Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics” (exposure drafts and consultation paper, IAASB, September 2016), 6. 26. IAASB Data Analytics Working Group, “Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics” (exposure drafts and consultation paper, IAASB, September 2016), 8. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 22 auditors may face difficulties in substantiating the judgments It is evident from the research undertaken in preparing this they make and the procedures they use when they rely on paper that private sector audit companies and SAIs continue the use of audit data analytics. This in turn may deter auditors to experiment and innovate in the use of IT-based methods from “using and experimenting with data analytics” and also and procedures to improve audit efficiency and audit quality. result in the views of audit oversight authorities evolving “in an Clearly, incorporating these methods in the framework of inconsistent manner—within and between jurisdictions.”27 international audit standards, while ensuring that the standards themselves do not inhibit better, more efficient audits, remains a challenge. 27. IAASB Data Analytics Working Group, “Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics” (exposure drafts and consultation paper, September 2016), 10. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 23 >>> Conclusions and Recommendations for the Way Forward In conclusion, IT-based methods and procedures create the identified in this paper and those for the medium- to long-term opportunity for SAIs to improve the quality, efficiency, and that will take longer to organize and implement. SAIs are the effectiveness of their audits. And certainly, the experience to- main driver of these initiatives with the help of the International date of SAIs in responding to the challenges of the COVID-19 Organization of Supreme Audit Institutions (INTOSAI), regional pandemic shows the opportunity that remote working coupled bodies, donors, and development organizations. with the ingenuity of professional auditors can create to improve productivity and encourage innovation and new The initiatives to address the challenges faced by SAIs include working methods. the following: However, SAIs face three broad challenges in responding to the • A key gap in the information currently available is the evolution of and growth in the increasingly sophisticated IT tools absence of firm data about the IT capabilities of individual and applications available to them for the audit of governments SAIs, particularly those in developing countries. A survey of and other public bodies and entities. The first challenge is INTOSAI members could be conducted to identify the level the need to develop techniques and procedures for the audit of maturity of each SAI in the use of audit data analytics of, for example, cloud-based accounting and management and IT-based audit tools. The survey could build on the information systems, and for the increasing digitization and work done by INTOSAI to assess how SAIs are responding automation of government and public sector processes. The to the challenges of the COVID-19 pandemic. The results second challenge revolves around the challenge of harnessing of a survey of this kind could then be used to support the opportunities created by IT to conduct audits more efficiently INTOSAI in developing its approach to supporting SAIs’ and to develop audit procedures that will enhance an SAI’s use of IT and in facilitating exchange of experiences and effectiveness and that will contribute to more impactful audit sharing of good practices among and between SAIs. The outcomes. The third, and arguably most difficult challenge, survey results would be a valuable source of information is to find the resources, experience, and expertise needed to for other development partners in designing the support develop, implement, and use audit data analytics and IT-based they provide to SAIs at the national or regional level. audit tools and procedures. • Another gap that could be closed is that of the absence of As for providing support to SAIs to address the challenges they firm guidance for SAIs on the successful implementation of face and, therefore, enable them to capitalize on the benefits IT-based audit methods and, subsequently, the conditions and opportunities created by rapid advances in the technology, for the efficient and effective use of those methods and it is possible to sketch out a way forward for the support of tools. A set of diagnostic or case studies could be designed the World Bank and development partners. The initiatives and sponsored. The aim here should be to identify the set out below fall into two broad categories: those in which necessary conditions for success and the key risks that quick actions could be taken to address some of the issues SAIs should manage. The outputs of this work could EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 24 then encompass lessons learned, the potential pitfalls • Also, in the medium-term, and building on short-term to be avoided, quick fixes to common problems, and the initiatives, detailed technical guidance could be developed key components of successful longer-term IT strategies for all SAIs (but specifically targeted for those in developing for SAIs. countries) that focuses on best practice in developing, implementing, and using audit data analytics and IT-based • Also, in the short-term, targeted events and seminars audit methods and tools. encouraging the use of IT could be organized to explore the potential for SAIs to make better, more effective use • In the longer-term, the fundamental issue of the chronic lack of audit data analytics and IT-based audit tools and to of resources and expertise available to the majority of SAIs identify ways to facilitate that process. This paper has worldwide could be addressed. An option of establishing touched on potential topics for study and discussion. a fund that international partners can contribute to could These include, for example, the use of smartphones be explored. The fund would support professional training and drones for audit purposes; the audit of blockchain for SAI auditors in the use and application of audit data transactions; the implications for audit of the Internet analytics and other IT-based audit tools. If sufficient of Things; the use of augmented intelligence or artificial resources were made available, the proposed fund could intelligence for audit purposes (for example, to contribute also support the provision of the relevant audit software to risk assessments); and the lessons learned from SAIs’ (such as licenses) as well as the required hardware. The responses to the challenges of COVID-19. There is also proposed fund could be freestanding or, if it is judged more scope for sponsoring events that consider how SAIs can appropriate, it could be set up within the framework of the use IT to support new or evolving types of audit, such as existing INTOSAI-Donor Cooperation. work focusing on social audit mechanisms. • Starting from the short-term, complementary technical • In the medium-term, the following activities could be assistance to SAIs alongside the financing of public sector pursued: support for the development of SAIs’ use of IT and IT capacity strengthening could be promoted. For example, IT-based audit processes and procedures customized to when a donor-funded project includes financing of low-capacity settings; the development of appropriate case automation of public procurement, revenue administration, studies to encourage SAIs to make more extensive use of, integrated financial management information systems, for example, audit data analytics; and encouraging more etc., the project design is encouraged to include a advanced SAIs to establish a mentoring role with individual component for SAI capacity strengthening to effectively SAIs in developing countries to provide continuing support audit those systems. and expertise. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 25 >>> Appendix A: How SAIs Are Responding to the Challenges of Implementing Advanced IT Audit Methods and Procedures Appendix A contains a wide range of examples of individual challenges associated with evolving science and technology supreme audit institutions (SAIs) using advanced, sophisticated issues; the risks and management needs to address complex IT-based audit methods and procedures for three broad and growing cyber security developments; increased objectives: to improve the efficiency and quality of their audits, investments in the Department of Defense; and rising to respond to the needs and expectations of their stakeholders, health care costs.” In establishing its Science, Technology and to keep pace with developments in the private sector. Assessment, and Analytics (STAA) team, the GAO’s aim was to expand the support and advice it provides to the US Congress by Data Analytics SAI Example 1: US • Conducting technology assessments and providing Government Accountability Office technical services; • Reviewing science and technology programs and initiatives to assist in oversight of federal investments in research, The subject of data analytics has been, and continues to development, and advanced manufacturing; be, a topic of significant interest for the US Government • Compiling and utilizing best practices in engineering Accountability Office (GAO).28 sciences, including cost, schedule, and technology readiness assessments; and In particular, two broad developments encouraged the GAO • Establishing an audit innovation lab to explore, pilot, to develop its capability in this regard. The first development and deploy new advanced analytic capabilities, conduct flowed from the bipartisan Fraud Reduction and Data Analytics research in information assurance, and explore emerging Act, 2015 (Public Law No. 114–186), which required federal technologies that will impact future audit practices. agencies to use the GAO A Framework for Managing Fraud Risks in Federal Programs and to implement control activities The GAO has a track record in carrying out studies on related to fraud risk management. The goal was to prevent, technological developments. For example, in 2018, it issued detect, and respond to fraud in the context of an estimated reports on “the emergence and implications of artificial US$100 billion of reported improper payments. intelligence; the benefits, risks, and regulatory issues The second development flowed from the pressure on the US concerning financial technology; and the need for revised cost Congress to improve its oversight of the federal government’s estimation and scheduling policies by the National Science science and technology programs. In 2018, the US Congress Foundation for large facilities, major National Aeronautics passed spending legislation that directed the GAO to increase and Space Administration (NASA) projects, and critical its capacity in science and technology analysis. And in January infrastructure protection.” Also, the GAO has developed 2019, the GAO established “a new Science, Technology “technology readiness assessment best practices guides” and Assessment, and Analytics team to better meet Congress’s has used these guides to evaluate major technical systems growing need for information on science and technology acquisitions such as those found at the Department of issues.” In doing this, the GAO combined and enhanced Defense, Department of Homeland Security, the Department “its technology assessment functions and its science and of Energy, NASA, and other agencies. At this stage it is unclear technology evaluation into a single, more prominent office to what extent the GAO is developing and using “new analytic that will expand GAO’s support to lawmakers on cutting-edge capabilities” and “emerging technologies” to enhance its audit topics, such as artificial intelligence, regenerative medicine, methods and procedures. 5G wireless communication, and quantum computing.”29 This new unit also encompasses a specialist audit innovation lab “to In relation to managing the risk of fraud in federal programs explore, pilot, and deploy new advanced analytic capabilities, and the application of data analytics, Timothy Persons, head of conduct research in information assurance, and explore the GAO STAA team, has cited the following example: emerging technologies that will impact future audit practices.”30 The US Department of Agriculture (USDA) saw a In turn, this development flowed from the GAO’s concern dramatic drop in fraudulent claims for crop insurance “to increase [its] capabilities to review the opportunities and after they implemented data analytics approaches. 28. For example, in 2016, the GAO hosted two high-level meetings to consider the issues raised by data analytics. See GAO, “Data and Analytics Innovation: Emerging Opportunities and Challenges,” GAO-16-659SP, September 2016; and GAO, “Data Analytics to Address Fraud and Improper Payments,” GAO-17-339SP, March 2017. 29. GAO, “Technology and Science,” Press release, January 29, 2019. 30. American Institute of Physics, “Congress Bolstering Its Access to S&T Expertise,” May 9, 2019, https://www.aip.org/fyi/2019/congress-bolstering-its-access-st-expertise. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 27 Their Crop Insurance Program Compliance and Integrity In relation to its proposed investment in information technology, Data Warehouse utilized multiple datasets to prevent the budget proposal commented that the GAO was developing fraudulent claim payments, which resulted in billions the requirements for replacing its 30-year-old content and of dollars in savings. Specifically, USDA data analysts records management system that was used to do its audits with utilized access to 170 data sources—including several a modern and secure cloud enterprise content management terabytes of policy information, 120 terabytes of weather, solution. The GAO believes that, in the long term, this new satellite, and other remotely sensed data, and 1.3 million system would allow its employees to work more efficiently and crop insurance policies across 3,200 counties—to look at the same time improve the GAO’s efficiency by eliminating for atypical patterns among insurance claims, cross- custom code and using commercial off-the-shelf software. In checking them with data from high-solution satellite 2021, the GAO plans to “build off this requirements analysis to images and weather records. Their approach involved procure this important new system that will truly modernize the both conventional prosecution activities as well as newer, agency’s fundamental tools and establish a solid foundation for “softer” approaches where letters of inquiry were sent to the coming years.” claimants who were suspected of fraud. This latter method resulted in a subsequent dramatic drop in claims simply In addition, in fiscal 2021, the GAO plans to complete the because the participants in the program quickly became development and deployment of its platform to edit, fact check, aware of USDA’s new ability to detect fraud or suspected and distribute its reports. Referred to as “New Blue,” the GAO fraudulent activities.31 comments that this new platform “will streamline the publishing processes to enable efficient and scalable publication of The GAO acknowledged that “to bolster its work in this products in a responsive web-based format intended to area” it would have to recruit additional staff with experience accommodate the rising demand for content that is accessible in “computer/systems/electrical engineering for digital and on mobile devices and can be quickly and easily navigated by communications technologies (that is, 5G wireless, blockchain, congressional staff during hearings or by users on the go.” quantum cryptography, artificial intelligence/machine learning systems).”32 The GAO is also focusing its developmental work on the audit implications of the accelerated growth of the use of, The GAO’s budget request for fiscal 2021 built on these and reliance on, artificial intelligence (AI) in citizens’ lives, themes.33 An important part of the request focused on the specifically governance-related considerations.34 In the resources required for its STAA team and to support “priority absence of governance structures, “entities that develop, investments” in information technology. According to the budget purchase, or deploy algorithmic systems will face potential request, in 2021 the STAA team will focus on the following: risks.” This in turn raises two important questions for SAIs: • Equipping a state-of-the-art collaborative space to conduct • How can SAIs purposefully integrate AI capabilities across data science experiments and to define, design, and audit activities in ways that move the needle toward execute advanced analytics pilot projects. continuous auditing and 100 percent sampling? • Building prototypes of emerging technologies such as • “How will the SAIs—absent of a uniform policy and blockchain/digital ledgers as well as machine learning/ regulatory framework—approach oversight of AI solutions artificial intelligence systems. when, not if, they are asked to provide assurance? • Developing a framework for algorithmic accountability in machine learning/artificial intelligence systems. In the GAO’s view, SAIs need to look ahead and “be prepared to audit algorithms in ways that include both empirical as well The GAO commented that the lab will “provide Congress as inferential evidences to draw assurance . . . Ushering in a with critical foresight, oversight, and insight of science new age of algorithmic accountability will require more from and technology issues to help ensure continued American SAIs. They will need to reconsider their approaches to auditing innovation, competitiveness, and security.” data-driven automated systems from a culture, workforce, 31. Interview with Timothy Persons published in Revista do TCU (English version) 48, no.135 (January/April 2016). 32. GAO, “Fiscal Year 2020 Budget Request,” GAO-19-451T, 2–4, 8. 33. GAO, “Fiscal Year 2021 Budget Request,” GAO-20-429T, 15–16. 34. Taka Ariga and Stephen Sanford, “A is for Accountability – Oversight in the Age of Artificial Intelligence” BIG DATA & Digital Audit ECA Journal 1/2020: 88–91. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 28 and infrastructure standpoint. There will likely be strong • Improving ANAO’s connective technology; demand from legislatures and the public for accountability and • Investing in “modern consumption-based cloud computing”; assurance about governmental use of algorithms. SAIs need to • Building high-capacity data storage and computer be ready to answer this call. The field of AI is rapidly evolving, platforms to support enhanced data analytics, modeling, and its known and unknown effects will have a growing impact and automation; and on government and society. SAIs need to start planning today • Enhancing ANAO’s security environment with a focus for what tomorrow will bring.” on maintaining organizational cyber resilience and streamlining secure data governance collection and The GAO is addressing these issues through the work of classification methods. its Innovation Lab and through its development of an AI oversight framework. Data Analytics Example 3: United Kingdom National Audit Office Data Analytics Example 2: Australian National Audit Office For the National Audit Office of the United Kingdom (NAO), a central element of its strategic planning is investment in the The Australian National Audit Office (ANAO) created its office’s in-house IT capability to make greater use of technology Systems Assurance and Data Analytics Group in 2017. The in its audit work.37 The NAO sees three benefits flowing from ANAO has stated that the focus of this group’s work will be this approach: on increasing the quality and productivity of its audit work “in an environment of increased data collection and generation in • Improvement in the quality of the office’s work through the the public sector.”35 The ANAO also expects its investment in use of automated tools that would help NAO auditors focus enhancing its data capability to support its audit planning and their efforts on the important risks to their audits provide improved audit outcomes and “new audit products and • A better understanding of business processes and services.” In addition, the ANAO will support its investment in supporting technology data analytics capability with the implementation of its Data • Enhanced technology that would help the NAO keep Analytics Strategy over the next four years. Initially, the strategy pace with other audit institutions and UK government will focus on delivering the following three projects: departments • A data analytics pilot that “will leverage technology to In this context of the increasing complexity and greater automate manual audit procedures and enable [ANAO] prevalence of IT audit and data analytics in government, the auditors to quickly focus audit attention on higher-risk NAO also recognizes the need to “upskill and invest more in areas across complete populations of procurement data” professional and technical training.” The NAO’s approach is • The identification of tools and systems to support data to develop its skills base through in-house training to ensure analysis in audit work that NAO auditors have the skills and knowledge to meet the • Electronic access to information and evidence office’s quality requirements. in government In essence, two interconnected themes run through the In July 2020, the ANAO issued its Corporate Plan for 2020– NAO strategy. The first concerns the NAO’s audit methods 21.36 The plan confirmed that, with regard to technology, and procedures, in particular its financial audit methods and the ANAO continues to be focused on “keeping pace with procedures. The second concerns the office’s need for a advances in technology.” The document notes that from 2018, secure and well-integrated digital environment. the ANAO had invested in and implemented “several IT, data, and security strategies.” These strategies had “empowered The NAO sees technology as a central part of its financial audit [the ANAO’s] workforce to work remotely.” Over the next four methodology. It notes that to advance its testing of financial years the ANAO plans to make investments in technology by systems and its understanding of audited bodies, it has been 35. Australian National Audit Office, Corporate Plan, July 2018. 36. Australian National Audit Office, “Corporate Plan 2020–21,” July 2020, https://www.anao.gov.au/work/corporate/anao-corporate-plan-2020-21. 37. National Audit Office, “NAO Strategy 2019–20 to 2021–22,” December 2018, 15–34. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 29 trialing new analytical techniques in its audits. The NAO’s cited by the NAO include the widespread adoption across work in this regard has identified opportunities to analyze UK government departments of “cloud based accounting transactions more completely, supporting additional insight systems, coupled with department specific system and value-adding activities. changes, such as a new customs declaration system, a new visa system and digital passport system, and the In this context, the NAO has commented that “in line with other automation of court systems with its knock-on effects on firms in the industry” it will continue to develop this work stream court fees and fines.” The NAO comments that its IT audit as a priority. It plans to maintain its annual investment in the work supports the assurance it gets over the income that use of technology to augment its financial audit approach. government receives from these systems. The NAO sees this as “essential” to it being able to audit new technologies such as cloud-based computing that are In relation to the NAO’s own IT environment, the office increasingly used by the entities that it audits. The NAO also completed its business improvement program in its 2017–18 sees this investment as necessary to ensure that its use of fiscal year. In the course of implementing the program, the NAO technology stays in line with developments in the wider audit replaced “legacy IT assets with a fully integrated cloud-based profession. The NAO concludes that “in this way, we can finance, human resources, and project system, introducing make sure that Parliament receives audit services that are new streamlined workflows and project management tools.” comparable in quality and capability to those offered by the top The NAO is now moving other technology office applications commercial audit firms.” to the cloud, including its communication and information management systems. The NAO plans to invest in the following: The NAO’s strategy to make this transition is based on the • Data analytics software that extracts, processes, and following three principles: highlights anomalies in large quantities of data. The NAO notes that this approach was deployed “with great • Security: The NAO’s access to information and success” in the first year of auditing the BBC. The NAO’s data in government is fundamental to discharging its view is that “the ability to quickly identify problem areas responsibilities. In the NAO’s view, with the risk of and potential control failures through data analytics means cybersecurity breaches increasing, cloud platforms offer a that our audits are of higher quality and provide better more secure environment than it can maintain in-house. insight and value to governing bodies.” • Capability: By giving itself access to much bigger IT • Training to make sure its audit teams have the skills platforms, the NAO believes it will increase the volume to audit emerging technologies and the increasing and speed at which it can process and store data. The digitalization and automation of process. Examples NAO believes that by purchasing this as a service, it will EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 30 benefit from continual updated technologies and good Data Analytics Example 4: National industry practices. • Value: In the NAO’s view, cloud services provide simplicity Audit Office of China and agility. It believes that as an organization it “can flex [its] IT and business processes as [it needs] them, rather than entering into long-term design commitments that need Since it hosted the 2013 INTOSAI Congress, the National Audit to be hosted and maintained on [the NAO’s] premises.” Office of China (CNAO) has played a key role in influencing the strategic direction set for the International Organization The NAO’s implementation of its business improvement of Supreme Audit Institutions (INTOSAI). In addition, it has program was the subject of an external value-for-money study. complemented the CNAO’s own strategic objectives, in This study concluded that the benefits were realized in line particular with regard to the use of data analytics for audit with the office’s plans. These benefits manifested themselves purposes. For example, the CNAO is chair of the INTOSAI in smarter planning, with more efficient allocations of staff Working Group on big data that was established at the 2016 between projects and this, in turn, meant that the NAO could INTOSAI Congress. manage the variable elements of its portfolio more effectively. The CNAO has used sophisticated IT tools for its audits.38 For The NAO’s Five Year Strategy for 2020 to 2025 was published example, in 2015, the Auditor General issued a report about in the spring of 2020. The strategy emphasized the “vital role” land transfer payments and the protection of arable land. In that technology and data play in supporting the NAO’s work undertaking this audit, the CNAO analyzed and correlated and in enabling it “to improve the quality of our work, efficiently structured and unstructured data from multiple sources. risk assess our audits, and create insights about how public These sources included remote sensing image data from the money is being managed that are not otherwise possible.” The Department of Land, planning data, data on and mapping of NAO’s strategy also comments that automation enables the forests, data from the agricultural sector, data from the Ministry NAO “to perform routine work quickly, freeing up our people for Development and Reform, as well as relevant industrial, to focus on areas of greater complexity where professional commercial, and tax data. judgement is required, enhancing the quality of our work.” The CNAO also used data matching for an audit of an affordable The strategy document notes that the NAO’s technology housing scheme. The audit examined a range of issues enables the NAO to run an efficient workplace and to share, including the construction of houses and their distribution collaborate, and communicate the office’s knowledge simply, to beneficiaries. In planning the audit, the CNAO concluded quickly, and securely. Specifically, the NAO commented that that reviewing each beneficiary’s application would be too it is using the lessons learned about “how to operate as an time consuming and focusing on a small, random sample of entirely remote workplace during the COVID-19 pandemic to applications would not provide the insights it wanted to obtain. accelerate our progress in becoming a more agile organization.” Instead, the CNAO opted for an approach that involved the From 2020 to 2025, the NAO plans to make more effective matching of data from different sources. It concentrated its use of technology, data, and knowledge. The specific initiatives examination on one key qualification for access to houses highlighted in the strategy encompass the following: constructed under the scheme, namely that beneficiaries did in fact have low incomes and did not own other assets such • A new digital audit platform incorporating automated as other houses or expensive motor vehicles. Accordingly, the methodologies and integrated data analytics auditors crosschecked details of the scheme’s beneficiaries • A new knowledge management approach connecting with data held by other state institutions on car ownership and financial audit and value for money programs property ownership. This crosschecking enabled the CNAO • A redesigned website to identify individuals who had obtained housing under the • A new secure and efficient approach to managing and scheme but who were not in fact entitled to that housing. The handling data from the bodies that the NAO audits SAI found that a number of these individuals were friends • A review of the NAO’s remote working practices during the or relatives of the individual in charge of the affordable COVID-19 pandemic, with more agile working practices housing scheme.39 incorporated into how the NAO operates 38. National Office of China, “National Audit in the Big Data Environment,” (Paper presented at the meeting of INTOSAI Working Group on IT audit, 2017). 39. Alexander Juliano, “The 10th ASOSAI Research Project Report: Audit to Detect Fraud and Corruption - Evaluation of the Fight against Corruption and Money Laundering,” Fraud Audit Office, Philippines Commission on Audit, Asian Organization of Supreme Audit Institutions, 2015, 197, 200. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 31 The CNAO believes that the availability and analysis of big institutions with regard to its thinking in relation to exploiting the data will bring profound changes to its audit methods and possibilities created by information technology, specifically the procedures.40 It is developing a range of initiatives in this use of data analytics. TCU’s strategic plan for 2015–21 provides regard with the aim of implementing them operationally by for the development of a “comprehensive organizational 2020. These initiatives include the following: competence to work with emerging technological resources and analyzing large databases (big data).” This strategy was • Strengthening the CNAO’s systems and methods for reflected in TCU’s 2015–17 audit plan, which provided for collecting data at the national and provincial level by TCU to undertake the continuous monitoring of databases establishing data centers, centralizing and standardizing to assess the use of public funds to detect and correct in a data requirements and data management, and facilitating timely manner possible improper diversion of public funds. In data sharing throughout the organization. 2015, TCU established its Center for Research and Innovation, • Strengthening the CNAO’s use of big data technology which in turn is linked to TCU’s Innovation and Co-participation through the use of cloud computing, data mining, and Laboratory to promote applied research within the SAI. The aim intelligent data analysis methods. of this innovation is “to support innovative projects, ensure the • Enhancing what it terms the CNAO’s “digital audit knowledge management of developed solutions, coordinate command” capability. This capability will encompass cooperation activities, and promote training activities and establishing platforms for big data analysis, integrated events in relation to topics at the frontier of knowledge” and in audit working, laboratory simulations, and comprehensive this way support creativity in the use of technology to support management support systems for the administration of the and strengthen the external audit function.42 organization. The following examples illustrate the innovative audit The CNAO sees all of these strategic initiatives as contributing approaches that the TCU has adopted through its use of to a series of “comprehensive transformations.”41 It defines information technology and exploiting the possibilities created these as follows: by data analytic techniques: • Transformation from single-point discrete audit to multipoint • The use of information technology has enabled the TCU linked audit to adopt a new approach that it describes as “continuous • Transition from partial audit to full-coverage audit monitoring” in auditing welfare programs that account for • Changing from static audit to a combination of static and expenditure of about R$500 billion a year. The process dynamic audit enables auditors to specify and set the parameters for • Transitioning from a wholly post-audit system to a their audit tests and audit procedures. In this way, the combination of post-audit and concurrent audit TCU believes its testing provides for a more selective and • Transitioning from on-site audit to a combination of on-site risk-based approach that has helped prevent frauds and audit and remote audit identified system weaknesses.43 • Transformation from micro-audit to a combination of micro- audit with macro-audit • In January 2016, the TCU created a Department of Special Operations for Infrastructure to oversee activities related to the “Operation Carwash” investigation. An important Data Analytics Example 5: Tribunal element of this initiative was to facilitate the collection and handling of a substantial volume of data and information, de Contas da União (Supreme Audit some of which was highly sensitive and confidential. Institution of Brazil) Accordingly, the working practices adopted for these audits have involved using information technology specialists to assist auditors in applying analytical techniques and The Tribunal de Contas da União (TCU, the supreme audit audit procedures to the various sets of relevant data and institution of Brazil) is one of the more advanced supreme audit information.44 40. National Office of China, “National Audit in the Big Data Environment,” (Paper presented at the meeting of INTOSAI Working Group on IT audit, 2017). 41. National Office of China, “National Audit in the Big Data Environment,” (Paper presented at the meeting of INTOSAI Working Group on IT audit, 2017). 42. Revista do TCU (English version) 48, no. 135 (January/April 2016): 41. 43. Revista do TCU (English version) (May/August 2015): 13. 44. Revista do TCU (English version) 48, no. 135 (January/April 2016): 13, 17. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 32 • The TCU also has experimented with other forms of new • The specification of roles and responsibilities for the technology to enhance the quality and impact of its audits. development of infrastructure and capacity building within One example is the work that the SAI is doing to assess the SAI to mainstream data analytics the contribution that georeferenced data could make to its audits in which environmental issues are a key aspect (such As well as developing its policy and processes in relation to as audits involving the assessment of schemes affecting the use of data analytics, SAI India has undertaken two pilot the environment in the Amazon basin or the construction studies to explore and test the potential contributions of “big of roads in remote areas). This approach involves using data” and the use of data analytics. These pilot studies are and analyzing data available from government and as follows: other sources based on GPS systems. Another striking innovation is the TCU’s initiative to encourage greater • The first pilot was a performance audit that focused citizen awareness of and participation in its work through on the administration in one Indian state of three social the release of mobile applications that can be downloaded assistance programs: old-age pension, widows’ pension, to Android and IOS devices.45 and disability pension. Although the structure of the audit followed a traditional, established approach to undertaking • Also, the TCU is a key source for developing and sharing performance audits, in the design and implementation of the innovative thinking about the use of new technology, the audit the SAI decided to make increased use of available use of “big data,” and the application of techniques such data and research evidence. Specifically, in addition to as data analytics. A crucial vehicle for this sharing is the analyzing and testing databases of beneficiaries, the SAI TCU’s journal, Revista do TCU, which is essentially a made use of external databases such as census data and publication with the hallmarks of an academic journal. data on people with incomes below the poverty line. This use enabled the SAI to go beyond a traditional outcome of this type of audit—with its focus on the payment of Data Analytics Example 6: Office of benefits to individuals who do not meet the appropriate eligibility criteria—and identify potential beneficiaries the Comptroller and Auditor General who, for whatever reason, had been excluded from the of India relevant scheme. • The second pilot was concerned with an analysis of infant The Office of the Comptroller and Auditor General of India (SAI mortality rates. A conventional audit on this type of subject India) provides another example of an SAI that is developing would have focused on the implementation of a scheme a sophisticated approach to the use of data analytics.46 In or schemes intended to reduce infant mortality rates and February 2016, SAI India adopted a big data management policy would probably have refrained from commenting on whether and established a task force to oversee the implementation of the examined schemes had factored into their design all the framework developed for this policy. That policy framework significant factors that affect the level of infant mortality comprises the following elements: rates. The study used a number of innovative techniques as far as SAI performance audits are concerned. The first • The identification of data sources covering sources that step was a text analysis of the relevant research literature are internal to the SAI as well as external sources using an IT tool that enabled the study team to produce a • The establishment of data management protocols to word cloud that identified the key variables that appeared cover aspects such as accuracy, security, privacy, and from the research to have the most significant impact on confidentiality as well as issues around compliance with infant mortality. The study team next used data from official legislative and regularity requirements sources (such as census databases about rates of infant • The establishment of arrangements in the SAI to develop mortality in different regions) and linked this data with the organization’s strategy and guidelines for digital the variables identified from the word cloud using linear auditing, data analytics, and visualization correlation analysis and multivariate regression analysis. 45. Revista do TCU (English version) (May/August 2015): 36, 37. 46. The source for the material about SAI India is a document entitled “A Paper on Big Data Analytics in SAI India.” The document is not dated but its contents indicate that it was prepared after February 2016. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 33 The results served to confirm which variables appeared to have the most impact on infant mortality rates in the In its detective role, the BAI has attempted to adopt a strategic districts being examined. In turn, this enabled the SAI audit approach in relation to how it monitors officials who may be team to concentrate its examination on those government susceptible to corruption. In particular, it uses data analytical interventions that had potentially the most effect and, tools such as data matching or data mining. In this context, conversely, to identify those interventions likely to be it draws on a range of skills from within the BAI including its less effective. special investigations bureau, its audit requests investigation bureau (this deals with matters received through its whistle- In assessing the potential contribution of big data and data blowing systems), its IT audit group, and its internal audit analytics, SAI India concluded that they can provide new and support group.47 deeper insights, enhanced risk assessment and better audit planning, the capability of examining an entire data population, identifying hidden links, and an improved presentation of Data Analytics Example 8: European results through the use of visualization tools. However, to achieve these outcomes, the SAI needs to address Court of Auditors issues such as data quality and security, confidentiality, the creation of the necessary infrastructure (encompassing both hardware and software), and training on the application of the In February 2019, in its document “Foresight for ECA,” the appropriate approaches and techniques. In SAI India’s opinion, European Court of Auditors (ECA) committed to establish a the range and nature of the challenges involved in identifying Strategic Foresight and Advisory Panel. In this context, the ECA and exploiting the opportunities created by data analytics identified an urgent need to carry out a digital transformation emphasized the importance of securing the commitment and of its audit work.48 To take this forward, in May 2019, the ECA support of the leadership of the SAI. set up its Digital Steering Committee (DSC) with a mandate “to drive the digital transformation of the ECA’s audit work in the coming years, in line with its current 2018–20 strategy.” The goals of the ECA’s digital transformation strategy encompass Data Analytics Example 7: Board increasing the efficiency of its audit work and focusing on the of Audit and Inspection (SAI of the results and the impact of the European Union budget and Republic of Korea) policies getting more value for money. The ECA has identified the following three steps for taking forward its commitment to digitalization: The Board of Audit and Inspection (BAI, the SAI of the Republic of Korea) has what it describes as both a preventive role and • First, it will map “where we are as an institution in our a detective role in relation to corruption and, in discharging its financial, compliance, and performance audits, as well as responsibilities in relation to both roles, the BAI makes use of for the Statement of Assurance work, both in terms of IT data analytics techniques. audit tools at our disposal and skills and competences.” In its preventive role, the BAI collects and analyzes data that • Second, the ECA plans to collaborate with SAIs that focuses on issues that may be indicative of corrupt behavior are “particularly advanced in digitalization to learn and on the part of public officials. This involves targeting senior collaborate”—specifically the SAIs of Estonia, Finland, officials and suspected officials in relation to the acquisition or France, the United Kingdom. The ECA also plans to sale of property, the ownership of luxury vehicles, gambling, analyze “the state of the art on digitalization of audit in litigation over the payment of taxes, and the recruitment of the private sector. Big private audit firms are investing family members. More widely, the BAI works in cooperation significantly in new technologies and new competences.” with other units involved in anti-corruption activities such as internal audit units, the police, and the prosecution services. 47. Alexander Juliano, “The 10th ASOSAI Research Project Report: Audit to Detect Fraud and Corruption - Evaluation of the Fight against Corruption and Money Laundering,” Fraud Audit Office, Philippines Commission on Audit, Asian Organization of Supreme Audit Institutions, 2015, 283–87, 289. 48. BIG DATA & Digital Audit ECA Journal 1/2020: 35–40 EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 34 • Third, the ECA proposes to stay in close contact with its by using tools such as Tableau, Power BI, Gephi (visualization auditees, in particular with the European Commission of networks), Maltego, and Plotly. Another area that ECALab because much of the ECA’s work “will depend on accessing data scientists have worked on is process mining because the data in our auditees’ (both the European Commission’s this “could play a significant role in the digital transformation and the member states’) databases and IT applications.” of audit. It combines data analytics and data visualization to allow auditors to analyze and detect patterns and irregularities At the operational level, the ECA has established the ECALab in both static data and entire procedures. It works by extracting to serve “as an in-house incubator for exploring new ideas event logs to reconstruct and visualize processes. Due to the in audit and offers auditors a variety of techniques that can increasing digitalization of auditees’ processes and records, assist in achieving individual audit goals.” The ECA explains its log data has become more accessible, and auditors can now rationale for this initiative as follows: examine very detailed audit trails.”49 For many years, the private sector audit firms have been In September 2017, in the context of an audit task that exploring how to make best use of data-driven audits. resulted in the report “Landscape Review: Putting EU Law The Big Four audit firms have embraced the opportunity into Practice,” a team of ECALab data scientists and auditors to explore areas such as artificial intelligence (AI) and worked together to merge data from various sources in a single blockchain, so why should supreme audit institutions not integrated model. In the ECA’s view, this allowed for innovative follow the same path? In 2017, the ECA responded to this ways of presenting data graphically and greatly facilitated the question by launching the ECALab, an in-house center task of data analysis. for research and innovation. This space for sharing ideas, exploring, testing, and implementing technologies in the The ECALab is also currently exploring the potential of text audit process is part of the ECA’s digital transformation mining and it is developing a new tool, Document Navigator, to initiative. The ECALab is comprised of data science help the ECA’s auditors in this regard. enthusiasts and expert auditors who cooperate on finding tailored solutions to audit tasks and audit-related projects. Finally, following a conference that it organized in November 2019, the ECA has established its Technology and Innovation One area that the ECALab has focused on is data visualization. Network for Audit, which brings together professionals from the The ECALab data scientists have created interactive visuals European Union SAIs and the ECA. 49. BIG DATA & Digital Audit ECA Journal 1/2020: 40. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 35 >>> Appendix B: Challenges Faced by SAIs in Developing Countries and the Experience of the Auditor-General of South Africa This appendix focuses on research undertaken by AFROSAI-E Within the tool, there are about 10 questions that are on the use of information technology (IT) by supreme audit concerned with the SAI’s use of technology. In completing the institutions (SAIs) in its region. It shows that these SAIs have self-assessment, the SAI uses the criteria provided for each a limited IT capability, with a lack of resources and expertise question to assess which category best reflects its current level cited as major impediments to their more extensive use of of development. The ICBF defines five development levels: technology. However, the material included in the appendix also explains how the Auditor-General of South Africa (AGSA) • Level 1 – The Founding Level: An SAI exists, but has managed the process of transformation toward becoming everything is very rudimental. The SAI is part of the an audit institution that is putting technology at the heart of executive government structure and not independent in what it does and is capitalizing on the opportunities that any area. The audit work is not organized according to technology creates. a strategic, annual operational or audit work plan. There is no human resource policy or development plan. The audits are not based on manuals aligned with international AFROSAI-E Survey Research standards (ISSAIs). No plans exist or are carried out to change the situation. • Level 2 – The Development Level: The SAI exists and Research and analysis conducted by AFROSAI-E, the regional has some legal provisions for its independence, but this group of predominantly (but not exclusively) English-speaking provision is not adequate and the SAI is dependent on African SAIs, provide the most comprehensive snapshot of the the executive for its human and financial resources. nature and extent of the use of IT by African SAIs. AFROSAI-E Strategic, annual operational, audit, development, and produces a wide range of technical and professional advice communication plans, and the thinking behind them, can and guidance for its members. It also actively promotes and be planned or are under development but they are not supports the capacity building initiatives its members take. implemented. Plans and development work to implement the ISSAIs are also underway. To track the region’s progress in developing its professional, organizational, and management capacity, AFROSAI-E • Level 3 –The Established Level: The plans prepared or encourages all of its members to complete on an annual basis under development at level 2 are implemented at level 3, the organization’s institutional capacity building framework however, improvement is needed in some areas or in the (ICBF).50 This framework is a self-assessment tool that implementation. The SAI has legislative, administrative/ covers all aspects of an SAI’s activities and consists of about managerial, and financial independence. The SAI 130 different questions or statements linked to the relevant implements its functional strategic and operational plans international standards of supreme audit institutions (ISSAIs), as well as its quality control requirements. The SAI has the International Organization of Supreme Audit Institutions implemented a management information system that (INTOSAI), or AFROSAI-E guidance and against which the has the capacity to track key management information SAI assesses itself. For each question or comment, the tool including costs, quality, and timeliness of audits. The SAI’s provides criteria against which the SAI places itself at the most audits are based on manuals complying with the ISSAIs’ appropriate level of development. requirements. The ICBF covers the following five “institutional domains”: • Level 4 – The Managed Level: Full compliance with all requirements on level 3 is achieved by the SAI—that • Independence and legal framework is, all of the requirements for the five domains and the • Organization and management individual elements in the domains are met. The SAI and • Human resources the key stakeholder are fully satisfied at level 4 with the • Audit standards and methodology implementation of plans, procedures, etc. At level 4, the • Communication and stakeholder management SAI has achieved full sustainable development. 50. AFROSAI-E, 2017 State of the Region: ICBF Self-Assessment Report, May 2018, https://afrosai-e.org.za/wp-content/uploads/2019/02/2017-State-of-the-Region_ICBF- Self-Assessment-Report1_00.pdf. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 37 • Level 5–The Optimized Level: The SAI complies fully with an IT/IS audit of systems producing financial statements. all requirements on level 4. On level 5, the SAI can scan the environment and position itself to use resources in the Taken overall, AFROSAI-E believes that these results show most proactive and value-adding way. To optimize its use that, as in previous years, funding constraints and inadequate of resources, the SAI must constantly evaluate, analyze, technical skills are limiting factors when it comes to the and assess its policies, objectives, strategies, systems, development of audit knowledge in the region and specifically procedures, and capacity as well as the skills of its staff in relation to the audit of information technology and information and the impact of its decisions. systems. In their view, this is illustrated specifically by the small number of specialist IT auditors employed by SAIs in the In relation to the individual ICBF questions about IT, the key region, by limited IT infrastructure and equipment, and by the findings for the most recent published assessment (2017) were shortage of technical IT audit skills in the region. as follows: Against this background, AFROSAI-E argues that SAIs in the • In relation to the ICBF question about information and region should prioritize the development of IT/IS audit capacity communication technology (ICT) support functions, 56 to mitigate the risk of giving governments the wrong audit percent of the SAIs that assessed themselves at level 3 opinions. Consequently, in parallel with INTOSAI, AFROSAI-E and above indicated that they had established IT support is also encouraging its members to use technology more to functions that included hardware, software, and network support and enhance their audit methods and procedures. And support maintenance. These SAIs indicated that they had in 2018, it established a working group on information systems at least one IT person per 30 staff members and that they audit and management. The group met for the first time in May had developed and implemented an ICT strategy that is 2019 and considered the following key issues: linked to the SAI strategy. • Information security challenges facing SAIs in Africa • In relation to the conduct of audits of information • Successes and challenges in SAIs’ digital transformation technology and information systems (IT/IS), during 2017, • The agenda, successes, and challenges of Smart data provided by the SAIs that completed the relevant Africa (an alliance providing leadership in accelerating section of the ICBF showed that between these SAIs, socioeconomic development through ICT) they had conducted approximately 685 IT/IS audits. About • Information security 284 IT/IS auditors worked on those audits. However, the • Incident management SAI of South Africa (the Auditor-General of South Africa) • Cyber security had conducted the bulk of these audits—about 486 audits • Potential areas of collaboration between SAIs and partners involving 141 IT/IS auditors. The next stage of the group’s activities will be to develop its • Only seven of the SAIs that completed the relevant sections work plans and to identify specific projects that focus on the of the ICBF had developed and implemented documented key issues the group has identified. IT/IS audit strategies. • Fourteen SAIs indicated that they had satisfactorily audited The Auditor-General of South Africa: the main integrated financial management expenditure and revenue systems of their countries and the other 10 A Data Analytics Case Study indicated that they have yet to consistently audit the IFMS annually. This may be attributed to the lack of technical capacity and resource constraints facing SAIs. The Auditor-General of South Africa (AGSA) is the most advanced African SAI in relation to the use of IT in its audit • Seventy-six percent of the SAIs that scored level 2 or and in its audit of IT systems. The AGSA began its information below indicated that they have yet to satisfactorily carry technology audits in the early 1990s and subsequently out integrated IT/IS audits for performance audits and established a specialist unit to support its work in this area.51 regularity audits with all regularity audits being preceded by 50. AFROSAI-E, 2017 State of the Region: ICBF Self-Assessment Report, May 2018, https://afrosai-e.org.za/wp-content/uploads/2019/02/2017-State-of-the-Region_ICBF- Self-Assessment-Report1_00.pdf. 51. Auditor-General of South Africa, “The Supreme Audit Institution (SAI) of South Africa’s Approach to Supporting Financial Audits on Human Resources, Payroll and Supply Chain Management through the Use of Data Analytics” (country paper, May 23, 2017). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 38 The AGSA Information Systems Auditing business unit provides added value to audited entities through independent and support for all of the organization’s other business units. It informed recommendations targeted on reducing the risk has a staff of 167, the majority of which hold post-graduate of project failure. degrees; 80 are certified information systems auditors (CISAs). • Enterprise resource planning system assurance: This The Information Systems Auditing business unit is an integral element involves the assessment of controls within the part of the AGSA’s audit processes and supports the office’s audited entity’s enterprise resource planning systems (for units that are responsible for financial audit by, for example, example, Oracle). audited entities’ IT internal controls. In addition, it supports the AGSA’s Investigations unit by using data analytics to monitor • Network security: This element focuses on the security of public entities’ payment systems to identify questionable the physical configuration and environment of the audited transactions. The Information Systems Auditing business unit entity’s IT systems and includes firewall reviews and also carries out financial audits in relation to entities at the database vulnerability assessments. national, provincial, and municipal levels.52 • IT research and development: To support the AGSA’s The AGSA describes the examinations that the Information integrated auditing approach, the Information Systems Systems Auditing business unit carries out as information Auditing business unit’s work in this area includes systems auditing.53 It defines this action as the examination researching and updating audit methodologies in line with of controls within an entity’s IT environment that focuses any changes in audit standards and taking account of other on “determining whether adequate controls are operating developments within the profession and among SAIs. effectively to mitigate risks to ensure that data is protected, reliable, and available.” The AGSA’s work in this area can be As part of its strategic plan for 2017 to 2020, the AGSA is further broken down to the following discrete elements: implementing a project to enhance its audits through the use of data analytics. The success of this project will revolve around • The audit of general IT controls: These are audit procedures the Information Systems Auditing business unit’s ability to that focus on the controls that should be present in the develop new IT-based tools that can be applied by auditors environment surrounding the audited entity’s application throughout the organization. In the AGSA, the main purpose of systems. These include aspects such as the organizational the use of data analytics has been to support auditors during and administrative structures of the IT function, IT policies the planning or execution of an audit. To date, the AGSA has and procedures, security management, change control, typically performed this work “under the banner of” computer- data backups and recovery, and infrastructure and assisted audit techniques (CAATs). In addition, the AGSA has environmental controls. used trend analysis tools and data visualization techniques “to identify areas of high risk and anomalies to improve audit • The audit of application controls: These controls are planning” and enhance the AGSA’s ability to focus its audit present within the relevant IT system and include the effort “on what matters.”54 validation of data input; adherence to the entity’s business rules; and access to the application systems, exception Building on this work, the AGSA is developing enhanced handling, and event logging. CAATs (that is, enhanced data analytics) to provide an automated “solution” for two key areas of its audit activities: the • The use of data analytics: For the Information Systems audit of human resources and payroll and the audit of supply Auditing business unit, this is the process of collecting, chain management.55 Because the management of human organizing, and analyzing large sets of data to identify resources and payroll is heavily centralized and procurement patterns and other useful information. management systems follow a common set of procedures at the national, provincial, and local level in South Africa, the • Project risk assurance: This element encompasses AGSA carries out what it describes as horizontal audits in both the assessment of controls in the development and areas. It defines horizontal auditing as “a process whereby a implementation of IT systems with the aim of providing single business unit or team performs audit processes and 52. Auditor-General of South Africa, 3. 53. Auditor-General of South Africa, 2. 54. Auditor-General of South Africa, “The Supreme Audit Institution (SAI) of South Africa’s Approach to Supporting Financial Audits on Human Resources, Payroll and Supply Chain Management through the Use of Data Analytics” (country paper, May 23, 2017), 4–9. 55. Auditor-General of South Africa, 10–12. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 39 procedures that are common to a number of engagements.” It The solution developed by the AGSA to respond to the sees three key benefits of this approach: challenge this creates for the audit of such a large number of entities has been to standardize its audit approach across the • Increasing audit efficiency board in this area and rely on automated data analytic tests. • Improving the quality of audit procedures However, the AGSA notes that while its data analytics tests • Improving consistency between audit engagements and are automated, its data gathering from audited entities is not promoting consistent reporting automated and it requires extensive manual intervention. In relation to the use of data analytics for the audit of human In summary, the AGSA specifies the data it requires from resources and payroll, the AGSA believes that “fewer hours each of the entities it audits. The office’s Information Systems are spent on the audit . . . and efficiencies are gained. The Auditing business unit then ensures the data provided is quality of the audit procedures is centrally monitored through properly formatted and usable. This business unit is also the development of CAATs and associated working papers. responsible for ensuring that the comparative information that Reporting . . . will be more consistent. The approach eliminates the AGSA uses for procedures for the audit of procurement the need for ad hoc audit procedure design and different and contract management is accurate and up-to-date. The approaches having to be followed for the same system and Information Systems Auditing business unit then runs a series similar business processes.”56 The processes that the AGSA is of tests that focus on the following: putting in place enable it to take payroll and human resources data on a monthly basis and subject it to a series of audit tests • The identification of fictitious suppliers, which encompasses that may draw on other sources of information. The purpose identifying employees who have the same bank account of these procedures is to focus on potential high-risk areas, details as the suppliers or who share the same address including the following: or telephone number as the supplier as well as identifying duplicate suppliers and suppliers with no physical address • The identification of employees with invalid identity and no value added tax numbers. numbers or incorrect surnames when compared with the • Awards made to individuals in the service of the audited national population register for South Africa entity or to family members of individuals in the service of • Employees who are deceased but still recorded on the audited entity. the payroll • Awards made to suppliers that have been prohibited from • Changes in the bank account details for individual doing business with the state or to companies that have employees been liquidated or deregistered. • Awards made to shell companies. The audit of supply chain management represents a major • Awards made to companies that have only been registered area of concern for the AGSA.57 It currently uses procurement in the past year. data analytics as a key element of its audit approach for 1,037 audits. Its planning for these audits is predicated on The AGSA’s audit of procurement has had a significant impact. the assumption that there is a significant risk of fraud in the In the period between 2012/13 and 2015/16, it identified procurement practices of the entities it audits and, in its audit questionable payments to the value of R$2.636 billion. risk assessments, it has identified a significant number of This discovery led to the introduction of new government audited entities displaying fraud indicators. procurement regulations intended to strengthen procurement and contract management in the public sector of South Africa. 56. Auditor-General of South Africa, “The Supreme Audit Institution (SAI) of South Africa’s Approach to Supporting Financial Audits on Human Resources, Payroll and Supply Chain Management through the Use of Data Analytics” (country paper, May 23, 2017), 12–17. 57. Auditor-General of South Africa, 18–26. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 40 >>> Appendix C: INTOSAI Strategy and Initiatives to Promote SAIs’ Use of IT This appendix describes the strategy and initiatives that the • To identify the opportunities and challenges that SAIs International Organization of Supreme Audit Institutions face in a big data world and to provide recommendations (INTOSAI) has put in place to facilitate the sharing of for SAIs knowledge and experience between supreme audit institutions • To summarize the know-how, experiences, and good (SAIs) about IT-based audit methods and procedures and to practices concerning big data audit to help SAIs boost promote and encourage the more efficient and effective use of relevant skills IT by SAIs. • To develop guidelines and research reports, to share knowledge, and support capacity building activities in big The INTOSAI Working Group on IT Audit was set up in 1989 data audit and is one of the organization’s oldest professional working • To strengthen bilateral, regional, and INTOSAI-wide group or committee. This working group is chaired by SAI cooperation between SAIs in big data audit India and 43 other SAIs participate in and support its activities. The working group acts as a forum for SAIs to share their The SAI of China chairs the working group and 18 other SAIs experiences about issues pertinent to IT audit. It also plays the participate in its activities. The available information indicates key role in developing INTOSAI advice and guidance about that the group has met twice since it was established but has IT audit. Its current work program consists of the following not issued any research, advice, or guidance. five projects: These two sets of interest intersected in 2017 at the 24th joint • Establishing general capacity requirements for SAIs for UN/INTOSAI symposium held in Vienna. This event focused conducting IT audit on sharing knowledge and experiences with digitalization, • Development of a roadmap for future INTOSAI guidance open data, and data mining to lay the foundation for SAIs to • Data analytics and IT audit techniques monitor progress toward the implementation of the SDGs. The • Updating ISSAI 5300 on guidance on IT audit and ISSAI symposium consisted of a number of presentations and panel 5310 on information systems’ security audit including discussions involving SAI representatives from around the cybersecurity world as well as contributions from multilateral organizations • Identifying the documentation requirements of an IT audit and international development partners. For INTOSAI, the key outcome was the definition of the themes for its 2019 INTOSAI Congress held in September in Russia. Since its 2016 Congress, which was held in Abu Dhabi, INTOSAI’s interest in data analytics has grown significantly. The following two themes were identified for the Congress: This interest was initially prompted by the prominence INTOSAI gave to encouraging SAIs to assess and report the • Theme 1: Information technology for the development of progress made by their respective nations toward achieving public administration including (a) data application in public the Sustainable Development Goals (SDGs). Elaborating administration, and (b) the role of big data analytics in the on the priority given to this in its strategic plan for the period activities of SAIs. 2017 to 2022, INTOSAI emphasized the scope for SAIs to strengthen their work in this area through the use of data • Theme 2: The role of SAIs in the achievement of the analytics and other audit techniques that exploit the power of national priorities and goals including (a) core strategic information technology.58 activities within audit and advisory functions, and (b) foresight and analytics in SAIs/advanced communication In parallel, as well as agreeing with its strategic plan at the strategy. 2016 Congress, INTOSAI also established a working group to consider the issue of “big data” and data analytics. The In line with the traditional method used by INTOSAI to organize establishment of this working group reflected a growing interest the technical content and discussion at the Congress, in in this subject among INTOSAI members. The working group November 2018 the Congress host (the SAI of Russia) issued on big data has the following objectives: a principal paper for each theme.59 58. In agreeing on the 2017–2022 Strategic Plan, discussion among INTOSAI members focused on how INTOSAI could contribute to the 2030 Agenda for Sustainable Development, including good governance, to strengthen the fight against corruption. The plan as agreed on at the 2016 Congress contains four strategic goals and five cross-cutting priorities including one cross-cutting priority to contribute to the follow-up and review of the SDGs within the context of each nation’s specific sustainable development efforts and SAIs’ individual mandates. 59. The principal paper for each theme is available on the website that SAI Russia has set up for the Congress: incosai2019.ru. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 42 The principal paper for theme 1 focused on two topics specified a serious problem in government due to the absence of for the theme. The paper listed in broad, general terms the proper institutional arrangements, integrated information potential benefits of SAI audit approaches and methods that systems, standards, and guidelines. SAIs should address draw on the possibilities of data analytics. It also recognized national (“whole-of-government”) capacity and public the issues that SAIs will have to address: entities capacity to effectively use performance information and its organization to improve and manage results. SAIs Despite different national conditions, the challenges should also support data-driven and evidence-based faced by SAIs are mainly manifested in two aspects. In culture and values in government by paying due attention to terms of external environment, SAIs usually face such the objective representation of performance measurement challenges as the lack of legal authorization and the low problems, public skills deficiencies, and transparency of level of digitization and big data industry; from a domestic performance information systems. perspective, they usually encounter a series of challenges in big data thinking, audit organization models, professional The traditional practice within INTOSAI is that the SAI hosting expertise, and IT infrastructure. Especially, the lack of the Congress (in this case Russia) circulates the two principal legal support and implementation measures pose major papers to all INTOSAI members and invites them to comment obstacles for SAIs to perform big data audit. That is to on the issues raised. The host then prepares a discussion say, SAIs have no mandate for obtaining electronic data paper for each theme and these papers provide the basis from audited entities or limited access to continuous and for the Congress’s deliberations. SAI Russia issued the two comprehensive data acquisition. The unavailability of data discussion papers at the end of July 2019.60 has become the prominent difficulty in the use of big data for SAIs. The discussion paper for theme 1 considered the characteristics of “big data” before briefly setting out the potential benefits for The principal paper for theme 2 of the Congress focused on SAIs of capitalizing on the opportunities offered by big data and the role of SAIs in monitoring and reporting on progress toward data analytics for enhancing SAI audits. The benefits identified achieving the SDGs. In this context, it considered the scope by the paper revolve around enhancing the efficiency and range for the use of data analytics and other advanced, sophisticated of SAI audits and demonstrating the value of an SAI’s work IT-based methodologies: by contributing, for example, to the SDGs and assessing the progress made within individual jurisdictions toward achieving Data analysis and analytics is an innovation that makes data the goals. a resource for promotion of the efficiency, accountability, effectiveness, and transparency of public administration. The paper also identified challenges facing SAIs: In order to realize SAIs’ full potential, it is crucial to build capacity to manage, analyze and interpret performance Despite the different conditions of countries, the and evaluation data for audit purposes, to nurture the challenges faced by SAIs are mainly manifested in two culture of evaluation and foresight within SAIs, to build aspects. The first aspect is from a domestic perspective. skills on program evaluation, data analysis and analytics, The SAIs usually encounter a series of challenges in big key national indicators, systemic thinking, assessment of data thinking, audit organization models, expertise, and IT policy coherence. infrastructure. The most common problems are insufficient professionals and technical challenges. The second aspect The paper advocated a role for SAIs in promoting transparency of challenges comes from the external environment. There in government decision-making and in ensuring the might still be some barriers related to obtaining data for reliability and accuracy of the data that governments use for SAIs.These barriers may be technical issues, costs, decision-making: data anonymization [sic], quality, and delays. In practice, obstacles occasionally emerge from the audited entities in Advances in technologies are close to allowing real-time order to avoid providing the SAI with data appropriate for a monitoring and early warnings, but the use of data and complete and in-depth analysis. evidence to inform and drive major decisions still remains 60. The discussion paper for each theme is available on the website that SAI Russia has set up for the Congress: incosai2019.ru. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 43 In response to these challenges, the paper set out aspirations, approaches, and a need for SAIs to rethink their role in the including the following: government accountability process. • Encouragement for SAIs “to step up” their use of IT In this regard, the paper explored the implications for SAIs that applications for audit purposes are strengthening their research and advisory capacity and • The observation that SAIs should make “greater efforts act, in effect, as a “trusted advisor” for their key stakeholders to enhance the overall competence” of their audit teams while at the same time maintaining their independence. The including establishing a “big data analysis team or paper focused in particular on the contribution that data agency to help transform the audit organization from on- analytics could make to enhancing the quality of SAI audits. site audit to the combination of on-site audit and off-site The paper commented: data analysis.” • The argument that “data should be recognized as an In order to realize … [the] full potential [of SAIs], it is important resource in the information era” and consequently crucial to build capacity to manage, analyze, and interpret the relevant legal framework should “explicitly” performance and evaluation data for audit purposes, to stipulate that the SAI is “entitled to obtain data from the nurture the culture of evaluation and foresight within SAIs, audited entities.” to build skills on program evaluation, data analysis and analytics, AI, system thinking, and assessment of policy However, the paper was sketchy when it came to specifying coherence etc. There is a need to embed data analytics how SAIs could make greater use of sophisticated audit data in the whole audit process, from planning to reporting. analytics; essentially, it asked for INTOSAI to encourage Data analysis, analytics, AI, and machine learning are greater cooperation between SAIs and more sharing of relevant innovations that make data a resource for the promotion experience between SAIs: of the efficiency, accountability, effectiveness, and transparency of public administration … [T]he skills of As a professional organization advocating for public problem-solving and data analytics that allow patterns sector auditing, INTOSAI should strengthen the sharing to be seen can be hidden by the sheer quantity of data of big data audit experiences. By strengthening bilateral available. Developing these skills are crucial for auditors. and multilateral cooperation among SAIs as well as coordination and cooperation with relevant international This paper was also vague on how SAIs can develop what organizations, INTOSAI can summarize big data audit it describes as “auditors of the future” that have the requisite experience and knowledge, develop relevant guidelines skills and organizational capability to carry out audits that and research reports, and encourage SAIs to improve big harness the potential created by new technology and new data audit, so as to promote the development of big data technological innovations. For example, it advocated working audit within the INTOSAI community. closely with the research community and initiatives at all levels within INTOSAI to share experiences and develop new, The discussion paper for theme 2 presented a set of arguments innovative audit approaches. for a potentially significant shift away from the traditional role of SAIs in response to changes in the wider political, social, In response to the questions raised by the theme papers, and technological environment in which they work. It argued individual SAIs might submit country papers that reflect that developments such as the 2030 Agenda for Sustainable their own relevant individual experiences and views. These Development and the adoption of the SDGs create new contribute to, and inform, the debate at the Congress. In light opportunities for SAIs to play a more influential role in of those discussions, the Congress agreed on a final statement assessing and encouraging progress toward achieving those that reflects INTOSAI’s position or views on the relevant issues goals within their respective jurisdictions. This opportunity in and sets out, in broad terms, how INTOSAI envisages moving turn intersected with a predisposition to assist on the part of forward with its collective work in these areas. SAIs (which INTOSAI encourages)—namely, that the work of SAIs should have a beneficial impact on the lives of citizens, Another key priority for INTOSAI that informed the discussion something that INTOSAI has enshrined in ISSAI 12 (the at the Congress was to encourage, and as far as possible value and benefits of supreme audit institutions). The paper facilitate, the capacity of development of SAIs. INTOSAI’s contended that, taken together, this discussion points to a initiatives in this regard were framed within the context of need for better quality audit work on the part of SAIs, new audit the international standards of SAIs (the ISSAIs). The SAI EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 44 performance measurement framework (SAI-PMF) developed World Bank. With the WGISTA established, INTOSAI now has and endorsed by INTOSAI provides the best illustration of a forward-looking working group that will monitor potential this. The SAI-PMF assessment tool covers the full range of impacts that science and technology may have on the audit SAI independence, audit work, management organization, profession; share this specialized knowledge and experiences; and administration. The criteria that it incorporates are framed and ensure INTOSAI’s work remains relevant and impactful.61 around the ISSAIs and these in turn reflect the standards for audit in the private sector (the international standards on At the XXIII INTOSAI International Congress of Supreme Audit auditing, ISAs). Institutions (INCOSAI), the Moscow Declaration was endorse.62 The declaration is intended to establish the Congress’s key Like the ISAs, the ISSAIs were largely formulated in the era outcomes and to help guide INTOSAI’s future. It comprises before the use of sophisticated IT tools and methods and, the results from the comprehensive thematic discussions therefore, they do not create an incentive for SAIs to adopt on “information technologies for the development of public new, innovative IT-based methods for their audits. In addition, administration” and “the role of supreme audit institutions in the unlike their private sector counterparts, SAIs do not work in achievement of national priorities and goals.” The declaration an environment in which they are subject to competitive also informs the future activities and strategies of INTOSAI pressure and in which competition acts as a spur to innovation and member SAIs while taking into account the SDGs as well and change. as fundamental changes in public auditing and public policy worldwide. The declaration states that SAIs and INTOSAI The Working Group on Impacts of Science and Technology commit themselves to provide independent external oversight on Auditing (WGISTA) was officially formed at the 2019 on achieving nationally agreed on targets (including those Congress. Chaired by the State Audit Institution of the United linked to the SDGs); to effectively respond to opportunities Arab Emirates (UAE) with the US Government Accountability brought by technological advancement; and to enhance the Office (GAO) acting as vice chair, the WGISTA will focus on impact of SAIs, including through the following commitments: key trends in areas, such as artificial intelligence, blockchain technology, cybersecurity, data analytics, 5G cellular network • Contribute to more effective, transparent, and informative technology, machine learning, and quantum computing. accountability for outcomes. WGISTA’s primary strategic objectives include conducting • Develop a strategic approach to public auditing to support environmental scanning; sharing best practices; maintaining achieving national priorities and the SDGs. expertise within SAIs; applying science and technology in • Enhance public auditing value by extending the provision auditing; and developing competencies required by SAIs of audit-based advice on important and strategic issues to and auditors. parliament, government, and public administration. • Promote the principle of availability and openness of data, The WGISTA concept evolved from the work of the Supervisory source code, and algorithms. Committee on Emerging Issues, which identified science and • Make better use of data analytics in audits, including technology trends as key issues that would increasingly affect developing adaptation strategies and experienced data governments and SAIs in the future. An interim task force was analytics teams and introducing new techniques in public established at the 2018 INTOSAI Governing Board meeting in audit practices. Moscow, Russia, to develop Terms of Reference for the new • Foster an experimental mindset. working group. The task force, led by the SAI UAE and US GAO, • Extend focus on identifying risk areas, raising awareness included the American Institute of Certified Public Accountants, of risks, and managing systemic ones. Audit Board of the Republic of Indonesia, National Audit Office • Nurture “auditors of the future” and act as strategic players, of China, Institute of Internal Auditors; INTOSAI Development knowledge exchangers, and foresight producers. Initiative, Office of the Comptroller and Auditor General of India, • Address inclusiveness when conducting audits and follow Organisation for Economic Co-operation and Development, the principle of leaving no one behind. Pacific Association of Supreme Audit Institutions, United • Establish productive interactions with auditees and enhance Nations Department of Economic and Social Affairs, and the cooperation and communication with stakeholders. 61. For more information, see the INTOSAI International Journal of Government Auditing website at http://intosaijournal.org/wgista/. 62. INTOSAI, “Summary of the Moscow Declaration,” International Journal of Government Auditing 46, no. 4 (Autumn 2019). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 45 >>> Appendix D: Private Sector’s Experience Using Audit Data Analytics and Other Advanced IT-Based Audit Methods and Procedures This appendix describes how audit firms in the private sector data completely and accurately into a format their automated and the internal audit function of some major companies tools and techniques can analyze.” This level of investment and corporations are using information technology (IT). The meets firms’ current requirements and, in the FRC’s view, research and linked examples cited in this appendix show that provides “a foundation for the needs of emerging technologies.” competition has spurred innovation and that the key objective In this context, the FRC made the important observation that in the application of more advanced, sophisticated IT-based “the auditor’s ability to extract useful data remains dependent audit methods and procedures has been to improve audit on the availability of granular data at the audited entity, and the efficiency and audit quality. capability and willingness of management to provide that data for auditor analysis.” The Experience of Private Sector As for these emerging technologies, the FRC distinguished between those that have the potential to improve audit quality External Audit Companies and those that “drive audit efficiencies.” The FRC identified three emerging technologies that may Recent reports issued by the UK Financial Reporting Council contribute to improving audit quality because they potentially (FRC) and the ACCA (the Association of Chartered Certified enable the auditor to deal with “increasingly complex corporate Accountants), in conjunction with Chartered Accountants entities, increasingly complex electronic transaction flows, and Australia and New Zealand, provide an overview of how private increasing volumes of data.” These technologies may enable sector audit companies are using increasingly sophisticated auditors “to assess and potentially interpret or extrapolate IT-based technology and techniques as integral parts of their entire populations of data.” audit methods and procedures.63 The specific technologies identified by the FRC were as follows: In its 2020 report “The Use of Technology in the Audit of Financial Statements,” the FRC noted that audit firms’ use of • Machine learning (ML): “Machine learning provides the audit data analytics (ADA) tools followed good practice “with opportunity to identify unusual patterns and exceptions the technology deployed as part of a package with associated in large populations of data that might not be discernible methodology, training, and support.”64 In the FRC’s view, this using more traditional techniques (which commonly include enables “audit teams to understand the audit evidence that setting expectations of what would be unusual in advance). the technique generates, and the additional audit procedures Journals testing is one area where firms are seeking to use necessary to address the relevant assertions appropriately.” machine learning to identify journals with characteristics Documentation of the evidence generated by these approaches that are unusual in relation to the population, without was “in general, much approved.” And the use of ADA had predefining such characteristics as noted in ISA (UK) 240 become “business as usual” for many audit teams, “with the para A23. This means the audit testing is less predictable current generation of audit recruits developing relevant skills and hence potentially more robust.”65 as an intrinsic part of their training.” • Natural language processing: The pilot use of this The FRC found that although firms had established different technique observed by the FRC involved extracting and priorities in their adoption and rollout of automated tools and structuring contract information from source documents techniques, all were continuing to develop and enhance their (largely unstructured text) into audit workpapers for use of technological resources. The FRC noted that the audit of further analysis. The FRC commented that it liked the revenue and the use of ADA for risk assessment were the two combination of using an intelligent automated tool to audit areas with the most prevalent use of ADA beyond journal perform a time-consuming and mundane task for a “human” entry testing. There has been substantial investment by firms auditor, leaving the auditor free to perform the more in “the infrastructure required to capture, collate, and organize qualitative analysis.66 63. Financial Reporting Council, “The Use of Technology in the Audit of Financial Statements,” AQR Thematic Review, March 2020; Association of Chartered Certified Accountants and Chartered Accountants Australia and New Zealand, “Audit and Technology,” June 2019. 64. Financial Reporting Council, “The Use of Technology in the Audit of Financial Statements,” AQR Thematic Review, March 2020, 2–3. 65. Financial Reporting Council, “The Use of Technology in the Audit of Financial Statements,” AQR Thematic Review, March 2020, 14. 66. Financial Reporting Council, 15. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 47 • Predictive analysis: The informed and effective challenge auditors inspecting and verifying assets in inaccessible of management is a cornerstone of a high-quality audit places. The FRC commented that it had seen the use of and an area in which auditors may be able to harness drones by auditors to assess the existence and condition the potential of a technique such as predictive analysis. of assets in the following industries: extractive, agriculture, “The use of predictive analysis can assist the auditor in construction, and infrastructure.70 forming independent expectations of events, conditions, or outcomes. This may be for current outcomes, such In the research paper “Audit and Technology,” published in as forming an independent expectation when performing June 2019, the ACCA and Chartered Accountants Australia substantive analytical procedures, or future events such and New Zealand comment that advances in technology as when auditing judgments around going concern or “promise significant benefit for the audit profession.” They go impairment. The auditor must identify which are the on to identify “the rapid increase in volume of data, changes significant scenarios to record on the final audit file to in business models, the shift towards automation, and the demonstrate the nature and extent of their work and the demand for a proactive and forward-looking approach to conclusions reached.”67 audit” as key drivers that show the need for technological change in audit.71 The paper then goes on to highlight the The FRC also identified three additional emerging technologies following new and evolving technologies that are changing the that have the potential to increase the efficiency of audit work audit landscape: that involves the collection and collation of audit evidence through administrative and repetitive tasks that do not require • Artificial intelligence (AI): The report describes the judgment. These tasks lend themselves to automation. “intelligence” in AI as often constituting a combination of processing power and access to data that enables the The specific technologies identified by the FRC in this regard analysis of entire populations of data to identify patterns were as follows: or exceptions. This technology in turn has the capability to free auditors from mundane tasks and enable them • Smartphones: This technology was the most widely to focus their time on deploying their skills, training, and deployed technology identified by the FRC with the most judgment, thus improving audit quality.72 popular application being in physical stock counts. Audit firms have developed their own “apps” that auditors can • Robotic process automation (RPA): The report use to follow and record the results of inventory count describes RPA as “software that can be easily programmed procedures, streamlining the stock counting process.68 or instructed by end users to perform high-volume, repeatable, rules-based tasks in today’s world where • Robotic process automation (RPA): RPA can be used multiple loosely integrated systems are commonplace.” to populate audit working papers from a set of underlying The report notes that because it mimics a process rather audit evidence. Under the FRC’s working definition of than analyzing data, RPA itself is not AI, which could be RPA, “no intelligence is applied, it is purely the automation used later to look for the anomalies that previously a of a repetitive task. The output from an RPA may be very human operator might have had to spot. RPA offers many similar to that from a tool that uses machine learning or potential benefits because it enables nonstop, faster, and natural language processing to populate workpapers. more accurate work. The report comments that there are The difference is the process. In general, the audit firms’ questions about accountability and ownership of the RPA approach to the implementation of RPA is to deploy limited process and about the security of the data that passes functionality but make it available to a wide number of through it.73 audit teams.”69 • Data analytics: The report notes that auditors have • Drones: The FRC noted the prime purpose of the use of long applied analytical tools to the data derived from drones by audit firms was as “eyes in the skies.” A key accounting and operational systems and it notes that benefit was to reduce practical issues associated with some firms are already using data analytics as part of their 67. Financial Reporting Council, 15. 68. Financial Reporting Council, “The Use of Technology in the Audit of Financial Statements,” AQR Thematic Review, March 2020, 16. 69. Financial Reporting Council, 16. 70. Financial Reporting Council, 16. 71. Association of Chartered Certified Accountants and Chartered Accountants Australia and New Zealand, “Audit and Technology,” June 2019, 6. 72. Association of Chartered Certified Accountants and Chartered Accountants Australia and New Zealand, “Audit and Technology,” June 2019, 9. 73. Association of Chartered Certified Accountants and Chartered Accountants Australia and New Zealand, 9–10. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 48 transactions testing, and they are gradually moving away documents. It goes on to quote a 2017 Forbes report that from traditional sampling techniques. Data analytics allow Deloitte’s use of NLP took contract review from a task auditors to use 100 percent of a population’s transactions keeping “dozens” of employees occupied for half a year to when performing their tests. The report quotes the following one that six to eight members could complete in less than 2015 comment from KPMG: “Using [data analytics] we a month.76 make the analysis of the past more insightful. Rather than sampling transactions data to test a snapshot of activities, • Deep learning (DL): The report comments that DL is we can now analyze all transactions processed, allowing a subset of ML and that “it more closely mimics human us to identify anomalies and drill down on the items that learning through the use of artificial neural networks show the greatest potential of being high risk. Our systems to perform more complex tasks such as visual object automate this process, increasing its ability to produce high recognition.” The report notes that “DL systems are quality audit evidence.” The report comments that the next commercially available and have already been deployed step for auditors is to apply AI and ML algorithms to improve by the Big Four accountancy firms: KPMG uses IBM’s the quality of analysis and forecasting and to increase the Watson to analyze commercial mortgage loan portfolios, rate of fraud detection. Developing this theme, it argues while Deloitte works with the Canadian-based legal AI that auditors can use data mining software “to drill down company Kira Systems to ‘read’ thousands of complex and identify anomalies—possibly aided by AI—focusing documents, such as contracts, leases, and invoices, resources on identifying risks in addition to monitoring extracting and structuring textual information such as key ‘business as usual’ activities.”74 words or phrases.”77 The report argues that the ability of DL to analyze a range of internal and external sources • Machine learning (ML): The report also elaborates on the means that “big data can potentially supply complementary potential of ML. This technology uses statistical analyses to audit evidence and feed into the narrative requirements generate predictions or make decisions from the analysis of audit.” of a large historical dataset. The report refers to an example of the use of ML in audit in PwC’s report “Confidence in • Drone technology, Internet of Things, and sensor the Future: Human and Machine Collaboration in the technologies: The report notes that unmanned drones Audit Report.” The PwC report describes the following are used in a variety of commercial projects, such as example: “company A was way out of line with the peer power line inspection, and it comments that the Big Four group benchmarks on a particular point. This data is then accountancy firms have spotted the potential for their use shared with the audit team, who can decide whether that in inventory inspection, “particularly where physical scale variance is really an anomaly and if so, what caused it. The or distribution is an issue.” The report refers to an example team’s decision about the anomaly and its cause is then from PwC, which announced in 2019 its first stock count fed back to the machine, which is ‘taught’ how to respond audit—of an open cast mine—using drone technology. The to similar relationships in [the] future. And the more this report comments that an example of a sector that is ripe for exercise is carried out, the better the machine will get at the adoption of such technologies in audit and assurance spotting real anomalies—meaning we’ll be better able to is agriculture.78 identify unusual patterns and anomalies in huge amounts of data in an instant.”75 • Distributed ledger technology (DLT): The report describes DLT as a family of technologies that includes • Natural language processing (NLP): The report notes blockchain and that is of great interest to both auditors and that NLP refers to the ability of the computer to recognize businesses. It comments that “for the auditor, distributed and understand human speech. It comments that the ledgers become a sort of universal bookkeeping service, most immediate impact is speed and that NLP has been removing the need to reconcile multiple databases of shown to achieve orders of magnitude improvements in records and providing a perfect audit trail... DLT offers due diligence exercises involving very large numbers of the possibility of generating exception reports that are 74. Association of Chartered Certified Accountants and Chartered Accountants Australia and New Zealand, 10–11. 75. PwC (2017), as quoted by the Association of Chartered Certified Accountants and the Chartered Accountants Australia and New Zealand in “Audit and Technology,” June 2019, 11–12. 76. Zhou (2017), as cited by the Association of Chartered Certified Accountants and the Chartered Accountants Australia and New Zealand in “Audit and Technology,” June 2019, 12. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 49 based on all transactions rather than on using sampling it must contain critical data, and that “a key benefit for audit techniques—a return to the roots of auditing.” The report is that organizations will increasingly be referring to a single notes that “it has even been claimed that blockchain will data source, which updates for everyone, everywhere with become the industry standard for accounting and reporting.” no time lags or inconsistencies.” However, it is recognized And in this context, according to Jon Raphael, audit chief that this comes with risks arising from the need to protect innovation officer at Deloitte & Touche LLP, “Blockchain critical data and comply with relevant regulation. “Cyber has the potential to be very transformative. By itself, risk is one of the most talked-about business risks. In our blockchain will likely change how records are maintained increasingly disrupted world, it is at the forefront of our and how value is transferred between counterparties... minds.”80 Although implementation of appropriate risk- Most compelling, however, is blockchain’s potential for mitigation strategies rests with the cloud provider, the risk transformative analytic capabilities. One of the beneficial itself remains with the data owner. In practice, however, outcomes of blockchain is easy access to structured data, many smaller entities argue that cloud vendors offer better which can then be used to generate advanced analytics security than, say, an in-house server. The report refers and accelerate machine learning. This will enable tools to to KPMG’s argument that auditors need to integrate more get smarter and drive us further and faster toward more cybersecurity capability in the audits and rethink and continuous auditing and assurance.”79 reevaluate their approach in providing assurance around cloud systems. In this context, the report quotes: “This can • Cloud technologies: The report comments that the rise be achieved by transforming audit approaches leveraging of cloud-based systems goes back a long way “arguably, data analytics-driven procedures in order to address the a mainframe computer connected to dumb terminals in less preventive controls in the system. In addition, the regional offices represented an early form of ‘cloud.’ Now, a processing and storing of data in the cloud for cloud-based cloud system will be hosted remotely and accessed remotely systems introduces new challenges around third party by generic devices such as tablets, PCs, or smartphones.” management and data security and confidentiality.”81 The report makes the key point that for cloud to be useful, 77. Association of Chartered Certified Accountants and Chartered Accountants Australia and New Zealand, 12. 78. Association of Chartered Certified Accountants and Chartered Accountants Australia and New Zealand, 13. 79. Deloitte (2018), as quoted by the Association of Chartered Certified Accountants and the Chartered Accountants Australia and New Zealand in “Audit and Technology,” June 2019, 13–14. 80. Association of Chartered Certified Accountants and the Chartered Accountants Australia and New Zealand, “Audit and Technology,” June 2019. 81. KPMG (2018), as quoted by Association of Chartered Certified Accountants and the Chartered Accountants Australia and New Zealand in “Audit and Technology,” June 2019, 15. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 50 Internal Audit Examples into the SAI’s audit approach with a relatively low investment. The more sophisticated techniques, however, require more investment with regard to development and training for their proper application. The research paper “Data Analytics: Is It Time to Take the First Step?” prepared by the Chartered Institute of Internal Auditors • The successful introduction and application of audit data and published in 2017 contains three case studies that are analytics requires close cooperation between audit staff particularly pertinent for SAIs that are planning to introduce and the SAI’s IT technical support staff. In addition, audit or to make more extensive use of audit data analytics. The staff need to develop their understanding of how these case studies focus on the internal audit function of three private tools work and also acquire experience and expertise in sector companies: Coca Cola, Credit Suisse, and Dublin areas such as coding and scripting. Airports Authority. They illustrate the challenges inherent in an audit function placing greater reliance on data analytic methods and procedures and then draw out the benefits of Coca Cola Case Study those methods and procedures for the audit function. The overall conclusion that the research paper draws from the Coca Cola case study is the increased assurance that the In summary, the case studies provide the following key practical audit function is able to draw from the use of whole-population lessons for SAIs: testing that data analytics enables and the need to assign specialist staff to the audit function to increase the requisite • Audit data analytics can be embedded in all aspects technological skills. of the audit cycle although the main benefits flow from their use at the planning stage and in the course of the The key points drawn from the Coca Cola case study are audit fieldwork. as follows: • The use of audit data analytics creates the opportunity • The ability to test 100 percent of the sample in various to rebalance the use of the resources invested in audits is where the real value lies, because the audit individual audits. For example, detailed audit procedures function is able to provide much greater assurance. involving the examination of individual transactions can be completed more quickly and efficiently and this • The challenge of getting people with a combination of enables auditors to focus more on key areas of strategic internal audit leadership and data analytics skills will be an risk and the implications for the audited entity and its increasingly important issue to overcome in the future. financial statements. • The head of internal audit assigns members of his team to • The capability to audit a larger volume of transactions, the company’s data team to help up-skill them in the more including the audit of an entire population of transactions, technological side of data analysis, such as scripting. means that the auditor can draw greater assurance from the audit tests and procedures carried out. • The audit function introduced the use of data analytics a few years ago because Coca Cola wanted to benefit fully • The increasing use of audit data analytics brings with it the from making use of data generated from the company’s need to ensure a proper understanding of the enhanced enterprise resource planning system. It now has an audit methods and procedures at all levels within the independent data analytics team (led by a data analytics SAI (and particularly among senior practitioners and expert) designing and developing a broad range of audit senior managers). It also needs effective engagement command language (ACL) scripts and working with the with the management of the audited entity to ensure they business to understand its key strategic concerns, and if understand the nature and benefits of these audit methods possible, design tools to help it. and procedures so that they cooperate with, and when necessary, facilitate, the application of these methods • The audit function uses data mining tools (for example, ACL) and procedures. to build and automate various queries over the available sets of data, which are good for identifying business • A wide range of potential audit data analytic tools are process exceptions, highlighting macro-level patterns or available to SAIs. Desktop applications can be incorporated risks. However, elements of this are a manual process EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 51 and require a unique combination of relevant business and • They go beyond making use of data analytics in financially technical knowledge on top of the understanding of writing oriented audits by utilizing it in strategic risk areas such as scripts that underpin the data mining tools. the “risk and control culture” aspects of their audits. • Data analytics are incorporated into the whole audit • Data analytics enables the audit function to do more with process from planning onwards. The design incorporates less data while increasing the quality of its output. In other risk-based internal auditing into the process. Each audit words, it helps to enhance the function’s efficiency and contains up to 300 standard working papers and there are effectiveness. The overall objective is to transform the certain things that are always tested. Approximately 40 function from a traditional judgment-based, sample-driven, percent of working papers have a built-in data analytics tool. time-intensive and reactionary audit process to one that is risk-based, continuous, and wholly data-centric and that is • The internal auditors use data analytics heavily in the carried out in real time. Today, data analytics is helping the fieldwork stage. organization to identify business areas with high-control risks due to anomalous, nonconforming events, and it is • Internal auditors need to learn to use data analytics facilitating the continuous monitoring of the risks. effectively. To help auditors build their technical skills they can spend six months on assignment with the company’s • Data analytics is used in the entire internal audit cycle but data analytics team where they are given the opportunity mostly as part of fieldwork. The use in the entire lifecycle is to write their own scripts and learn how the data systems as follows: work from a technical perspective. • Risk assessment: Can analytics help with ongoing • Continuous auditing would enable auditors to spot risk assessment, either via ad-hoc testing or in the anomalies in large quantities of data, but continuous context of continuous risk monitoring processes? auditing is at an early stage. They have started to run • Planning: Can analytics help to drive more targeted some scripts continuously but they have found that they auditing by helping auditors focus on areas or need to define the parameters accurately, otherwise too population segments with the highest risks? much data is generated. • Fieldwork: Can analytics be used to provide a higher degree of assurance by testing up to 100 percent of Credit Suisse Case Study the population, or perform testing more efficiently? The overall conclusion that the research paper draws from the • Reporting: Can more insightful findings and actionable Credit Suisse case study is the extensive use that the audit outcomes be provided through analytics by helping to function makes of data analytics. In addition to using data quantify risk or identify root causes? analytics in its financial audit work, Credit Suisse also applies them in its examinations of nonfinancial strategic risk areas. Dublin Airports Authority Case Study The overall conclusion that the research paper draws from The key points drawn from the Credit Suisse case study are the Dublin Airports Authority case study is that the use of data as follows: analytics enables the audit function to place a greater focus on strategic risk. • Adopting data analytics transformed the function from a traditional judgment-based, sample-driven, manually The key points drawn from the Dublin Airports case study are intensive and reactionary audit process to one that is as follows: truly risk-based, continuous/real-time, and data-centric. Buy-in from the function’s leadership and every auditor in • Prior to applying data analytics, 90 percent of the function’s the team was crucial. Data analytics is integrated into all time was spent on financial audits. Since incorporating parts of the internal audit cycle. However, it is mainly used data analytics, the focus has shifted, with 50 percent of in fieldwork. audits now concerning nonfinancial risks. This shift of focus improved the function’s overall effectiveness because the • Ideal auditors who use data analytics as standard in their key risks in the organization are largely nonfinancial and audit methodology need a blend of core analytics skillsets, the overall direction of internal audit is to operate on a business experience, and a solid understanding of risk. more strategic level by looking at strategic risks. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 52 • Data analytics enables continuous auditing on specific • One of the initial key challenges was getting buy-in from business cycles but not necessarily all of them, the organization’s IT department because it was concerned especially in the case of smaller audit functions. It is about the potential impact on the enterprise resource important to weigh the costs and benefits of introducing planning system’s performance. Access to the raw data at continuous auditing. the start of using data analytics poses some issues and accessing specific tables and reformatting them so that • Data analytics can be applied in all audit functions they are user-friendly can be a little tricky, but with the right regardless of size. Sophisticated applications require training and repeated use over time these issues become larger investment, but the basic application of using a less problematic. desktop function can be incorporated into all internal audit functions with relatively low levels of investment. • The overall benefits are that data analytics is carrying out a significant amount of extra work that it wasn’t doing a • Initially, data analytics was introduced as a desktop couple of years ago. Also, the organization has saved function enabling data analytical work to be used on significantly on headcount using data analytics—it is doing the work around financial risk. The use of data analytics more work with fewer bodies and achieving much deeper brought immediate benefits, for example, through being audit penetration and the work is much more insightful. able to identify potential duplicate transactions. Using audit software and reporting tools enables statistical analysis and the ability to consider many interrelated fields • Data analytics is mainly used in the fieldwork stage of and transactions over time. This then gives auditors greater audits, but sometimes it is used in the planning stages as scope to provide greater insight into what is happening part of the risk assessment process or, for example, when across the organization. trying to gauge the volume of transactions. Dublin Airports Authority’s auditors are now able to test 100 percent • Data analytics has the potential to be used in all audit of populations and more accurately identify areas of functions regardless of size. To use it in a more sophisticated noncompliance and control in certain areas. Data analytics way, a larger investment is needed but the basic is not used in all areas and some areas lend themselves application using a desktop function can be incorporated better to its use than others. Examples of areas in which into all internal audit functions with relatively low levels of the auditors use data analytics include the procure-to-pay investment. Scripting may be beyond the reach of some cycle and the time and attendance system. As time goes audit functions, but much can be done without it (such as on, data analytics can be used in more and more areas of point in time analytics). A great advantage of using data the audit universe. analytics, the team has found, is that it frees up time to take a more in-depth look at strategic areas. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 53 >>> Appendix E: International Audit Standards and IT-Based Audit Methods and Procedures The appendix sets out developments within the audit profession The development of standard tools for general use by audit about the way in which the use of IT-based tools and methods teams can be traced back to early 2005 following the introduction by private sector audit firms is overseen and regulated. In this of specific tests as part of the auditor’s responsibilities in regard, the analysis and research underpinning reports issued relation to fraud (International Standards of Auditing 240). by the International Auditing and Assurance Standards Board Firms introduced standard tools to facilitate the audit tests in (IAASB) and by the Financial Reporting Council (FRC) of the line with ISA 240 together with specialized support to aid in United Kingdom provide a useful starting point for a summary capturing the data and loading it into such tools. of the audit profession’s growing use of data analytics and the issues that this raises for the regulation and oversight of The FRC found that UK audit teams were developing bespoke the profession. audit data analytics in relation to specific auditing issues and that a key characteristic of the increase in the use of audit In relation to the impact of technology on the audit of private data analytics was the roll out of standard audit data analytics sector entities, the IAASB has commented that, in the course tools and techniques, coded and tested by specialist staff and of the history of the audit profession, there have been shifts deployed with central support. The FRC commented that this in how the audit is executed. These shifts have been a result meant the use of audit data analytics was becoming more of transformations in the environment in which companies efficient, consistent, and reliable. And, in this context, some of operate, and in which audits are performed. Prior to the current the most successful and widely used audit data analytics tools risk-based audit approach, companies operated in a far less had started out as bespoke CAATs for use on individual audits complex environment. As a result, the audit was carried out and were subsequently developed further and standardized for in a largely manual way with a relatively high proportion of the wider application. financial information underlying the financial statements being tested without any significant emphasis on the nature and Both the IAASB and the FRC found that competition had been extent of the risks of material misstatement. a key driver in private sector audit firms making more extensive use of audit data analytics. The FRC comments, for example, Over time, the risk-based audit approach has evolved. This that the introduction of mandatory retendering of audits in evolution was due to a range of factors including higher the United Kingdom has provided additional incentives to transaction volumes such that auditors were not able to test accelerate the development of audit data analytics because all transactions underlying the financial statements; increased companies’ audit committees increasingly see their use as a complexity; regulation stimulated by highly public failures key differentiator between competing bids from audit firms. of companies; and technological limitations. A risk-based audit focuses on the nature and extent of risks of material The FRC noted that firms’ strategies in respect of the roll out misstatement for the particular engagement, with greater of audit data analytics differ: some focus on the adoption of emphasis on obtaining an understanding of internal control a limited tool set while others provide a wider range of tools. established by an entity and, when appropriate, obtaining audit Within the UK market, the FRC found that audit data analytics evidence from the auditor’s testing of the effectiveness of such tools were being used to internal control.82 • Analyze all transactions in a population, stratify that In this context, the FRC found that professional audit firms population, and identify outliers for further examination; in the United Kingdom have used computers to analyze data • Re-perform calculations relevant to the financial in performing audits since companies first computerized statements; their accounting systems. Such procedures, referred to as • Match transactions as they pass through a processing cycle; CAATs (computer-assisted audit techniques), were typically • Assist in segregation of duties testing; used to analyze sets of data to identify data meeting certain • Compare entity data to externally obtained data; and characteristics for further testing by the audit team. These • Manipulate data to assess the impact of different CAATs tended to be tailored specifically to the entity being assumptions. audited, requiring a significant investment of time, and, as such, were not widely used across all of the firms’ audits.83 82. IAASB Data Analytics Working Group, “Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics” (exposure drafts and consultation paper, September 2016), 6. 83. Financial Reporting Council, “Audit Quality Thematic Review: The Use of Data Analytics in the Audit of Financial Statements,” January 2017, 6. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 55 The FRC found that some UK audit firms were considering • The classification of evidence obtained through the use additional audit data analytics techniques, but these had not of data analytics in the context of the current “risk and been developed or deployed for use on audits of financial response” nature of the ISAs statements. These techniques included, for example, • Deciding the appropriate amount of work effort for continuous control monitoring, benchmarking of data between exceptions identified—essentially should the auditor audit clients at a transactional level, and unstructured data examine all exceptions and outliers analysis (the automated analysis of unstructured data such as • The documentation requirements for the application of the content of emails or word processing documents).84 data analytics • The quality assurance arrangements to be applied to the Against this background, the key concern for the IAASB is how development of data analytics technology and tools used to accommodate audit data analytics and its potential benefits in an audit and related audit methodology including the within the structure of the international standards for audit— reliability of data analytics tools developed by third parties87 the ISAs—and facilitate the regulation of the audit profession in this rapidly changing environment. As they currently stand, the The most recent technical update issued by the IAASB identified ISAs “do not prohibit, nor stimulate, the use of data analytics. . two areas in which the IAASB is considering the impact of . . The ISAs acknowledge the use of technology by the auditor technology on audit standards. These areas are ISA 315 in executing the audit, through use of computer-assisted audit (identifying and assessing the risks of material misstatement techniques (CAATs). However, the reference to CAATs in the through understanding the entity and its environment) and ISAs was created in a completely different technological era two quality management standards (International Standard on and CAATs have evolved significantly into what is now being Quality Management 1 and ISA 220, quality control for an audit referred to as data analytics.”85 The IAASB acknowledges that of financial statements).88 this absence of reference to data analytics “beyond mention of traditional CAATs in the ISAs may be viewed as a barrier to In its discussions about the implications of audit data analytics their adoption more broadly.” And, the IAASB also recognizes for the audit of financial statements, the IAASB acknowledges that without a firm framework of standards, there is a risk that that the implications for SAIs may be different. It has stated: auditors may face difficulties in substantiating the judgments they make and the procedures they use when they rely on In public sector entity environments, where the existence the use of audit data analytics. This in turn may deter auditors of home-grown systems is more prevalent based on the from “using and experimenting with data analytics” and also specific nature of the role of these entities, data capture result in the views of audit oversight authorities evolving “in an could prove to be challenging and limiting with respect to inconsistent manner—within and between jurisdictions.”86 applying data analytics in the financial statement audit. Audits of public sector entities include the audit of the In the IAASB’s view, auditors need to consider the following financial statements, but there are numerous other types issues in relation to the application of data analytics within the of audits performed (such as performance audits) that may ISA regulatory framework: lend themselves well to data analytics.89 • General IT controls within both the audited entity and the In responding to the IAASB’s initial assessment of the external auditor implications of developments in data analytics for audit oversight • The completeness and reliability of the information and regulation, two aspects of the views expressed by the produced by the audited entity wider audit profession are particularly pertinent for this paper. • The relevance and reliability of externally produced data • The nature and relevance of information derived from data First, many firms expressed “privately, at least, a strong analytics for risk assessments sense that some aspects of regulation are stifling innovation 84. Financial Reporting Council, “Audit Quality Thematic Review: The Use of Data Analytics in the Audit of Financial Statements,” January 2017, 7. 85. IAASB Data Analytics Working Group, “Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics” (exposure drafts and consultation paper, September 2016), 8. 86. IAASB Data Analytics Working Group, “Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics” (exposure drafts and consultation paper, September 2016), 10. 87. IAASB Data Analytics Working Group, “Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics” (exposure drafts and consultation paper, September 2016), 11–14. 88. IAASB Tech Talk, May 2019. 89. IAASB Data Analytics Working Group, “Exploring the Growing Use of Technology in the Audit, with a Focus on Data Analytics” (exposure drafts and consultation paper, September 2016), 15. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 56 [with regard to the use of audit data analytics]. The regulatory Second, there was a poor response from the SAI community approach to inspection is causing some firms not to develop and no public, formal response as such from INTOSAI. The new applications simply because they do not think regulators Office of the Auditor General of New Zealand was the only SAI will like it or will find it difficult to slot into the existing ISA mind- to comment on the IAASB’s initial assessment. It noted that set.” There was also a specific concern relevant to smaller it was experiencing some of the challenges identified by the audit firms. This concern related to the cost of developing IAASB including specifically confirming the completeness of and implementing audit procedures involving the use of data data sets and the use of data generated or owned by a third analytics. For firms in this category, “if the market demands party. It suggested a gap in the IAASB’s coverage. This related the use of data analytics within external audit, but auditing to the nonfinancial information that accompanies financial standards do not admit that data analytics has value, and no information in the financial statements. It commented “analysis consideration is given to what firms do not now need to do as of both financial and other datasets offers the potential of a result of these enhanced capabilities, the economics of the providing analytical support to the judgment an auditor is called provision of smaller audits will be further skewed in favor of upon to make in reaching an audit opinion . . . our view is that large firms.”90 data analysis within an audit will increasingly encompass other entity datasets.”91 90. Institute of Chartered Accountants in England and Wales, “ICAEW Representation 22/17 – Exploring the Growing Use of Technology in the Audit with a Focus on Data Analytics,” 3. 91. Letter from Office of the Auditor-General New Zealand (dated February 3, 2017), addressed to chair of the IAASB Data Analytics Working Group, reference AU/APS/2- 0002. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 57 >>> Appendix F: Reference Websites 92 www.accaglobal.com www.icaew.com https://www.intosaicommunity.net/ www.audit.gov.cn/en/ www.nao.org.uk www.intosai.org www.eca.europa.eu www.iaasb.org www.ifac.org www.oag-bvg.gc.ca www.intosaijournal.org www.wgea.org www.frc.org.uk www.cpsc.gov www.afrosai-e.org.za www.agsa.co.za www.iia.org.uk www.barclaysimpson.com www.oagnep.gov.np www.sayistay.gov.tr www.gao.gov www.concorindia.com www.pergamos.lib.uoa.gr www.regulations.justia.com www.ggi.com https://www.asosaijournal.org/ www.icas.com https://enb.iisd.org/ www.ifiar.org www.mdpi.com www.anao.gov.au www.oecd.org 92. Websites visited during the research part of the report and during the development of the appendixes. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 58