June 2023 Centre for Financial Reporting Reform (CFRR), Governance Global Practice, Europe and Central Asia Region, The World Bank Address: Praterstrasse 31, 1020 Vienna, Austria Web: www.worldbank.org/cfrr Email: cfrr@worldbank.org Phone: +43-1-217-0700 The Public Sector Accounting and Reporting Program Web: www.pulsarprogram.org © 2023 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Direc- tors, or the governments they represent. The World Bank does not guarantee the accuracy, completeness, or currency of the data included in this work and does not assume responsibility for any errors, omissions, or discrepancies in the information, or liability with respect to the use of or failure to use the information, methods, processes, or conclusions set forth. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be construed or considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Cover design: Kora Reichardt TOOL USER MANUAL 1 Table of Contents TABLE OF CONTENTS 1. INTRODUCTION  4 1.1. PULSE Methodology  4 1.2. User Roles and Responsibilities  4 1.3. PULSE Check  5 1.4. Lifecycle of an Assessment  5 1.5. Documents of an Assessment  8 1.6. Document Templates  8 2. LOGIN AND SECURITY  9 2.1. Login  9 2.2. Forgot Password  9 2.3. Change Password  11 3. ASSESSMENT MANAGEMENT  12 3.1. Create an Assessment (Add Command)  13 3.2. Edit an Assessment (Edit Command)  14 3.3. Set Deadlines for Steps in the Lifecycle (Configuration Command)  14 3.4. Managing Users and Roles  15 3.4.1. Management in the Matrix Display  15 3.4.2. Add Existing Users to an Assessment  16 3.4.3. Add New Users to the Assessment  16 3.5. Move an Assessment to the Recycle Bin (Remove Command)  18 3.6. Dashboard of an Assessment  19 3.6.1. Methodology  20 3.6.2. Users  20 Table of Contents 2 3.6.3. QA Steps  20 3.6.4. Document Repository  21 3.6.5. Document Share  22 3.6.6. AuditLog  23 4. ASSESSMENT – CONCEPT NOTE PHASE  24 4.1. Prepare Concept Note  24 4.1.1. Download Concept Note Template  25 4.1.2. Upload Concept Note  26 4.1.3. Manage Reviews from Follow-up Window  27 4.2. Review Concept Note  28 4.2.1. Peer Reviewer Feedback  29 4.2.2. No Observations or Comments  29 4.2.3. Notification of Newly Reviewed Document  29 4.2.4. Submit Review  30 4.3. Revise Concept Note  30 4.4. Approve Concept Note  31 4.4.1. Review All Processes for Approval  32 4.4.2. Return of the Concept Note  33 4.4.3. Additional Revision of the Concept Note  33 4.4.4. Approval of the Concept Note  34 5. ASSESSMENT – SCORING PHASE  35 5.1. Assign Users to the Assessment  35 5.2. Start the Actual Assessment Phase  35 5.3. Follow up the Scoring Dimensions Phase  36 5.4. Score Dimensions  37 5.5. Automatic Score Updates  40 3 Table of Contents 6. ASSESSMENT – VALIDATING SCORES  41 6.1. Submit Scores for Validation  41 6.2. Manage the Validation Process  42 6.3. Validate Scores  43 6.4. Submit Validation  45 6.5. Multiple Reviews of the Same Dimension  45 7. ASSESSMENT – REVISING SCORES  46 7.1. Revise Dimensions  47 7.2. Enable Pulse Check  48 8. ASSESSMENT – REPORTING PHASE  50 8.1. Prepare Draft Report  50 8.2. Review the Draft Final Report  51 8.3. PULSE Report - Revision  52 8.4. Approve Final Report  53 8.5. PULSE Check Cycle – Revised Report  54 8.6. Review the Final Report for Compliance with PULSE Check  55 1. Introduction 4 1 INTRODUCTION The PULSE web-based tool facilitates the management The methodology will be updated on a regular basis; of a PULSE assessment. Since the assessment process however, only one version of the methodology can be may take up to six months and involves many experts associated with a given assessment. The PULSE Tool to perform tasks (like scoring and quality assurance), allows the PULSE administrators to manage different the PULSE Tool enables the process to be managed versions of the methodology. Definitions can be found easily in a logical, methodical way. It guides the in the PULSE Handbook. Assessment Team Leader and other experts through the assessment lifecycle such that no important step can be overlooked. The PULSE Tool is available 24x7 and also contains a communications module that keeps all the experts informed of progress. 1.2. User Roles and Responsibilities The PULSE Handbook contains all the details and definitions of the assessment process. This manual To initiate the assessment process, the Lead Agency or assumes that users are at least familiar with the Institution assigns an Assessment Team Leader (ATL) Handbook and the assessment process. In the to lead the assessment. The ATL provides oversight introductory paragraphs that follow, important of the entire assessment process and manages the concepts that pertain to use of the PULSE Tool will be PULSE Tool. reiterated for the convenience of users. After assignment of the ATL, to establish the participation of relevant experts in the assessment process, the Assessment Team Leader (ATL) must 1.1. PULSE Methodology assign subject matter experts to work on the assessment. Each expert is given a defined role which indicates what sort of tasks s/he is to perform. The PULSE methodology consists of 29 indicators Possible roles and responsibilities of the experts are and 102 dimensions, classified in six pillars. The shown in Table 1. methodology also contains rules for scoring each dimension. 5 1. Introduction Table 1. Responsibilities in the PULSE Tool Role Responsibilities in the PULSE tool Assessment Team Provide oversight of the entire assessment process and manage the PULSE Tool. Leader (ATL) Create the assessment record in the PULSE Tool. Assign roles to other participants who will be working on the assessment in various capacities. Perform the change of statuses in the assessment to follow the assessment lifecycle. Assessment Team Assist the ATL in carrying out the Quality Assurance (QA) process, including revision of assessment’s raw data, draft and final PULSE reports, and preparation of matrix of comments and responses, based on the feedback received from Peer Reviewers. Peer Review Team Review draft PULSE Concept Note (CN). External Participate in the QA process, including review and validation of assessment’s raw Validation Team data, and review of feedback on the first draft of the complete PULSE report. Oversight Team Review and approve the Concept Note and the final PULSE report. Pulsar Team Provide feedback on the Concept Note. Conduct the PULSE Check. 1.3. PULSE Check 1.4. Lifecycle of an Assessment An important aspect of the Quality Assurance (QA) process is the PULSE Check. The PULSE Check is a QA The lifecycle of an assessment is the list of process touchpoint performed by the PULSAR Team which steps from the beginning to end. An assessment has ensures that the PULSE methodology has been applied up to 12 steps (the exact number depends on whether correctly, scoring is properly evidenced, and the report there is a PULSE Check). The lifecycle is described in structure follows the PULSE Handbook, among other detail in the Handbook (page 5), but is summarized in things. Although the PULSE Check may be optional in Table 2 for convenience. some circumstances, it is very strongly recommended. The PULSE Tool accommodates the PULSE Check – it Figure 1 provides a process flow diagram showing all is part of the lifecycle of an assessment. Please see the steps in the lifecycle in logical order. The start and the dark green rows in Table 2 of the next section. end of the process are marked with double circles. Further details are also included in the Handbook (page 9). At each step in the lifecycle, one or more of the experts perform certain work. Table 2 gives a description of each step and the work that is performed in that step. Rows in light blue color pertain to the Concept Note; rows in light green color pertain to the performance of the actual assessment. Rows in dark green color pertain to the PULSE Check. 1. Introduction 6 Table 2. Lifecycle Steps of an Assessment # Step / Status Description 1 Create The ATL sets up the administration of the assessment, assigns experts to work on the various steps, and establishes the timeframe. 2 Prepare The system enables the ATL to upload a draft of the Concept Note. The system Concept Note allows this status only if relevant experts were assigned in the previous step. 3 Review In this status, the reviewers can review the Draft Concept Note and submit Concept Note their feedback. The system allows this status only if the assessment has a Draft Concept Note and Peer Reviewers have been assigned. 4 Revise The ATL can upload a final version of the Concept Note. The system allows this Concept Note status if the assessment has reviews of the Concept Note and these reviews were submitted. 5 Approve The Designated Approver can review all the documentation and approve or reject Concept Note the CN. This status is possible only if there is a final version of the Concept Note in the system. 6 Assessment In this step, Assessment Team members score the dimensions. This status is only possible after approval of the Concept Note in the prior step, and after assigning all the Assessment Team members. 7 Validate In this step, the External Validators verify the reliability and completeness of all Assessment assessment data, and suggest new scores and add comments. 8 Revise In this step, the system allows the Assessment Team members and to rescore any Assessment dimension, provide additional information, and revise the original assessment data as needed. The system allows this status if all of the dimensions were validated and reviewed at least once. 9 Prepare The system allows upload of a draft of the report. This status is only possible after Report all assessment data was revised in the prior step. 10 Review Report In this step, the reviewers can review the draft report and submit their feedback. The system allows this status only if the assessment has a draft report. 11 Revise Report In this step, the ATL can upload a revised version of the report. The system allows this status if the draft report was reviewed by at least one External Validator. 11a Review Pulse This step is only enabled when the assessment has been marked as PULSE Check. Check The system will automatically assign the report previously submitted as revised as the new Draft for this review stage. Only the PULSAR Team may provide feedback during the PULSE Check. 11b Revise Pulse This step is only enabled when the assessment has been marked as PULSE Check. Check The system allows this state only when the PULSAT team has provided comments, and the draft PULSE report is reviewed by the ATL and Assessment Team. 12 Approve Once the ATL has uploaded a revised or PULSE Check version of the final report, Report the system allows the report to be approved. The Designated Approver can login and approve or reject the document. This step is the last step in the assessment lifecycle. After approval, the assessment is completed, and no further actions are necessary. 7 1. Introduction Figure 1. Lifecycle Process Flow CREATE ASSESSMENT Users defined Reject Prepare Review Revise Approve Concept Concept Note Concept Concept Concept Concept Note Draft Reviews Concept Note Note Note Note Note Final Dimensions Review Scores Revise Scores Prepare Assessment scored Assessment reviewed Assessment final Report Report Draft Review Report Revise Submitted Review Pulse Check Revise Report Reviews Report to Pulse Pulse Check Reviews Pulse Check Report Pulse Final Final Reject APPROVE Reject REPORT 1. Introduction 8 1.5. Documents of an 1.6. Document Templates Assessment The CN and the PULSE Report are detailed documents that must follow already established guidelines. These Many types of documents can be “attached” to an guidelines are already programmed in the system assessment. There is a system storage folder called in the form of template documents. Although the the Document Repository into which all forms of Handbook describes exactly what these guidelines documents may be placed. These documents may be are for each document, the PULSE Tool allows these uploaded as support evidence for the assessment. guidelines to be changed. Established templates can Later, during the scoring process, these documents be downloaded for an assessment. Instructions for may be used to support the dimension scores. Every downloading are included later in this manual. document in this repository may be assigned to a category to make it easier to organize and locate. For further details on the Document Repository and the storage categories please see section 3.6.4 of this manual. Types of documents that may typically be stored in the Document Repository include: (a) Supporting Evidence (these documents may be attached or linked to specific dimensions); (b) Concept Notes; and (c) Reports. 9 2. Login and Security 2 LOGIN AND SECURITY 2.1. Login 2.2. Forgot Password To perform tasks in the PULSE Tool, users must login If the user forgets his/her password it can be to the system using a Username and a Password recovered directly by clicking on the option Forgot your (Figure 2). The Username is the registered email password? as depicted in Figure 3. address. The user will receive an email with instructions as shown in Figure 4. When the user clicks on the link in the email the system will display a page requesting the user to enter and confirm a new password (Figure 5). Figure 2. Login Screen 2. Login and Security 10 Figure 3. Reset Password Dialog Figure 4. Notification to Reset Password Figure 5. Reset a Password 11 2. Login and Security 2.3. Change Password To change the password, select the Change Password option from the menu on the left side of the display (Figure 6). Figure 6. Change Password Dialog 3. Assessment Management 12 3 ASSESSMENT MANAGEMENT The assessment list (Figure 7) displays the current À CN - Concept Note assessments to which a user/role has access. À % NA - Percentage of dimensions scored as Not Any user / role may view the list of assessments Available window. However, the system will only display the assessments to which the user has been assigned by À % D* - Percentage of dimensions scored with D* the Assessment Team Leader. In some cases, only one À % Comp - Percentage of scored dimensions assessment will be displayed. For the ATL role, clicking on the button ( ) in The columns on the right give an overview of the the left-most column of the assessment list opens status of the assessment and its progress towards a contextual menu (Figure 8). The contextual menu completion. The column abbreviations on the right provides several options. These options are described stand for: in the next several pages. Figure 7. List of Assessments 13 3. Assessment Management Figure 8. Contextual Menu Figure 10. Create Assessment Form 3.1. Create an Assessment (Add Command) The system automatically assigns the current methodology version and the correct dimensions for The first step in the assessment process is the creation the methodology. Typically, this function will only be of a system record of the assessment. For the ATL role, performed once. the link to create an assessment + Add Assessment is The ATL should have the following information at hand displayed on top of the list of assessments. to create an assessment: Figure 9. Add Assessment Link À The institution at which the assessment will be performed. À The period of time of the assessment. À A description of the assessment. The ATL selects the institution that will lead the assessment. The system automatically populates the Click on +Add Assessment and the system opens a country and region fields of the institution. Please form (Figure 10) to create a new assessment. note that the country and region fields are “grayed out,” meaning that they do not accept data entry. 3. Assessment Management 14 3.2. Edit an Assessment (Edit Command) Click on Edit from the contextual menu to revise the Figure 11. Edit Details of the Assessment general information of the assessment, if needed. The same dialog as in the creation process appears (Figure 11). Make the appropriate revisions and then click the Save button to save the changes. Please note that the methodology was assigned automatically earlier and cannot be changed in this form. 3.3. Set Deadlines for Steps in the Lifecycle (Configuration Command) This feature enables notifications to users about deadlines for different steps in an assessment cycle (Figure 12). The system takes the number of days specified and sends an appropriate message to users when the days specified have elapsed. Enter values in whole days (integers) on each line. Do not enter other values, text, or punctuation. The total number of days for all steps should not exceed 180 (approximately 6 months). The system counts the total number of days and displays a message in the lower right corner of the form. Figure 12. Configure Deadlines of the Assessment 15 3. Assessment Management 3.4. Managing Users and the expert’s role. It is also possible to create the user account directly here via +Create User if the user does Roles not yet exist in the system. Instructions on the form remind the ATL about the 3.4.1. Management in the Matrix applicable rules for defining users/roles. There are Display also two important rules for users/roles that are always relevant: After the ATL has created the assessment and À Each role must have at least one expert assigned configured the reminders, the system asks the ATL to it. to assign the experts who will work on it in various À An expert can have more than one role, e.g., a Peer capacities (roles). The display (Figure 13) enables Reviewer and an External Validator could be the the ATL to manage the user setup process. This form same person. allows the ATL to assign existing and new users, and to view the assignments of users and roles in a matrix format. When the users/roles have been selected and all rules have been complied with, the system enables the To select the experts and their roles, use the first (left) green button Finalize Defining Users. Click this button dropdown list options to pick the name of an expert to save the selections. and then the second (middle) dropdown list to choose Figure 13. Assign Users and Define Roles 3. Assessment Management 16 3.4.2. Add Existing Users to an 3.4.3. Add New Users to the Assessment Assessment Use the first (left) dropdown list options to select Click on the Create User link to define the name the name of an expert and then the second (middle) and email address of a new user. A dialog opens dropdown list to select the expert’s role (Figure 14). (Figure 15). Enter appropriate data in all the fields and Click Save users and roles. then click Save users and roles. All fields are required. Figure 14. User and Role Dropdown Menus Figure 15. Add New User Dialog 17 3. Assessment Management The system will automatically send an email to the Once the new user sets a password, the system will new user asking that the email address be verified, redirect to the login page. The system automatically and the user will be required to set a password checks whether there is at least one assessment linked (Figure 16). When clicking the link, the user will be able to this user. If there is more than one assessment the to set the password and will be confirmed as system’s system redirects to the list of assessments. If there user (Figure 17). is only one assessment the system redirects to the assessment Dashboard as shown in Figure 18. Figure 16. Notification to New User with Link to Password Figure 17. Reset Password Dialog 3. Assessment Management 18 Figure 18. Assessment Dashboard 3.5. Move an Assessment an assessment to the “Recycle Bin” using the Remove command from the contextual menu (Figure 19). to the Recycle Bin (Remove Assessments in the Recycle Bin are not permanent deletions and can be restored later. Command) Once the user confirms the move into the recycle bin Because an assessment record is the fundamental unit the assessment can be viewed immediately in the list of the system, the system will not allow an immediate, of recycled assessments (Figure 20). full deletion of any assessment. The ATL can move Figure 19. Move Assessment to the Recycle Bin 19 3. Assessment Management Figure 20. Assessments in the Recycle Bin It is possible to undo this operation by going to the the Dashboard, click on the magnifying glass button list of recycled assessments and restoring these on the left side of the assessment list. The Dashboard assessments if necessary. for the selected assessment opens. In Figure 21, the assessment is “MFGMFG20.” The Dashboard contains vital information about the status of the CN, assessment, and PULSE report. All 3.6. Dashboard of an users can view the Dashboard. However, the content Assessment and options displayed vary according to the user’s role. The information presented in the Dashboard is an The following several paragraphs refer to the option in overview of the status of the assessment. To reach blue in the middle of the Dashboard. Figure 21. Assessment Dashboard Showing Configure Step 3. Assessment Management 20 3.6.1. Methodology users was discussed earlier in the section, “Managing users and roles.” Please refer to that section for detailed instructions. To view all the details of the methodology for this assessment, click on the Methodology link. A display opens with details of the methodology for that assessment (see Figure 22). Click on any title or 3.6.3. QA Steps phrase to view the description of the relevant pillar, QA stands for quality assurance. This is a small indicator, or dimension. window with information on the progression of the changes in status that have occurred from the beginning. The window indicates the date and user 3.6.2. Users who caused the change in status (Figure 23). This is accessible from the link QA Steps in the middle of the Assessment users and roles can be managed from the Dashboard. Users link on the Dashboard. Management of system Figure 22. Looking Up Methodology Details Figure 23. Status Change Display 21 3. Assessment Management 3.6.4. Document Repository To upload a document simply click on the +Add Item link. The window opens as shown in Figure 25. The experts in charge of the assessment need Select the category from the dropdown list, select a to collect evidence and documentation from the file, type the description, and click Upload. This will institution to support the scores. For that purpose, put the document on the list available later for scoring there is a set of pre-defined categories in the PULSE the dimensions. In the Description, be as specific as Tool to which documents may be assigned. To browse possible – this will be helpful later when reviewing the the documents and the categories, click on the documents. Document Repository link. The display shown in Figure 24 opens. In this example, there are 14 possible In the list of documents click to view the categories categories. that do not already contain uploaded documents. This allows the user to quickly identify the documents that The display will only show the categories for which the have been uploaded. assessment actually has some documents uploaded. Figure 24. Assessment Dashboard Showing Configure Step 3. Assessment Management 22 Figure 25. Upload Document to a Category 3.6.5. Document Share À Final Report À Final Report after Pulse Check The system allows users to distribute specific documents via an email distribution list. After selecting the document to share, the ATL must next create the email distribution list. The text The dropdown list Document to Share (Figure 26) box Emails (Figure 27) allows a list of formatted or will show the following documents only when they unformatted email addresses to be pasted into the are available to be shared. Select the appropriate box. Simply copy and paste email addresses into the document from the dropdown list. box. The system will extract the emails from the text À Draft Concept Note (and ignore other text) and send once to every unique address found in the text. Every identified email À Final Concept Note address in the list will receive the message shown in À Draft Report Figure 28. Figure 26. Selecting a Document to Share Figure 27. Creating the Email Distribution List 23 3. Assessment Management Figure 28. Document Shared Message 3.6.6. AuditLog assignment of users, changes of experts on an assessment, etc. The audit log is only available to the ATL. To view the log, click on Auditlog. A sample audit The audit log is a list of all the events that have taken log display is shown in Figure 29. place in the system, for example, the creation of an assessment, change in status of an assessment, Figure 29. Audit Log 4. Assessment – Concept Note Phase 24 4 ASSESSMENT – CONCEPT NOTE PHASE The next step in the assessment lifecycle is to prepare Please note that the Prepare Concept Note button is a CN. In order to initiate this step, it is assumed that only available to Assessment Team Leaders. an assessment has already been created (see section 3.1. “Create an Assessment”), configured, and that The concepts “upload” and “download” are very experts were assigned to the assessment (see section important to the following processes: 3.4. "Managing Users and Roles"). À Upload = Move a document from the user’s own system (e.g., laptop) to the PULSE Tool À Download = Move a document from the PULSE Tool to the user’s own system 4.1. Prepare Concept Note An upload documents dialog opens (Figure 31). The Once the ATL has defined the experts that will ATL may upload a CN, if one is already available. To do participate in this assessment, s/he can mark the so, select a file and then click Upload. assessment and proceed to the next step, Prepare Concept Note. This button can be found on the right side of the assessment Dashboard (Figure 30). Figure 30. Dashboard with Prepare Concept Note 25 4. Assessment – Concept Note Phase Figure 31. Upload a Concept Note Dialog 4.1.1. Download Concept Note A template is an empty document that provides guidance on the required structure of the CN, Template which helps the author to develop the CN more easily (Figure 32). The user will only have to fill in If there is no CN developed yet, an option allows appropriate text in spaces provided, as explained in the ATL to download a Concept Note template. the instructions in the template. Figure 32. Concept Note Template with Instructions 4. Assessment – Concept Note Phase 26 4.1.2. Upload Concept Note in the assessment lifecycle, Review Concept Note (Figure 34). Alternatively, the ATL can also upload a new version of the CN. Once the author has completed a draft of the CN (in accordance with the instructions / guidelines), s/he Once the ATL is sure that this CN is ready to be can upload it to the system using the Upload button reviewed, it can be sent to the assigned Peer (Figure 33). Reviewers. The ATL will first confirm the action by clicking Proceed to Review (Figure 35). Uploading the first draft of the CN will enable the Assessment Team Leader to move to the next step Figure 33. Upload a Concept Note Figure 34. Review Concept Note with Current Version Figure 35. Confirmation Dialog 27 4. Assessment – Concept Note Phase The system will recognize that Peer Reviewers now Figure 37. Follow-up Actions must complete their tasks, and it will temporarily disable any possible action for the ATL. The system now displays the Dashboard and a new follow-up window appears below the Dashboard (Figure 36). Figure 36. User Assignments and Status 4.1.3. Manage Reviews from Follow-up Window This follow-up window indicates who has been In the follow-up window the ATL can choose to release assigned to review the CN and the status of their a Peer Reviewer from his/her duty, mark the task review. as reviewed without any comments, or review the À The CN has been ASSIGNED to be reviewed. document (Figure 37). À The Peer Reviewer has STARTED reviewing the In Figure 36 “Rem.” (in the first column) stands for document by accessing the details. “remove.” For example, if the ATL chooses to remove À The Peer Reviewer has SUBMITTED the review to a Peer Reviewer from the list, the system will clearly the system. show this event and add it to the Audit Log of the assessment. 4. Assessment – Concept Note Phase 28 4.2. Review Concept Note From the link in the email notification, the reviewer can login to the system and access the assessment. Once in the assessment Dashboard, Peer Reviewers Every Peer Reviewer of the CN receives an email only will see a button to review the CN. When the Peer notification when the CN is ready to be reviewed Reviewer clicks on this button, the review is started (Figure 38). (Figure 39). Figure 38. Start Review Message to Peer Reviewers Figure 39. Peer Reviewer Display with Comments Box 29 4. Assessment – Concept Note Phase 4.2.1. Peer Reviewer Feedback 4.2.3. Notification of Newly Reviewed Document The Peer Reviewer can now download the Draft Concept Note and review it. When the CN has been It should be noted that even if the reviewer has not reviewed, the Peer Reviewer typically uploads the yet submitted the review, the document uploaded revised CN and provides his/her feedback directly is already available to the ATL via the download in the system. However, it is also possible for Peer icon in the follow-up window from the assessment Reviewers to give their written feedback directly in the Dashboard. system without uploading any document. The Peer Reviewer writes comments, if desired, in the Main In the list of reviewers of the Concept Note, the Comments field (Figure 39, lower left). Finally, the ATL already has access to download the reviewed reviewer clicks Save to save the review. document and the comments that were introduced by the reviewer (Figure 40). This is available even before The reviewer can upload a new version of the review the user submits the review. at any time while the CN is still in the Review Concept Note step. Figure 40. Status Display 4.2.2. No Observations or Comments It is also possible to submit a review without any comments, thereby accepting the CN as it is. To submit without comments, leave the Main Comments field blank (Figure 39), and click on the gray button Submit Review with No Observations on the upper right. The practice of submitting a review without comments is discouraged. Once the Peer Reviewer provides his/her feedback and/or uploads the revised CN, the ATL will receive a notification of this event (Figure 41). Figure 41. Notification Message to ATL 4. Assessment – Concept Note Phase 30 4.2.4. Submit Review The system will change the status of the assessment and will allow the ATL only to revise the document by clicking on the Revise Concept Note button (Figure 43). To submit the review, the reviewer clicks on the green button. This button appears on the same page where In the revision display (Figure 44), the ATL can upload the user has uploaded the review (Figure 42). the revised version of the CN and add observations or comments to indicate whether all the comments from the review were included in the document. From this page it is easy to see the reviewed documents and 4.3. Revise Concept Note analyze the main comments made by the reviewers. The user must upload the final version of the CN to be Once all the reviewers have finished the review of the able to submit the document for approval. document (with or without observations) the ATL will revise the CN and upload its final version. The system will change the status of the assessment to Revise Concept Note automatically. Figure 42. Submit Review Button Figure 43. Ready to Revise the Concept Note 31 4. Assessment – Concept Note Phase Figure 44. Revise Concept Note with Documents 4.4. Approve Concept Note Additionally, the Designated Approver from the Oversight Team will receive an email notification about the required action (Figure 47). The Designated Once the revised CN has been uploaded, the system Approver logs in and the system will display the will open the Submit for Approval dialog (Figure 45). interface to start the approval process of the CN. On the Dashboard the Approve Concept Note step is highlighted in orange color (Figure 46). Figure 45. Submit for Approval Button Figure 46. Dashboard Showing Approve Concept Note Step 4. Assessment – Concept Note Phase 32 Figure 47. Notification to the Designated Approver to Approve the Concept Note 4.4.1. Review All Processes for À The initial draft of the CN, along with the initial comments. Approval À The reviews submitted by the Peer Reviewers. In the approval dialog (Figure 48), the user can À The revised version of the CN. download all the documents related to this CN, for À The comments/observations for the revised example: version of the CN. Figure 48. Approve Concept Note 33 4. Assessment – Concept Note Phase 4.4.2. Return of the Concept Once returned, the ATL receives a notification with the request to submit a new revised version of the Note CN (Figure 50). The assessment also goes back to the Revise Concept Note step. The Designated Approver can approve or return the CN. In case of return, the Designated Approver provides comments/explanation about the reasons for the return. Comment/explanation is mandatory. In 4.4.3. Additional Revision of the case of return, the Designated Approver confirms the Concept Note step by clicking Return Concept Note (Figure 49). The ATL can now submit again a new revised version Figure 49. Confirmation Dialog of the CN. The interface will display the observations submitted by the Designated Approver (Figure 51). After the new revised CN is uploaded, the ATL clicks Submit for Approval. Figure 50. Notification of Returned Concept Note for Revision Figure 51. Submit for Approval with Observations 4. Assessment – Concept Note Phase 34 4.4.4. Approval of the Concept Note The Designated Approver can now login to the system and approve the CN (Figure 52). All experts who have been involved in the production of the CN receive a message about the approval (Figure 53). Now the system moves the assessment to the next step in the assessment lifecycle. Figure 52. Confirm Approval of the Concept Note Figure 53. Notification of the Approval of the Concept Note 35 5. Assessment – Scoring Phase 5 ASSESSMENT – SCORING PHASE Once the CN is approved, the ATL can proceed to the next steps in the assessment lifecycle. 5.2. Start the Actual Assessment Phase Once the users have been defined and the CN 5.1. Assign Users to the has been approved, the ATL can start the actual assessment phase. Assessment The phase is started by clicking on the green button To perform the actual assessment, at this stage the Assessment (Figure 55). All experts assigned to system allows the assignment of additional experts the Assessment role will be notified about their (who were not involved earlier with the CN) to the participation. Assessment Team, External Validation Team, and Pulsar Team (Figure 54). The ATL should make these assignments now before moving to the next steps. Figure 54. Assign Users and Roles 5. Assessment – Scoring Phase 36 Figure 55. Initiate the Actual Assessment 5.3. Follow up the Scoring The system also displays a small visualization with the status of the assessment, including the number and Dimensions Phase percent of dimensions scored (Figure 57). The Dashboard presents a follow-up window with the number of dimensions scored by the Assessment Figure 57. Dimensions Scoring Overall Team experts (Figure 56). Figure 56. Number of Dimensions Scored by Expert 37 5. Assessment – Scoring Phase 5.4. Score Dimensions À “NE” means that the dimension is currently Not Evaluated. The scoring of dimensions is a key feature of the À “NA” means a score for this dimension is Not PULSE Tool. Assessment Team members can score the Applicable. dimensions easily from the Dashboard, as depicted in Figure 58. Click on Score Dimensions to initiate the Not Applicable means that in certain cases, an process. indicator or dimension may not be applicable to the reporting entity being assessed. This situation might During the scoring process, a small display appears arise, for example, if certain laws and regulations that provides a snapshot of the assessment process prohibit a reporting entity from being in possession of (Figure 59). certain assets or liabilities. Note that if the assessment team uses the “NA” option, a clear and rigorous In addition, a list of all the dimensions / indicators disclosure is required about the reason(s). For more of the methodology appears in its logical structure detail about the NA option, please see the Handbook, (Figure 60). The displays show the current scoring for page 24. all the dimensions / indicators. Figure 58. Initiate the Scoring Process Figure 59. Scoring Snapshot 5. Assessment – Scoring Phase 38 Figure 60. Structure of the Methodology To score a dimension, click on the corresponding link. methodology has already defined the way a score After clicking, the three parts of a dimension open in should be given. The scoring guidelines are already a new page. The first part (Figure 61) is the dimension loaded in the system and can be viewed next to the header (in this case, “2.1 Accounting Policies"). score. Select the appropriate score by clicking on the value (Figure 62). To view the complete description of the dimension, click on the plus sign at right. The system displays all Underneath the scores is an open box into which the information regarding that dimension. users should record the observations that support the score. The second part of each dimension concerns scoring. For each possible value of a score (A, B, C, etc.) the Figure 61. Description of the Dimension 39 5. Assessment – Scoring Phase Figure 62. Scoring Display with Scoring Guidelines Figure 63. Upload Support Documents In the third part of a dimension, Assessment uploaded under this particular category. Click on the Team members should upload or refer to specific corresponding plus sign button to select the specific documents that support the observations and justify document to be referenced (Figure 64). the score which was given (Figure 63). The system allows the addition of documents from the Document Figure 64. Upload Document Dialog Repository or the upload of documents from a local device or desktop. When selecting a document from the Document Repository, users will see only the categories that have at least one document. When a category is selected, it will display the list of all documents previously 5. Assessment – Scoring Phase 40 It is also possible to upload a completely new etc. This technique will save a lot of time for External document from a local device, e.g., user’s laptop. Validators later during the assessment data validation This document will only be associated with the process. dimension, and will not be placed in the Repository. Select the document in the dialog and click on Upload (Figure 65). 5.5. Automatic Score Figure 65. Upload Document Dialog Updates The higher-level scores (indicator, pillar) that depend on the scoring of a dimension are automatically updated each time the score of a dimension is added or changed. In Figure 67 example the indicator (at left), having only one dimension (at right), gets its After selecting or uploading a document Assessment score calculated. Team members should state the reference inside the document that supports the score given (Figure 66). The automatic update feature calculates up to the Please be as specific as possible. Cite pages pillar level and can be viewed on the Dashboard numbers, paragraph numbers, section numbers, (Figure 68). Figure 66. Uploaded Documents with References Figure 67. Score of a Dimension Figure 68. Scores Displayed on the Dashboard 41 6. Assessment – Validating Scores 6 ASSESSMENT – VALIDATING SCORES 6.1. Submit Scores for Assessment for Validation as seen in Figure 69. This action can also be performed from the Dashboard. Validation The system sends an email notification (Figure 70) to all External Validators to begin the validation process. Once the Assessment Team members have completed The External Validators can start the validation scoring ALL dimensions, the ATL can submit process immediately via the link in the message. the assessment scores for validation via Submit Figure 69. Submitting Scores for Validation 6. Assessment – Validating Scores 42 Figure 70. Notification to External Validators 6.2. Manage the Validation At the bottom of this display, the link List of dimensions that have NOT been reviewed yet opens a Process useful interface (note: not depicted). It is important to note that a dimension can be validated more The ATL can view the status of the External Validators’ than once because each validator can have different work at any point after the assessment phase is observations for the same dimension. By clicking on initiated. The status is viewed via a window in the the name of the validator the system displays the list Dashboard (Figure 71). This window is only available of dimensions and their status (Figure 72). to the ATL. A validator may review all dimensions or just specific ones to which s/he was assigned. But between all Figure 71. View the Status of the Validation validators the dimensions must all be reviewed to proceed to the next step. The External Validators may discuss and agree on who validates which dimensions depending on their professional expertise/ background. Each dimension should be validated by at least one validator, but some dimensions could be validated by more than one validator. 43 6. Assessment – Validating Scores Figure 72. External Validation Overview Display From the same follow-up window, the ATL can end the Figure 74. Reopen the Validation Process Dialog validation task by submitting the review (Figure 73). Figure 73. Confirm Submission of Validation 6.3. Validate Scores If an External Validator has submitted the results of When the External Validator opens the Dashboard the his/her validation and omitted some dimensions that button Validate Scores appears (Figure 75). s/he was expected to perform, the ATL can click on the button to reopen the review process for that specific The list will show all the dimensions and their status in validator (Figure 74). a small label below the dimension name (Figure 76). Figure 75. Dashboard with Validate Scores Button 6. Assessment – Validating Scores 44 Figure 76. Dimension Showing the Measured Label À Measured: The dimension has been scored but By clicking on the name of the dimension, the user not yet validated. will be able to review the dimension scoring. For that, the system will present the same scoring page but À Validated: The dimension has been validated by also open a window on the bottom for the validator the External Validator. to be able to submit the validation comments and/ À Revised: The dimension has been revised by an or suggest new scoring (Figure 77). The validator Assessment Team member. can click on the button to scroll down to the review section. In the example, there is a small green label “Measured” which indicates that the dimension has Once the validation review is finished, the validator not yet been validated. clicks on the Save button. After that, the list of dimensions is automatically updated, and the dimension is labeled with the Validated label (Figure 78). Figure 77. Review Section with Observations Box Figure 78. Dimension Showing the Red Validated Label 45 6. Assessment – Validating Scores 6.4. Submit Validation 6.5. Multiple Reviews of the Same Dimension When the validator has finished reviewing all the dimensions that s/he can review, the validator clicks It is possible for two or more validators to review the on Submit Validation (Figure 79). The system will ask same dimension. Each one of them may have different for confirmation of this action. If for some reason observations and/or comments on the scoring. Every the validator needs to review other dimensions after validator can login to the system and see the status of this confirmation, s/he can ask the ATL to restore his/her review without seeing the status of the other the review status of the assessment for that specific reviewers. validator. Only the ATL can see all the validations. Every validator’s comment and suggested score appears in different tabs in the revision interface. Figure 79. Submit Validation Display 7. Assessment – Revising Scores 46 7 ASSESSMENT – REVISING SCORES Figure 80. Status Display It is only possible to move to the next step if all the dimensions have been reviewed at least once. The ATL can see if there are dimensions that are still to be validated from the status display by clicking on the link at bottom, List of dimensions that have NOT been reviewed yet (Figure 80). A small window opens to display the result (Figure 81). If all dimensions have been validated, the ATL can Figure 81. Results Display proceed to the next step by clicking the green Revise Scores button on the Dashboard (Figure 82). Now the assessment is in the Revise Assessment step. The ATL and Assessment Team members may revise all the observations made to all the dimensions. Figure 82. Dashboard with Revise Scores Button 47 7. Assessment – Revising Scores 7.1. Revise Dimensions This action will display the list of dimensions (Figure 84), similar to the scoring and validation processes. Besides each dimension the Assessment In the assessment Dashboard the ATL can click on Team member and ATL can view the labels Validated / the blue Revise Scores button to start the revision Revised to check the status of each dimension. (Figure 83). Assessment Team members and the ATL may click on each dimension to move to the revisions page. Figure 83. Dashboard with Blue Revise Scores Button Figure 84. List of Dimensions with Status Labels 7. Assessment – Revising Scores 48 The link in the header section Scroll to Review Info (Figure 85) allows the user to scroll down to the review 7.2. Enable Pulse Check section (Figure 86) to view the actual information to be revised. Note that the review section has tabs – After the revision, the system will show the option to each tab represents the name of the reviewer that enable the PULSE Check for an assessment. The PULSE reviewed this particular dimension. Check is very strongly recommended. If there are any changes to the dimension scoring, the The main objective of the PULSE Check consists of information can be changed just like when the user ensuring that: (i) the PULSE methodology has been assessed the dimension the first time. applied correctly; (ii) general scoring is evidenced; (iii) the current overall IPSAS adoption status and financial reporting basis are properly documented; (iv) the report structure and content follow the PULSE Handbook; and (v) the proposed recommendations and action plan are reasonable and feasible for implementation. Figure 85. Revise a Dimension Display Figure 86. Review Section with Tabs for Each Reviewer 49 7. Assessment – Revising Scores Every assessment can comply with the PULSE Check The system will always ask for confirmation of this or not. If the assessment complies with the conditions action (Figure 88). established in the Handbook, the system will enable the PULSE Check from a button on the Dashboard. In Figure 87 the button is visible. This is a toggle button that changes from DisablePulseCheck to EnablePulseCheck and vice versa. Figure 87. Dashboard with PulseCheck Option Figure 88. Confirmation of PulseCheck 8. Assessment – Reporting Phase 50 8 ASSESSMENT – REPORTING PHASE 8.1. Prepare Draft Report The ATL now has the possibility to prepare the draft The ATL can now upload a draft report document in version of the PULSE report. Note that this is only a dedicated page (which has the same interface as possible when all the dimensions have been revised. the preparation and upload of the draft CN). A new The ATL clicks the green Prepare Draft Report button Dashboard opens (Figure 90). to initiate the process (Figure 89). Figure 89. Dashboard with Prepare Draft Report Button Figure 90. Dashboard with Review Report Button 51 8. Assessment – Reporting Phase Similarly to the CN preparation phase, the user can also download a report template to get an idea of the 8.2. Review the Draft Final minimum content requirements of the PULSE report Report (Figure 91). To use this feature, click on Download Report Template. The reviewers are notified by email of the review task (Figure 93). Once the draft report has been uploaded, to continue with the process, the ATL submits the report to External Validators now start the review process, External Validators for review. External Validators are which consists of uploading a reviewed file and/or automatically notified to review the document. The entering the general observations / comments (as was ATL can follow up on this process from the status the case earlier with the CN). The Assessment Team display (Figure 92). members and ATL will receive a notification as soon as the review has been uploaded so that they can start working on the revised version of the report. Figure 91. Download Report Template Link Figure 92. Status Display Figure 93. Notification to External Validators 8. Assessment – Reporting Phase 52 When the External Validators have completed their like in the CN, the user will start the revision process, reviews and uploaded necessary documentation upload a revised report, and submit it to Oversight and comments, the validators now proceed with the Team for approval (Figure 95). submission by clicking the Submit your Review button (Figure 94). This will change the status of the assessment and will inform the Oversight Team which is responsible to approve the final version (Figure 96). Please note that the approval process for assessments 8.3. PULSE Report - Revision that are NOT Pulse Check is being described. Later in this document the process for PULSE Check will be The ATL and Assessment Team members should now described. start with the revision of the report. To do that, just Figure 94. Submit a Review Display Figure 95. Submit for Approval Display Figure 96. Dashboard with Awaiting Approval Message 53 8. Assessment – Reporting Phase 8.4. Approve Final Report After the Oversight Team conducts its review process and is satisfied with the quality of the final PULSE report, the Designated Approver clicks the The Designated Approver views the Dashboard which green Approve Report button. The ATL will receive a now displays a button to start the approval process notification of this approval and can then proceed to (Figure 97). the next step (Figure 98). The system will provide the Designated Approver from The next step is the final step and it is only available the Oversight Team with all the information to be when the PULSE Report has been approved. The ATL able to perform the approval. This includes the draft can now close the assessment when ready (Figure 99). report, all the reviews, and the final version of the PULSE Report. Figure 97. Dashboard with Approve Report Button Figure 98. Notification of Approval 8. Assessment – Reporting Phase 54 Figure 99. Dashboard with Final Closure Button Figure 100. Final Dashboard Summary Once clicked, the assessment is finished, and no more actions can be performed by any user / role 8.5. PULSE Check Cycle – (Figure 100). Revised Report When the PULSE report is being revised, the system starts displaying information about the fact that the process will follow a PULSE Check (Figure 101). 55 8. Assessment – Reporting Phase Figure 101. Submit for PulseCheck Display After the ATL uploads the revised version of the PULSE Additionally, just like the CN and the first review of the report, and prior to the approval phase, the system report, the ATL has a window to follow up the PULSE will ask for confirmation to start the PULSE Check Check review process. All PULSAR team members have (Figure 102). been notified to perform the review. Figure 102. Confirmation of PulseCheck Dialog 8.6. Review the Final Report for Compliance with PULSE Check The status of the assessment will be changed to PULSE To start performing the PULSE Check review the PULSE Check (Figure 103). team member(s) login to the system and start the PULSE Check with the PULSE Check Review button on the Dashboard (Figure 104). Figure 103. Change of Status Figure 104. Initiating the PulseCheck Review 8. Assessment – Reporting Phase 56 The PULSE Team member can submit the review The ATL will be notified of the new reviewed document without any observations but typically s/he will and can start preparing the final version of the report. download the report and start working on the PULSE Once all PULSAR Team members have finished Check (Figure 105). submitting their feedback, the system will display the button to continue with the next step to start the Once this review is uploaded, the system will display revision of the document (Figure 107). the button to submit the review (Figure 106). Figure 105. Submit Review with No Observations Figure 106. Dashboard with Submit Review Button Figure 107. Dashboard with PulseCheck Кevision Button 57 8. Assessment – Reporting Phase This will enable the Revise Report button in the The system will now automatically detect that the Dashboard to start the revision process (Figure 108). assessment is in the final stages of a PULSE Check process and will enable three links besides the report Just like the other documents, the PULSE Check (Figure 110): revision process needs a revised (PULSE Checked) À Draft of the PULSE report version of the document to be uploaded. Once it is uploaded, the user can submit it for approval À Final (revised) version after the first review process (Figure 109). À PULSE Check version after the PULSE Check Figure 108. Dashboard with Revise Report Button Figure 109. Submit for Approval Figure 110. Dashboard with Draft, Final, and PulseCheck Links 8. Assessment – Reporting Phase 58 The approver can now login into the system and start Once it is approved the system will recognize the the approval process (Figure 111). action and will display the corresponding message below the label of the final report (Figure 113). This process is exactly the same as previous documents and the user has all the previous information to analyze it and approve the report (Figure 112). Figure 111. Dashboard with Approve PULSE Check Button Figure 112. Dashboard with Approve Report Button Figure 113. Dashboard with Approved by Message 59 8. Assessment – Reporting Phase