69805 v. 1 Technical Assistance on Poverty Analysis and Social Statistics Mission on the Census of Buildings and Dwellings March 21 – April 19, 2009 Assessment - Draft - Consultant Report - Document I May 5, 2009 Contents Introduction 1. Assess the capacities of CAS in conducting Censuses 2. Needs for staff, equipment and other resources 3. Assess the quality of previous census work 4. Assess the quality of data 5. Assess data archiving and dissemination activities related to the Census 6. Examine possibilities of generating a maximum of relevant socio-demographic data through the census of buildings and dwellings 7. Recommendations for upcoming census based on findings in points 1-5 above. 8. Work Program for the Technical Assistance Phase Document I-Assessment-Draft.doc 2 Introduction1. Poverty monitoring, a priority area for the Government of Lebanon, requires an array of social and demographic statistics unavailable or not readily available in Lebanon. The World Bank is developing a project to develop the capacity of the Central Administration for Statistics (CAS) to produce demographic and social statistics for this and other purposes. The census of buildings and dwellings that CAS is planning for 2010-2011 has been identified as an initial candidate for implementation-technical assistance under the current project. In the absence of population censuses in Lebanon, the census of buildings and dwelling is one of the few sources of comprehensive data and information at the national level. The CAS has requested technical support for the Census of Buildings and Dwelling from the World Bank. This technical assistance will run over the life of the Census, from the initiation of the census to the analysis of data, which would extend over the current and the coming fiscal years. In this context, the World Bank organized a one month mission to explore the prospects for this technical assistance. The first two weeks of the mission, Monday 23, March 2009 to Saturday 4, April 2009 focused on assessment. The second two weeks, Monday 6, April 2009 to Saturday 18, April 2009, focused on technical assistance. This note reports the assessment made during the first half of the mission. 1. Assess the capacities of CAS in conducting Censuses Capacity to conduct censuses involves multiple dimensions including qualified and experienced staff; sufficient numbers of staff at each level; administrative infrastructure to necessary to support field operations; legal basis for the census; and government support of the census in the form of, for example, sufficient budget, creation of a national census committee, and provision for support from appropriate government entities. It is a hard fact of life that a serious lack in any of these areas can imperil a census even if all the other areas are well provided for. CAS capacity for census taking is high only in the first of these categories, qualified and experienced staff. There are more or less serious problems on the remaining dimensions, as the experience of the 2004 census demonstrated. The specific problems derive mainly from systemic issues that cannot be addressed from within CAS. The staff shortage, for example, reflects seriously inadequate rates of remuneration that cannot be changed within the present legal framework. The same framework imposes administrative rigidities that severely impair the organization’s capacity for census taking. The systemic problems have been addressed in detail in the Statistical Master Plan (SMP) prepared several years ago. Absent implementation of the SMP, there is little CAS can do to strengthen its capacity for census taking.  Strong leadership. Director-General is highly motivated, forceful and articulate, has lead the staff successfully through formidable challenges.  Highly educated and motivated staff. CAS is severely understaffed (see below) at the higher grades, which results in unreasonably heavy workloads for many of the staff in 1 The note is based on meetings held by Griffith Feeney with the Staff and the Director General of the Central Administration for Statistics in their offices in Beirut during the period from March 21 to April 4, 2009. Document I-Assessment-Draft.doc 3 the posts that are filled, and salaries are not competitive. Despite this, the core professional and administrative staff is highly motivated.  Core staff experienced in field work. Proficiency in designing and managing field operations for data collection is critical to any national statistical office, but it can only be learned from experience. CAS is fortunate to have numerous staff highly competent and experienced in this area.  Ability to innovate demonstrated by use of PDAs (“Personal Digital Assistant,� handheld digital device in which survey responses are entered by field workers) for data collection in the MICS survey now ongoing (though probably not relevant for census data collection)  Severely understaffed; of 94 authorized posts at the top 3 levels (Civil Service or CAS Levels 1, 2 and 3), only 21 are filled, a deficit of more than 75%  Staff morale is an issue because of shortage of staff, low pay, long delays in promotion, inability to recruit and pay field workers on a timely basis, and perceived lack of government support for the census and its results.  Project Design and Management practices need development to, for example: facilitate organizational and staff development, keep projects on schedule, adapt to changed circumstances, and facilitate negotiations for increased resources (cf. SMP)  In the field of Project Design and Management, more focus is needed on Budgeting and Financial Management practices, especially if CAS is able to increase staff and develop capacity (cf. SMP)  Capacity to analyze data quality needs development (this is true of many statistical offices, see Engracia ESCAP meeting paper noted below)  More attention to metadata issues, e.g., documentation of operations. 2. Needs for staff, equipment and other resources The most severe impairment of CAS capacity to conduct censuses is due to the deficit of mandated staff. Current staffing is below the critical mass required for sustainability. There are few if any areas in which CAS can advance its capacity without additional staff. a- Staff Needs CAS headquarters needs a reasonably full complement of professional, technical and managerial staff stipulated by the statistics law and subsequent administrative documents. As of 02 April 2009, only 21 of 94 authorized posts at CAS and Civil Service levels 3 and above (3, 2) are filled. Note is taken that Article 51 of Decree No. 27289 issued on February 28, 1980, which authorizes CAS to “get the assistance of persons from outside its cadre� for its work. Additional headquarters staff will be required for the census (minimum of 10 permanent staff available for work at least one year prior to the beginning of the enumeration), and of course large numbers of field work (estimate 450 minimum to carry out the enumeration in 8 weeks) and office staff will be needed. Specific needs can be identified only in the context of a project plan for the census as a whole. A useful overview census planning is provided in Document I-Assessment-Draft.doc 4 Chapter I of the United Nations Statistics Divisions Handbook on Census Management for Population and Housing Censuses (available free online in PDF format, and in CAS library). b- Equipment Needs Vehicles are needed for field work. CAS staff has had to use their personal vehicles for field work travel, as CAS has no vehicles. This should not be necessary. Specific needs will be derived from a project plan for census. Data entry equipment will most likely be needed, but it need not necessarily be purchased; renting may be preferred (note importance of scrubbing hard disks to maintain confidentiality if rented machines are used). The magnitude of the need depends in part on how the field work is expected to last. Specifics can be developed only in the context of a project plan for the census as a whole In most census work, data entry is a bottle neck. This was not the case for the 2004 census in Lebanon because of the long delays in field work. This resulted from a change in administrative procedures that required the Prime Minister’s office to approval each field worker recruited by name. Previously, the Director-General of CAS was able to authorize these recruitments. CAS submitted the names promptly, but approval took over one year. c- Other Resource Needs Staging centers for distribution and collection of forms are needed. Regional offices, which do not exist, though they are stipulated in the legal framework, would serve this purpose. The alternative is either to rent space commercially or to agree with local authorities on having spaces dedicated to CAS teams. In the past census, meetings were sometimes held in restaurants. This is clearly inappropriate if only on account of confidentiality concerns. 3. Assess the quality of previous census work The quality of previous census work was poor, as CAS staff very well recognize, on account of insufficient budget, insufficient staff, insufficient high level support from the government, and extreme administrative inflexibilities with respect to government support that made successful field operations next to impossible. Under the conditions prevailing at the time, no other outcome is imaginable. One can of course point to some shortcomings on the part of CAS in the conduct of the census, but the larger point is that it is impossible to address these successfully without first addressing the larger systemic problems CAS faces.  Census operations were funded from general CAS operating funds; there was apparently no central government budget allocation specifically for census operations.  More than one year elapsed between the training of the enumerators for the 2004 census and the beginning of field work, for the reason described above.  High attrition of enumerators (apparently over 50%; exact statistics would require unreasonably time consuming manual tabulation of transaction level records), necessitating further recruitment and training and long delays in execution of field work. This resulted from the initial delay in recruitment, during which many field workers who had been trained dropped out of the work.  Field work, planned for eight weeks, extended over more than 30 months, this based on records of forms entered at headquarters (exact statistics would require manual Document I-Assessment-Draft.doc 5 tabulation of transaction level records). Again, this reflected the initial 13 month delay in deploying field workers.  Supervision of field work was unsatisfactory due to the interposition between field supervisors and CAS central office staff of prefectural level supervisors. These prefectural level supervisors did not perform effectively. This extra tier of supervision was an international advisor recommendation that CAS subsequently regretted following.  The “Controller� system used seems not to have allowed for correction of enumerator lapses while field work was underway; with this system, enumerator errors are corrected if found, but enumerator behavior may in the meantime have generated more errors.  There was no formal, government appointed, national census committee with representation from pertinent government entities to support the census operation, as there generally is for population and housing censuses. There was effective communication and coordination between CAS, the Ministry of the Interior, the Lebanese Army, the General Security Directorate, and the State Security Directorate. 4. Assess the quality of data There are two broad approaches to assessing data quality. One approach is to assess the processes that generated the data, this on the grounds that well designed, well planned and well implemented processes are more likely to generate good quality data than less well planned, designed and implemented processes. The other approach is to analyze the data itself for evidence of quality, whether by tests of internal consistency or by comparison with other sources of data, as for example comparison with the results of a post-enumeration survey.  The field work issues just noted suggest that data quality is likely to be problematic.  No post-enumeration survey (PES) was conducted, so no evidence on data quality is available from this source.  CAS has not attempted to estimate coverage error.  There appears to have been little analysis of data for the purpose of assessing data quality. No reports containing conclusions on data quality appear to have been produced. It should be noted that insufficient attention to analysis of data quality is a problem in many statistical offices (Luisa T. Engracia, “The 2000 Round of Population and Housing Census in the ESCAP Region: Lessons Learnt and Emerging Issues,� UN ESCAP Expert Group Meeting on Population and Housing Censuses, 9- 10 December 2004, Bangkok, Thailand.)  It should be noted also that internal assessments of the quality of dwelling unit data are inherently more limited than internal assessments of the quality of population data. This speaks to the importance of a post-enumeration survey, which will provide direct evidence of data quality.  Bottom line: there is little evidence to attest to data quality, whether coverage error, content error, or errors in the numbers that appear in tabulations. Document I-Assessment-Draft.doc 6 5. Assess data archiving and dissemination activities related to the Census The data archiving and dissemination aspects of capacity are less vulnerable to the staff shortages, administrative inflexibilities, and other problems that CAS experienced. Capacity in these areas is relatively strong, though substantial improvements are possible in the dissemination.  The 2004 data files are available for active processing; tables for internal CAS use may be rapidly produced on demand; this is excellent.  Backup/disaster recovery procedures are in place, including off site backup, but there seems to be no document detailing what specifically these procedures are, where offsite backups are kept, what tests of the system have been made, etc.  Documentation of preparatory phases of the census (project management plan, national mobilization, identifying and documenting user needs, form design and testing, etc., as per the UN Census Management Handbook) is minimal.  Documentation of field operations is minimal. The administrative department has transaction level records, but these do not lend themselves to statistical summary without extensive manual work.  Dissemination activities included publication of prefecture level reports only (no report for the country as a whole and are in Arabic only (1996 census reports, were published in Arabic and French).  The census reports are available on the CAS website.  CAS can and does respond to user requests for special tabulations; transaction level records are available in the administrative office, but there is no regular compilation of requests per year, date received, to who assigned, work required, date delivered, resources required, etc. 6. Examine possibilities of generating a maximum of relevant socio-demographic data through the census of buildings and dwellings The possibilities for generating socio-demographic data by adding questions to the census of buildings and dwellings questionnaire were considered in some detail before the risks of adding such questions became apparent. It was assumed initially that census enumerators obtained the very limited information that they obtained on dwelling units from residents of these units. From the familiar perspective of population and housing censuses, this is a reasonable assumption,, so reasonable that it is difficult to imagine assuming anything else. As discussions with CAS staff progressed, however, it became clear that for dwelling units in multi-story buildings, which constitute a majority of all dwelling units, it cannot be assumed that enumerators interviewed residents. The frequency with which they did so cannot be established statistically because the questionnaire provides no information on whether or not the respondent was a resident of the dwelling in questions. This being the case, adding questions that could only be answered by residents of dwelling units is far too risky to contemplate. Doing so would run the risk of making the enumeration unworkable as well as the risk of generating poor quality data. Document I-Assessment-Draft.doc 7 a- Population Counts  Population counts are theoretically obtainable in a census of dwelling units by asking “number of people living in this dwelling unit.�  There are several potentially severe problems with such questions. One is the question of the target population to be counted in each dwelling unit. A “usual residence� definition may be impractical, and a de facto question (e.g., “slept here last night�) may lead to inconsistencies with other data sources.  A second problem is the ease with which answers may be falsified if there is any incentive to do so. In a population census, household members are listed individually and numerous questions asked of each. Exaggerating the number of members in the household is therefore problematic. No such constraint exists on a “number of persons in this household� (or dwelling unit) question.  A third problem is that the most important risk cannot be effectively pre-tested because it may arise only in the context of the full census enumeration, with attendant publicity and, sometimes, controversy. If CAS field work results in grossly exaggerated population numbers, everyone will recognize this and CAS’s reputation will be impaired.  A fourth problem is the public embarrassment that might occur if such a question is included and results in exaggerated statements of population numbers.  For these reasons, it is recommended that CAS not include such questions in the census of buildings and dwellings. b- Dwelling Unit Questions  Another possibility for generating socio-demographic data are questions pertaining to dwelling units or to households.  Such questions are common in population censuses (which are often also censuses of housing) and population surveys; examples of population and housing census questionnaires for many countries may be found on the United Nations Statistics Division website (unstats.org).  Given that poverty monitoring is a priority area of the Government of Lebanon (as stated in the background section of the TOR), the aim would be to identify questions that would discriminate the economic condition of households, particularly at lower income levels.  Three broad foci may be distinguished: characteristics of the dwelling unit proper (e.g., construction materials), characteristics of the dwelling unit relating to its connection to supporting systems (e.g., electricity), and consumer durables possessed by persons living in the dwelling unit.  Unfortunately, uncertainty over how often census enumerators receive information on dwelling units from residents of these units, as distinct from, for example, a neighbor or building concierge, make it inadvisable to include any such questions in the census of buildings and dwellings. 7. Recommendations for upcoming census based on findings in points 1-5 above. Document I-Assessment-Draft.doc 8 To have a reasonable chance of a successful census, the capacity issues noted in section 1 above must be addressed: insufficient numbers of staff at each level; administrative infrastructure to unable to support field operations; and insufficient government support of the census. It may be emphasized again that a serious lack in any of these areas can imperil the census even if all the other areas are well provided for. As noted above, these capacity issues are systemic. They cannot be addressed from within CAS only. The appropriate way to address the systemic issues is to proceed with implementation of the Statistical Master Plan. Absent this, it will be difficult or impossible to keep the next census from being overtaken by the same issues that overtook the last census. In this connection it should be emphasized how unusual the Lebanon census of buildings and dwellings is. Nearly all countries of the world conduct censuses of population and housing to collect socio-demographic data, both directly, and indirectly by providing a sampling frame for the many household surveys on which modern statistical systems rely. Sampling frames for surveys of business establishments and enterprises are constructed by other means (often, in developing countries, by construction of a business register from records kept by the Ministry of Finance). The census of buildings and dwellings questionnaire is an intricate and ingenious solution to the collection of data, mainly for the purpose of sampling frames for surveys of households and business establishments, under the peculiar legal and institutional arrangements prevailing in the country when CAS was established in 1979, 30 years ago. At the same time, it is an inefficient instrument for these purposes and should be abandoned as soon as new legal and institutional arrangements can be put into place that allow more internationally standard, rational and efficient approaches to data collection. In this context also, then, we come back to the fundamental importance of implementing the Statistical Master Plan as a basis for creating a modern statistical system in Lebanon. a- Staffing and Human Resource Development  Move aggressively to fill authorized but vacant posts in CAS, especially at the higher levels, CAS and Civil Service levels 2 and 3.  Develop orientation program for new staff.  Short term training in modern corporate management for senior CAS staff (cf. SMP). Willie Pietersen’s book Reinventing Strategy provides an approach developed for the corporate sector that has been successfully applied in non-profit sector.  Short term training in basic Project Design and Management for CAS staff at level 3 and above (CAS and civil service levels 1-3), along the lines of Peter Hobbs’s book Project Management (AMACOM, 2000). b- Project Design and Management  Initiate development of a project plan for the census project that details, for example, time, cost and quality objectives; project sponsor, stakeholders, project manager, and project team; tasks, deliverables, time estimates, milestones, dependencies, and critical path; and management process, including periodic reviews and go/no go decision points.  Incorporate into the plan a detailed risk analysis based on the experience of the last census (e.g., risk of being unable to recruit or pay field workers on schedule). Document I-Assessment-Draft.doc 9  Incorporate into the plan measures toward the creation of a National Census Committee to support the next census. Creation requires GoL action, but CAS needs to initiate the process. c- Infrastructure and Organizational Change  Initiate development of an ongoing, systematic program of user consultations aimed at projecting a new awareness CAS’s role as leader and coordinator of the national statistical system, improving understanding of CAS’s role and capabilities, as well as learning about user needs. Prepare a two years plan identifying key user segments, which to start with, what kinds of meetings, preparation, follow up, etc. This work is related to but not purely concerned with the census of buildings and dwellings.  Continue development of metadata documentation systems that facilitate assessment of organizational performance and development of organizational strategy. This would cover, for example, staff responsibility for documentation, filing, and access to documentation, etc. Initiate a project to test use of the World Bank’s Survey Data Dissemination Tool Kit for survey metadata.  Continue development of a CAS intranet to promote and facilitate information sharing. This will help develop capability in web technology, design, standards, and related areas that CAS can use to develop its public website (even if large parts of the work are contracted out).  Draft concept paper for new statistics law.  Draft concept paper for national census committee.  Draft concept paper for national statistical committee. 8. Work Program for the Technical Assistance Phase The ultimate objective of the technical assistance phase of the mission was to develop, in collaboration with CAS staff, and work plan for future technical assistance. As a result of extensive and intensive discussions with CAS staff, two principle areas for technical assistance for the census of buildings and dwellings were identified: questionnaire development, and project design and management. The following paragraphs describe the deliverables to be produced during the second half of the mission. a- Questionnaire development The deliverable would be a document specifying (i) questionnaire content and structure (not format) and (ii) design notes covering, for example, list of information used as input to the development process, identification of pertinent statistical aggregates, risk assessment of each questionnaire item, proposed tabulations, list of options considered and rejected, (iii) scenarios of questionnaires related to the choice of carrying out a large scale population survey concurrently with or shortly following the census, etc … b- Project Design and Management2 2 Reference: UN Statistics Division’s Handbook on Census Management for Population and Housing Censuses (Chapter I, Section B, “Census Planning�). Document I-Assessment-Draft.doc 10 The deliverable will be a draft project plan for census of buildings and dwellings. The draft will identify tasks, deliverables, time scales, dependencies, critical path, milestones, progress reports, review meetings, risk analysis, and go/no go decision points. The plan will identify: (i) preparatory work already completed or underway (e.g., cartography); (ii) key elements on which decisions need to be made, including especially the option of carrying out a large scale population survey concurrently with or shortly following the census, but also, e.g., formation of a national census committee and conducting a post-enumeration survey; next steps for further development of the project plan; and next steps for defining processes for management of the project, (iii) identifying administrative flexibility issues and their impact on the project, (iv) identifying field work and human resource development issues and would sketch a plan for filling authorized but vacant CAS staff positions and for dealing with associated issues (e.g., orientation), (v) issues related to data processing and quality, (vi) a dissemination plan taking account of, for example, publications in print and digital format, the CAS website, and special services requests and, (vii) an archiving plan aimed at securing critical CAS data for the long term future. Document I-Assessment-Draft.doc 11 SWOT Analysis: Census of Buildings and Dwellings Strengths Weaknesses 1. Capacities of CAS in Conducting 1. Capacities of CAS in Conducting Censuses: Censuses: For all the problems, CAS staff did gain valuable Disastrous experience in carrying out 2004 experience while conducting the 2004 census. census left CAS staff with no experience of what it would be like to conduct a well run census. 2. Staff, Equipment and Other Resources: 2. Staff, Equipment and Other Resources: Educated, experienced, motivated staff capable Staff below critical mass required for of doing first rate work in a supportive sustainability. Loss of one or two senior staff environment. could make it impossible for CAS to carry on work. 3. Quality of Previous Census Work: 3. Quality of Previous Census Work: No strengths. Poor, due to impossibly restrictive legal framework and other external circumstances. 4. Quality of Data: 4. Quality of Data: No strengths. Cannot be determined. Pertinent analysis has not been carried out. Based on problems encountered during enumeration, one would expect the quality of the data to be uneven. 5. Data Archiving and Dissemination 5. Data Archiving and Dissemination Activities Activities Data archiving 2004 data readily available for Data archiving No written policy/procedures on special tabulations. data security was available. Dissemination activities System of responding Dissemination activities Little analysis of data, to special requests from users. rudimentary tabulations, no hard copy publication. 6. Generating a maximum of relevant socio- 6. Generating a maximum of relevant socio- demographic data through the census of demographic data through the census of buildings and dwellings: buildings and dwellings: No strengths. Not possible. Likely that respondents for a large proportion of dwelling units were not residents of units; no data available. Document I-Assessment-Draft.doc 12 SWOT Analysis: Census of Buildings and Dwellings (concluded) Opportunities Threats 1. Capacities of CAS in Conducting 1. Capacities of CAS in Conducting Censuses: Censuses: Implementation of the SMP would make it Absent implementation of the SMP, the possible to create a new legal framework and restrictive legal framework and externally remove externally imposed administrative imposed administrative rigidity may lead to rigidities. serious problems in census operations similar to what was experienced in 2004. 2. Staff, Equipment and Other Resources: 2. Staff, Equipment and Other Resources: Implementation of the SMP would make it Absent implementation of the SMP, and critical possible to address critical staff shortages. staff shortage would lead to a situation where Government provision of a sufficient budget and the same issues that plagued the 2004 census other support would address other problems will threaten the success of the next census. encountered in the 2004 census 3. Quality of Previous Census Work: 3. Quality of Previous Census Work: No opportunities. Nothing can be done about As noted above, the quality was poor, but this the quality of previous census work. per se does not pose a significant threat to higher quality work on the next census if the problems that plagued the last census are addressed, as they would be by implementation of the SMP. 4. Quality of Data: 4. Quality of Data: Implementation of the SMP would begin to A repeat of the experience of the 2004 census create conditions that would make it possible to would result in data with uneven quality. collect good quality data. 5. Data Archiving and Dissemination 5. Data Archiving and Dissemination Activities Activities Additional budget and additional staff will make Absent additional budget and staff, it possible to improve performance in both dissemination and outreach activities will areas, especially dissemination activities. continue to be weak. 6. Generating a maximum of relevant socio- 6. Generating a maximum of relevant socio- demographic data through the census of demographic data through the census of buildings and dwellings: buildings and dwellings: A large-scale household survey conducted With the absence of a large-scale household concurrently with the next census of buildings survey conducted concurrently with the census, and dwellings would make it possible to collect there will likely be no change in the availability socio-demographic data of the kind that would of socio-demographic data. be collected by a population census. Document I-Assessment-Draft.doc 13