JOBS NOTES Issue No. 17 PILOTING A MACHINE LEARNING-BASED JOB-MATCHING ALGORITHM: SUMMARY OF RESULTS FROM POMERANIA BACKGROUND 1 The World Bank developed a task-based job- Labor Office (LO) portals. 6 The similarity score was matching tool to assess viable job pathways for used as a metric to approximate the ease for work- workers in the mining sector affected by the coal ers to perform other occupations based on their task transition. It had developed an algorithm based content: on the one hand, the higher the similarity on machine learning (ML) to assess the similarity of scores, the easier the transition of workers from one tasks across occupations as part of the job diagnostic occupation to the other; and on the other hand, the analysis for the coal regions in Poland (Wielkopolska, lower the similarity score, the costlier in terms of Silesia, and Lower Silesia).2 The text mining algorithm acquiring necessary experience and the skills to per- generates similarity scores for each pair of occupa- form the new task. To account for other job attri- tions 3 based on their task content, as described in the butes, results were filtered to consider occupations by national taxonomy of occupations (at six-digit level) their local labor demand based on the occupational published by the Polish Ministry of Family and Social Wielkopolska barometer 7 and by the average wage Policy 4 and consistent with the ISCO-08.5 For about level (based on Labor Force Survey [LFS] 2018) with 5 percent of occupations (about 100 occupations), respect to average wages in occupations in the mining the text mining analysis combined the task descrip- sectors. For Silesia and Lower Silesia, high-potential tion from the national (ISCO) classification with infor- professions were identified based on a desk review of mation from job postings available in online national published regional labor market analysis. 1 Prepared by Maddalena Honorati (Senior Economist, World Bank), Céline Ferré, and Tomasz Gajderowicz (Consultants, World Bank). Draft for comments. 2 https://www.worldbank.org/en/country/poland/publication/support-for-polish-coal-regions-in-transition. 3 Latent Semantic Indexing (LSI) and similarity score between 0 and 1 were constructed for each pair of occupations yielding a 2.700 by 2.700 similarity score matrix. A similarity score close to zero indicates that the tasks performed for the two occupations are different, while a similarity score close to one indicates similar tasks. LSI is an indexing and retrieval method that uses a mathematical technique called singular value decomposition (SVD) to identify patterns in the relationships between the terms and concepts contained in an unstructured collection of text. It is based on the principle that words that are used in the same contexts tend to have similar meanings. A key feature of LSI is its ability to extract the conceptual content of a body of texts by establishing associations between those terms that occur in similar contexts. Applying the LSI model with 100 topics (number established using trial and error approach), the team was able to describe each job/occupation in our corpus as a mixture of topics. 4 https://psz.praca.gov.pl/rynek-pracy/bazy-danych/klasyfikacja-zawodow-i-specjalnosci/wyszukiwarka-opisow-zawodow. 5 ISCO — International Standard Classification of Occupations. 6 https://oferty.praca.gov.pl/, accessed December 15, 2022. 7 Occupational barometers are produced by powiat labor offices (PUP — Powiatowy Urza ˛d Pracy) and published on an annual basis by the respective RLO. Occupational barometers provide information on occupations in demand, balanced, or in deficit on a five-point scale at the powiat level. Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 The technical work done for the Polish coal and efficiency of service delivery from both clients and regions raised interest from the Gdan ´sk LO to advisers; and Section 5 concludes. tailor and operationalize the tool for its career advisers. The collaboration with the Gdan ´sk Regional Labor Office (RLO) was shaped by an agreed road map 2. ADAPTATION OF THE JOB-MATCHING of activities and timeline to adapt and pilot the tool TOOL FOR THE POMERANIA PILOT in Pomerania to improve the quality and efficiency of the career advisory services of the RLO considering The job-matching tool was adapted to fit the the data and capacity requirements. As part of the needs of career advisers and the data require- Supporting Poland Just Transition technical assistance, ments of the RLO. Specifically, the algorithm was cus- the World Bank and the Gdan ´sk RLO partnered to tomized to (a) generate similarity scores based on the adapt the tool and develop, implement, and evalu- combination of up to three job positions/occupations ate a three-month pilot in two RLOs in Pomerania (at the six digit level)9 rather than just one; (b) map voivodeship, where Gdan ´sk is located. the barometer data (at four digits) to similarity scores (defined at six digits) based on a systematic publicly Further developments of the job-matching tool available cross-walk; (c) present similarity scores for have been proposed and discussed with the jobs in ‘surplus’, ‘deficit’, and ‘balanced’ demand with Gdan ´sk RLO to improve the efficiency and quality respect to applications at the powiat levels for each of service delivery of both powiat and regional selected region and at the national level based on the LOs. Looking forward, there is shared interest to fur- most recent 2022 data from local occupational barom- ther develop the tool to assess and identify skills needs eters; and (d) update the wage information based on for selected occupations and to match them with the 2020 LFS and 2020 Wage Structure Survey. jobseekers’ skills and preferences (possibly in part- nership with other existing tools such as SkillsCraft The user-friendliness of the interactive dash- and SkillLab8), as well as to tailor the tool to specific board was improved. The World Bank team hired a vulnerable groups of jobseekers (workers affected by web designer to improve the dashboard’s visuals and the energy transition, Ukrainian refugees, and other to include an initial filter for users to select the region socially vulnerable jobseekers). As these further devel- of interest, so that results of similar occupations opments require additional resources and commit- could be presented for each powiat in the selected ments, it was agreed to pilot the simple version of regions, tailored to mobility preferences. 10 Users the tool with minor adaptations (as discussed in the could also select the entire country, and results will following section) and plan for the next phase build- be presented for the whole country. Importantly, the ing on the pilot results. A rigorous evaluation of the dashboard only presents occupations that are active impact of the job-matching tool on the efficiency and in the local labor markets, and job descriptions are effectiveness of public employment service (PES) deliv- detailed enough to allow the algorithm to generate ery and on jobseekers’ outcomes (probability of finding meaningful similarity scores. Out of the 2,700 occupa- a job, job search length, wage, satisfaction with job tions in the national classification, around 1,300 were found, and so on) was also discussed and planned. included in the dashboard. The remainder of the report is structured as follows: The underlying data on local labor demand and Section 2 details the adaptations made to the job- wages presented along with the similarity scores matching algorithm to match the capacity and needs of were updated and systematically mapped to the the LOs; Section 3 describes the pilot objective, design, national occupation classification. The regional and implementation arrangements; Section 4 presents barometers provide qualitative information on local the findings from the pilot in terms of the satisfaction labor demand for each occupation at a four-digit level 8 More info at https://skilllab.io/en-us and skillcraft.ml. 9 Based on simple aggregation, each of the three positions was given equal weight in the similarity score. 10 The tool outputs are publicly available at https://zmiana-pracy.herokuapp.com/. 2 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 of disaggregation. Since occupations in the regional added as the extreme cases (1 and 5) were never used barometer are labeled with position titles that are dif- in the barometer. Career advisers requested to add ferent from the ones used in the national classification another filter to flag formal occupational certification of occupations (consistent with ISCO-08), a cross-walk requirements for similar occupations. This solution matrix publicly available on the ministry website11 was could not be implemented as the underlying database used to map labor demand information based on on formal certification requirements by occupation the 2022 barometers defined at the four-digit level does not exist. The revised tool that was piloted is to the similarity scores defined at the six-digit level. publicly available (screenshot displayed in Figure 1). Average wages by occupation at the six-digit level were estimated based on the October 2020 Wage In sum, the revised tool has the potential to Structure by Occupations Survey12 published by the assist career advisers in the provision of orien- national Statistical Office in 2022. For other occupa- tation services, though the scope of the assess- tions that were not available in the Wage Structure ment and match remain simple. The revised tool by Occupations Survey, average wages were estimated produces a list of occupations matched with jobseek- at the four-digit level based on the 2020 LFS,13 which ers’ work experience based on the similarity of tasks relies on the ISCO-08 classification. performed in up to three previous jobs, filtered by geographical areas and average pay of interest. The Other customizations to the tool were discussed tool, in its current design, does not allow to assess but not implemented due to the lack of data. It jobseekers’ skills (cognitive and noncognitive) nor was envisaged to add a functionality to present sim- does it match them with the skills required by employ- ilarity scores by demand trends based on a five-point ers. The tool is also not designed to read and match scale rather than on a three-point scale (surplus, defi- education and training certificates or diplomas with cit, and balanced). However, the functionality was not occupational requirements. Figure 1 A screenshot of the interactive career advisory tool Source: https://zmiana-pracy.herokuapp.com/#wyniki. 11 https://psz.praca.gov.pl/rynek-pracy/bazy-danych/klasyfikacja-zawodow-i-specjalnosci/wyszukiwarka-opisow-zawodow. 12 https://stat.gov.pl/download/gfx/portalinformacyjny/pl/defaultaktualnosci/5474/4/10/1/tablice_struktura_wynagrodzen_wedlug_ zawodow_w_pazdzierniku_2020_r.xlsx. 13 Wage estimates are based on year 2020 to gain more granularity (from 2021 onward, LFS only collects data for occupations defined at the three-digit level). 3 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 3. PILOT DESIGN AND IMPLEMENTATION the consultation. Testing the performance of the tool compared to traditional career advice on employment The pilot design was adapted to the mandate outcomes was initially contemplated as another and responsibilities of career advisers in the objective of the pilot, though deemed too ambitious RLOs. In Poland, RLOs are responsible for managing given the responsibilities of the RLO advisers (who and supervising the budget allocations from national provide career advice, not actual intermediation and European Union (EU) funds, coordinating PUPs services) and the scope of the pilot. for the entire voivodeship and training their staff, pro- ducing regular publications on labor market trends in In terms of process, some principles were defined the region and analytics (such as the job barometers), to select the clients participating in the pilot and maintaining the registry of vocational training and to guide the consultation. Career advisers providers in the region partnering with PUPs.14 The expressed interest in retaining the discretion to RLOs are also responsible for on-demand counseling choose when and for which client to apply the tool. and advisory services. Unlike powiat-level LOs that It was agreed that the general guiding principles are mandated to improve the employability of regis- to select when to apply the tool would be the tered jobseekers and their matching with employers, following: (a) the reason the client approached the the RLOs provide career advisory services, professional RLO and (b) whether the client had work experience. orientation, guidance on how to prepare a CV, and Advisers would apply the tool when the reason of clarifications on labor regulations. The RLO clients are the counseling appointment was to seek professional generally not served on a walk-in basis but based on orientation and career advice rather than seeking appointments previously scheduled. support to prepare CVs and for job interviews. The tool would not be applied for clients with no The objective of the pilot was to assess whether professional experience. Advisers would ask clients even simple ML-based tools contribute to improve for up to three occupations for which they either had the efficiency of PES delivery and job-seeking professional experience or think they are skilled for behaviors compared to rule-based, knowledge- or passionate about. No minimum time frame was driven approaches. The potential (and risks) of defined for the duration of the consultation. data-driven approaches based on ML, and even more broadly on artificial intelligence (AI), is unlocked and Monitoring and evaluation (M&E) activities were under-evaluated. The opportunities and challenges planned ahead to allow regular data collection are being addressed in the EU’s draft AI Act expected from both career advisers and RLOs clients. The to be approved in 2024. Within the PES domain, the World Bank developed two short questionnaires to question is whether employment services (that is, collect feedback from the RLO career advisers and referrals, orientation, advice, and job offers) delivered clients. The questionnaire for advisers included ques- by job counselors based on ML-based algorithms tions on the technical use of the matching tool, per- have higher performance (in terms of employment ceived changes in the job offers suggested by the outcomes and delivery process) than rule-based and matching tool, and perceived efficiency in improving knowledge-driven approaches. The main objective jobseekers’ counseling and outcomes. The ques- of the pilot was to assess the impact of the revised tionnaire for jobseekers included questions on their job-matching algorithm on improving the job search demographic characteristics, employment and unem- behaviors and upskilling efforts among jobseekers ployment history, and education and feedback on and the efficiency of the orientation and career advice the job offers suggested by the matching tool (see services by the RLO advisers in terms of the number Annex 1 for the questionnaires). The World Bank also of suitable job pathways provided, adequate and developed a longer questionnaire for a follow-up sur- accepted skills training pathways, and the duration of vey to assess jobs outcomes among clients about one 14 Honorati and Banaszczyk, World Bank Jobs Notes #16. 4 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 or two months after the first consultation. Finally, a 4. FINDINGS FROM THE JOBSEEKERS’ focus group discussion with the advisers participating AND COUNSELORS’ FEEDBACK SURVEY in the pilot was designed and implemented by the World Bank. This section describes the results of the two surveys administered to (a) the RLO career advisers partic- M&E protocols were established and imple- ipating in the pilot and to (b) the jobseekers they mented by the RLO advisers. The advisers were selected to assess the usefulness of the World Bank asked to fill in a survey to express their opinions of job-matching algorithm in terms of efficiency of service the job-matching tool, from both their own and cli- delivery and job-seeking behavior based on jobseekers’ ents’ perspectives. Both questionnaires were filled expectations, evaluations, and intentions regarding the immediately after the interview by the advisers and proposed jobs and upskilling. It has to be noted that jobseekers. 15 The questionnaires were completed career advisers filled the survey based on the feedback electronically through an online site developed by the of their overall interaction with the client, not just World Bank team, which allowed real-time data anal- the part related to the discussion of the outcomes of ysis. In total, 66 surveys were completed by 11 advis- the tool. ers, and 64 surveys were completed by jobseekers participating in the pilot. The focus group discussion 4.1 Clients’ Satisfaction with the took place on April 29, 2023, to identify issues and Job‑Matching Recommendations potential improvements. Pilot participants were skewed toward prime- The ML-based job-matching tool was piloted by age, more educated, and experienced females. the RLO career advisers for about three months. In total, 66 clients were offered career advice based The pilot ran between March 1 and June 15, 2023, in on the outcomes of the tool during the pilot while the Gdan ´sk and Słupsk RLOs. In total, 11 career advis- 64 completed the client satisfaction survey: 75 per- ´sk and ers participated in the pilot: 6 advisers in Gdan cent of the pilot participants were female, 63 percent 5 in Słupsk. The decision on whether to use the tool were 36–50-year-olds, 46 percent had a tertiary edu- to formulate the advice was left entirely up to the cation degree (bachelor’s, master’s, or PhD degree), career advisers who applied the tool with 66 clients and an additional 28 percent graduated from tech- (about 30 percent of the total RLO clients)16 accord- nical university (postsecondary non-tertiary educa- ing to the pilot guidelines discussed above. The RLO tion) (Figure 2). Compared to the distribution in the career advisers were trained by the World Bank team Pomerania working-age people by education (based in February 2023 to use the tool and read the out- on regionally representative LFS 2022), pilot partici- comes and were asked to fill the monitoring surveys. pants are mostly higher educated. The advisers entered the information and discussed the outcomes of the tools with the clients asking Half of the pilot participants were employed about their interest in proposed occupations one by when they sought advice from the RLO. Half were one, looking at the job descriptions in the tool. employed at the time that they visited the RLO, out 15 Some advisers filled the survey on behalf of the client, while others allowed the client to fill the survey from their computer. 16 The career advisers reported that the number of appointments they schedule varies between 10 and 30 per week, on average. About one in five clients is Ukrainian. 5 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 Figure 2 Distribution of pilot jobseekers by the highest level of completed education 50 46% 45 39% 40 35 30 28% Percentage 24% 25 22% 20 15 11% 11% 11% 10 4% 5% 5 0 Lower secondary Primary Upper-secondary Post-secondary or Tertiary or below vocational general upper-secondary vocational Pomerania working age population (LFS) Pilot jobseekers Source: Authors’ own analysis of the survey results; LFS 2022 is the data source for the average statistics related to the Pomerania working-age population. of which 53 percent had accumulated over 10 years upskilling and elevating professional qualifications by of work experience, suggesting that a considerable 11 percent, and advice to changing profession by just portion of individuals seeking career advice want to 5 percent (Figure 3). Contrary to the pilot protocol, transition to another job, possibly due to career aspi- support in the job-seeking process was the main rations, job dissatisfaction, or other personal reasons. reason to reach out to the RLO career advisers among Only 9 percent of the pilot participants are registered jobseekers selected as pilot participants by the career with the PUPs as either unemployed (6 percent) or advisers. The RLO career advisers reported that the jobseeker (3 percent). Among those registered with main reason clients participating in the pilot requested the PUPs, almost one-third are long-term unemployed the counseling meeting was to get support in creation (had registered over 12 months before), while the of applications, preparation for job interviews, motiva- majority had been enrolled for up to 6 months. In tion in preparing for a job interview, and preparation addition, most respondents looked for a job near their of their individual action plan. place of living: 87 percent limited their job search to their municipality or powiat. See Annex 2 for a com- The vast majority of pilot jobseekers declared plete description of the jobseekers’ profile. that the service provided by the career advisers met their expectation and provided them with In fact, only half of the pilot participants job options they had not considered before. approached career advisers to seek support to A majority, comprising 52 percent of the pilot clients, find a job, change job, and upgrade their qual- stated that their expectations were somewhat met ification. Specifically, selecting the location of the while an additional one-third reported being highly job and analyzing the job offers received was indi- satisfied with the suggested occupations (Figure 4). cated as the most prevalent issue by 26 percent of This indicates a significant level of satisfaction among the pilot participants. 17 Support to determine own the customers, as only 7 percent of the respondents professional skills (skills assessment) was indicated as expressed discontent. Part of the satisfaction was the main challenge by 14 percent of the participants, because the tool output gave them new ideas on 17 The statistics refer to the subsample of clients selected by the advisers to participate in the pilot, hence those to which they provided orientation services based on the ML-based tool. 6 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 Figure 3 Main professional challenge faced by pilot participants according to career advisers In need of support in the process of job 45% seeking (help with creation of application… Change of profession 5% Elevating professional qualifications / skills 11% Determining their own professional skills 14% (self-discovery) Finding employment (choice of job location, 26% position, analysis of job offers) 0 10 20 30 40 50 Percentage Source: Authors’ own analysis of the survey results. Figure 4 Alignment of job proposals based on the tool with jobseekers’ expectations Not at all 3% Not really 5% Difficult to say 8% Somewhat 52% Definitely 33% 0 10 20 30 40 50 60 Percentage Source: Authors’ own analysis of the survey results. potential job options. In fact, most clients admitted occupations matched by the ML-based tool. When that the proposed occupations went beyond previously asked at the end of the advisory meeting, most clients considered options. About 70 percent were at least (57 percent) admitted that they planned to follow the somewhat surprised by the proposed jobs. However, advice received by looking at and analyzing vacancies 24 percent did not receive any propositions which they related to the proposed occupation; one-third of them had not previously considered. The remaining 6 per- were not sure and 11 percent said that they would not cent found it difficult to say. look for a job in the proposed fields proposed by the adviser based on the ML-based tool. It has to be noted Male and lower-educated jobseekers were more though that most jobseekers requested advice to pre- open to follow up and expand their job search to pare CVs and job applications, rather than support for 7 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 actual job search. In any case, males were more open 4.2 Career Advisers’ Satisfaction with the and keener than women to expand their job search Job-Matching Tool to the occupations proposed by the adviser based on the tool.18 Those with high school and lower education The RLO career advisers are neutral to slightly attainment were also more likely to follow up than job- positive about the job-matching tool, which they seekers with postsecondary vocational education and recognized to be more useful for job counselors training (VET) and tertiary education. As noted in the in the PUPs. Advisers found the job suggestions to focus group discussion, this could be explained by the be at least somewhat useful in almost 40 percent fact that the proposed occupations matched to high- of the interactions and neutral in 36 percent of the er-educated jobseekers are more likely to require some cases, stating that they found the job suggestions additional specialization: for example, a pharmacist neither useful nor useless (Figure 6). For 26 percent who is offered to become a doctor (based on the tool of their interactions with clients, the advisers had outcomes) needs to spend years on specialization not reservations about the effectiveness or relevance of just a short-term professional skills training. To the con- the algorithm-generated job recommendations, with trary, low-skilled jobseekers are given a range of pos- 18 percent saying the job suggestions were not very sibilities that more often require short-term technical useful and 8 percent stating that they were not at all upskilling and re-skilling. useful. It was noted that the tool would work better for people looking for specific jobs rather than those Few jobseekers expressed readiness and willing- seeking general advice and support to prepare job ness to retrain. When advised to consider retrain- applications, who represented almost half of the pilot ing, only 27 percent of the respondents confirmed participants. In that sense, the tool is more appropri- their intention to train in the recommended direction. ate for job counselors in the PUPs who are mandated Others are not yet sure (43 percent) or have already by law to provide intermediation and job-matching decided not to retrain (30 percent). Lower-educated services, among others, while the RLO staff provide jobseekers were more likely to enroll in skills training general advice and career guidance. (Figure 5). Figure 5 Clients interested in enrolling in training courses suggested by the RLO career advisers 100 90 80 70 Percentage 60 50 40 30 20 10 0 Secondary and below Post-secondary VET Higher education Yes No / not sure Source: Authors’ own analysis of the survey results. 18 About 70 percent of males said they would follow up and expand their job search to the suggested occupations as opposed to 53 percent of women. 8 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 Figure 6 Usefulness of the job suggestions generated by the algorithm to advisers Not at all useful 8% Not very useful 18% Neutral 35% Somewhat useful 38% Very useful 2% 0 5 10 15 20 25 30 35 40 Percentage Source: Authors’ own analysis of the survey results. Similar to what the pilot jobseekers report, the perspective in almost half of their applications. For advisers’ satisfaction with the tool is mostly 41 percent of the applications, advisers did not have related to the proposed job matches that they a strong opinion and selected ‘neutral’ on the useful- would not have thought of. The job-matching ness of the list of proposed professions generated by tool performs well in suggesting occupations that are the tool. ‘original’ in the sense that they are not closely linked to the initial experience of the jobseeker based on a However, there was consensus on the limitation standard rule-based approach. In 35 percent of the of the tool with respect to matching the educa- responses, advisers declared that the tool was very tion profile, skills, and preferences of jobseek- successful in suggesting job positions they would not ers. By design, the tool only proposes occupations have considered, while in 50 percent of the responses, that match work experience based on task similarity the advisers were neutral. The tool was judged as without considering information on the jobseekers’ not very and not at all successful in providing unique education, skills, and preferences. In practice, in some suggestions by 12 percent and 3 percent of the cases, the occupations suggested by the tools were responses, respectively. inadequate matches in the sense that the correspond- ing job offers require higher education than those of According to most advisers, the correspondence the clients. In case of young jobseekers without work between the client’s professional experience and experience, advisers entered ‘aspirational’ occupa- the professions suggested by the tool was log- tions and proposed similar occupations according to ical and understandable. The similarity score indi- the tool. cated how well a job aligns with an individual’s work experience. In approximately 76 percent of the surveys, Advisers were rather satisfied with the local advisers expressed that the similarity score scale at barometer information on labor demand for least somewhat made sense to them. In the remaining each suggested occupation, though it was noted 12 percent of the responses, they indicated a neutral that it is important to have other sources of stance, and in 11 percent of the cases, the similarity labor demand, including skills demand would score did not make sense. Lastly, the counselors found be needed, and that the official taxonomy of the similarity score to be completely illogical in only occupations is becoming outdated. The metric 2 percent of the applications. According to the coun- on whether the suggested matched occupation is in selors, the list of proposed professions resulting from demand, balance, and surplus was considered useful the tool was somewhat useful also from their clients’ by the advisers in 12 percent of the tool applications 9 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 and at least somewhat useful in 27 percent of the clients were positively surprised by the results; the cases. Additionally, its impact was neutral according interaction was quick, and they did not have to go to advisers in 26 percent of the cases and not really through more questions to get advice on potential job useful in 35 percent of applications. The Advisers matches. Some noted that the use of the ML-based recognized the importance of other sources of labor tool helped them transition to a smoother discussion, demand than the barometer: just the suggestion of and the element of surprise in the outcomes improved the jobs in demand is not enough for some jobseek- the atmosphere. ers; they want to see real job vacancies. In addition, during the focus group discussion, it was noted that The information about salary for each suggested the ministry’s official taxonomy of occupations in occupation was considered to be of little use. Poland (used as the main data source to generate sim- Salary information was outdated and hence judged ilarity scores) does not keep up with the times and by the advisers as not useful in almost half of the that some jobs are not included in the classification, applications of the tool (47 percent). In one-third for example, the occupations more exposed to AI. of the applications, salary information did not really matter, while in 20 percent of the cases, the coun- The use of the tool had no or positive effect on selors judged the information about the salary to be the duration of counseling interactions with cli- somewhat useful. Only in 5 percent of the tool appli- ents. The majority, or 56 percent, responded with a cations, salary information was definitely useful from neutral perspective, indicating that they did not per- the clients’ perspective. ceive a significant change in the time spent with each client (Figure 7). In contrast, 39 percent of the advis- Finally, there is scope to improve the user-friend- ers viewed the impact as ‘rather positive’, suggesting liness of the online dashboard. Some advisers com- that the tool had a beneficial effect on streamlining plained that not all browsers work with the tool and and enhancing the efficiency of their sessions. A that the Excel reports are too long and could be short- small percentage (2 percent) considered the impact ened by including just the relevant information. as ‘very positive.’ According to the advisers, most Figure 7 Judgment of the impact of the algorithm on the time spent counseling a jobseeker Very positive 2% Rather positive 39% Neutral 56% Negative 3% 0 10 20 30 40 50 60 Percentage Source: Authors’ own analysis of the survey results. 10 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 5. SUMMARY AND CONCLUSIONS The Pomerania pilot was very small in scale and not assessed through rigorous impact evalua- The objective of the pilot was to assess whether tion techniques; however, the feedback data career advise based on an online ML algorithm collected pointed to some promising positive would contribute to improve the efficiency impact on supporting both advisers and jobseek- of PES delivery and beneficiaries’ job-seek- ers to expand the menu of possible job matches. ing behaviors compared to rule-based, knowl- The revised ML-based tool piloted in the two RLOs in edge-driven approaches. In many PESs, the Pomerania had the objective to support the career provision of career advice and job search assistance advice function of the RLO career advisers and to is usually done by trained counselors who meet the improve the jobs-seeking behaviors of their clients, jobseekers on a regular basis. In the Organisation for presenting them with proposals of professions that Economic Co-operation and Development (OECD) best suit clients’ competencies and professional expe- countries, it is common to have systems supporting rience. No follow-up data among jobseekers were col- jobseeker profiling. However, the low capacity in lected as planned to monitor job search activities after terms of financial resources and limited trained per- the consultations as well as on job search success and sonnel often mean that such advice can only be lim- jobs outcomes in the short and medium terms. ited in scope. Our online ML-based tool was meant to be a cheap way to support the career advisers in the The jobseekers participating in the pilot were RLOs to expand the advice on job search. overall satisfied with the proposed occupations resulting from the tool and willing to expand While the evidence on the impacts of ML algo- job search efforts and possibly participate in the rithms on employment service delivery and training courses needed for the suggested occu- employment outcomes is scant, existing evidence pations. For most jobseekers, the proposed solutions shows mixed results. An experimental evaluation of went beyond their current thinking and the career an online platform helping jobseekers in Edinburgh, options they had considered before. Slightly more UK, expand their search through a ‘field-in-the-lab’ than half of the jobseekers selected by the advisers approach did contribute to expand the set of jobs that to participate in the pilot reached out to the RLO as the jobseekers would consider and increased their they were seeking job search assistance; the other half number of job interviews after 12 weeks. 19 Similar were seeking support to prepare the documentation to to the tool piloted in Pomerania, the online platform apply to existing vacancies. Those who were seeking tested in Edinburgh suggests occupations and avail- suitable vacancies reported following up, searching, able jobs based on an ML algorithm drawing on the and applying to the proposed occupations suggested representative labor market statistics. By contrast, in by the tool. Although only a few pilot jobseekers France, the online job search assistance platform Bob would consider retraining in the direction of the pro- Emploi had no significant impacts on the time spent fessions proposed by the adviser, 59 percent are inter- looking for a job, search scope (occupational or geo- ested in the specific training courses suggested by the graphical), or any employment outcome, whether in adviser to be better prepared for job offers. However, the short or medium run based on randomized control we cannot conclude that the tool contributed to trial (RCT).20 The platform provides jobseekers with tips increase in job search efforts and willingness to upskill to improve their search and recommendations regard- in the short and medium terms due to lack of data. ing new occupations and locations to target, based on their personal characteristics and preferences, as well According to the career advisers, the tool helped as labor market data provided by the French PES. them expand the menu of potential job pathways 19 Belot, M., P. Kircher, and P. Muller. 2019. “Providing Advice to Jobseekers at Low Cost: An Experimental Study on Online Advice.” Review of Economic Studies 86 (4): 1411–1447. https://academic.oup.com/restud/article/86/4/1411/5115940. 20 Ben Dhia, A., B. Crépon, E. Mbih, L. Paul-Delvaux, B. Picard, and V. Pons. 2022. “Can a Website Bring Unemployment Down? Experimental Evidence from France.” https://drive.google.com/file/d/13dUTDC85KsV622eOwGl725YnrdVjlHxH/view. 11 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 and training referrals options; however, the lack user-friendliness and design of the online dashboard. of information on jobseekers’ education, skills, These improvements require a substantial expansion and preferences limited the efficiency of the of the online platform so that additional input data proposed job matches. Almost 40 percent of the (jobseekers’ skills and preferences, updated average advisers declared that the tool is somewhat useful wages, and employers’ skills needs, among others) for them in determining professional proposals and can be either generated in the expanded platform or matching them to the work experience of clients. linked with existing information systems. Every third client also admitted that the tool indi- cated professions that they would not have thought Going forward, it is recommended that the of themselves. The tool either had no major impact tool is further expanded, used by the PUPs at on the time of the advisory service or shortened it. scale, and rigorously evaluated in the context The disadvantage of the tool, pointed out by the of tight labor markets and the greening econ- counselors in the focus group discussions, was the omy as in Pomerania. As a low-cost tool to assist fact that the matched occupations are not based on intermediation and job-matching services, the utility the jobseekers’ skills and preferences, resulting in of the tool should be assessed with the PUPs that are inadequate job offers. It was noted that the informa- mandated by law to provide job-matching services tion on salary is important and should be up-to-date. to registered jobseekers. As an immediate next step, In addition, the tool should be up-to-date when it it is recommended to train the PUP job counselors comes to salaries and ongoing changes in the labor in Pomerania to apply the tool in their services and market demand. agree on a clear implementation protocol. In the medium term, the tool could be integrated with the The tool worked better for people looking for management information system of the PUPs that specific suitable jobs rather than those seeking profile registered jobseekers in terms of their educa- general advice and support to prepare job appli- tion, work experience, and other sociodemographic cations. The RLO advisers admitted that the tool characteristics. The expansion of the PUP operational worked better for clients coming in to seek job search tools and information technology (IT) system to per- assistance rather than general advice and support to form skill assessment consistently with the European prepare job applications, though the RLO’s job is to Skills, Competencies and Occupations (ESCO) classi- advise people on several issues. In this sense, the job fication as it is being piloted in few PUPs in Poland counselors in the PUPs would make the best use of on a voluntary basis would support the upgrade of it. It would be important to train the PUP staff in the the job-matching tool from purely matching ‘tasks’ use of the tool, as well as implement and evaluate an to also matching ‘skills’ as per the ESCO skills taxon- expanded version of the tool as a next step. omy. Finally, as the evidence on the impacts of the ML-based operational tools for PESs is limited to a To sum up, the tool may have a positive impact couple of European countries, it is recommended to on the time and quality of career advisory ser- expand the knowledge base by designing a rigorous vices at the RLO, but it requires several improve- impact evaluation in a country and region where the ments in terms of (a) linking and revising the ML unemployment rate is among the lowest in Europe algorithm to individual-level information on education, and in a region—Pomerania—that is investing in pro- skills assessments, and preferences of jobseekers; (b) moting job creation in sectors that are key for the linking to up-to-date information on average salaries greening of the economy such as energy, informa- and labor demand (not just on barometers but also tion and communication technology (ICT), transport, on real-time data from online job portals) so that cli- logistics, health, and tourism. An experimental impact ents always receive up-to-date information; (c) updat- evaluation would quantify the potential impacts on ing the official taxonomy of occupations to include, job search behaviors and jobs outcomes of ‘tradi- for example, newer occupations more exposed to AI; tional’, knowledge-based ways to provide job-match- (d) linking it to the database of available professional ing services by the PUP job counselors versus training providers (BUR – Baza Usług Rozwojowych) ML-based approaches. in the region once completed; and (e) improving the 12 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 ANNEX 1: FEEDBACK QUESTIONNAIRES QUESTIONNAIRE FOR JOBSEEKERS We would like to get your feedback on the use of the new job-matching algorithm developed in collaboration with the World Bank. Age <25 25–34 35–44 45–54 >54 Gender Male Female Prefer not to answer Highest level of education completed Basic / primary Lower secondary Upper-secondary vocational Upper-secondary general Postsecondary non-tertiary Tertiary Initial labor market situation when visiting the PES Employed Unemployed (not working, Inactive (not working, receiving benefits) not receiving benefits) Did you have work experience? Yes, 1 position only Yes, several different No positions How long have you been registered with PUP? Up to 6 months From 6 to 12 months Over 12 months What is your total work experience? Up to 1 year From 1 year to 5 years From 5 to 10 years (incomplete) (incomplete) From 10 to 20 years 20 years and more (incomplete) Are you looking for a job? If so, for how many months? Yes, ____ months No In how many positions do you have professional experience? In one position In several different positions In what area were you looking for a job? In the town/powiat of In the voivodeship of In other voivodeships residence residence 13 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 Were the directions of development/potential occupations proposed by the RLO adviser in line with your previous expectations? Definitely Somewhat Difficult to say Not really Not at all Did the proposals for development directions/potential occupations proposed by the RLO adviser go beyond the options previously considered by you? Definitely Somewhat Difficult to say Not really Not at all Do you plan to follow and look for job offers in the professions proposed by the RLO adviser? Yes No I am not sure yet Are you thinking about retraining in the direction of the professions proposed by the RLO adviser? Yes No I am not sure yet Are you interested in taking part in the courses/trainings suggested by the RLO advisers in order to be better prepared for job offers in the professions suggested by the adviser? Yes No Difficult to say What did you like about the counseling process, and what needs to be changed/improved? Liked Needs to be changed/improved QUESTIONNAIRE FOR COUNSELORS We would like to get your feedback on the use of the new job-matching algorithm developed in collaboration with the World Bank. Overall Impression of the Interface Attractiveness: What was your overall impression of using the new interface? Very unsatisfied Unsatisfied Neutral Satisfied Very satisfied Efficiency: How easy was it to use the new interface? Not very easy Not easy Neutral Easy Very easy Dependability: How much did you feel you were in control of the new interface? Not very much Not much Neutral Much Very much 14 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 Novelty: How much do you think the new interface is innovative/creative (regarding suggestions on job placement)? Not very much Not much Neutral Much Very much Improvements Compared to Previous Interface Potential positions suggestions: Was the list of suggested positions useful for you? Not very much Not much Neutral Much Very much Viability/creativity of suggestions: Did the list suggest positions you would not have thought of? Not very much Not much Neutral Much Very much How / why? Viability of similarity score: Did the similarity score between jobseekers’ experience and suggested positions make sense? Not at all Not much Neutral Somewhat Very much How / why? Occupational barometer: Was the information on labor demand (position in demand/balance/ surplus) useful? Not at all Not much Neutral Somewhat Very much How / why? Wage information: Was the information on salary useful? Not at all Not much Neutral Somewhat Very much How / why? Interaction with jobseeker: Did the dashboard help you counsel jobseekers better than in the past? Not at all Not much Neutral Somewhat Very much How / why? Overall: What did you like about the interface, and what needs to be changed/improved? Liked Needs to be changed/improved 15 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 ANNEX 2: PILOT JOBSEEKERS’ PROFILE The pilot participants were mainly women and prime-age workers. A significant majority of clients (63 percent) who participated in the RLO advisory program were above the age of 36 (Figure A2.1, left). Clients under the age of 25 accounted for only 12 percent of the participants. The age distribution of the pilot par- ticipants mirrors the age distribution of the working-age population in Pomerania based on LFS. There was a notable gender disparity among the clients, with women making up 75 percent of the total pilot participants (Figure A2.1, right). Figure A2.1 Distribution of pilot jobseekers by age (left) and gender (right) Panel A Panel B 40 35 30 25% 25 Percentage 20 15 75% 10 5 0 <25 26–35 36–50 >50 Age, years Man Pomerania working age population Survey jobseekers Woman Source: Authors’ own analysis of the survey results. Respondents varied in terms of education; however, most of them had a tertiary education. Specifically, 46 percent of the participants reported completing tertiary education, which includes bachelor’s, master’s, or doctoral degrees. An additional 28 percent indicated they had graduated from postsecondary general education or upper-secondary vocational schools. Moreover, the survey highlights that 11 percent of the respondents completed their education at the upper-secondary general level, equivalent to high school education. Similarly, 11 percent finished their schooling at the primary vocational level, indicating specialized vocational training or certification. Only 5 percent of the respondents did not pursue formal education beyond lower secondary school. Most clients seeking advice from the RLO were experienced workers. More specifically, 53 percent of the clients surveyed had accumulated over 10 years of work experience (Figure A2.2, left). Additionally, a notable 35 percent of the clients reported having more than 20 years of work experience, highlighting a high level of professional expertise. A significant majority of clients (86 percent) have worked in various positions within their chosen profession (Figure A2.2, right). 16 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 Figure A2.2 Clients’ overall work experience (left) and number of positions they worked on (right) Panel A Panel B 20 years or more 35% 14% Between 10 and 20 years 18% Between 5 and 10 years 17% 86% Between 1 and 5 years 20% Up to 1 year 9% 0 5 10 15 20 25 30 35 40 On one position Percentage On several different positions Source: Authors’ own analysis of the survey results. About half of the clients were employed when they sought advice from the LO This suggests that a considerable portion of individuals seeking guidance were already in the workforce and sought additional support, possibly due to career transitions, job dissatisfaction, or other personal reasons. In contrast, approximately 40 percent of the clients were classified as inactive, indicating they were unemployed and not registered with the PUP. The rest were registered with the PUP as unemployed (6 percent) or job-seek- ing (3 percent). Among those registered with the PUP, 71 percent had been enrolled for up to 6 months, while the remaining 29 percent had registered before over 12 months (Figure A2.3, right). Figure A2.3 Clients by their labor market situation when visiting the RLO (left) and periods of being registered with the PUP (right) Panel A Panel B 29% 40% 51% 71% 3% 6% Employed Registered with PUP as unemployed For up to 6 months Registered with Inactive (unemployed and PUP as job-seeking not registered with PUP) For longer than 12 months Source: Authors’ own analysis of the survey results. 17 Piloting a Machine Learning-Based Job-Matching Algorithm: Summary of Results from Pomerania JULY 2023 Additionally, 63 percent of the clients reported looking for a job while visiting the LO (Figure A2.4, left). Most of them reported searching for employment for 3 months. On average, the duration of job-seeking among the respondents was found to be approximately 3.71 months. Most respondents looked for a job near their place of living. About 87 percent of the respondents limited their job search to the town or powiat of residence (Figure A2.4, right). In contrast, 17 percent of the respondents considered their entire voivodeship a job search area. Others (6 percent) considered also searching in other voivodeships. Figure A2.4 Share of clients who were looking for a job at the time of the survey (left) and the areas in which they reported searching (right) Panel A Panel B 100 90 80 70 37% Percentage 60 63% 50 40 30 20 10 0 Yes In the In the In other town / powiat voivodeship voivodeships No of residence of residence Source: Authors’ own analysis of the survey results. 18 The policy note was prepared by Maddalena Honorati (Senior Economist, World Bank), Céline Ferré, and Tomasz Gajderowicz (Consultants, World Bank). Draft for comments. All Jobs Group’s publications are available for free and can be accessed through the World Bank or the Jobs and Development Partnership website. Please send all queries or feedback to Jobs Group. Join the conversation on Twitter: @WBG_Jobs #Jobs4Dev.