Abstract

This report provides a mapping of existing research that employs online labour market data, covering both online job vacancies (demand side) and online applicant data (CVs) (supply side). We discuss and assess a variety of tools and empirical methods that have been used to address specific disadvantages of this data, such as non-representativeness or fluctuations in data quantity and structure; these may be due to external shocks, such as the COVID-19 pandemic. We find that while this research field has expanded rapidly, including with respect to geographical coverage, many empirical studies do not engage with the methodological aspects and weaknesses of online labour market data and take them at face value. We highlight that there are legitimate research approaches, which are inductive in nature, focused on discovering patterns and trends in underlying data. These are by definition less concerned with generalizability of findings, as they have different objectives. For this body of research, online labour market data open new avenues for understanding developments in labour markets. We also argue that biases in online labour market data emerge due to multiple factors. With respect to the order of discrepancies between online labour market data and representative data sources, these are typically not paramount. Different techniques have been adopted to deal with the non-representativeness problem, such as statistical techniques; adapting the research questions and research focus to the quality of data; and use of mixed methods, including qualitative methods, to increase the robustness of results.

Introduction

Mismatches between workforce skills and employers’ demands represent a key obstacle to economic growth; they are closely associated with factors such as productivity, unemployment, labour force participation and informality (Acemoglu and Autor 2010; Beblavý, Maselli, and Veselková 2014; 2015; CEDEFOP 2014; Ernesto and Francesco 2016). Skill and task demand is changing due to rapid technological advancement, automation and digitalization – processes that are relevant not only to developed economies, but also to emerging and developing ones (Comyn and Strietska-Ilina 2019). The ability to respond to the changing skills demand is considered key to successful economic transitions that are inevitably sought by economies and individuals throughout the world, and for developmental catch-up between higher- and lower-income countries. The improved understanding of how required skills and work tasks are changing, and the quest to better align skill supply to employer demand, have inspired efforts to create more demand-driven labour market policies. Timely and reliable data are a key prerequisite for these efforts.

The online data on labour markets have in the past years become an important source of information for better understanding how labour markets function. This process has been affected by the spread of the Internet and emergence of online labour market intermediary platforms (e.g. Babajobs in India, Glassdoor in the United States (US), Profesia in Slovakia), online vacancy aggregators (e.g. Burning Glass Technologies), and professional websites and social media (e.g. LinkedIn, Twitter, Facebook). These types of labour market data currently provide a source of timely, granular and often comprehensive data that has been increasingly used by academics and policy makers to analyse labour markets around the world.

The use of such data has been driven throughout the world by the fact that traditional representative surveys might not be available, or do not cover more specific aspects of labour markets in sufficient detail and frequency. The absence of high-frequency, detailed survey data has led researchers to revert to a second-best solution of using online data to study diverse questions. More traditional micro-economic questions (focusing on the behaviour of firms and individuals, skill supply, skill demand, matching, and skill-biased technological change) or macro-economic questions (such as predictions of the unemployment rate, aggregate demand, broader phenomena, and changes at the national, regional or local levels) have been analysed with the use of online data (Boselli, Cesarini, Marrara, et al. 2018). Furthermore, new questions or approaches to a structured understanding of labour market characteristics or changes have also emerged in relation to online data availability (for instance, building skill or task taxonomies, building curricula based on identified demand, a deeper understanding of job changes, etc.).

In general, research using online data to study labour market issues has been organized around five related aspects of research: labour market monitoring and analysis; studying demand for workforce skills; observing job search behaviour and improving skill matching; predictive analysis of skill demand; and experimental studies (Nomura et al. 2017). Due to the granularity of the data, research in these areas has been conducted also at sub-national levels, examining regional or local labour markets (e.g. Azar et al. 2019; 2020; Marinescu and Rathelot 2018).

The use of online labour market data, however, is not without disadvantages. The key concern is the non-representativeness of online data, and the implications this has for various aspects of research and policy making. Other data quality issues relate to data validity, reliability, scalability, generalizability, integrity or privacy, and legal issues (Blazquez and Domenech 2018). This paper situates itself within the debate on the methodological appropriateness of using online labour market data for academic and policy purpose, and provides a systematic review and discussion of: (1) the types and forms of biases present in online labour market data, and ways in which these are understood, discussed and addressed by research; and (2) measures and tools – statistical and other that have been used in past academic and policy research to remedy biases of online labour market data, with a particular focus on two dimensions: representativeness, and fluctuations in online labour market data.

We build on previous studies that have discussed methodological aspects of big data more generally, including implications for the development of new analytical approaches and tools (Blazquez and Domenech 2018; Einav and Levin 2014; Mezzanzanica and Mercorio 2019; Varian 2014). We differ from these studies by providing a narrower focus on online labour market data, such as job vacancies and applicant data. Nevertheless, in particular parts of this paper we refer to broader conceptual issues of different motivations for research: for instance, deductive versus inductive approaches to gathering and analysing information.

Methodology

This analysis builds on an earlier article co-authored by one of the authors (Kureková, Beblavý, and Thum Thysen 2015), which discussed the methodological aspects of using online vacancy data and voluntary Web-based labour market data (i.e. WageIndicator). In this analysis we concentrated on empirical works published after 2015, meaning new empirical papers focusing on or using online labour market data, and a set of methodological and conceptual papers about online labour market data and big data more generally, also prior to 2015. Our approach to a systematic analysis in this paper comprised two related consecutive steps.

In the first step, we conducted a Google Scholar search to identify empirical (applied) studies that have used online labour market data to study the skills demand or supply in the labour market. We focused on the abstracts of the retrieved documents in order to identify papers of interest to us. We paid particular attention to the fast-growing body of studies in developing countries, where representative survey data might be less readily available, and where online data has a specific potential for further expanding the analysis of these labour markets (Table 2). Importantly, the first-step mapping showed that many empirical studies do not engage with the methodological aspects and weaknesses of online labour market data, and take them at face value. The global expansion of this research field, thus, has mainly proceeded through empirical applications, without sufficiently understanding the methodological problems of using such data.

In the next step, we therefore restricted our search by imposing a number of criteria, in order to gather a varied and robust sample of studies which have taken a methodologically engaged approach to applying online data. The criteria for selecting studies for methodological discussion were the following: (1) studies were published in peer-reviewed journals or by recognized international research institutions, such as the Organisation for Economic Co-operation and Development (OECD), the European Centre for the Development of Vocational Training (CEDEFOP), and World Bank; (2) studies were varied geographically, to cover developed and developing countries; and (3) they focused primarily on skills analysis at the micro-level. This search yielded nearly 40 empirical papers, which we systematically reviewed with the aim of understanding how biases were identified and accounted for (see the Annex). In addition, our discussion and analysis is also informed by more conceptual papers, especially those explicitly concerned with and addressing methodological aspects of online data (e.g. Blazquez and Domenech 2018; Einav and Levin 2014; Gandomi and Haider 2015; Kotu and Deshpande 2019; Mezzanzanica and Mercorio 2019; Varian 2014).

We mapped a range of studies using online labour market data, covering both online job vacancies (demand side) and online applicant data (CVs) (supply side). Accordingly, we will discuss and assess a variety of tools and empirical methods that have been used to address specific disadvantages of this data, such as non-representativeness or fluctuations in data quantity and structure, including those caused by external shocks, such as the COVID-19 pandemic. Whereas most research using online labour market data has drawn on sources in advanced economies, we conduct our analysis with an enhanced focus on the specific needs and varied contexts of emerging and developing economies. We are particularly interested in research related to different aspects of skills, such as skill demand, skill supply, matching, changes in skills; and in the relationship between skills and other changes in society and economy, such as automation, digitalization and technological change.

The remainder of the paper is organized as follows. Chapter 2 presents the characteristics and trends in research that relies on labour market data, Chapter 3 discusses the advantages, while Chapter 4 addresses the main disadvantages of online labour market data. Chapter 5 offers an overview of methodological approaches to address non-representativeness and fluctuations in data, and the final chapter concludes.

Online data in labour market research: Trends and characteristics

Efforts to understand labour market matching date back to at least the 1960s. Nonetheless, until relatively recently, the empirical work on job vacancies has been quite sparse. Even today, basic questions remain difficult to answer: for instance, how workers are assigned to jobs; what share of jobs is filled through a formal application process; how many people apply to a typical advertised vacancy; how many applications a typical job seeker submits; and how job applicants decide where to apply. This situation reflects the fact that these aspects of matching are rather time- and context-specific (Kuhn 2014).

One likely reason for the slow progress in the field is that before the Internet started taking on the role of “labour market matchmaker”, obtaining reliable data about job vacancies and applicants was relatively difficult. Regardless of whether the data were obtained through surveys (Abraham 1983; Barron and Bishop 1985; van Ours and Ridder 1992), public employment office databases (van Ours 1989), or by painstakingly collecting individual advertisements published in newspapers (Jackson 2007; Dörfler and van de Werfhorst 2009; Álvarez and Hofstetter 2014), the data suffered from selection bias and other assorted representativeness issues.

Over the course of the 2000s and 2010s, labour market matching shifted increasingly towards the Internet; this has benefited job seekers by improving access to information about vacancies, and allowed employers to benefit from a larger pool of job candidates (Autor 2001; Kuhn and Mansour 2014; Piróg 2016). This shift to an online labour market has also opened new opportunities for research and policy applications. At the same time, newspapers stopped being relevant as a data source. In the US, since the 1950s the Conference Board organization had systematically surveyed the number of vacancies posted in the “Help Wanted” section of the newspapers, but it stopped publishing its findings in 2008, after failing to see any increase since the 1990s boom. This failure was attributed to the migration of vacancies to the Web (Anastasopoulos et al. 2021).

The Internet allows researchers and policy practitioners to access a large volume of information about jobs and job candidates. Most commonly, the data used originate from individual job sites (Beblavý, Kureková, and Haita 2016; Drahokoupil and Fabo 2022; Marinescu and Wolthoff 2020), as well as commercial websites where people post their personal profiles or CVs, such as LinkedIn or Indeed (Kureková and Žilinčíková 2018; Mamertino and Sinclair 2019; Apaza, Vidal, and Chire 2021; Pejic-Bach et al. 2020). In some cases, however, data are instead taken from an aggregator, such as the European Public Employment Services network, EURES (Kureková et al. 2016), or companies such as Emsi Burning Glass, which collect and extract information from large volumes of job vacancies from many websites (Hershbein 2016; Deming and Kahn 2018; Fabo and Kahanec 2020; Acemoglu et al. 2020). A major advantage of this data source is that the data collection is regular rather than a one-off event. Thus, the data can be analysed as time series and not just cross-sectionally (e.g. Acemoglu et al. 2020; Leitner and Reiter 2020).

Social media is a comparatively less developed alternative. Nonetheless, several notable studies have used social media data to analyse the labour market (see the more detailed overview of alternative online data sources in Box 1). Specifically, Twitter has been used to predict labour market flows by counting the incidence of searches such as “lost my job” over time (Antenucci et al. 2014). Additionally, there has been growing research conducted with LinkedIn data (Mamertino and Sinclair 2019; Tambe 2014; Tambe et al. 2020). Overall, social media remains a powerful but underutilized tool for studying labour markets; this is due to difficulty in obtaining data, as well as “making sense” of often rather opaque signals present in the social media content (Lenaerts, Beblavý, and Fabo 2016).

Importantly, the Internet has a dual role in labour market research: it serves as an vital source of data, but at the same time it is transforming the labour market (Horton 2011). A large number of studies have tried to estimate and/or forecast important macroeconomic variables such as unemployment using Google Trends data; mostly in developed countries (Askitas and Zimmermann 2009; Fondeur and Karamé 2013; McLaren and Shanbhogue 2011), but also in China (Su 2014, using Baidoo). Empirical analyses have shown that online job vacancies are suitable data sources for measuring aggregate economic activity in the labour market (Hershbein and Kahn 2016; de Pedraza et al. 2019).

As a consequence, using online job vacancies as a data source for calculating common labour market indicators (such as number of vacancies, labour market tightness, degree of skill mismatch) has become a relatively common practice for scholars and practitioners (Japec and Lyberg 2020; Štefánik, Lyócsa, and Bilka [2022]; Turrell et al. 2019). Interesting micro-level applications of online labour market data, beyond skills analysis, include those focusing on the value of the migration experience in employers’ demands (Kureková and Žilinčíková, 2018); the role of occupational mismatch in explaining the productivity puzzle (Turrell et al. 2021); the relationship between firm credit crunch and employee job search behaviour (Gortmaker, Jeffers, and Lee 2021); discrimination against women in the labour market (Kuhn and Shen 2013); or links between the introduction of unemployment benefits, job searches and job postings during the Great Recession in the US (Marinescu 2017).

Internet labour market data typically cover a specific labour market, such as a country; but in some cases they include multiple countries (or sub-country units, such as the US states), in a comparative design (Azar et al. 2020; Fabo, Beblavý, and Lenaerts 2017; Modestino, Shoag, and Ballance 2020). Most studies focus on the entire labour market, but several investigate specific occupations or industries, such as logistics (Kotzab et al. 2018), nursing (Kobayashi et al. 2016), or data science (Debortoli, Müller, and vom Brocke 2014; Ecleo and Galido 2017).

A clear majority of published studies of online labour market data focus on the US, UK or EU labour markets, which is in line with the general over-representation of research focusing on developed countries in academia (Das and Do 2014). Nevertheless, the Western-centric focus in the literature should not overshadow the growing number of important publications that cover developing and emerging nations’ economies. In particular, they deal with big labour markets such as China (Fang et al. 2020; Kuhn and Shen 2013; Maurer-Fazio and Lei 2015; Xu et al. 2017; Zhu et al. 2016), India (Chowdhury et al. 2018; Nomura et al. 2017), Russia (Pitukhin, Astafyeva, and Astafyeva 2020; Skhvediani et al. 2021), and Pakistan (Bilal et al. 2017; Matsuda, Ahmed, and Nomura 2019).

The large size of these markets generates a huge amount of labour market data: for instance, a recent study focusing on the Chinese market identified 20 million job adverts, offering 105 million job vacancies, posted on just one online platform in four months (Fang et al. 2020). Other emerging and developing countries that have been studied include countries as diverse as the Philippines (Ecleo and Galido 2017), Ukraine (Muller and Safir 2019), Belarus (Vankevich and Kalinouskaya 2020), Kosovo (Brancatelli, Marguerie, and Brodmann 2020), Peru (Apaza, Vidal, and Chire 2021), and Mexico (Campos-Vazquez, Esquivel, and Badillo 2021).

Lastly, research using online vacancy data appears more frequently than research relying on job applicant/CV data; these two aspects of labour markets are rarely studied simultaneously, although some examples exist (Fabo and Kahanec 2020; Matsuda, Ahmed, and Nomura 2019). This imbalance might be due to the generally easier access to information about vacant jobs, rather than applicant information, which some portals offer on a paid basis (e.g. Profesia.sk). However, it could also reflect the greater interest in labour demand research, for which there are fewer alternative data sources of vacancy data.

Box 1: Alternative online data sources relevant for labour market analysis

Advantages of online labour market data

Notwithstanding the well-known limitations, the online labour market data represent a powerful tool that allows researchers to study the labour market. A key motivation for using online labour market data is their granularity and detail; together with a large number of observations, this allows researchers to access detailed information on the demand and supply of skills in the labour market, and generate insights on important topics. These include the skill mismatch (Beblavý, Kureková, and Haita 2016), school-to-work transition (Buchs and Helbling 2016), the skills and educational characteristics of new occupations (Acemoglu et al. 2020; Beblavý et al. 2016; Rios et al. 2020), the evolution of skill demand over time (Blair and Deming 2020), and lifelong learning (Kotzab et al. 2018). Box 2 presents a discussion of the importance of skills as a variable in the analysis of online labour market data, and provides selected findings about skills, tasks and occupations based on the analysis of online data.

Box 2: Skills, tasks, occupations: What do we see in online vacancies?

An important characteristic of online labour market data is their real-time availability. Real-time labour market data have been used to anticipate, for example, unemployment trends (Askitas and Zimmermann 2009; Simionescu and Zimmermann 2017) or GDP growth. Real-time availability implies that data can be analysed with a much shorter time delay than survey-based data about labour markets, while it can also capture the impact and dynamics of unexpected events and shocks, such as the COVID-19 pandemic (Campos-Vazquez, Esquivel, and Badillo 2021; Fang et al. 2020). For instance, an OECD study identified not only the sizable dip in vacancy rates at the onset of the pandemic (March–April 2020), but also the variance between countries, sectoral differences in fluctuations, and the change in skill requirements connected to the switch to working from home (OECD 2021); this implies that the impact of the COVID-related shock varied across a set of dimensions. Moreover, timely information is useful not only for research, but particularly for policy makers and educators. For instance, it is well known in economics that unsuccessful school-to-work transition has long-term implications for career outcomes (Bloom, Freeman, and Korenman 1988). It is, therefore, important to be able to advise young people which sectors are growing despite the recession, and which are the most demanded skills that might improve their chances in a difficult labour market.

While real-time availability is a key advantage, in some cases, past online labour market data can also be reconstructed (for example, the job portal Profesia.sk stores past vacancies and has provided researchers with vacancy and CV data over a retrospective time period: see e.g. Beblavý et al. 2016). Such data can in principle be used as longitudinal data, to study trends in skills supply and demand, the emergence of new occupations and their skill requirements, or transformations such as skill-biased technological change. This enables cost-effective access to longitudinal and sometimes cross-sectional data about skills (such as via the EURES platform, Skills Panorama). Moreover, this longitudinal aspect allows the study of changes within occupations, which is rarely possible with other data sources.

Large quantities of online labour market go hand-in-hand with comprehensiveness that online labour market data possibly entail. While data are typically unstructured (Gandomi and Haider 2015), in principle the information that can be extracted from the content of job vacancies or from individual CVs is very comprehensive. In terms of online job vacancies, information about educational or qualification requirements, skills or tasks, and required experience is often detailed, and can be systematically analysed (Beblavý, Kureková, and Haita 2016). Likewise, online job applicant data in fact contain detailed professional and educational experience, and personal information (i.e. key socio-demographic characteristics). With rising competition for jobs, applicants increasingly highlight key competences or skills – especially IT and language skills – but also soft skills; this provides comprehensive input for studying profiles of job candidates (Haddad and Mercier-Laurent 2021; Kureková and Žilinčíková 2018).

It is also important to consider trends that are likely to positively impact the quality and usability of online labour market data. First, the recruitment market is moving online in a dynamic way, and in some developed economies, labour market matching is organized fully online. In an ILO report, Van Loo and Pouliakis (2020) reviewed online job markets in the EU28 countries during 2019, and concluded that in countries such as Estonia, Sweden and Finland, the proportion of vacancies published online approached 100 per cent, while in others such as Denmark it accounted for around 50 per cent. Secondly, in part driven by mismatches, firms are forced to broaden their recruitment processes to include other occupations and regions, thus in effect enlarging their candidate base (which in turn mitigates demand for higher wages). Both these trends work in favour of increasing representativeness, and reducing measurement error (van Loo and Pouliakas 2020). There is now also strong evidence that the Internet has become an effective tool for matching workers to jobs (Kuhn and Mansour, 2014), and that poorer strata of society are increasingly turning to the Internet to search for jobs because their social capital is weaker (Kuhn, 2014). It is also worth mentioning that academic research has already influenced concrete policy initiatives, some of which are summarized in Box 3.

Box 3: Policy initiatives using online labour market data

Sources of biases in online labour market data

Notwithstanding the evident advantages discussed in the previous chapter, the online labour market data suffer from various biases, particularly a lack of representativeness. Non-representativeness in applicant data might have different causes to that affecting vacancy data. With respect to job applications and job searches, the main source of non-representativeness is linked to the fact that the universe of jobs intermediated online is not equal to the universe of new jobs that exist. Internet access is an important driver of this, as individuals’ ability to access the Internet remains unequal across and within countries, and varies by socioeconomic status, age or skills. Other aspects that intervene in decisions about online job seeking include a sector’s level of informality, as well as the level of social capital that sustains referrals, which are more widely used in lower-skilled jobs and in smaller enterprises.

Regarding vacancy data, the individual labour market segments are unlikely to advertise open positions to an equal extent. While Internet access has become less of an issue for firms, factors such as the intensity of labour demand, nature of work, level of informality in a given sector, or aspects such as firm size, all affect the likelihood of a vacancy being published online in the first place (Sostero and Fernandez-Macias 2021). Sectors such as construction or agriculture are in some countries less amenable to the use of online labour intermediation platforms, while micro and small enterprises are more likely to rely on informal and non-advertised hiring processes. We now turn to discussing these aspects in greater detail.

The extent to which the population is connected to the Web varies greatly between different countries, but also within them. As evident from Figure 1, Internet access is close to universal in high-income countries, and is available to a majority in most upper-middle income countries and some lower-middle income countries. However, the majority of the population in low-income countries and many lower-middle income countries are still without access to the Internet. The poorest, least educated and the most distant from the labour market, even in high-income countries, are typically digitally disconnected (Warschauer 2003; van Dijk 2006; 2020; Scheerder, van Deursen, and van Dijk 2017). Furthermore, other specific groups such as females, older workers and rural populations are likely to be unable to go online in countries where Internet access is not widespread (Birba and Diagne 2012). Hence, information about online labour market matching is likely to have biases in less developed and emerging economies, due to limited or skewed Internet access.

Figure 1: Share of Internet users per population, correlated with GDP in 2017 on a log scale.

Data source: World Bank: World Development Indicators (extracted 25 February 2022)

In countries where Internet access is close to universal, the bias of analysis using Internet labour market data might be less significant (Askitas and Zimmermann 2015); however, there is still an observable bias in the data, leading to an over-representation of tertiary educated workers and job opportunities for better-educated workers (Muller and Safir 2019; Štefánik 2012). For example, Carnevale, Jayasundera and Repnikov (2014) studied Burning Glass Technologies data for the US labour market, and estimated that 80–90 per cent of job vacancies requiring a tertiary degree (bachelor’s and higher) are posted online, compared to about 40–60 per cent of job advertisements requiring a high-school diploma. In spite of this limitation, some researchers have used online labour market data to understand demand in low-skilled occupations or in unstable, typically less skilled jobs (Beblavý, Kureková, and Haita 2016; Kureková and Žilinčíková 2016).

Another important dimension to consider is that of informal labour. From the existing ILO analyses, we know that six workers out of ten work in the informal economy. Unlike some past predictions, we know that this number is not necessarily decreasing. The issue is not limited to emerging countries, and particularly affects vulnerable populations such as women, uneducated people, or migrants (ILO 2021). The reasons for informal employment vary; for instance, enterprises might opt to operate informally to avoid regulations applying to a formal employment relationship. Additionally, even formal firms might employ workers informally; in some cases this reflects the preference of workers, as in the case of online crowdworkers preferring to make some quick money on the side. Thus, informal employment might be associated with lower numbers of vacancies being published either online or offline. Nevertheless, we see that some online job portals also cover the informal labour market, such as Babajobs in India (see Nomura et al. 2017).

Next, particularly in the developing and emerging markets, a major part of the workforce finds itself in a self-employment arrangement, due to necessity or choice; even though their work is similar to that performed by employed workers (ILO 2016; Poschke 2019). Self-employed work does not typically generate vacancies (Dunlop 1966), and people might be particularly prone to being inaccurate when describing their self-employment experience in CVs (Jones 1984).

In the formal economy, there are also reasons for not advertising jobs publicly. Enterprises and job seekers might opt for an informal (internal) approach for multiple reasons, including lower search costs, ability to avoid initial screening, and because seeking workers or work through informal networks is likely to result in opportunities and applicants located in the near vicinity (van Ours 1989). The sheer size of the online job markets demonstrates that there are many situations where a formal job search is nonetheless initiated; but we need to be mindful of the limited generalizability of any patterns identified in the job postings, even in countries with a high share of Internet users and an insignificant informal economy. That being said, the relatively low cost of advertising job vacancies (or of finding a worker via a CV posted online) might empower actors who would not have initiated formal recruitment in the pre-Internet era (Sodhi and Son 2010).

In addition to non-representativeness, validity and reliability might also be of concern when using online data. Both vacancy and CV data are self-reported, and there are no tools embedded to check the validity and reliability of information provided. For example, Internet job boards can be flooded by resumés that in fact no longer correspond to people who are searching for a job – known as “stale” resumés (Kuhn 2014) – while the same might be true in the case of vacancies. Nonetheless, some researchers consider online information about job applicants to be more truthful and accurate (van Loo and Pouliakas 2020). Moreover, vacancies might be posted online even after the position has been filled, or one posted vacancy can in practice mean more job openings. These specificities warn of a measurement error due to duplicates and the lifetime of a vacancy. Research has also identified that firms might use vacancies as an advertising or company branding tool, which is likely to affect the choice of vacancy content (Winzenried, 2020).

Particular concerns might also arise for cross-country comparative research. Existing studies have shown that employers in different countries seem to use very different strategies in terms of their expressed expectations for skills or education; this might be due to underlying differences in the functioning and institutional underpinning of national labour markets across Europe (Kureková et al. 2016). Similar differences have been identified among formally identical jobs advertised in different sectors, occupations and skill levels (Beblavý, Kureková, and Haita 2016; Brandas, Panzaru, and Filip 2016; van Loo and Pouliakas 2020). Brandas et al. (2016), who studied academic vacancies worldwide, also pointed out a lack of semantic and structural compatibility of data mined from different sources. Winzenried (2020) provided examples of job vacancies that greatly vary in the “density” of skills they require, and emphasized the importance of “implicit” knowledge in vacancy posting, which can be country-, sector- or occupation-specific (such as the significance of education or experience).

Methodological aspects of online labour market data

In this chapter we present how the methodological weaknesses of online data have been addressed in existing research, by covering a range of statistical and other approaches. First, we describe data processing techniques, and explore more conceptual questions regarding the philosophy of research and its aims; we then discuss these in light of the research objectives, using online labour market data.

Conclusion and implications

This paper has studied how online data on labour markets can be used to describe, analyse, understand, refine or predict labour market trends. This relates to the more general aspect of labour demand, but also more specifically to skills demand and skills changes, in light of the existing deficiencies of online data: specifically, a lack of representativeness.

The research field that relies on online labour market data has expanded rapidly in recent years. Our review revealed that this expansion is characterized by several features. First, research continues to focus on single countries, with a small number of attempts at comparative work. Second, research using online vacancy data appears more frequently than research relying on job applicant/CV data. This might be caused by the generally easier access to information about vacant jobs, than to applicant information, which some portals offer on a paid basis. However, it could also reflect a greater interest in research on the demand for labour, for which fewer alternative survey data sources are available. Third, while this research field continues to be driven by a focus on advanced economies, mainly the US and EU, there is also an evident trend towards expanding this research to developing countries´ labour markets. These studies, on average, tend to be less concerned with data biases than research focusing on advanced economies. Fourth, somewhat to our surprise, the data limitations of online sources used to study labour markets often remain undiscussed in terms of the biases, non-representativeness, or other potential pitfalls of these data. Moreover, this situation does not seem to be improving with time – in contrast to those studies that have taken a rigorous approach to understanding the qualities of online labour market data, and have addressed them using various methodological or research design approaches.

This paper advances the current debate by offering a mapping of biases recognized in online labour market vacancies and CV data, and an overview of approaches and techniques to address the identified biases. We highlight that legitimate research approaches exist, which are inductive in nature, focused on discovering patterns and trends in underlying data. These methods are by definition less concerned with generalizability of findings, as they have different objectives. For this body of research, online labour market data open new avenues for understanding developments in labour markets. (Near) real-time availability, granularity, relative affordability and size represent some of the key qualities which make online labour market data uniquely suitable for many forms of analyses – traditional as well as novel ones.

Biases in online labour market data emerge due to a myriad of factors, including populations’ varying levels of Internet access; different resources, motivations or opportunities for advertising a vacancy, among different sectors or firms of different size; as well as higher levels of informality in some economies or sectors. Most evaluations of biases pertain to developed countries, and these have identified over-representation of some sectors (ICT, finance) and under-representation of others, mainly the public sector and manufacturing. While there are more skilled than non-skilled vacancies found in the online world, there is no clear evidence of white-collar vacancies’ over-representation in relation to blue-collar, as public jobs are prevailingly white-collar. In addition to the type of work, additional factors such as hiring practices, job turnover or the levels of informality can shape the probability of vacancies appearing online.

With respect to the nature of discrepancies, however, these are typically not paramount to hinder research and reliability of findings. Different techniques have been adopted to deal with the non-representativeness problem. These include statistical approaches such as weighting, and tools applied at the data preparation phase (de-duplication, data matching). Dummy variables can be used to statistically account for the effect of fluctuations in data over time. An alternative approach to using this data has relied on adapting the research focus and research objectives to the quality of data. Hence, a multitude of studies focus on studying a narrower aspect of the labour market, such as academic jobs, the IT sector, or students, as these groups are well covered in the online segment. Lastly, mixed-method approaches essentially aim to validate or understand the biases of online data with the use of alternative research methods, including qualitative interviews (such as with HR managers), in order to correctly interpret the biases. Triangulation of data sources has also been applied to study a particular question, when online data or Web-based data represent only one source.

In conclusion, addressing the biases of online labour-market vacancy data is a strenuous task, as the population of vacancies is inherently unknown. Nevertheless, representativeness problems are likely to vary in different contexts; they should be evaluated at the country level, and with respect to specific research objectives, research questions, and existing alternatives, in terms of accuracy, granularity, costs and timeliness. In essence, research ethics and transparency are key. Any analysis using online labour market data should be embedded in a particular context, and analytical steps and decisions need to be clearly defined and specified. Ideally, analytical “protocols” should be recorded and made available upon request, and also to enable replication of the analysis, as is possible with any survey data.

Although from the point of view of representativeness and generalizability, representative surveys are the first-best option in many developed countries, the accessibility in many developing countries is very limited. For these countries, online data, including web-collected surveys, represent the preferred alternative, opening new horizons and opportunities in research. Web-collected data also provide information that is difficult to gather even from representative data, such as wages (real and expected) or aspects of gender biases in labour markets. Not least, it is important to realize that representative data also suffer from biases, such as non-response and coverage bias.

Importantly, the nature of online labour market data analysis, which lies in the intersection of research, policy and industrial applications, makes it possible to pool substantial resources to increase the pace of progress. CEDEFOP (2019), for example, is committed to making training sets and ontologies public, under creative commons licences. Leading data vendors are working closely with the academics, even co-authoring papers, to a far greater extent than in other domains of social science. As a result, online labour vacancy research is one of the key areas where there is rapid advancement in the application of machine learning and artificial intelligence in social science. Furthermore, in the recent past, systematic efforts have emerged to consider biases, such as the ESSnet Big Data project,4 which will assist further efforts to address biases.

Finally, beyond the issue of representativeness, there are other key challenges of working with big data in the labour market, which we did not discuss but would like to raise in conclusion. These are linked to issues of privacy and confidentiality (Einav and Levin 2014). While the sheer number of collected vacancies and CVs ensures on the one hand a form of anonymization, on the other hand, ethical concerns emerge regarding the (explicit) consent of firms and individuals that their information can be used for analytical and research purposes. While data are typically anonymized, monitoring of data management and research ethics appears more problematic for online (labour market) data than with surveys, as access to the online data is much less centralized, and ad hoc. More general regulations, such as the GDPR regulation adopted and enforced within the European Union countries, might be considered a gold standard case, as it provides specific guidelines for the protection of individual personal details (Mezzanzanica and Mercorio 2019).

Annex

Table 2: Overview of studies using online labour market data

References

Abraham, Katharine G. 1983. “Structural/Frictional vs. Deficient Demand Unemployment: Some New Evidence.” The American Economic Review 73 (4): 708–24.

Acemoglu, Daron, and David Autor. 2010. “Skills, Tasks and Technologies: Implications for Employment and Earnings.” Working Paper 16082. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w16082.

Acemoglu, Daron, David Autor, Jonathon Hazell, and Pascual Restrepo. 2020. “AI and Jobs: Evidence from Online Vacancies.” Working Paper 28257. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w28257.

Álvarez, Andrés, and Marc Hofstetter. 2014. “Job Vacancies in Colombia: 1976–2012.” IZA Journal of Labor & Development 3 (1): 15. https://doi.org/10.1186/2193-9020-3-15.

Anastasopoulos, L. Jason, George J. Borjas, Gavin G. Cook, and Michael Lachanski. 2021. “Job Vacancies and Immigration: Evidence from the Mariel Supply Shock.” Journal of Human Capital 15 (1): 1–33. https://doi.org/10.1086/713041.

Antenucci, Dolan, Michael Cafarella, Margaret Levenstein, Christopher Ré, and Matthew D. Shapiro. 2014. “Using Social Media to Measure Labor Market Flows.” Working Paper 20010. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w20010.

Apaza, Honorio, Américo Vidal, and Josimar Chire. 2021. “Job Recommendation Based on Curriculum Vitae Using Text Mining.” In , 1051–59. https://doi.org/10.1007/978-3-030-73100-7_72.

Askitas, Nikolaos, and Klaus F. Zimmermann. 2009. “Google Econometrics and Unemployment Forecasting.” Applied Economics Quarterly 55 (2): 107–21.

———. 2015. “The Internet as a Data Source for Advancement in Social Sciences.” International Journal of Manpower 36 (1): 2–12. https://doi.org/10.1108/IJM-02-2015-0029.

Autor, David H. 2001. “Wiring the Labor Market.” Journal of Economic Perspectives 15 (1): 25–40. https://doi.org/10.1257/jep.15.1.25.

Azar, José, Emiliano Huet-Vaughn, Ioana Marinescu, Bledi Taska, and Till von Wachter. 2019. “Minimum Wage Employment Effects and Labor Market Concentration.” Working Paper 26101. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w26101.

Azar, José, Ioana Marinescu, Marshall Steinbaum, and Bledi Taska. 2020. “Concentration in US Labor Markets: Evidence from Online Vacancy Data.” Labour Economics 66 (October): 101886. https://doi.org/10.1016/j.labeco.2020.101886.

Barron, John M., and John Bishop. 1985. “Extensive Search, Intensive Search, and Hiring Costs: New Evidence on Employer Hiring Activity.” Economic Inquiry 23 (3): 363–82. https://doi.org/10.1111/j.1465-7295.1985.tb01773.x.

Bauer, Anja, Tobias Hartl, Christian Hutter, and Enzo Weber. 2021. “Search Processes on the Labor Market during the Covid-19 Pandemic.” CESifo Forum 22 (04): 15–19.

Beblavý, Miroslav, Mehtap Akgüc, Brian Fabo, and Karolien Lenaerts. 2016. “Occupations Observatory-Methodological Note.” CEPS Special Report, no. 144.

Beblavý, Miroslav, Lucia Kureková, and Corina Haita. 2016. “The Surprisingly Exclusive Nature of Medium- and Low-Skilled Jobs: Evidence from a Slovak Job Portal.” Personnel Review 45 (2): 255–73. https://doi.org/10.1108/PR-12-2014-0276.

Beblavý, Miroslav, Ilaria Maselli, and Marcela Veselková, eds. 2014. Let’s Get to Work!: The Future of Labour in Europe. Brussels: Centre for European Policy Studies.

———, eds. 2015. Green, Pink and Silver?: The Future of Labour in Europe. Brussels: Centre for European Policy Studies.

Bilal, Muhammad, Nadia Malik, Maham Khalid, and M. Ikram Ullah Lali. 2017. “Exploring Industrial Demand Trends in Pakistan Software Industry Using Online Job Portal Data.” University of Sindh Journal of Information and Communication Technology 1 (1): 17–24.

Birba, Ousmane, and Abdoulaye Diagne. 2012. “Determinants of Adoption of Internet in Africa: Case of 17 Sub-Saharan Countries.” Structural Change and Economic Dynamics 23 (4): 463–72. https://doi.org/10.1016/j.strueco.2012.06.003.

Blair, Peter Q., and David J. Deming. 2020. “Structural Increases in Skill Demand after the Great Recession.” Working Paper 26680. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w26680.

Blazquez, Desamparados, and Josep Domenech. 2018. “Big Data Sources and Methods for Social and Economic Analyses.” Technological Forecasting and Social Change 130 (May): 99–113. https://doi.org/10.1016/j.techfore.2017.07.027.

Bloom, David E., Richard B. Freeman, and Sanders D. Korenman. 1988. “The Labour-Market Consequences of Generational Crowding.” European Journal of Population / Revue Européenne de Démographie 3 (2): 131–76.

Boselli, Roberto, Mirko Cesarini, Stefania Marrara, Fabio Mercorio, Mario Mezzanzanica, Gabriella Pasi, and Marco Viviani. 2018. “WoLMIS: A Labor Market Intelligence System for Classifying Web Job Vacancies.” Journal of Intelligent Information Systems 51 (3): 477–502. https://doi.org/10.1007/s10844-017-0488-x.

Boselli, Roberto, Mirko Cesarini, Fabio Mercorio, and Mario Mezzanzanica. 2018. “Classifying Online Job Advertisements through Machine Learning.” Future Generation Computer Systems 86 (September): 319–28. https://doi.org/10.1016/j.future.2018.03.035.

Brancatelli, Calogero, Alicia Marguerie, and Stefanie Brodmann. 2020. “Job Creation and Demand for Skills in Kosovo: What Can We Learn from Job Portal Data?” Working Paper. Washington, DC: World Bank. https://doi.org/10.1596/1813-9450-9266.

Brandas, Claudiu, Ciprian Panzaru, and Florin Gheorghe Filip. 2016. “Data Driven Decision Support Systems: An Application Case in Labour Market Analysis.” Romanian Journal of Information Science and Technology 19 (1–2): 65–77.

Buchs, Helen, and Laura Alexandra Helbling. 2016. “Job Opportunities and School-to-Work Transitions in Occupational Labour Markets. Are Occupational Change and Unskilled Employment after Vocational Education Interrelated?” Empirical Research in Vocational Education and Training 8 (1): 17. https://doi.org/10.1186/s40461-016-0044-x.

Burke, Mary A., Alicia Sasser, Shahriar Sadighi, Rachel B. Sederberg, and Bledi Taska. 2020. “No Longer Qualified? Changes in the Supply and Demand for Skills within Occupations.” Working Paper 20–3. Working Papers. https://doi.org/10.29412/res.wp.2020.03.

Cammeraat, Emile, and Mariagrazia Squicciarini. 2021. “Burning Glass Technologies’ Data Use in Policy-Relevant Analysis: An Occupation-Level Assessment.” Paris: OECD. https://doi.org/10.1787/cd75c3e7-en.

Campello, Murillo, Gaurav Kankanhalli, and Pradeep Muthukrishnan. 2020. “Corporate Hiring under COVID-19: Labor Market Concentration, Downskilling, and Income Inequality.” Working Paper 27208. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w27208.

Campos-Vazquez, Raymundo M., Gerardo Esquivel, and Raquel Y. Badillo. 2021. “How Has Labor Demand Been Affected by the COVID-19 Pandemic? Evidence from Job Ads in Mexico.” Latin American Economic Review 30 (May): 1–42.

Capiluppi, Andrea, and Andres Baravalle. 2010. “Matching Demand and Offer in On-Line Provision: A Longitudinal Study of Monster.Com.” In 2010 12th IEEE International Symposium on Web Systems Evolution (WSE), 13–21. https://doi.org/10.1109/WSE.2010.5623576.

CEDEFOP. 2014. “Briefing Note - Skill Mismatch: More than Meets the Eye.” March 19, 2014. https://www.cedefop.europa.eu/en/publications/9087.

———. 2019. “Online Job Vacancies and Skills Analysis: A Cedefop Pan-European Approach | VOCEDplus, the International Tertiary Education and Research Database.” https://www.voced.edu.au/content/ngv:82496.

Chowdhury, Afra Rahman, Ana Carolina Areias, Saori Imaizumi, Shinsaku Nomura, and Futoshi Yamauchi. 2018. “Reflections of Employers’Gender Preferences in Job Ads in India: An Analysis of Online Job Portal Data.” SSRN Scholarly Paper ID 3150092. Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=3150092.

Colombo, Emilio, Fabio Mercorio, and Mario Mezzanzanica. 2018. “Applying Machine Learning Tools on Web Vacancies for Labour Market and Skill Analysis.”

Comyn, Paul, and Olga Strietska-Ilina. 2019. Skills and Jobs Mismatches in Low- and Middle-Income Countries. Geneva, Switzerland: ILO. https://www.semanticscholar.org/paper/Skills-and-jobs-mismatches-in-low-and-middle-income-Comyn/1a2c34a7e6c337b46fc0b004291c2c8d27f5c85f.

Das, Jishnu, and Quy-Toan Do. 2014. “US and Them: The Geography of Academic Research.” VoxEU.Org (blog). February 11, 2014. https://voxeu.org/article/geographical-bias-top-journal-publication.

Das, Marcel, Peter Ester, and Lars Kaczmirek. 2018. Social and Behavioral Research and the Internet: Advances in Applied Methods and Research Strategies. Routledge.

Davis, Steven J., R. Jason Faberman, and John C. Haltiwanger. 2013. “The Establishment-Level Behavior of Vacancies and Hiring *.” The Quarterly Journal of Economics 128 (2): 581–622. https://doi.org/10.1093/qje/qjt002.

Debortoli, Stefan, Oliver Müller, and Jan vom Brocke. 2014. “Comparing Business Intelligence and Big Data Skills.” Business & Information Systems Engineering 6 (5): 289–300. https://doi.org/10.1007/s12599-014-0344-2.

Deming, David, and Lisa B. Kahn. 2018. “Skill Requirements across Firms and Labor Markets: Evidence from Job Postings for Professionals.” Journal of Labor Economics 36 (S1): S337–69. https://doi.org/10.1086/694106.

Dijk, Jan van. 2006. “Digital Divide Research, Achievements and Shortcomings.” Poetics, The digital divide in the twenty-first century, 34 (4): 221–35. https://doi.org/10.1016/j.poetic.2006.05.004.

———. 2020. The Digital Divide. 1st edition. Cambridge, UK ; Medford, MA: Polity.

Dörfler, Laura, and Herman van de Werfhorst. 2009. “Employers’ Demand for Qualifications and Skills.” European Societies 11 (5): 697–721. https://doi.org/10.1080/14616690802474374.

Drahokoupil, Jan, and Brian Fabo. 2016. “The Platform Economy and the Disruption of the Employment Relationship.” ETUI Research Paper-Policy Brief 5.

———. 2022. “The Limits of Foreign-Led Growth: Demand for Skills by Foreign and Domestic Firms.” Review of International Political Economy 29 (1): 152–74.

Dunlop, John T. 1966. “Job Vacancy Measures and Economic Analysis.” In The Measurement and Interpretation of Job Vacancies, 27–47. NBER. https://www.nber.org/books-and-chapters/measurement-and-interpretation-job-vacancies/job-vacancy-measures-and-economic-analysis.

Ecleo, Jerina Jean, and Adrian Galido. 2017. “Surveying LinkedIn Profiles of Data Scientists: The Case of the Philippines.” Procedia Computer Science, 4th Information Systems International Conference 2017, ISICO 2017, 6-8 November 2017, Bali, Indonesia, 124 (January): 53–60. https://doi.org/10.1016/j.procs.2017.12.129.

Einav, Liran, and Jonathan Levin. 2014. “Economics in the Age of Big Data.” Science 346 (6210): 1243089. https://doi.org/10.1126/science.1243089.

Ernesto, Caroleo Floro, and Pastore Francesco. 2016. “Overeducation: A Disease of the School-to-Work Transition System.” In Youth and the Crisis. Routledge.

ESCO. 2015. “European Skills, Competences, Qualifications and Occupations.” (https://ec.europa.eu/esco/ portal/home#modal-one.

Fabo, Brian, Miroslav Beblavý, and Karolien Lenaerts. 2017. “The Importance of Foreign Language Skills in the Labour Markets of Central and Eastern Europe: Assessment Based on Data from Online Job Portals.” Empirica 44 (3): 487–508. https://doi.org/10.1007/s10663-017-9374-6.

Fabo, Brian, and Martin Kahanec. 2018. “Can a Voluntary Web Survey Be Useful beyond Explorative Research?” International Journal of Social Research Methodology.

———. 2020. “The Role of Computer Skills on the Occupation Level.” European Journal of Business Science and Technology 6 (2): 87–99. https://doi.org/10.11118/ejobsat.2020.006.

Fang, Hanming, Chunmian Ge, Hanwei Huang, and Hongbin Li. 2020. “Pandemics, Global Supply Chains, and Local Labor Demand: Evidence from 100 Million Posted Jobs in China.” Working Paper 28072. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w28072.

Ferber, Robert, and Neil Ford. 1966. “The Time Dimension in the Collection of Job Vacancy Data.” In The Measurement and Interpretation of Job Vacancies, 447–61. NBER. https://www.nber.org/books-and-chapters/measurement-and-interpretation-job-vacancies/time-dimension-collection-job-vacancy-data.

Fondeur, Y., and Frédéric Karamé. 2013. “Can Google Data Help Predict French Youth Unemployment?” Economic Modelling 30 (C): 117–25.

Forsythe, Eliza, Lisa B. Kahn, Fabian Lange, and David Wiczer. 2020. “Labor Demand in the Time of COVID-19: Evidence from Vacancy Postings and UI Claims.” Journal of Public Economics 189 (September): 104238. https://doi.org/10.1016/j.jpubeco.2020.104238.

Gandomi, Amir, and Murtaza Haider. 2015. “Beyond the Hype: Big Data Concepts, Methods, and Analytics.” International Journal of Information Management 35 (2): 137–44. https://doi.org/10.1016/j.ijinfomgt.2014.10.007.

Gortmaker, Jeff, Jessica Jeffers, and Michael Lee. 2021. “Labor Reactions to Credit Deterioration: Evidence from LinkedIn Activity.” SSRN Scholarly Paper ID 3456285. Rochester, NY: Social Science Research Network. https://doi.org/10.2139/ssrn.3456285.

Haddad, Rabih, and Eunika Mercier-Laurent. 2021. “Curriculum Vitae (CVs) Evaluation Using Machine Learning Approach.” In Artificial Intelligence for Knowledge Management, edited by Eunika Mercier-Laurent, M. Özgür Kayalica, and Mieczyslaw Lech Owoc, 48–65. IFIP Advances in Information and Communication Technology. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-80847-1_4.

Hensvik, Lena, Thomas Le Barbanchon, and Roland Rathelot. 2021. “Job Search during the COVID-19 Crisis.” Journal of Public Economics 194 (February): 104349. https://doi.org/10.1016/j.jpubeco.2020.104349.

Hershbein, Brad. 2016. “Is College the New High School? Evidence from Vacancy Postings.” https://research.upjohn.org/projects/169.

Hershbein, Brad, and Lisa B. Kahn. 2016. “Do Recessions Accelerate Routine-Biased Technological Change? Evidence from Vacancy Postings.” Working Paper 22762. National Bureau of Economic Research. https://doi.org/10.3386/w22762.

Holt, Charles, and Martin David. 1966. “The Concept of Job Vacancies in a Dynamic Theory of the Labor Market.” In The Measurement and Interpretation of Job Vacancies, 73–110. NBER. http://www.nber.org/chapters/c1599.pdf.

Horton, John J. 2011. “The Condition of the Turking Class: Are Online Employers Fair and Honest?” Economics Letters 111 (1): 10–12. https://doi.org/10.1016/j.econlet.2010.12.007.

Huang, Hayan, Lynette Kvasny, K.D. Joshi, Eileen Trauth, and Jan Mahar. 2009. “Synthesizing IT Job Skills Identified in Academic Studies, Practitioner Publications and Job Ads.” In Proceedings of the Special Interest Group on Management Information System’s 47th Annual Conference on Computer Personnel Research. Ireland: ACM.

ILO. 2016. “Disguised Employment / Dependent Self-Employment.” Document. November 11, 2016. http://www.ilo.org/global/topics/non-standard-employment/WCMS_534833/lang--en/index.htm.

———. 2021. “Transition from the Informal to the Formal Economy - Theory of Change.” Briefing note. January 29, 2021. http://www.ilo.org/global/topics/employment-promotion/informal-economy/publications/WCMS_768807/lang--en/index.htm.

Jackson, Michelle. 2007. “How Far Merit Selection? Social Stratification and the Labour Market1.” The British Journal of Sociology 58 (3): 367–90. https://doi.org/10.1111/j.1468-4446.2007.00156.x.

Japec, Lilli, and Lars Lyberg. 2020. “Big Data Initiatives in Official Statistics.” In Big Data Meets Survey Science, 273–302. John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118976357.ch9.

Jones, Lyndon. 1984. “Lies, Damned Lies, and CVs.” Education + Training 26 (4): 124–26. https://doi.org/10.1108/eb002125.

Kahn, Lisa B., Fabian Lange, and David Wiczer. 2020. “Labor Supply in the Time of COVID19.” 06–2020. Cahiers de Recherche. Cahiers de Recherche. Centre interuniversitaire de recherche en économie quantitative, CIREQ. https://ideas.repec.org/p/mtl/montec/06-2020.html.

Khaouja, Imane, Ismail Kassou, and Mounir Ghogho. 2021. “A Survey on Skill Identification From Online Job Ads.” IEEE Access 9: 118134–53. https://doi.org/10.1109/ACCESS.2021.3106120.

Kobayashi, Vladimer, Stefan T. Mol, Gábor Kismihók, and Maria Hesterberg. 2016. “Automatic Extraction of Nursing Tasks from Online Job Vacancies.” In Professional Education and Training through Knowledge, Technology and Innovation, 51. Siegen, Germany: Universitätsverlag Siegen.

Kotu, Vijay, and Bala Deshpande. 2019. “Chapter 9 - Text Mining.” In Data Science (Second Edition), edited by Vijay Kotu and Bala Deshpande, 281–305. Morgan Kaufmann. https://doi.org/10.1016/B978-0-12-814761-0.00009-5.

Kotzab, Herbert, Christoph Teller, Michael Bourlakis, and Sebastian Wünsche. 2018. “Key Competences of Logistics and SCM Professionals – the Lifelong Learning Perspective.” Supply Chain Management: An International Journal 23 (1): 50–64. https://doi.org/10.1108/SCM-02-2017-0079.

Kuhn, Peter. 2014. “The Internet as a Labor Market Matchmaker.” IZA World of Labor, May. https://doi.org/10.15185/izawol.18.

Kuhn, Peter, and Hani Mansour. 2014. “Is Internet Job Search Still Ineffective?” The Economic Journal 124 (581): 1213–33. https://doi.org/10.1111/ecoj.12119.

Kuhn, Peter, and Kailing Shen. 2013. “Gender Discrimination in Job Ads: Evidence from China.” The Quarterly Journal of Economics 128 (1): 287–336. https://doi.org/10.1093/qje/qjs046.

Kureková, Lucia, Miroslav Beblavý, Corina Haita, and Anna Elisabeth Thum Thysen. 2016. “Employers’ Skill Preferences across Europe: Between Cognitive and Non-Cognitive Skills.” Journal of Education and Work 29 (6): 662–87. https://doi.org/10.1080/13639080.2015.1024641.

Kureková, Lucia, Miroslav Beblavý, and Anna Elisabeth Thum Thysen. 2015. “Using Online Vacancies and Web Surveys to Analyse the Labour Market: A Methodological Inquiry.” IZA Journal of Labor Economics 4 (1): 18. https://doi.org/10.1186/s40172-015-0034-4.

Kureková, Lucia, and Zuzana Žilinčíková. 2016. “Are Student Jobs Flexible Jobs? Using Online Data to Study Employers’ Preferences in Slovakia.” IZA Journal of European Labor Studies 5 (1): 20. https://doi.org/10.1186/s40174-016-0070-5.

———. 2018. “What Is the Value of Foreign Work Experience for Young Return Migrants?” International Journal of Manpower 39 (1): 71–92. https://doi.org/10.1108/IJM-04-2016-0091.

Lazer, David, Ryan Kennedy, Gary King, and Alessandro Vespignani. 2014. “The Parable of Google Flu: Traps in Big Data Analysis.” Science 343 (6176): 1203–5. https://doi.org/10.1126/science.1248506.

Leitner, Sandra M., and Oliver Reiter. 2020. “Employers’ Skills Requirements in the Austrian Labour Market: On the Relative Importance of ICT, Cognitive and Non-Cognitive Skills over the Past 15 Years.” Working Paper 190. wiiw Working Paper. https://www.econstor.eu/handle/10419/240633.

Lenaerts, Karolien, Miroslav Beblavý, and Brian Fabo. 2016. “Prospects for Utilisation of Non-Vacancy Internet Data in Labour Market Analysis—an Overview.” IZA Journal of Labor Economics 5 (1): 1. https://doi.org/10.1186/s40172-016-0042-z.

Lewis, Phil, and Jennifer Norton. 2016. “Identification of ‘Hot Technologies’ within the O*NET® System.” https://www.onetcenter.org/reports/Hot_Technologies.html.

Loo, Jasper van, and Konstantinos Pouliakas. 2020. “Cedefop and the Analysis of European Online Job Vacancies.” In The Feasibility of Using Big Data in Anticipating and Matching Skills Needs. Geneva, Switzerland: ILO.

Lovaglio, Pietro Giorgio, Mario Mezzanzanica, and Emilio Colombo. 2020. “Comparing Time Series Characteristics of Official and Web Job Vacancy Data.” Quality & Quantity 54 (1): 85–98. https://doi.org/10.1007/s11135-019-00940-3.

Mamertino, Mariano, and Tara M. Sinclair. 2019. “Migration and Online Job Search: A Gravity Model Approach.” Economics Letters 181 (August): 51–53. https://doi.org/10.1016/j.econlet.2019.05.005.

Marconi, Gabriele. 2022. “Content Removal Bias in Web Scraped Data: A Solution Applied to Real Estate Ads.” SSRN Scholarly Paper ID 4031466. Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=4031466.

Marinescu, Ioana. 2017. “The General Equilibrium Impacts of Unemployment Insurance: Evidence from a Large Online Job Board.” Journal of Public Economics 150 (June): 14–29. https://doi.org/10.1016/j.jpubeco.2017.02.012.

Marinescu, Ioana, and Roland Rathelot. 2018. “Mismatch Unemployment and the Geography of Job Search.” American Economic Journal: Macroeconomics 10 (3): 42–70. https://doi.org/10.1257/mac.20160312.

Marinescu, Ioana, and Ronald Wolthoff. 2020. “Opening the Black Box of the Matching Function: The Power of Words.” Journal of Labor Economics 38 (2): 535–68. https://doi.org/10.1086/705903.

Marrara, Stefania, Gabriella Pasi, Marco Viviani, Mirko Cesarini, Fabio Mercorio, Mario Mezzanzanica, and Marco Pappagallo. 2017. “A Language Modelling Approach for Discovering Novel Labour Market Occupations from the Web.” In Proceedings of the International Conference on Web Intelligence, 1026–34. WI ’17. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3106426.3109035.

Masso, Jaan, Raul Eamets, Pille Mõtsmees, and Kaia Philips. 2012. “The Impact of Interfirm Labor Mobility on Innovation: Evidence from Job Search Portal Data.” In Innovation Systems in Small Catching-Up Economies: New Perspectives on Practice and Policy, edited by Elias G. Carayannis, Urmas Varblane, and Tõnu Roolaht, 297–321. Innovation, Technology, and Knowledge Management. New York, NY: Springer. https://doi.org/10.1007/978-1-4614-1548-0_16.

Masso, Jaan, Lucia Kureková, Maryna Tverdostup, and Zuzana Žilinčíková. 2016. “Return Migration to CEE after the Crisis: Estonia and Slovakia.” https://style-handbook.eu/contents-list/migration-and-mobility/return-migration-to-cee-after-the-crisis-estonia-and-slovakia/.

Matsuda, Norihiko, Tutan Ahmed, and Shinsaku Nomura. 2019. “Labor Market Analysis Using Big Data: The Case of a Pakistani Online Job Portal.” SSRN Scholarly Paper ID 3491253. Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=3491253.

Maurer-Fazio, Margaret, and Lei Lei. 2015. “‘As Rare as a Panda’: How Facial Attractiveness, Gender, and Occupation Affect Interview Callbacks at Chinese Firms.” International Journal of Manpower 36 (1): 68–85. https://doi.org/10.1108/IJM-12-2014-0258.

McLaren, Nick, and Rachana Shanbhogue. 2011. “Using Internet Search Data as Economic Indicators.” Bank of England Quarterly Bulletin 51 (2): 134–40.

Mezzanzanica, Mario, and Fabio Mercorio. 2019. “Big Data for Labour Market Intelligence: An Introductory Guide.” European Training Foundation. https://www.etf.europa.eu/en/publications-and-resources/publications/big-data-labour-market-intelligence-introductory-guide.

Modestino, Alicia Sasser, Daniel Shoag, and Joshua Ballance. 2020. “Upskilling: Do Employers Demand Greater Skill When Workers Are Plentiful?” The Review of Economics and Statistics 102 (4): 793–805. https://doi.org/10.1162/rest_a_00835.

Mukoyama, Toshihiko, Christina Patterson, and Ayşegül Şahin. 2018. “Job Search Behavior over the Business Cycle.” American Economic Journal: Macroeconomics 10 (1): 190–215. https://doi.org/10.1257/mac.20160202.

Muller, Noël, and Abla Safir. 2019. “What Employers Actually Want: Skills in Demand in Online Job Vacancies in Ukraine.” Working Paper. Washington, DC: World Bank. https://doi.org/10.1596/31884.

Nomura, Shinsaku, Saori Imaizumi, Ana Carolina Areias, and Futoshi Yamauchi. 2017. “Toward Labor Market Policy 2.0: The Potential for Using Online Job-Portal Big Data to Inform Labor Market Policies in India.” Working Paper. Washington, DC: World Bank. https://doi.org/10.1596/1813-9450-7966.

OECD. 2021. “OECD Skills Outlook 2021: Learning for Life.” https://www.oecd.org/education/oecd-skills-outlook-e11c1c2d-en.htm.

Ours, Jan van. 1989. “Durations of Dutch Job Vacancies.” De Economist 137 (3): 309–27. https://doi.org/10.1007/BF02115697.

Ours, Jan van, and Geert Ridder. 1992. “Vacancies and the Recruitment of New Employees.” Journal of Labor Economics 10 (2): 138–55.

Pedraza, Pablo de, Stefano Visintin, Kea Tijdens, and Gábor Kismihók. 2019. “Survey vs Scraped Data: Comparing Time Series Properties of Web and Survey Vacancy Data.” IZA Journal of Labor Economics 8 (1). https://doi.org/10.2478/izajole-2019-0004.

Pejic-Bach, Mirjana, Tine Bertoncel, Maja Meško, and Živko Krstić. 2020. “Text Mining of Industry 4.0 Job Advertisements.” International Journal of Information Management 50 (February): 416–31. https://doi.org/10.1016/j.ijinfomgt.2019.07.014.

Piróg, Danuta. 2016. “Job Search Strategies of Recent University Graduates in Poland: Plans and Effectiveness.” Higher Education 71 (4): 557–73.

Pitukhin, Eugene, Marina Astafyeva, and Irina Astafyeva. 2020. “Methodology for Job Advertisements Analysis in the Labor Market in Metropolitan Cities: The Case Study of the Capital of Russia.” In Intelligent Algorithms in Software Engineering, edited by Radek Silhavy, 413–29. Advances in Intelligent Systems and Computing. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-51965-0_37.

Poschke, Markus. 2019. “Wage Employment, Unemployment and Self-Employment across Countries.” https://www.iza.org/publications/dp/12367/wage-employment-unemployment-and-self-employment-across-countries.

Profesia. 2022. “Covid19 Recruitment Report.” 2022. https://public.tableau.com/app/profile/profesia.analytics4840/viz/ProfesiaReport_V2/Covid?publish=yes.

Rios, Joseph A., Guangming Ling, Robert Pugh, Dovid Becker, and Adam Bacall. 2020. “Identifying Critical 21st-Century Skills for Workplace Success: A Content Analysis of Job Advertisements.” Educational Researcher 49 (2): 80–89. https://doi.org/10.3102/0013189X19890600.

Scheerder, Anique, Alexander van Deursen, and Jan van Dijk. 2017. “Determinants of Internet Skills, Uses and Outcomes. A Systematic Review of the Second- and Third-Level Digital Divide.” Telematics and Informatics 34 (8): 1607–24. https://doi.org/10.1016/j.tele.2017.07.007.

Simionescu, Mihaela, and Klaus F. Zimmermann. 2017. “Big Data and Unemployment Analysis.” 81. GLO Discussion Paper Series. GLO Discussion Paper Series. Global Labor Organization (GLO). https://ideas.repec.org/p/zbw/glodps/81.html.

Skhvediani, Angi, Sergey Sosnovskikh, Irina Rudskaia, and Tatiana Kudryavtseva. 2021. “Identification and Comparative Analysis of the Skills Structure of the Data Analyst Profession in Russia.” Journal of Education for Business 0 (0): 1–10. https://doi.org/10.1080/08832323.2021.1937018.

Smyk, Magdalena, Joanna Tyrowicz, and Lucas van der Velde. 2018. “A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator.” Sociological Methods & Research, July, 0049124118782538. https://doi.org/10.1177/0049124118782538.

Sodhi, M S, and B-G Son. 2010. “Content Analysis of OR Job Advertisements to Infer Required Skills.” Journal of the Operational Research Society 61 (9): 1315–27. https://doi.org/10.1057/jors.2009.80.

Sostero, Matteo, and Enrique Fernandez-Macias. 2021. “The Professional Lens: What Online Job Advertisements Can Say About Occupational Task Profiles.” JRC Working Papers on Labour, Education and Technology 2021–13. Joint Research Centre (Seville site). https://econpapers.repec.org/paper/iptlaedte/202113.htm.

Štefánik, Miroslav. 2012. “Internet Job Search Data as a Possible Source of Information on Skills Demand (with Results for Slovak University Graduates.” In , 258–72. Thessaloniki: CEDEFOP.

Štefánik, Miroslav, Štefan Lyócsa, and Matúš Bilka. 2022. “Using Online Job Vacancies to Predict Key Labour Market Indicators.” Social Science Computer Review. DOI: https://doi.org/10.1177/08944393221085705

Su, Zhi. 2014. “Chinese Online Unemployment-Related Searches and Macroeconomic Indicators.” Frontiers of Economics in China 9 (4): 573–605. https://doi.org/10.3868/s060-003-014-0027-3.

Tambe, Prasanna. 2014. “Big Data Investment, Skills, and Firm Value.” Management Science 60 (6): 1452–69.

Tambe, Prasanna, Lorin Hitt, Daniel Rock, and Erik Brynjolfsson. 2020. “Digital Capital and Superstar Firms.” Working Paper 28285. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w28285.

Tijdens, Kea. 2010. “Measuring Occupations in Web-Surveys: The WISCO Database of Occupations.” https://pure.uva.nl/ws/files/1491455/118626_1000_WP86_Tijdens_Measuring_occupations_WISCO_database.pdf.

Tijdens, Kea, and Stephanie Steinmetz. 2016. “Is the Web a Promising Tool for Data Collection in Developing Countries? An Analysis of the Sample Bias of 10 Web and Face-to-Face Surveys from Africa, Asia, and South America.” International Journal of Social Research Methodology 19 (4): 461–79. https://doi.org/10.1080/13645579.2015.1035875.

Turrell, Arthur, Bradley Speigner, David Copple, Jyldyz Djumalieva, and James Thurgood. 2021. “Is the UK’s Productivity Puzzle Mostly Driven by Occupational Mismatch? An Analysis Using Big Data on Job Vacancies.” Labour Economics 71 (August): 102013. https://doi.org/10.1016/j.labeco.2021.102013.

Turrell, Arthur, Bradley J. Speigner, Jyldyz Djumalieva, David Copple, and James Thurgood. 2019. “Transforming Naturally Occurring Text Data Into Economic Statistics: The Case of Online Job Vacancy Postings.” Working Paper 25837. Working Paper Series. National Bureau of Economic Research. https://doi.org/10.3386/w25837.

Vankevich, Alena, and Iryna Kalinouskaya. 2020. “Ensuring Sustainable Growth Based on the Artificial Intelligence Analysis and Forecast of In-Demand Skills.” E3S Web of Conferences 208: 03060. https://doi.org/10.1051/e3sconf/202020803060.

Varian, Hal R. 2014. “Big Data: New Tricks for Econometrics.” Journal of Economic Perspectives 28 (2): 3–28. https://doi.org/10.1257/jep.28.2.3.

Visintin, Stefano, Kea Tijdens, Stephanie Steinmetz, and Pablo de Pedraza. 2015. “Task Implementation Heterogeneity and Wage Dispersion.” IZA Journal of Labor Economics 4 (1): 20. https://doi.org/10.1186/s40172-015-0036-2.

Warschauer, Mark. 2003. “Demystifying the Digital Divide.” Scientific American 289 (2): 42–47.

Xu, Haoyu, Chongyang Gu, Han Zhou, Sengpan Kou, and Junjie Zhang. 2017. “JCTC: A Large Job Posting Corpus for Text Classification.” ArXiv:1705.06123 [Cs], June. http://arxiv.org/abs/1705.06123.

Zhu, Chen, Hengshu Zhu, Hui Xiong, Pengliang Ding, and Fang Xie. 2016. “Recruitment Market Trend Analysis with Sequential Latent Variable Models.” In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 383–92. KDD ’16. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/2939672.2939689.

Acknowledgements

We would like to thank Verónica Escudero, Hannah Liepmann and Janine Berg for their excellent comments on earlier versions of this working paper. The authors acknowledge that this publication has received financial support from ILO, and L. M. Kureková also acknowledges financial support from the VEGA 2/0079/21 project, provided by the Ministry of Education, Science and Sports of the Slovak Republic and the Slovak Academy of Sciences.