Tech-driven, inclusive, and trustworthy: Envisioning the future of national census agencies
To adapt to technological and societal shifts, national statistics agencies are transforming operations and practices to provide faster, better, and real-time population insights
Mahesh Kelkar
Adithi Pandit
Randy Turkel
William D. Eggers
Glynis Rodrigues
In 2012, as Hurricane Sandy hurtled toward the New Jersey coast, the state urgently needed accurate local information to devise evacuation, shelter, and rescue plans. After the storm, agencies needed additional detailed information to rebuild communities, infrastructure, and businesses. A data-sharing program between the state and the federal government helped make this possible.1
As a part of this program, the US Census Bureau had access to state-collected data on local businesses and jobs. This “administrative” data from the state included information about the status of commercial enterprises—when they started, shut down, or added or lost employees. The Census Bureau also tapped into localized data collected through its American Community Survey to help identify the number of people living in disaster areas, where they work, the type of businesses in those areas, demographic details of residents, and the languages they speak.2
By linking these federal and state data sets, New Jersey developed a new mapping tool, OnTheMap for Emergency Management, to direct aid and help local businesses and communities quickly recover.3
The Hurricane Sandy emergency response is just one example of how invaluable census data can be for government leaders, policymakers, researchers, academia, and businesses. Central statistical agencies, like the Census Bureau, are critical in collecting population insights beyond the decennial or five-year surveys they publish. These agencies create hundreds of data products weekly, monthly, quarterly, and annually.
Change is afoot, both at a societal and technological level, which could shape the future of census agencies in the coming decades. Nationwide censuses are being conducted in different societal and technological contexts than in previous decades. Many countries have moved to a primarily digital-first census survey, with in-person surveys conducted mainly in hard-to-reach communities.
At the same time, data availability across government has exploded, and data governance guidelines have matured. Gone are the days when government data was often unstructured and inaccessible due to government data silos.4 With the growing ubiquity of digital systems, data has been freed from its traditional jurisdictional confines. More and more data is available across government systems in machine-readable formats that can be integrated and ingested into large data analysis platforms. This can give census organizations a better starting point to get population insights beyond traditional surveys.
Beyond data ubiquity, advances in digital technologies like geographic information systems, remote sensing, and artificial intelligence are upending census operations. Getting the most value from these technologies requires census agencies to redefine processes and operational systems to reflect the changing technological and digital landscape better.
Increasingly declining public trust and the growing equity imperative will continue to challenge census agencies to ensure a complete population survey or insights into hard-to-reach communities. This could be seen in voluntary response rates on the decennial census across geographies. With each decennial census exercise, the voluntary response rate from citizens—both online and offline—has declined.5 There are many possible reasons for this decline, ranging from rising distrust in government institutions to survey fatigue to rising privacy concerns in individuals and businesses.6
Some of these shifts, such as advances in digital technology and data ubiquity, can be transformational for national statistics agencies, while other shifts, like declining public trust and a growing equity imperative, may challenge census activities altogether as they strive to provide population insights faster, better, and in almost real time. (See “My take: Getting future-ready for the 2030 Decennial Census.”) National census agencies should adapt to these changes to remain effective and relevant. Not understanding these paradigm shifts could result in inaccurate population insights, which could, in turn, lead to less effective policymaking.
An era of tech-infused census
Advances in technology and data analysis will play an important role in transforming census counts in 2030 and beyond. Although many countries have transitioned to an online census over the past two decades, the decennial or five-year census counts are still a labor-intensive, “knock-on-doors” activity in many countries, and understandably so. There is a growing public distrust in some communities that hinders voluntary survey intake, and data suggests that voluntary survey responses have declined over the last few decades.7 Moreover, challenges persist regarding access to the internet or digital devices and the ability of some citizens to complete a survey online. Additionally, census enumeration and normalization of the address records are still largely manual exercises.
Census enumeration is a critical clerical task that precedes any big decennial census exercise to ensure that the addresses on record match the current dwelling status of households. Census enumerators sometimes have to physically go to the field to check addresses and compare them with the records, adding new real estate developments and deleting records of old demolished dwellings. This process is highly time-consuming and requires a significant workforce to normalize address records. For instance, the US Census Bureau uses lengthy administrative data modeling methods to support its data collection operations.8
However, advances in geographic information systems and remote sensing technologies can help census agencies create accurate and up-to-date maps of large enumeration areas. These technologies can help census organizations analyze and visualize changing spatial patterns in data such as population density, urban or semiurban sprawl, changes in physical structures, and more. For instance, by comparing snapshots of maps from different years, enumerators can identify demolished dwellings, newly developed homes, recent physical infrastructure like schools and hospitals, and new businesses opened between the two snapshots. This can allow census organizations to identify geographic areas they may need to target during physical enumeration activity.9
Meanwhile, AI and machine learning tools can further automate the overall census process. As census agencies move toward more administrative data to improve operations, machine learning models can compare data across a wide array of datasets, automate data-processing, identify data inconsistencies, and integrate data quickly with high accuracy. This involves merging data snippets from different datasets like tax records, education, and health data to create a partial or complete picture of an individual or household. Automating such tasks can help reduce high-volume but low-value manual tasks and reduce error rates in data-processing.10
Machine learning algorithms can analyze large volumes of census data and identify patterns, trends, and anomalies that may be difficult to detect using traditional statistical methods. For instance, census agencies can use machine learning to understand representation in ongoing surveys and allocate resources more efficiently to ensure better coverage of hard-to-reach communities.11 Similarly, AI can analyze survey responses with other administrative datasets to check for anomalies, errors, or deliberate incorrect data inputs from citizens. Additionally, AI and machine learning can be used to develop predictive models based on historical census data, helping to forecast population growth, migration patterns, and other demographic trends.12 (See “My take: Technology can help reimagine and improve census operations.”)
The emergence of generative AI offers new opportunities for making census data accessible to citizens, communities, and research organizations. A primary prerequisite for a generative AI application is the availability of a high volume of training data—something census agencies have in spades. Generative AI solutions can help democratize census datasets for the public and non-technical users by allowing them to retrieve knowledge through simple natural language questions (figure 1). Census agencies may harbor concerns about losing control over population data and ensuring data privacy and security. However, some of those apprehensions can be addressed with advances in generative AI solutions and more secure cloud environments.13
National population data as a knitwork of administrative data sets
Government agencies have always collected data specific to their mission needs and used it to make critical decisions and shape policy. Large swathes of such data are available across departments and levels of government. With advances in data governance processes, many new administrative data sets can now be standardized, anonymized, and made machine-readable—a prerequisite for deeper integration and analysis. (See “My take: Transforming the US Census Bureau into a data-centric organization.)
Using administrative data as the primary source for national population statistics is not a new concept. Nordic countries were pioneers in using administrative data or register-based census, which refers to data obtained from various registers or administrative sources like the population register, building or dwelling register, tax register, social security register, business register, and more.14 The data in these registers can be linked, generally through a personal identification number, to generate population insights.15
Today, more countries are considering using administrative data as the starting point for generating national population statistics and not as a replacement for traditional surveys. “Many countries will be using administrative data by 2030 census round. That’s not to say they’ll necessarily be using it as census data, they might use it as a supportive means of building their address register and sampling frames, and so on,” expressed Fiona Willis-Núñez, a statistician at the United Nations Economic Commission for Europe.16
There are likely a few reasons for this shift in many national statistics agencies:
- The availability of data and the technology needed to create such a data infrastructure is now often more accessible.
- There is growing survey fatigue among residents, and such administrative data can allow agencies to reduce the number of questions and surveys.
- National population survey costs have increased significantly over the years. For instance, New Zealand spent nearly US$200 million on its 2023 five-year census survey,17 while Scotland spent US$176 million on its decennial census in 2022,18 and England and Wales spent US$1.1 billion.19 In the United States, the 2020 census will have cost roughly US$13.7 billion by the time its activity ends in 2024.20
“Many countries will be using administrative data by 2030 census round. That’s not to say they’ll necessarily be using it as census data, they might use it as a supportive means of building their address register and sampling frames, and so on,” expressed Fiona Willis-Núñez, a statistician at the United Nations Economic Commission for Europe.
The advances in administrative data integration capabilities are a first step toward supporting and supplementing traditional census surveys. However, administrative data may be insufficient in providing information about ethnicity, physical attributes of a household, or personal opinions and choices. “So everything around ethnicity is very, very valuable because the census enables people to provide granular information about their ethnicity,” says Jonathan Wroth-Smith, Director of 2022 Census Statistics, National Records Scotland. “Administrative data may not be able to provide that level of detail on ethnicity but, of course is much more timely. This represents some of the challenges involved in balancing frequency and fidelity,” adds Jonathan.21
National statistical agencies acknowledge that gaps exist in such data platforms, particularly affecting those without digital access or footprint. Such discrepancies could lead to an overrepresentation of individuals or businesses who frequently interact with the government digitally while underrepresenting constituents who don’t. This is where surveys become essential and valuable in ensuring that census data is representative and without bias.
“Administrative data may not be able to provide that level of detail on ethnicity but, of course is much more timely. This represents some of the challenges involved in balancing frequency and fidelity,” says Jonathan Wroth-Smith, Director of 2022 Census Statistics, National Records Scotland.
Through administrative data, census agencies aim to bring more specificity to their data collection efforts: use the available data to reduce survey burden and survey hard-to-reach communities and cohorts without digital access. In New Zealand, a similar initiative is underway to bring together varied data in government silos into a single platform called the Integrated Data Infrastructure. (See “My take: Moving away from a one-size-fits-all census survey.”)
Evolving a security-by-design mindset
A census database is one of the most comprehensive records of citizens in a country, making it a valuable national resource and, at the same time, very vulnerable to significant security and privacy risks if it falls into the wrong hands.
Census agencies are working to build confidence in their data protection methods, which can help them collect data more effectively and gain public trust in the census process. “We should talk about privacy measures, security safeguards, and the ethical side of the census to give citizens more confidence,” says Mark Sowden, chief executive government statistician of Stats New Zealand.22
Many national census agencies adhere to a set of policy principles to safeguard and protect data and are also governed by law:
- Confidentiality: In many countries, census data is encrypted and stored in a private cloud managed by government, and only authorized personnel within government can access it. Physical forms stored in a secure government facility are under constant video surveillance. Once the documents are digitized, they are destroyed to ensure complete protection of the information.23
- Data access restrictions and non-release clauses: Although census data is made accessible to the public, many organizations are legally prohibited from providing any personally identifiable information. Many census organizations keep data secure and confidential for about 100 years, and there are penalties for disclosing personal data without lawful authorization.24
- Encryption and anonymization: Census agencies have multilayered security systems, encryption facilities, and processes in place to avoid data breaches. Many census agencies also anonymize data by removing or modifying personally identifiable information.
That said, data hackers and technological advances are sometimes ahead of the data protection and security curve. For instance, in 2019, the US Census Bureau acknowledged that privacy controls used in the 2010 census were not robust enough. Internal tests revealed that census officials could match census data with publicly available information and commercial datasets from social media companies to get the accurate age, gender, location, race, and ethnicity information of one in six Americans.25
“We should talk about privacy measures, security safeguards, and the ethical side of the census to give citizens more confidence,” says Mark Sowden, chief executive government statistician of Stats New Zealand.
The growing ubiquity of data systems and the advent of AI are expected to make data security and privacy increasingly challenging for census agencies. Traditional techniques and protocols may not be strong enough anymore. Census agencies should continuously evolve their data protection methods and tools to stay ahead of hackers and malicious actors. Census agencies could experiment with a new evolving class of techniques, such as privacy-enhancing computation, to protect sensitive information.26
Privacy-enhancing computation is an umbrella term for emerging privacy protection techniques, including cryptographic methods, homomorphic encryption, zero-knowledge proofs, secure multiparty computation, and differential privacy.27 These techniques can be used when multiple parties contribute confidential data and collaborate on tasks.
The US Census Bureau started implementing a privacy-enhancing computation technique called differential privacy in 2020 to protect identities and personal information. The technique involves intentionally injecting “noise” into the raw data so that there is a variance from the actual data.28 For instance, the total population of each state will be the actual number, but all other levels of geography, including census blocks, townships, and congressional districts, could have some variance from the raw data. However, these techniques may create data inconsistencies for rural areas, small subpopulations, smaller states, and longitudinal studies. Census organizations must strike the right balance between accuracy and risk of disclosure to ensure data protection and to build trust in the census process.29
Leading with trust in a fractured public trust environment
Public trust has been declining in many countries for years, according to the annual Edelman Trust Barometer.30 Although government interactions with citizens play a big role in building greater public trust, they might not be the only factor driving it down. The Edelman Trust Barometer states that disillusionment with rising inequality can undermine public trust.31 Moreover, there is a growing disparity in trust between the informed and the general populace, partly fueled by the rampant growth in misinformation or disinformation. These external factors dent public confidence in government institutional processes, systems, data-collection initiatives, and much more.32
One outcome of this declining trust is that people are less willing to share personal data with the government, including for census surveys. Deloitte’s Global Citizen Survey, conducted across 13 countries to gauge the public’s perceptions of government digital services, indicates the close relationship between trust and a willingness to share data. For instance, the analysis suggests that citizens who trust the government are twice as likely to share personal data and 1.9 times more likely to allow interagency data-sharing than citizens who report low levels of trust in government (figure 2).33
National statistics agencies produce a range of statistics for different government domains and population groups—the decennial or five-year census population count is just one of these projects. The success of these initiatives hinges in part on residents’ responding to census counts (constitutionally mandated in some countries) as well as other vital population and economic surveys.
Active engagement and communication with communities and community leaders and influencers are crucial, especially with communities that are hard to reach. For example, during the 2020 census, the US Census Bureau established a unique Trust and Safety team. The team’s main focus was to combat disinformation and misinformation about the 2020 census that could potentially harm response rates. This was achieved by working closely with government agencies, fact-check organizations, civil society organizations, and technology companies (figure 3). The Census Bureau’s efforts in building trust were important in helping drive participation and achieving a 99.9% response rate in the 2020 count.34
Overcoming the trust deficit with certain population cohorts or hard-to-reach communities requires a broader approach to building trust with these communities, including tapping into and working with trusted networks and partners in these communities. (See “My take: Tapping into networks of trust.”)
Showcasing the value of census
Filling out a census form can be a time-consuming task. Households may have to carve out 20 to 30 minutes, depending on the number of household members, to fill out lengthy surveys for which they might not see immediate benefits. In many countries, residents participate in the census due to a constitutional mandate and to avoid hefty penalties and fines for not responding to the survey.
Census agencies seek to increase awareness and motivate people and businesses to care about the census rather than simply enforcing it as a legal measure. One way to nudge people to share their data is to demonstrate the value that their data generates. Understanding an individual’s or community’s unique needs or requirements can help make a better case for census. For low-income populations, it may be financial aid or support benefits. People with disabilities need accessible services for a healthy lifestyle. For young parents, services for their children in the future can strike a chord.
The census data has an economic impact on a wide range of areas, including land-use designs, transportation planning, and infrastructure investment decisions, among others. As a key part of the country’s overall statistical infrastructure, the census has a ripple effect on other surveys. For example, the Labor Force Survey in Canada, which calculates the monthly employment and unemployment rate to make job creation decisions, is benchmarked to the census.36 Every five years, when the census is conducted, the Labor Force Survey gets rebased on the new population estimate recorded in the census. These inputs are critical for shaping labor market policies and reimagining unemployment services in the region.37
Government statistical agencies should keep demonstrating and reiterating the value of census data to individuals, communities, businesses, and other government agencies. Penny Pritzker, former secretary of commerce, noted in the 2014 US Census Bureau newsletter, “Few of us think about it, but Commerce Department data touch and benefit all Americans daily … whether through our cellphones, the weather report, or the economic and demographic characteristics of our nation and communities.” She added, “Yet, too much valuable data may fly under our radars,” indicating the need for greater awareness and usage of such data.
People who understand how census data directly affects their daily lives are more likely to complete it on time. For example, remote island areas such as Orkney and Shetland were the quickest to respond to the recent census survey in Scotland. They had the highest response rate overall because they understood the importance of that data for their communities. Over the years, they have realized that the census data is critical for funding local infrastructure projects and other services like schools and hospitals.38
Designing for equity
Census agencies strive to gather comprehensive data on every individual in the country, especially those traditionally undercounted or excluded in the past. Some of these groups include individuals and communities who live in rural or remote areas, the aging population, people experiencing homelessness, immigrants who face language barriers, Indigenous communities, racially and ethnically diverse people, and the LGBTQIA+ community.39
Census agencies are employing various strategies to obtain more diverse and inclusive data to ensure the representation of marginalized communities in policymaking. For instance, in Canada, one in four people counted in the 2021 census were immigrants. Since their first language is often not English or French, Statistics Canada translated census questions into 25 languages, including 13 Indigenous languages.40
In the 2020 census, the Census Bureau made a concerted effort to ensure that it counted people experiencing homelessness. To achieve this, the bureau partnered with local groups to identify locations where people were known to sleep.41 The operation involved specially trained census takers who counted people in shelters, soup kitchens, and mobile food van stops and those who lived outdoors or at transit stations—essentially, any location where homeless individuals were known to reside.42
Community-driven census could hold the key to a better count
Many hard-to-reach communities may not feel the need to participate in census surveys since they may not be aware of the importance of the census.43 Also, a history of exclusion, unfair treatment, and discrimination can lead to trust deficits. Census agencies can alleviate the lack of trust and apprehension in these groups by working with networks of individuals from these communities. By doing so, census agencies can develop a more locally driven model that taps into the existing trust within these networks.
Statistics New Zealand has leveraged trust networks to help improve the census experience and participation of Indigenous communities. The agency deployed a Māori-first marketing campaign to ensure census messaging reached these communities nationwide. It hired twice the number of engagement staff, with more than half belonging to the Indigenous community, to work locally with Māori people. It partnered with organizations such as Te Matatini Society Inc. that fosters and protects Māori performing arts to spread awareness of census activity. Besides a fully bilingual census website and forms, other kinds of support (including call centers) were available in the Māori language.44
The count is a first step toward systemic inclusion
Although understanding and acceptance of gender diversity and sexual orientation have grown over time, making it a part of the census exercise is important to acknowledge the evolving social and demographic landscape. In the 2021 census, the UK Office of National Statistics included sexuality and gender identity questions for the first time in the country. The question was voluntary for individuals above the age of 16.45
To address privacy concerns, household members could ask for a separate questionnaire to ensure their answers remained confidential.46 “The data could help decision-makers understand the extent and nature of disadvantage which people may be experiencing in terms of educational outcomes, health, employment, and housing,” said Jen Woolford, director of the Office of National Statistics.47 Including the community in the count is a first step toward greater social inclusion and helping them feel more seen and supported.
The US Census Bureau collects data on same-sex couples through the American Community Survey’s Household Pulse Surveys. The 2019 survey counted nearly a million same-sex couple households—the data showed that 58% were married. The District of Columbia, Delaware, Oregon, Massachusetts, and Washington state had the highest percentage of same-sex household couples.48 This level of detailed data helps to deepen the understanding of family composition, characteristics, and economic circumstances. Similarly, the Australian Bureau of Statistics also tracks same-sex marriage and plans to include new questions about gender and sexual orientation in the 2026 national census.49
Moving forward
Census agencies globally are on the brink of a major transformation in collecting, analyzing, and sharing data. Technological advancement—especially in AI—coupled with progressively improving data integration capabilities, could further democratize data in the coming decade. However, the primary challenge lies in maintaining data integrity—ensuring representation, reducing bias, and enhancing data security and privacy. Rebuilding trust in individuals and communities will help ensure census agencies can deliver on their mission.
- Lead with trust: A country’s baseline public trust can impact the trust in government institutions like census agencies. However, this agency-level trust can still be strengthened irrespective of the overall trust in government. Focusing on data security and privacy, building systemic transparency on the collected data, and improving civic participation and engagement can help census agencies rebuild trust in their activities. Census agencies can tap into trust networks, including community organizations and influencers, to rebuild trust in certain communities.
- Broaden the scope of administrative data: Integrating data across government agencies and levels could be a step toward more holistic population insights. However, there may be an opportunity to broaden the scope by integrating private sector datasets, including social media, banking, and retail transactions. This would require agencies to partner and develop agreements with such institutions. It can also provide opportunities to monetize data beyond research institutions and policymaking to drive business and economic value.
- Enhance data protection for digital connections: Deloitte’s digital government and citizen experience survey finds that concerns about privacy and security of personal data are some of the top challenges constituents face in accessing digital government services.50 Census agencies should not only work to bolster data security but also build cybersecurity skills internally to improve the protection and security of data.
- Use technology to catalyze data transparency: Census agencies should explore generative AI solutions to help democratize census datasets for the general public and non-technical users by allowing them to retrieve knowledge through simple natural language questions.
