In the United States, leading HR software company Workday is currently facing a lawsuit over its use of job screening tools powered by AI, which allegedly discriminated against applicants based on factors such as age, disability, and race.
The company, whose hiring software is widely used by large employers around the world, has denied the allegations.
The case is one of many examples of AI systems alleged to have caused discriminatory harm. When AI systems replicate and amplify discrimination, they blur the boundary between technical error and systemic injustice, turning bias into a digital harm.
And while our first instinct might be to blame the algorithms, they don’t decide what they can generate, what safeguards are built into them, or how a company responds when incidents of discrimination are reported. People make those calls long before an AI produces any output.
That is why technical fixes to AI systems are not enough. What is needed is an overhaul of AI ecosystems to ensure they are more inclusive.
A broader pattern
AI systems quietly narrow who gets seen as competent, employable, or fit to lead.
For example, in a 2025 study, we tested how two AI models, OpenAI’s GPT-4 (which has now been retired) and Microsoft Copilot, represented software engineers in a simulated recruitment exercise: 300 candidate profiles for four job roles, followed by recommendations and generated images of each AI model’s preferred candidates.
Both models favored male profiles, especially for senior roles. Their images also skewed towards engineers who were younger, slimmer, and lighter-skinned. The models were reproducing associations embedded in language, imagery, employment records, and assumptions about who belongs in the profession.
These outputs don’t stay contained to a research study. AI-generated recommendations are entering hiring, education, and public services.
This matters when certain demographics and women remain underrepresented in AI development and leadership, while being disproportionately exposed to its harms.
The problem is not limited to gender and race.
Even when AI systems operate across different languages and cultures, they often reproduce predominantly Western values, assumptions, and ways of understanding the world. Wealthy countries have become the main beneficiaries of AI, which widens global inequality.
In another study from 2025, we manually reviewed reported AI incidents.
Almost half involved a diversity or inclusion issue, with racial, gender, and age discrimination most prominent. The harms traced back to different points in the AI development lifecycle: non-diverse training data and neglected diversity and inclusion principles during design, development, and deployment.
Why technical fixes are not enough
Technical work matters, including bias identification, rebalancing datasets, and adjusting outputs. But these fixes often treat bias as a property of the AI model, when much of it originates from outside the system.
Data does not enter an AI system as a neutral record of reality.
People decide what data to collect, how to label and categorize it, and whose experiences are important. These decisions are shaped by history, cultural norms, institutions, and existing power imbalances.
Wherever society has linked leadership with men, technical skill with lighter skin, or innovation with youth, AI models learn from those associations and formalize, automate, and repeat them at a larger scale.
Bias also usually appears through the intersection of multiple identities, such as gender, race, age, disability, and class. A system that looks fair when each identity is tested separately can still disadvantage people at the overlap of several.
Building a more inclusive AI ecosystem
That’s why building a more inclusive AI ecosystem requires interdisciplinary knowledge, such as educating AI engineers about social science theories to help them understand the social origin of bias.
Inclusive AI is not about political correctness; it is about upholding human rights, preventing harm, ensuring justice, and building trust.
It also requires genuine participation from affected groups and sustained attention to the power structures these systems operate within. AI development teams should test not just whether a model is accurate, but whether its benefits, errors, and harms distribute fairly across different groups.
Together, this would help ensure tech companies better understand the nature of a bias once it’s manifested through AI and therefore develop new methods or tools to minimize the harm it causes.
Organizations that adopt AI also need stronger governance to monitor how the technology behaves. This could include, for example, having someone accountable for reviewing risk and responding to incidents and monitoring systems once they are live.
Algorithms don’t decide which data matter or what level of risk is acceptable. People make these choices. It’s high time tech companies remember that. The focus should not just be on fixing a biased algorithm, but rather on examining the human decisions that allowed the risk of harm, and who was missing when those decisions were made.
Muneera Bano, Principal Research Scientist, CSIRO and Didar Zowghi, Professor, Senior Principal Research Scientist, CSIRO
This article is republished from The Conversation under a Creative Commons license. Read the original article.
Follow us on X, Facebook, or Pinterest
