When we talk about aligning AI with human values, there is an assumption hidden inside that phrase: that there is some well-defined set of human values for an AI system to align with.
I find that assumption increasingly difficult to accept.
Humans disagree about almost everything that matters. We disagree about politics, morality, fairness, religion, freedom, equality, punishment, privacy, and even what it means to live a good life. These disagreements are not necessarily mistakes that disappear once everyone has enough information. Reasonable people can look at the same facts and still reach very different conclusions.
So what exactly should it mean for an AI system to be "aligned with human values"?
One tempting answer is to collect enough human feedback, combine everyone's preferences into some aggregate objective, and train the model to behave according to it. This idea sits behind many preference-based alignment methods, including Reinforcement Learning from Human Feedback (RLHF).
But there is a problem even before we ask how those preferences should be aggregated:
Who are the humans providing the feedback?
The people teaching AI are not necessarily "the public"
In practice, obtaining feedback from everyone who might eventually use an AI system is impossible. Developers instead rely on pools of human raters who compare different model responses and indicate which ones they prefer.
But those raters are a sample of society, and not necessarily a representative one.
Imagine that an AI system is intended to serve millions of people, while the preferences used to align it come disproportionately from a particular demographic or social group. The resulting model might be very good at learning the preferences in its training data while still failing to represent the population it is supposed to serve.
This made me realize that pluralistic alignment is not only a question of what values we teach AI. It is also a question of who gets to participate in teaching them.
And that led me to one of the first approaches I explored: sortition.
My first approach: let a representative mini-public teach the AI
Sortition has a surprisingly old history.
Instead of electing representatives, sortition selects ordinary citizens through a lottery. Modern citizens' assemblies use versions of this idea to construct what are sometimes called mini-publics: relatively small groups selected to resemble the broader population across characteristics such as age, gender, geography, or other relevant dimensions.
I became interested in a simple question:
What if we applied the same principle to AI alignment?
Suppose we have preference data from thousands of people, but the people in that dataset do not accurately represent the population our AI system is supposed to serve. Rather than treating every observation as if it came from a perfectly representative population, we could explicitly correct for who is represented.
In our work on sortition-weighted preference learning, we explored two ways of doing this.
The first, which we call Hard Panel, is closest to a citizens' assembly. We use algorithmic sortition to select a smaller panel of raters whose demographic composition satisfies representation targets, and train the AI only from the preferences of this mini-public.
The second, Soft Panel, keeps everyone's preference data but changes how much influence each person's feedback receives. People are weighted according to their probability of being selected into the representative panel.
Conceptually, I like the distinction because it gives us two different answers to the same problem. We can either construct a representative public and listen to that public, or keep the larger dataset while correcting the influence different groups have on the learning process.
When we tested these ideas using real preference data and fine-tuned language models, the results were encouraging, although not universal. The representative Hard Panel performed best in one of our main experimental settings, while Soft Panel was more useful in settings where restricting ourselves to a small representative panel meant throwing away too much high-quality data.
That qualification matters.
I do not think sortition "solves" pluralistic alignment. What these experiments convinced me of is something more basic: the composition of the people providing feedback actually matters.
And importantly, it is something we can make explicit.
Instead of saying that a model is aligned with "human preferences," we can ask a much more concrete set of questions: Which humans? From what population? How were they selected? Who was underrepresented? How much influence did each group have?
Those questions make alignment more auditable—and, in my view, more honest.
Representation doesn't solve disagreement
But even if sortition gave us a perfectly representative miniature version of society, we would still have another problem.
The people inside that representative panel would disagree.
Suppose 60% prefer one behavior and 40% strongly prefer another. Should the AI simply follow the majority? What if the minority has a much stronger preference? What happens when there are many competing groups rather than two? And what if the same minority repeatedly loses across thousands of decisions?
Representation tells us whose preferences should enter the process.
It does not tell us how conflicting preferences should become a collective decision.
This is where I think social choice theory becomes particularly relevant to AI. For centuries, economists, mathematicians, and political theorists have studied essentially the same problem: how can the preferences of many individuals be transformed into collective decisions?
AI alignment is beginning to encounter its own version of this old problem.
Should AI even have one value system?
There is an even more fundamental possibility.
Maybe aggregation itself is sometimes the wrong goal.
Imagine asking an AI about a genuinely controversial social question. Instead of producing whichever answer happens to win under its preference-aggregation procedure, perhaps a well-aligned system should understand that legitimate disagreement exists. It could recognize competing perspectives, explain where they agree and disagree, and respond appropriately to context without pretending that humanity has reached a consensus that does not exist.
In that sense, pluralism is not simply about making AI more "balanced."
It is about making AI's representation of humanity more accurate.
Human society is pluralistic. An AI system claiming to represent human values should somehow reflect that fact.
But pluralism creates its own problems
Pluralistic alignment is not a magic solution.
Some preferences are mutually incompatible. Some people may want AI systems to behave in ways that harm others. Personalized alignment can conflict with collective welfare. And treating every expressed preference as equally legitimate can itself lead to deeply undesirable outcomes.
At some point, an AI system must draw boundaries.
And then we encounter perhaps the hardest question in pluralistic alignment:
Who gets to decide those boundaries?
There is no purely technical answer. Better algorithms can help us represent preferences, measure disagreement, aggregate information, and understand trade-offs. But mathematics cannot independently tell society which values deserve protection or which compromises are morally legitimate.
That realization has changed how I think about alignment itself.
Alignment is a collective-choice problem
I increasingly think that AI alignment is not only a machine-learning problem. It is also a problem of collective decision-making.
As AI systems become more capable and are deployed across education, government, healthcare, workplaces, and everyday life, the question will not simply be whether these systems follow our values.
We first have to answer a much more uncomfortable question:
Who are "we"?
My work with sortition was one attempt at making that "we" more concrete. Instead of treating humanity as an abstract source of preferences, we can explicitly define the population an AI system is meant to serve and think carefully about how that population should be represented in the feedback used to train it.
But representation is only one piece of the puzzle.
We still need to understand how disagreements should be aggregated, when AI should preserve disagreement rather than resolve it, how minority preferences should be protected, and where the boundaries of acceptable pluralism should lie.
Pluralistic alignment does not make these questions disappear. If anything, it forces us to confront them directly.
And I think that is precisely its value.
Instead of pretending that humanity possesses a single coherent objective waiting to be learned, we can begin with the world as it actually exists: billions of people holding overlapping, conflicting, evolving, and sometimes irreconcilable values.
The challenge is not to erase those differences.
The challenge is to build AI systems that can live with them.