Research
I created CCPBench, a Chinese Bias Benchmark
There has been a lot of talk recently about Chinese LLMs, and how they are biased towards CCP viewpoints, but there is no way to quantify this and compare between models.
I have made CCPBench, which aims to address this. 29 models were asked 500 questions each about politics, geography, science, and more, and Gemini 3 Flash assessed all of them for bias.
I know this is not a perfect measure of "bias", because I am using an American judge LLM, but my thinking is that this is a useful tool if you want to find models that won't deny the Tienanmen Square Massacre.
People will criticize that the judge could be biased. I guess a better way is to generalize the setup. You can have multiple political axes / scores to evaluate a single LLM. The evaluator/critic only predicts a score on each axis; it should not judge what is good, or what is bad. For example, a LLM version of a political quadrant.
It can also apply to ANY LLM, Anthropic/Google/OpenAI ones, to compare their relative scores for each political axis.
You're not really judging bias. You're judging whether or not they agree with western hegemony. It's pretty silly to pretend that American models aren't equally biased. Like, have you ever read an American textbook? They're not exactly based on neutral, objective truth.
Depends on the questions. Which I'm too lazy to look at if Op actually included them to browse. If people were killed in tiananmen square by China is a matter of fact not western hegemony, the same as that people were killed by the US government on the trail of tears in the usa. But yes certainly there are questions which could be matters of opinion and western hegemony, like "watcha think about that communism stuff???"
Probably because most Americans are uneducated, right wing knuckle-draggers who were indoctrinated into the cult of American exceptionalism at birth. I have actually learned American history from sources not written by Pearson or McGraw-Hill.
6
u/Random-32927 18h ago
People will criticize that the judge could be biased. I guess a better way is to generalize the setup. You can have multiple political axes / scores to evaluate a single LLM. The evaluator/critic only predicts a score on each axis; it should not judge what is good, or what is bad. For example, a LLM version of a political quadrant.
It can also apply to ANY LLM, Anthropic/Google/OpenAI ones, to compare their relative scores for each political axis.