Racial categories like ‘Black’ and ‘White’ mean different things in different countries. I show that these categories can arise endogenously in a model of worker-elite conflict in the presence of slavery. When elites attempt to extract surplus from workers, they face the risk of a revolt removing them from power. If free and enslaved workers can find common cause, this threat is stronger than when they are divided. Promises that individuals carrying certain physical traits are protected from harsher exploitation reduces the amount of surplus elites must offer to buy the support of free workers against slave revolts. Thus, racial categories reflect the coalitions the elites needed to stay in power in different colonial circumstances.