The essential question it boils down to is, are people responsible for the actions of their ancestors?
Should modern day Italians should feel eternal ignominy about the Roman Empire? After all, at least 25% of the population of the ancient Roman Empire were slaves. (Gladiators didn't go into the coliseum to fight lions of their own volition.)
Should today's Greeks be self-reproachful about the Greek empire? It's estimated that in Classical Athens, between 40 and 80% of the population were slaves. (It would be hard to find a Greek today who didn't have both slaves and slave owners in his ancestry.)
Both the Mongolians and the Han Chinese have, at different times, owned each other as slaves. (Nobody volunteers to become a court eunuch.) Perhaps both groups should both pay reparations to each other.
Slavery has been pretty much universal. According to Wikipedia:
Slavery remained a major institution in Russia until 1723, when Peter the Great converted the household slaves into house serfs. Russian agricultural slaves were formally converted into serfs earlier in 1679. Russia's more than 23 million privately held serfs were freed by the Emancipation reform of 1861. State-owned serfs were emancipated in 1866...
In Algiers, the capital of Algeria, captured Christians and Europeans were forced into slavery.... According to the Encyclopedia of African History, "It is estimated that by the 1890s the largest slave population of the world, about 2 million people, was concentrated in the territories of the Sokoto Caliphate [in today's Nigeria]. The use of slave labour was extensive, especially in agriculture." The Anti-Slavery Society estimated there were 2 million slaves in Ethiopia in the early 1930s out of an estimated population of 8 to 16 million...
The Aztecs had slaves. Other Amerindians, such as the Inca of the Andes, the Tupinamba of Brazil, the Creek of Georgia, and the Comanche of Texas, also owned slaves.
Yes, sub-Saharan blacks were sold by other sub-Saharan blacks to white slave traders who then transported them to the New World. And yes, it was a terrible injustice. But why is that the only slavery we ever hear about? Especially when there are estimated to be between 12 and 30 million people in slavery even today:
A report by the Walk Free Foundation in 2013 found India had the highest number of slaves, nearly 14 million, followed by China (2.9 million), Pakistan (2.1 million), Nigeria, Ethiopia, Russia, Thailand, Democratic Republic of Congo, Myanmar, and Bangladesh; while the countries with the highest of proportion of slaves were Mauritania, Haiti, Pakistan, India and Nepal.
Mauritania was actually the last country to outlaw slavery, in 1981, but they didn't actually criminalize it until 2007. Since then, though many slaves have escaped, only one owner has ever been prosecuted.
Other forms of slavery, such as forced marriage, are still widespread in parts of Asia and Africa and the Middle East as well.
Apart from the issue of slavery, the central question is, are we responsible for actions we didn't take?
Ted Bundy managed to father a daughter before he was executed. Is she somehow guilty because of his crimes? What Bundy did -- torture and kill at least 36 young women -- is unquestionably worse than owning slaves. And his daughter is only one generation removed. Does anyone suggest she bear some responsibility for his actions?
These days, guilt for what one's ancestors have done is a concept that's applied only selectively, in the service of certain groups who want to keep other groups on the defensive.