It seems that it was built on much older vestiges of a strong hold of the Duke of Waiffre, a duke of Aquitaine, during the war against Pepin the Short (King of the Franks) in the 8th century.
In the 13th century the castle belonged to the Roussillon family (sometimes written as Roussilhon), a modest family of horsemen, from whom the castle probably takes its name. It then became the property of the Cahors in bankers until the 14th century.
During the 100 year war between the French and the English, Raymond d’Antéjac, the lord of Roussillon, recognised the King of England’s rights to the French Crown and opened the castle to the English in 1355.
The consuls of Cahors paid the english to depart.