The castle was built on Roman remains in the 11th century for the lords of Durban, first mentioned in a document of 1018. The Lords of Durban were vassals of the viscounts of Narbonne. In the 12th century the powerful Durban family owned Leucate and Fabrezan as well as lands and rights over Fontjoncouse and Villeseque. The castle was the property of Bernard de Durban in the 12th century. In 1229, Guillaume de Durban swore allegiance to the king of France and thus kept his lands, despite his ancestors earlier links with the Cathars.
In the 16th century, influenced by Renaissance architecture, the north and west walls were pierced and decorated with magnificent windows, with marble columns and sculpted lintels, which were restored in 1972. The last direct descendant of the Durbans to own the castle was Joseph de Gleon, Baron of Durban, who died aged 82 without heirs in 1787.
The last inheritor of the castle sold it in 1873 to Paul Combes, who intended to use the stone as building material and had destroyed large parts of the structure by the end of the 19th century. It was abandoned and then occupied by a Spanish soldier, who lived in the one room remaining. The castle subsequently became public property and continued to be used as a source of stone. The only remains today are the north face of the main building, a round staircase tower and a square tower.
In 1967, archaeological excavations were carried out. In 1980, efforts were made to preserve the remains and in 1989 the castle was acquired by the commune. Today, the castle belongs to the commune and is cared for by the Comite de Sauvegarde du Vieux Durban (Committee for the Safeguarding of Old Durban). It has been listed since 1926 as a monument historique by the French Ministry of Culture.