This 10th-century castle was constructed by the Bonacolsi family, the leading family of Mantua before Gonzaga. Later, the castle was used by the Gonzaga and Scaligeri families. The fame of the castle is also linked to the history of Modena.

In 1311, Francesco I Pico was designated by the emperor to manage the city of Modena. After only one year, he was hated by both the two major factions of the city (Guelph and Ghibelline) and was betrayed with the help of the Bolognese army. He was able to escape the ambush and went back to Modena. From that day, legend says that he took revenge on his enemies day by day, one by one. 

Because of the violent revenge, the people of Modena asked Rinaldo Bonacolsi, the nobleman in charge in Mantua, to intervene. Eventually, Bonacolsi took Francesco Pico and his two sons, got undressed them, and tied them on donkeys allowing people to beat and hit them with rocks on the way to Castel d'Ario. They were locked in the main tower, where they starved to death because they were left without food or water.

During an excavation of the main tower in 1851, seven skeletons were uncovered, one of which was still in stocks. Four of the corpses were identified as belonging to the Bonacolsi family. The tower is often called the Torre della Fame, or "Tower of Hunger," because of the horrific imprisonment it was used for.