Legend suggests that Visigoth Wamba (672-680) erected a castle on this site sometime during the 7th century, but archaeological excavations don't confirm this perspective.
The construction of the current structure and watchtower was erected between 11th and 12th century, likely over the pre-existing castro through the initiative of the Knights Templar: the tower erected could have been either a watchtower or keep tower. In 1199, the Herdade da Açafa was donated by King D. Sancho to the Templars.
By 1505, circling walls had already been greatly destroyed.
By the middle of the 18th century, the castle was used as an artillery spot for the protection of the Tejo river passage. The battlements and artillery posts were constructed on the flank overlooking the Tagus, in the area referred to as Batarias.
At the beginning of the 19th century, the castle was reconstructed, during the preparation for the first French invasion of the peninsula, during the Napoleonic Wars, through the initiative of the Marques de Alorna.