The primitive human occupation of its site dates back to prehistory, to a Neolithic period, later romanized, linked, it is believed, to the alluvial gold mining on the Ponsul River, practised until the end of the 20th century.

The medieval castle:
At the time of the Christian Reconquest of the Iberian Peninsula, it is accepted that the medieval castle was initiated by D. Sancho I, within the fortification policy he developed in Beira, facing the threats posed by the Lion Kingdom in the East, and the Muslims, To the south.
The domains of Penha Garcia and its castle were donated in 1220, by D. Afonso II (1211-1223), to the Order of Santiago, so that he would populate and defend them.
Penha Garcia received a Foral Charter, passed by D. Afonso III (1248-1279), on October 31, 1256. This document assured the residents of Penha Garcia, the customs of Penamacor.
D. Dinis (1279-1325) donated the domains of the village and his castle in 1303 to the Order of the Templars, in the person of his master in the country at the time, Vasco Fernandes. Faced with the extinction of the Order, they passed on to the Order of Christ.
In the sixteenth century, with the integration of military orders into the Crown, it returned to royal possession again. It is portrayed by Duarte de Armas, and the village received New Foral passed by D. Manuel I (1495-1521) in Santarém, on June 1, 1510.

From the seventeenth century to our days:
From the seventeenth century, its commendation passed to the counts of São Vicente da Beira.
It was the "couto" of the kingdom, or a place for outlaw people, until D. Maria I (1777-1816) extinguished them in 1790.
In the 19th century, with the extinction of the Council on November 6, 1836, the process of degradation of the castle began, aggravated by the activities of treasure hunters.
Patrimony not classified by the public power, the medieval castle currently has some sections of walls, in good condition (recently recovered), and a cannon in the village.