The history of the Villa has for centuries been linked with the ancient family of Altoviti, one of the oldest noble families in Florence. The origins of the family are uncertain and the subject of controversy among scholars: in excavations on Mount Fiesole a Roman inscription was also found that mentions a certain "Furio Cammillo Altovita", grandson of Furio Cammillo, leading to the assumption of Roman origins.

If we go by the coat of arms the flayed wolf that appears on one of the chimneys of the ground floor of the Villa the first traces are from the end of the 10th century and allow us to go back to a certain Longobardo di Corbizzo who in 1192 lived in Borgo Santi Apostoli in Florence and had seven children, including one named Altovita. The latter may have been the founder of the house and was appointed Knight by Emperor Federico II, perhaps with the symbol of the wolf that symbolised daring captains.

The Dominican friar Jacopo Altoviti was bishop of Fiesole in late 1300 and became the protagonist of a series of constructions and expansions (the Bishop's Palace, the Cathedral and the Convent of San Domenico), so it is easy to still see the distinctive Altoviti emblem on these monuments and even in several smaller oratories and chapels. His descendant Neri Altoviti was bishop and in 1637 founded the Episcopal Seminary of Fiesole. In Florence they had many houses in the area of Borgo Santi Apostoli and held the patronage of the nearby church of the Holy Apostles.