Scotland's oldest free public lending library, the library and school at Innerpeffray were founded by David Drummond, 3rd Lord Madertie around 1680.

Madertie was a member of the Drummond Family, one of the most important landowning families of the area, friend and brother-in-law to James Graham, First Marquis of Montrose. The original library was "partly in the west end of the chapel of Innerpeffray and partly in that little new house" which Madertie built. Making books available to ordinary people free of charge was unprecedented. Madertie wished the library and school to benefit the community "in time coming" and leaving them with a legacy of 5,000 Scottish Merks, he charged his successors with the responsibility.

In 1739 Robert Hay Drummond inherited the Innerpeffray estate, and with it the responsibility for the library and school. Drummond raised the funds and commissioned architect Charles Freebairn to design and build a new library building immediately adjacent to the Chapel. This handsome Georgian edifice still houses the library today.