The original entrance leads to a stair known as the 'Left-Handed Staircase', the story being that when Sir Andrew Kerr, who was himself left-handed, returned from Flodden in 1513 he had his followers trained to use their weapons with their left hands. This is said to be the origin of 'Corrie-fisted' or 'Kerr handed'. The basement is vaulted, and the hall has a 16th-century fireplace.

Ferniehirst was a property of the Kerrs and first built by Sir Thomas Kerr in 1476 on the remains of an earlier foundation, but was sacked by the English in 1523. It was recaptured with French help in 1549, and the leader of the English garrison was beheaded. Sir Thomas Kerr, protector of Mary, Queen of Scots invaded England in 1570, hoping to have her released, but all that resulted was an raid on Scotland, during which Ferniehirst was damaged. James VI destroyed the castle in 1593 because of help given by the family to the Earl of Bothwell. The castle was rebuilt about 1598. As late as 1767 the house was occupied and used by the Lord Lothian of that day but even then it was showing signs of delapidation. Between 1934 and 1984 it was leased by the Scottish Youth Hostel Association, except for during World War II when it served as a army billet.

A 'Green Lady' is said to haunt the castle in some tales, and unusual occurrences were reported during its time as a youth hostel, although the story is refuted.