Robert C. Anderson was born in Prince Edward County in 1817, the son of Robert H. Anderson and Henningham C. Venable. Robert C. Anderson inherited, from his parents, the Providence Plantation in Prince Edward County. The house and farm had been given to his parents as a wedding gift in 1816, by Anderson's maternal grandfather, Samuel Woodson Venable, Sr.

The land and home, originally known as Rose Hill, was located 1 ½ miles east of Worsham, Virginia. The property was owned by the Episcopal Church, situated in nearby Kingsville, Virginia, and served as the rectory for the Reverend Archibald McRobert. Local, unverified legend states that during the American Revolution, when Tarleton's troops were raiding the nearby area, they destroyed much of the furniture and interior of the home and then set fire to the house. Shortly after they rode off, a sudden rain put the fire out, prompting Reverend McRobert to change the name of the home from Rose Hill to Providence.

Prior to purchase by Samuel Woodson Venable, Sr., the land was also home to Dr. Francis Joseph Mettauer, who had been quartered near Kingsville when he was a surgeon with Count Rochambeau's troops during the Revolution. At that time, he came to the attention of local influential citizens, Patrick Henry and Nathaniel Venable, who, realizing the need for a doctor to serve the community and Hampden-Sydney College, induced him to settle there by offering him the land on which to build a home.