In the 1830s through the 1850s, Anthony Robinson Jr. purchased over 170 acres in the center of what would become the city of Richmond, including the property on which VMFA now stands. The estate, comprised of woods and open countryside, was cultivated and improved, no doubt through the labor of the enslaved African Americans listed in Robinson's tax records and will. Little is known about the earliest Robinson residence, but the imposing farmhouse still standing on the museum property was built by Anthony Robinson Jr. in the mid-1850s. In April 1865, during the final weeks of the Civil War, Union troops occupied the house and grounds at the invitation of Robinson's widow, Rebecca, in exchange for protection from looting. In 1883, the couple's son Channing sold the residence and thirty-six surrounding acres to establish a Confederate soldiers' home.

Between 1885 and 1941 the property was the site of a large residential complex for poor and infirm Confederate veterans of the Civil War. Established by R. E. Lee Camp, No. 1, Confederate Veterans, the camp was built with private funds, including donations from former Confederate and Union soldiers alike. At peak occupancy, residents numbered just over three hundred; altogether a total of nearly three thousand veterans from thirty-three states called the camp home. For the next half century, Robinson House, renamed Fleming Hall during the soldiers' home era, served as the compound's administration building and war museum. After the camp's closing, the Commonwealth granted use of the building to the Virginia Institute for Scientific Research in the 1950s and to the Virginia Museum of Fine Arts from 1964 to the present.

The green space in the central grounds of today's VMFA property was once the commons of the Confederate soldiers' home. Around the oak-filled park stood the administration building, barracks, dining hall, hospital, recreation hall, steam plant, and assorted outbuildings. The superintendent's house, nine residential cottages, and a chapel formed an arc to the west. With the exception of Robinson House and the Confederate Memorial Chapel, the structures were demolished or moved in the early 1940s. From the camp's earliest years, the Commonwealth of Virginia helped fund the institution. In 1892, Lee Camp No. 1 agreed that the property would revert to the Commonwealth in twenty-two years. A later agreement extended that transition to the time when the original purpose of the home was no longer needed. When the last resident died in 1941, the Commonwealth gained ownership of the site. By that time, it had been designated as the Confederate Memorial Park.

Dedicated in 1887 to the Confederate war dead, the nondenominational Confederate Memorial Chapel (also referred to as the Pelham Chapel) served as a place of worship for the residents of R. E. Lee Camp, No. 1. Funded by donations from veterans and private citizens of the Commonwealth, it was designed by architect Marion J. Dimmock in the Carpenter-Gothic style. In the postwar era of reconciliation, Union veterans from Lynn, Massachusetts, donated the organ. By the time the camp closed fifty-four years later, the chapel had hosted approximately 1,700 funeral services for the former soldiers.

The monumental limestone building to the west of the present museum grounds was built in 1932 as a residence for destitute female relatives of Confederate veterans. After relocating the final inhabitants of the Home for Needy Confederate Women to a nursing facility in 1989, the Commonwealth set aside the property for use by the Virginia Museum of Fine Arts.

Source: History of the VMFA Grounds