The Mt. Zion Old School Baptist Church was founded in 1850 when a group of members of Little River Baptist Church withdrew from the church because of their incompatible doctrinal beliefs. They built their own church in Aldie, Virginia in 1851 at the intersection of the Old Carolina Road (now U.S. Route 15) and the Little River Turnpike (U.S. Route 50).

The church served both whites and African-Americans, including freedmen and slaves. In 1852, the church began a cemetery that is still in use. Only five pastors served at the church during its 129 years of service. Some of these were circuit pastors who travelled between Mt. Zion Church and its sister churches, New Valley Church and Frying Pan Church, holding only one or two services at each per month.

The church held services continuously from 1851-1980, except for a short break during the Civil War when the church changed hands multiple times between the Union and the Confederacy. In 1863, Colonel John S. Mosby (1833-1916) used the church as a meeting place for the "Partisan Rangers," a group of Confederate soldiers who waged guerrilla warfare against Union troops. The church was also used as a field hospital for both Union and Confederate troops. Twelve Union soldiers and thirteen Confederate soldiers are buried in the cemetery.

On 6 October 1996, the church was named an historic site, and the Mount Zion Church Preservation Association, Inc. (MZCPA) was founded. According to the MZCPA 2010 Annual Report, the purpose of the association is to "preserve, protect, enhance and interpret the historical and natural resources of its properties for future generations." Starting in 1996, the MZCPA began to create opportunities for tourists to learn about the history of the church, holding annual events such as Colonel Mosby's birthday celebration and "Eyewitness to War," a living history program.

In 1999, the church became the property of Loudoun County, and in 2001, it became an affiliate member of the Smithsonian Institution.