Shortly after the town of Huntsville was incorporated, founder Pleasant Gray donated five acres of land for the creation of a school. The new institution began operations as early as 1845 and received a charter on April 11, 1846. Originally, the school offered education to both boys and girls and was known as the Huntsville Academy. Citizens in Huntsville raised the funds to build what was later described as a schoolhouse “built elegantly of red brick.” After serving as a coed school for several years, the Huntsville Male Institute broke away in either 1847 or 1848 (sources differ). From then on, they were separately known as Huntsville Male Academy and Huntsville Female Academy. When Austin College replaced Huntsville Male Institute, the remaining female school became know as the Female Academy or simply, the Brick Academy. It had an average enrollment of approximately 70 young women.
William Viser, Thomas Gibbs, M. Barrett, F. L. Hatch, and M. C. Rogers served as trustees of the school, and many famous residents of Huntsville worked for the institution. Dr. Samuel McKinney, who later became the first President of Austin College, taught at the Brick Academy as did the missionary-turned author Melinda Rankin. While in Huntsville, Rankin completed her famous book, Texas in 1850, in which she spoke very highly of Huntsville and the surrounding county.
According to records, the community occasionally used the sturdy school building for other purposes. The organizational meeting of the First Baptist Church took place in the Brick Academy, and the congregation appears to have used it to hold regular church services until a new sanctuary was built.
The Brick Academy served the community of Huntsville as a school for young women only for a short time. In 1853, the Methodist Church chartered Andrew Female Academy, and the older school quickly fell into disuse.