In 1833 he arrived in Texas with his brother Harvey, who later participated in the battle of San Jacinto. Between January and October 1834 Swisher and his family settled at Tenoxtitlán in Robertson's colony, in what is now Burleson County. He successfully led a retaliatory attack after a Comanche Indian raid on the settlement in April 1834. By October 1834 Swisher and his family had moved to Chriesman Settlement in what became Washington County. On July 2, 1835, Swisher was one of the petitioners who requested a separate municipality for the area that later became Washington Municipality.
|Independence Hall, location of the Declaration signing|
Between 1839 and 1841 he served as a justice of the peace in Washington County. He was also an incorporator and trustee of Union Academy, a Washington County school chartered in February 1840. In 1846 Swisher moved to Austin, where he operated a tavern, a hotel, and after 1852 a ferry. In his later years he also farmed. In 1848 he was registrar of an Austin high school. He was a member of the building committee of the First Presbyterian Church in Austin in 1851 and one of five members of a vigilance committee formed by the Austin vigilante movement in October 1854 to enforce slave-control laws. After Swisher's death in Austin on November 14, 1862, his wife, Elizabeth, continued to operate the important ferry transportation link on the Austin-San Antonio Road. Swisher had four children who lived to adulthood, including John Milton Swisher, who held many appointive offices in the republic and state of Texas, and James Monroe Swisher, an Indian fighter and later a state legislator. Swisher County and a street in Austin were named for James Swisher.
30° 16.608, -097° 43.625