In 1853 he acquired land six miles southwest of Indianola, where he tried to develop the town of Cayloma. Five years later he served as commissioner of the Indianola Railroad Company. By 1860 Milby was living in Indianola, where he owned $55,000 in real and personal property, including two slaves. Milby supported secession and was badly hurt by the Civil War and a fire that destroyed his mercantile house in 1867. By 1870 his property was valued at only $1,600. The Indianola storm (1875) destroyed his home. Ironically, he had been one of a three-man commission that had cooperated with the federal signal service's coastal reporting station. Milby and his wife, Mary, a native of Pennsylvania, had at least eight children. Milby was a Mason and Episcopalian. He died on February 2, 1887 and was buried in Glendale Cemetery in Harrisburg.
29° 43.180, -095° 16.458