Kelly hoped that a change of climate would aid his failing health, so he, Fanny, and her seven-year-old niece and adopted daughter, Mary Hurley, along with two " colored servants ", Franklin and Andy, set out on May 17, 1864 from Geneva for the region that is now Idaho or Montana.
