He married Mary Ann James, daughter of Joshua James, on June 7, 1856 in the Deer Creek circuit of the Methodist Church in Harford County. She was born in 1835 in Harford County.
They had four children:
- Oleita Mitchell born 1857
- John E. Mitchell born December 4, 1859
- Charles Harvey born July 29, 1862 in Maysville, Indiana
- Bertha (Birdie) Mitchell born 1871 in Maysville, Indiana died 1879 Baltimore, Maryland
James and Mary moved their family to Indiana around 1861 at the time that the Civil War broke out. James decided to join the 88th Regiment of the Indiana Volunteer Infantry, Company D on August 8, 1862. He fought in several battles, including the Battle of Chickamauga in Georgia during September, 1863. He got sick in Stevenson, Alabama on November 12, 1863 and was discharged on July 5, 1865 from the invalid corps. He spent many years writing letters and applying for his war pension, but he was never successful.
James' wife Mary died in 1874 from an illness and was buried June 22, 1874 at Old Harland Cemetery in Maysville, Indiana. After her death James moved his family back to Maryland and near his family in Baltimore City on Caroline Street. He started a carpenter business. He married his second wife
Frances before 1880. She was originally from Vermont.
In 1895 James moved to Mullhall, Oklahoma and later moved to a Soldiers Home in Tennessee in 1904. He returned to Baltimore afterward and died on July 7, 1914.