Ella Maude Baker was born on August 16, 1918 at a hospital in Chase, British Columbia, Canada. Her parents, Walter Scott Baker (b. 1893 in Burtrum, Minnesota) and Jesse Maude Leib (b. 1900 in Hot Springs, Arkansas), were married in 1916 in Lincoln County, Montana. In 1917, her parents moved to British Columbia to homestead. Ella Maude Baker was their only child.

From the mid-1920s through the mid-1930s, the family lived in Chehalis, Washington and Medford, Oregon, where Walter Baker worked in agriculture and the lumber industry. In 1934, Walter Baker and Jesse Leib Baker separated. Ella Baker moved to Eureka, Montana to live with her grandmother. There she met and married William Wayne Workman in 1935. They had four children: Alma Lee (b. 1936), Barbara Ramona (b. 1937), William Rodney (b. 1938), and Patricia Marie (b. 1940). After living in Idaho and Washington, Ella Baker and William Workman separated in 1947.

After the divorce, Ella Baker and her three living children moved to LaGrande, Oregon. In 1954, Ella Baker was remarried to Howard O. Zumwalt, a rural mail carier. Ella Baker Zumwalt worked for many years as a proofreader for the LaGrande Evening Observer.

In 1977, after both Ella and Howard Zumwalt had retired, they moved to Milton-Freewater, Oregon. For twenty years, they enjoyed golfing and fishing together. In 1997, after Howard’s death, Ella Baker Zumwalt moved to Clarkston, Washington to live near one of her children.