James graduated from Northwestern Bible School in Minneapolis, earning a diploma in Bible studies. There he met Eileen Tollefsen, whom he married in Zion, Illinois, on July 29, 1949. Two years later, they moved to his hometown of Fifield, Wisconsin, when James took over the family farm.
In 1965, he was called into the ministry and was accepted into Continental Baptist Missions (then called Hiawatha Baptist Missions). The first church he pastored was in Brillion, Wisconsin, starting in 1967. He also pastored churches in Woodruff from 1975 to 1980; Swan Valley, Montana, from 1980 to 1989; and Manitowoc, Wisconsin, from 1990 until his retirement in 1993, at which point the Pattersons moved back to Fifield.
After retirement, James continued to do pulpit supply as often as he could. In November 2015, the Pattersons were honored for 50 years of service with Continental Baptist Missions.
He is survived by his wife of 66 years, Eileen; their children, Jef, Barbara, Dik, Faith Cochrane, and Steve; 10 grandchildren; and eight great-grandchildren. He was preceded in death by a grandson.
Services will be held at Woodruff (Wisconsin) Baptist Church on Feb. 6, with visitation from 10:00 a.m. until the time of service at 11:00 a.m.