Welsh was born in Coaldale, Pennsylvania. He graduated from the U.S. Naval Academy in 1956 where he was an All-American quarterback. After serving as an assistant coach under Rip Engle and Joe Paterno at Penn State from 1963 to 1972, Welsh began his head coaching career at his alma mater in 1973. He inherited a program that had only had one winning season since the Roger Staubach era. He led the Midshipmen to three bowl game appearances and their first nine-win season in 16 years. In nine seasons, Welsh compiled a record of 55-46-1, making him the school's winningest coach.