The following year, he scored another Billboard Top 40 hit when "Sunday Mornin'" reached #35, but the follow-up, "Angelica" stalled at #97. From there he teamed up with '60s hitmaker Lesley Gore for a remake of The Fleetwood's "Come Softly To Me". The record, credited to "Billy and Sue", went nowhere and Oliver quickly disappeared from the charts. Resuming the name Bill Swofford, he toured hundreds of college campuses in the eastern and southern United States in 1976 and 1977, but a short-lived teaming with Karen Carpenter during the same period proved unsuccessful. He kept busy singing small parts on Broadway for a brief period before leaving the music business in the early 1980s. In 1983, People magazine published an article that looked back at Swofford's career, describing him as a happily married father who kept his distance from the music industry. He sold real estate for a while before taking a job with a major American pharmaceutical company.
Oliver's name came into the public eye again in the '90s when he became one of the few artists, along with Paul Anka and Captain and Tennille, to have more than one song named among The Worst 100 Singles of the Last 25 Years, by David Browne and David Hinckley for The New York Daily News. Both "Jean" and "Good Morning Starshine" made the list. This compilation however can't be taken too seriously, as such Rock legends as Mick Jagger and Michael Jackson ("State of Shock"), Paul McCartney ("Silly Love Songs" and "Ebony & Ivory") and Stevie Wonder ("I Just Called to Say I Love You" and "Ebony & Ivory") also made the tally. Someone once said that "A critic writing about music is like a man with no legs writing about shoes".
Swofford was diagnosed with Non- Hodgkin's Lymphoma in the mid-1990s. His brother John donated bone marrow in hopes of saving Bill's life in 1999, but William died ten months later on February 12th, 2000, at LSU Hospital in Shreveport. He was buried at Laurel Land Memorial Park in Dallas, Texas.
In 2009 North Carolina legislators introduced a resolution in the North Carolina General Assembly to honor Swofford and his contributions to music. On July 7th, 2009, that resolution was passed. On the 40th anniversary of Swofford's first success, his home town hosted a musical tribute in his honor called OliverFest. William Oliver Swofford was inducted into the North Carolina Music Hall of Fame in 2010.