After touring in 1971 to promote their previous album, Trafalgar, the Bee Gees worked quickly to complete another album. They recorded the song "Paper Mache, Cabbages and Kings" on 3 January 1972 which was the last song recorded with the Australian drummer Geoff Bridgford. He left the group before their tour of East Asia and was replaced on tour by Chris Karon. Recording resumed in April, 1972 with a Robin song called "Never Been Alone" and a song Barry did on his fan club recording from 1971 called "I Can Bring Love". The drummer on the April sessions was a veteran session player, Clem Cattini, who has appeared on many top recordings going back to 1960.
The album was primarily recorded between June 1971 and April 1972 (except for "We Lost the Road", recorded in January 1971 during the Trafalgar sessions). The Bee Gees saved a non-album single, "My World", from the sessions which was released in January 1972, becoming a UK/US Top 20 hit. The album came in November 1972 supported by very positive reviews, and preceded by the UK Top 10 hit "Run to Me", in July. Despite the critical acclaim of the variety of styles and strong tracks, To Whom It May Concern only reached #35 in the US and became their third consecutive studio album to fail to appear in the UK album charts. It performed better in other European countries. In Italy reached #10, and peaked at #6 in Spain. The subsequent single "Alive" was a modest sized hit in the US, reaching the Top 40, and a major hit in Australia, reaching #4.