Whiteman started out as a classical violinist and violist, then started leading a jazz-influenced dance band. He started making recordings for Victor Records in 1920, and became the most popular band leader of the decade. In the late 1920s he recorded for Columbia Records.

In the late 1920s and early 1930s Whiteman was billed as The King of Jazz. While most of what Whiteman's band played had little if any true jazz, many Americans of that era thought of the term "jazz" as simply referring to any peppy dance music.

While today most fans of jazz consider improvization to be essential to the musical style, Whiteman thought the improvisation of early jazz was sloppy and uncouth, and thought the music could be improved by eliminating improvisation except in small amounts in certain parts of elaborate arrangements. While it is easy to sneer at Whiteman in retrospect, his notions were critically popular and commercially successful at the time. Whiteman claimed that he was "making a lady out of jazz".

Whiteman commissioned George Gershwin's Rhapsody In Blue, which was premiered by Whiteman's Orchestra with Gershwin at the piano in 1924

Whiteman seems to have some appreciation of jazz musicians, since he hired many of the best white jazz men for his band, including Bix Beiderbecke, Frankie Trumbauer, Joe Venuti, Eddie Lang, Steve Brown, Gussie Mueller, Jack Teagarden, and Bunny Berigan. While Whiteman gave them very limited chance to improvise with his band, he paid them top salries and encouraged them to make small band jam recordings on the side.

Bing Crosby got his start singing with the Whiteman Orchestra.

In 1931, Whiteman married motion picture actressMargaret Livingston.

After he disbanded his Orchestra, in the 1940s and 1950s Whiteman worked as a music director for the ABC Radio Network.