After the war, Cosell began practicing law in Manhattan, primarily in union law. Some of his clients were actors, and some were athletes, including Willie Mays. Cosell’s own hero in athletics was Jackie Robinson, who served as a personal and professional inspiration to him in his career. Cosell also represented the Little League of New York, when in 1953 an ABC Radio manager asked him to host a show on New York flagship WABC featuring Little League participants. The show marked the beginning of a relationship with WABC and ABC Radio that would last his entire broadcasting career.
Cosell hosted the Little League show for three years without pay, and then decided to leave the law field to become a full-time broadcaster. He approached Robert Pauley, President of ABC Radio, with a proposal for a weekly show. Pauley told him the network could not afford to develop untried talent, but he would be put on the air if he would get a sponsor. To Pauley’s surprise, Cosell came back with a relative’s shirt company as a sponsor, and “Speaking of Sports” was born.
Cosell took his “tell it like it is” approach when he teamed with the ex-Brooklyn Dodgers pitcher “Big Numba Thirteen” Ralph Branca on WABC’s pre- and post-game radio shows of the New York Mets in their nascent years beginning in 1962. He pulled no punches in taking members of the hapless expansion team to task.
Otherwise on radio, Cosell did his show, Speaking of Sports, as well as sports reports and updates for affiliated radio stations around the country; he continued his radio duties even after he became prominent on television. Cosell then became a sports anchor at WABC-TV in New York, where he served in that role from 1961 to 1974. He expanded his commentary beyond sports to a radio show entitled “Speaking of Everything”.
Cosell rose to prominence covering boxer Muhammad Ali, starting when he still fought under his birth name, Cassius Clay. The two seemed to be friends despite their very different personalities, and complemented each other in broadcasts. Cosell was one of the first sportscasters to refer to the boxer as Muhammad Ali after he changed his name and supported him when he refused to be inducted into the military. Cosell was also an outspoken supporter of Olympic sprinters John Carlos and Tommie Smith after they raised their fists in a "black power" salute during their 1968 medal ceremony. In a time when many sports broadcasters avoided touching social, racial, or other controversial issues, and kept a certain level of collegiality towards the sports figures they commented on, Cosell did not, and indeed built a reputation around his catchphrase, “I’m just telling it like it is.”
Cosell’s style of reporting very much transformed sports broadcasting. Whereas previous sportscasters had mostly been known for color commentary and lively play-by-play, Cosell had an intellectual approach. His use of analysis and context arguably brought television sports reporting very close to the kind of in-depth reporting one expected from “hard” news reporters. At the same time, however, his distinctive staccato voice, accent, syntax, and cadence were a form of color commentary all their own.
Cosell earned his greatest interest from the public when he backed Ali after the boxer’s championship title was stripped from him for refusing military service during the Vietnam War. Cosell found vindication several years later when he was the one able to inform Ali that the United States Supreme Court had unanimously ruled in favor of Ali in Clay v. United States.
Cosell called most of Ali’s fights immediately before and after the boxer returned from his three-year exile in October 1970. Those fights were broadcast on taped delay usually a week after they were transmitted on closed circuit. However, Cosell was passed over for perhaps his biggest assignment of his career, the first Ali-Joe Frazier bout in March 1971. Promoter Jerry Perenchio selected actor Burt Lancaster, who had never provided color commentary for a fight, to work the bout with longtime announcer Don Dunphy and former light-heavyweight champion Archie Moore. Cosell attended that fight as a spectator only. He would do a voiceover of that bout, when it was shown on ABC a few days before the second Ali-Frazier bout in January 1974.
Perhaps his most famous call took place in the fight between Joe Frazier and George Foreman for the World Heavyweight Championship in Kingston, Jamaica in 1973. When Foreman knocked Frazier to the mat the first of six times, roughly two minutes into the first round, Cosell yelled out
“ Down Goes Frazier! Down Goes Frazier! Down Goes Frazier!”
His call of Frazier’s first trip to the mat became one of the most quoted phrases in American sports broadcasting history. Foreman beat Frazier by a TKO in 2 to win the World Heavyweight Championship.
Cosell provided blow-by-blow commentary for ABC of some of boxing’s biggest matches during the 1970s and the early 1980s including Ken Norton's upset win over Ali in 1973 and Ali's defeat of Leon Spinks in 1978 recapturing the Heavyweight title for the third time. His signature toupee was unceremoniously knocked off in front of live ABC cameras when a scuffle broke out after a broadcast match between Scott LeDoux and Johnny Boudreaux. Cosell quickly retrieved his hairpiece and replaced it. During interviews in studio with Ali, the Champion would tease and threaten to remove the hairpiece with Cosell playing along but never allowing it to be touched.