But Worsley, who died this week at 77 in hospital after suffering a heart attack last Monday, used his five-foot-seven 180-pound frame to forge a Hall of Fame career and help the Montreal Canadiens win four Stanley Cups.
"It was just his body shape," former teammate Gilles Tremblay said. "He was real quick in the net.
"He did his exercises. But some people are tall and thin like (former defenceman) Ted Harris and some are built like Worsley."
The Gumper, who got his nickname as a child because his hair stuck up like cartoon character Andy Gump, won the Vezina Trophy as the NHL's top goaltender in 1966 and 1968, when he was also a first-team all-star. He entered the Hockey Hall of Fame in 1980.
He was among the select few to play in net when the NHL had only six teams and teams carried only one, maskless goaltender. He went head-to-head with greats like Plante, Detroit's Terry Sawchuk, Chicago's Glenn Hall and Toronto's Johnny Bower.
Tremblay recalls a teammate who always had a smile and a joke in the dressing room and who was "very well liked all through the league.
"He'd walk through the room past guys with perfect builds and he'd say 'I've been in the league a lot of years with this belly, so I hope you guys can do as well as I did.' He always made us laugh."
Worsley's physique moved Rangers coach Phil Watson, who Gump detested, to tell him "you can't play goal with a beer belly."
Worsley, recalling the story on the Hall of Fame website, shot back: "I'm strictly a rye man."
What is less known about him was that he was also a pretty good centre half in soccer.
Canadian soccer historian Colin Jose said that while playing hockey in the minor leagues for the Saskatoon Quakers in the early 1950s, Worsley played soccer in the summer for the Saskatoon Legion.
He played for the Saskatchewan all-stars against the touring Tottenham Hotspur in 1952 and, when he moved home to Montreal the next year, reached the Canadian championship soccer final with Montreal Hakoah.
But hockey was Worsley's passion from his childhood in Montreal's Pointe St. Charles district.
He grew up in a family that worshipped the defunct Montreal Maroons, who disbanded during the Depression in the 1930s, and didn't like the Canadiens. His favourite player was Rangers goalie Dave Kerr.
In his teens, he signed with the junior Verdun Cyclones, who were owned by the Rangers and, in those pre-draft days, became Rangers property.
He played minor league hockey for the New York Rovers, the St. Paul Saints, Saskatoon and the Edmonton Flyers before he was called up for the start of the 1952-53 NHL season after goaltender Charlie Rayner was injured.
Despite finishing last, Worsley won the Calder Trophy as the league's best rookie, only to be shocked when he was sent down to the Vancouver Canucks (of the defunct minor Western Hockey League) the next season when the Rangers signed Bower to play goal.
Worsley was back up with the Rangers in 1954-55 and played brilliantly for nine more seasons on mostly weak New York teams.
Lounging at home in the off-season in 1963, Worsley got a call from a friend to tell him he had been traded to Montreal along with Leon Rochefort, Dave Balon and Len Ronson for Plante, Phil Goyette and Don Marshall.
He turned on the radio and heard it himself.
"To this day, the Rangers have never told me I was traded," Worsley told the Hall of Fame.
He went from facing 40-to-50 shots per game in New York to a team that was a perennial powerhouse, still with some of the players from the team that won five straight Stanley Cups from 1956 to 1960.
Injuries caused him to spend most of the next two seasons with the Quebec Aces, but he was called up in 1964-65 to help Montreal win four Cups in a five-year span, interrupted only by Toronto's last Stanley Cup triumph in 1966-67.
"With the trade, he got his reward by playing for a very good team," said former goaltender Ken Dryden, who joined the Canadiens in 1971.
"I played against him his last couple of seasons in Minnesota. He still wasn't wearing a mask, which was unbelievable."
Worsley was sold to the expansion North Stars for cash in 1970 and retired, but was talked into playing four more years in Minnesota.
He wore a mask only for the final six games before he retired in 1974 to his long-time home in Beloeil, Que. He then worked many years as a scout for the North Stars.
Dryden, whose No. 29 is to be retired Monday night at the Bell Centre, knew Worsley only from watching him play on TV while growing up.
"He was in that first generation of goaltenders that I watched - Hall and Sawchuk and Plante and Gump," he said.
There was a family connection, however.
Until well into the 1960s, teams carried only one goalie and, if he was hurt, the home team had a replacement on hand who would go in for either team. Dryden's brother Dave, then a junior, served that role in Toronto.
"He'd get $10 and watch the game, which I thought was just heaven," Ken Dryden said. "One game, early in the second period, Worsley got hurt and my brother had to play.
"He had to wear Gump's jersey, and they weren't exactly the same size."
Toronto won Dave Dryden's NHL debut game 4-1.
Worsley retired with a career record of 335-352-150 with 43 shutouts. In the playoffs, he was 40-26 with five shutouts. When he left the NHL, only one goalie, Andy Brown of the Pittsburgh Penguins, was still not wearing a mask.