Kovalchuk's winner came on a breakaway 3:02 into the extra period. He didn't have a goal in any of Atlanta's first five games.
Scott Mellanby scored twice for Atlanta, and Kovalchuk assisted on the second, 2:10 into the third period. That put the visitors ahead 3-2, but Ovechkin - presented with his Calder Trophy during a five-minute pre-game ceremony - assisted on Dainius Zubrus' tying goal with 5:40 remaining in regulation.
Jon Sim also scored for Atlanta, his fifth goal of the season.
Chris Clark and Jamie Heward had Washington's other goals, and Ovechkin mostly was kept in control. He didn't have a shot in the first period, and only two in the second.
Ovechkin was presented with his award as the league's top rookie in 2005-06 by general manager George McPhee as the player's parents and brother stood nearby. The above-ice video screens showed a highlight reel from last season, with Ovechkin - only the second rookie in NHL history to produce at least 50 goals and 100 points - scoring goals, celebrating them and slamming opponents against the boards.
Taped congratulatory messages from teammates were played between periods, and commemorative coins honouring Ovechkin were handed out. Fans also could pose for photos with his trophy.
And, when play began, it took all of 10 seconds for Ovechkin to create excitement, placing a pretty pass in front of the net for Clark, who couldn't convert the chance. Clark later did get his first goal of the season, when he flicked in a rebound to tie the game at 2-2 at the 15:38 mark of the second period.
The score stayed that way until Kovalchuk's slap shot was blocked but not held by Capitals goalie Olaf Kolzig, allowing Mellanby to tap in the rebound for his fourth goal of the season.
Mellanby's first goal of the night came when he deflected Marian Hossa's shot to tie the score 15:25 into the opening period. Washington had led 1-0 on Heward's power-play goal 5:41 in.
Heward, a defenceman, knocked in a cross from Alexander Semin, who earned his first assist of the season. Semin entered the day leading the NHL with five goals; he had at least one in each of Washington's previous three games.
Notes: Some fans booed after Washington failed to score while they had a two-man advantage for about 1 1/2 minutes in the second period. ... Heward's goal gave him 100 points in the NHL; this is his seventh season. ... The Thrashers are off until they host Washington on Thursday; the Capitals are at home against Florida the day before. ... Capitals C Boyd Gordon missed a third consecutive game with a lower body injury.