The 16th-seeded Alexandrova, who beat Australia's Talia Gibson in the round of 32, ensured Sabalenka's quest for a first Canadian Open title remains unfulfilled.
After losing the first set tiebreaker, Sabalenka fought back to win the second, before Alexandrova clinched a gutsy 7-6 (7-3) 4-6 6-4 win in Toronto on Saturday (Sunday AEST).
It's not the first time the Russian has beaten four-time grand slam singles champion Sabalenka, with her latest victory squaring their head-to-head record at 5-5.
Sabalenka will be looking to fine-tune her game ahead of the year's final major, the US Open in New York, where she has twice claimed the singles crown.
"I just tried to play every single point as if it was the last one because with her, you don't a lot of chances during the match," Alexandrova said in her on-court interview.
"I had a lot today, but I couldn't realise some of them.
"But honestly, I was trying not to think about the score or anything, just hit the ball and that's it.''
Alexandrova's victory set up a quarter-final against Ukraine veteran Elina Svitolina, who beat eighth-seeded American Amanda Anisimova 6-2 6-4.
Earlier, Coco Gauff overcame a late surge from Maria Sakkari to win their round-of-32 clash 6-1 6-4.
It was Gauff's fourth-straight victory over the Greek and she closed out the match in 76 minutes.
The American's only scare came in the second set after comfortably leading 5-1.
Sakkari fought back to cut the deficit to 5-4, but Gauff held her nerve, earned a second match point and converted it to seal the win.
"I'm happy with how I played today. Maria is a tough opponent," said Gauff, a two-time grand slam champion.
"Her level definitely raised toward the end of that second set, but I'm glad I was able to close it out. That last game was a big test for me."
Gauff, who won the 2023 US Open with a victory over Sabalenka in the final, will now face Russia's Alina Korneeva in the round of 16 in Toronto.
While Gauff's progress was pretty smooth, it was a different story for Swiatek, the No.7 seed.
Nothing went right for the Polish six-time slam champion in the first set, when she won just one point out of 17 on serve and was broken four times.
Kostyuk, the 10th seed, won the other 16 points on the way to taking the set 6-3.
But Swiatek bounced back to win the next two sets comfortably for a 3-6 6-1 6-2 victory.
Swiatek next faces Russia's Diana Shnaider, who knocked out third-seeded American Jessica Pegula.
With PA.
11°C