Boyce's five-wicket haul capped one of the finest spells in the BBL's 11 seasons, overshadowing Usman Khawaja's 77 on return from the Ashes in the Thunder's 6-170.
Aaron Finch hit 82 from 64 balls in the chase, appearing to steer the Renegades home and hand Brisbane the spoon when the Renegades needed 13 off the final two overs.
But a late collapse left Boyce needing two runs off the last ball, only for the legspinner to be caught at gully off Gurinder Sandhu.
The result leaves the Renegades in last place on the BBL ladder, while the Thunder's tight win left them in third spot, missing the chance to host a home final.
Regardless of the result, it was Boyce's afternoon at the MCG.
His figures of 5-21 were the best in the Renegades' history, while it also marked the fourth-best return by an Australian player in the BBL.
His double hat-trick was also the first in BBL history, removing Alex Hales, Jason Sangha, Alex Ross and Daniel Sams in four successive balls.
The deliveries to remove Hales (44 off 22) and Sangha were particularly brilliant.
Boyce beat them in flight to have Hales caught and Sangha stumped.
After trapping Ross lbw, he also claimed Daniel Sams lbw to turn the match after the Thunder had shot out to 0-80 in the seventh over.
Boyce later also had Matthew Gilkes caught at cover for two and ran out Ben Cutting at the non-striker's end with a ball that flicked his thumb on the way through.
The Queenslander played seven T20s for Australia between 2014 and 2016, taking eight wickets at an average of 19.
He was also at one stage regarded by Ashley Mallett as Australia's best spinner since Shane Warne.
But he lost his spot on the eve of the 2016 T20 World Cup, the victim of India chasing down 197 in a match at the SCG where Boyce took 2-28 and had an easy stumping chance missed.
Adam Zampa was called in for the World Cup in India and Boyce never regained his spot.
The 32-year-old has since battled health issues, missing last year's entire Big Bash and the start of this season after suffering oesophageal spasms and overcoming mental health issues.
Boyce admitted on Wednesday there had been times he thought his career was over.
"Last year was obviously quite tough, things popped up I never experienced before," Boyce told Fox Sports.
"We've got a young family now, so it's always in the back of your mind what will happen in the future.
"I missed out last year, which obviously sucked ... but thankfully everything came back clear."
12°C