In a 4-3 decision, the state Court of Appeals said the trial judge made a mistake by letting women testify that Weinstein assaulted them, even though their accusations were not part of the charges he faced.
"The remedy for these egregious errors is a new trial," Judge Jenny Rivera wrote for the majority.
Weinstein, 72, has been serving a 23-year sentence in a prison in upstate Rome, New York.
He is not expected to be freed immediately, as he also faces a separate 16-year prison term in California after being convicted there last year for the 2013 rape of an actress at a Los Angeles hotel.
"We will do everything in our power to retry this case, and remain steadfast in our commitment to survivors of sexual assault," Emily Tuttle, a spokeswoman for Manhattan District Attorney Alvin Bragg, said in an email.
Bragg's predecessor, Cyrus Vance, brought the New York case.
Arthur Aidala, a lawyer for Weinstein, called the decision a victory for his client and any person in the US charged with a crime, "no matter how popular or unpopular they are".
Weinstein has denied having non-consensual sexual encounters with anyone.
Ashley Judd, the actress and one of Weinstein's accusers, at a press conference called the decision "an act of institutional betrayal".
Judge Madeline Singas dissented, saying the decision "perpetuates outdated notions of sexual violence" and makes holding defendants accountable for sexual assault "significantly more difficult".
She also accused the majority of whitewashing the facts and continuing what she called a "disturbing trend" of overturning jury verdicts in sexual violence cases.
Lawyer Arthur Aidala says the decision is a win for any person charged with a crime. (AP PHOTO)
"New York's women deserve better," Singas wrote.
The New York conviction had been considered a milestone for the MeToo movement, in which women accused hundreds of men in entertainment, media, politics and other fields of sexual misconduct.
"Some thought the (Los Angeles) case to be superfluous," Elizabeth Fegan, a lawyer for some Weinstein accusers, said in an email.
"Now we realise how important it was."
Aidala, Weinstein's lawyer, said he believed the New York decision could influence a California appeals court which is considering whether to overturn that conviction.
The California trial similarly featured testimony about his alleged past misconduct as part of the prosecutors' case.
The New York appeals court also said the trial judge compounded its error by letting Weinstein be cross-examined on a broad range of "loathsome" conduct, including bullying and fits of anger toward associates, that portrayed him in a "highly prejudicial" light.
"It is an abuse of judicial discretion to permit untested allegations of nothing more than bad behaviour that destroys a defendant's character but sheds no light on their credibility as related to the criminal charges," the ruling said.
Weinstein was convicted in February 2020 in Manhattan of sexually assaulting former production assistant Miriam Haley in 2006, and raping aspiring actress Jessica Mann in 2013.
The charges on which he was convicted were first-degree sexual assault and third-degree rape.
Jurors acquitted him on other charges.
Weinstein's trial judge, James Burke, was not reappointed to the bench after his term expired at the end of 2022.
Some states, including New York, responded to the MeToo movement by passing laws that let women bring civil lawsuits seeking damages for sexual misconduct that occurred many years earlier even if statutes of limitations had already passed.
Weinstein co-founded the Miramax film studio, whose hit movies included Shakespeare in Love and Pulp Fiction.
His own eponymous film studio filed for bankruptcy in March 2018.
12°C