The AfD was set to win 37 per cent in the state of Mecklenburg-Western Pomerania, ahead of the Social Democrats on 35.5 per cent and far in front of Merz's Christian Democrats, who received 5.2 per cent and barely crossed the five per cent threshold to enter the state parliament, according to exit polls on ARD public television.
But the AfD is unlikely to govern as the other main parties rule out working with it.
If confirmed, Sunday's result would be the worst in any state election for the Christian Democrats in the party's 81-year history.
"It's a disaster," Merz said in televised remarks after the exit polls were published, but signalled that he would fight on as chancellor after speculation that a bad result could prompt him to resign.
Merz conceded it was not possible to "sugar-coat" the result.
But he vowed to "to prepare our country for the future," pledging to press ahead with his unpopular economic and social reforms.
"I am taking on this responsibility because I want to move our country forward," he said.
"I am firmly convinced that the success of our reforms will not only improve the way we live together, but also our self-image and our image in the world. Germany is capable of reform."
In another election in the capital Berlin that also took place on Sunday, the anti-capitalist Left party was on 25.3 per cent while Merz's conservatives were on 19.1 per cent and the AfD also gained in strength, hitting 16.2 per cent.
Sunday's results may not prove a knock-out blow for Merz, who last week received backing from eight conservative state premiers.
But his low approval ratings, unpopular reforms and the triumph of the AfD in an earlier state election in Saxony-Anhalt have left his future in the balance.
Merz has called a party meeting for Sunday to discuss the way forward.
Elected last year on a promise of reforms to return Germany's stagnant economy to growth, Merz has struggled to win over voters, compounding resentment over unpopular pension, tax and healthcare reforms with a stream of communication missteps.
Merz presented a consumer relief package on Friday that included tax cuts on petrol and diesel.
Several people who voted in Mecklenburg-Western Pomerania said living costs were a major concern.
"Here in Schwerin, with these wages, you can hardly afford anything anymore," voter Carlotta Heick told Reuters in the state capital.
One elderly man criticised funding for foreign aid.
"We need our money here, for our country, for children and pensioners," he said, asking not to give his name.
with AP and Reuters
