The anti-migration and Russia-friendly Alternative for Germany, or AfD, now has a very good chance of forming the first far-right state government since the Nazi era.
Merz said his centre-right Christian Democratic Union was "deeply shocked" by what he called "the most serious election defeat that the CDU has had for years, for decades".
But he indicated that he's determined to stay at the helm and said that "giving up is not an option for me".
The AfD, fell just short of a majority in Sunday's vote in Saxony-Anhalt, a small eastern state that gained outsized prominence as AfD eyed power there.
Its regional branch is one of several considered a proven right-wing extremist group by Germany's domestic intelligence agency.
AfD's victory is a major distraction for Germany and Merz at a time when Europe faces several challenges, including Russia's continued aggression against Ukraine, a suspected sabotage campaign targeting NATO countries, and difficult dealings with US President Donald Trump.
Merz said he wanted to assure Germany's allies that "our institutions are resilient and stable" and the country will stick to its values and constitution after the AfD win.
"We know that Germany carries special responsibility because of its history," he said.
He vowed that he won't "budge one millimetre from my fundamental conviction that we must defend our freedom against Russia now".
Germany is one of Ukraine's biggest supporters in its war against Russia.
Because of Germany's Nazi past, far-right movements have always come under intense scrutiny.
Other major parties have refused to co-operate with AfD in a so-called "firewall".
Unlike many countries in Europe, modern Germany hasn't yet had a far-right or right-wing populist party leading a national, state or major-city government.
Merz insisted "we need fundamental reforms in Germany".
He said it's a question of "nothing less than the future of our country and the opportunities of our children and grandchildren, and we must not squander those. That is why my determination to continue the reform course is unbroken."
While Merz continues to push back against resistance to reforms, AfD - whose victory was cheered by far-right parties across Europe - faces potentially tricky manoeuvring to put its candidate for governor in Saxony-Anhalt, Ulrich Siegmund, in power.
Siegmund had pushed for an absolute majority in the state legislature that would allow AfD to sidestep mainstream parties' shunning of it.
After the vote, he signalled that AfD still isn't willing to make compromises, insisting that "we don't want to become like the parties that were punished" by breaking pre-election promises.
He also rejected the idea of a minority government.
- with Reuters
