Just 16 per cent of Kiwis have Mr Luxon as their preferred leader in the latest TVNZ-Verian poll, which also showed his right-wing coalition would be turfed from office if an election were held in April.
"I get it. Not everyone will want me at their barbecue," Mr Luxon told Newstalk ZB on Monday.
Mr Luxon's National has been trending downwards since coming to office at the 2023 election, when the party won 38 per cent of the vote.
The latest figures put the centre-right party on 29.7 per cent - just below a psychologically important 30 per cent threshold - and behind Labour's 36 per cent result.
Should the result transpire at the November 7 election, it would be the first time National had lost government after one term.
Other polls suggest the right-wing coalition of National, NZ First and ACT might hold onto government, but with fewer National members, sparking concern among Mr Luxon's MPs.
The NZ Herald has reported that earlier in April, chattering from the backbench about Mr Luxon's performance had reached such a level that party whip Stuart Smith sought a meeting with the prime minister.
However, in what the Herald called a "rare and extraordinary" rebuke, Mr Luxon reportedly "ghosted" Mr Smith and didn't meet with him to discuss those issues.
It is not clear how firm or widespread dissatisfaction with Mr Luxon was then, or will be at a showdown caucus meeting on Tuesday.
Mr Luxon said he understood "there were five people that are moaning and frustrated ... of people that I think that are possibly speaking to media about their frustrations".
This is a shift, with Mr Luxon arguing as recently as Friday that he had "full support" of his caucus.
In March, Mr Luxon shook up his cabinet and re-election team, removing moderate MP Chris Bishop from the key posting of campaign chair in favour of conservative MP Simeon Brown.
Mr Bishop has denied being part of a plot to remove Mr Luxon.
"Some people have been talking out of school ... that is unhelpful and untidy and indicates that the National Party is focused on ourselves rather than focused on the country," he told TVNZ.
TVNZ has also reported an anonymous National MP stating the partyroom numbers were "probably there" to remove Mr Luxon, but the party wanted him to resign to avoid a messy confrontation.
Mr Luxon has vowed not to resign and contest the next election.
"We know we want to do better. We need to do better as a National Party. We know it's also been very volatile and challenging times," he said.
"What this country needs is strong economic management and a stable coalition, and that's what we're delivering."