The result in the Mornington Peninsula seat of Nepean could flag how much of a threat One Nation is to the Liberals across the state.
After poll booths closed on Saturday, results around 7.30pm showed Liberal candidate Anthony Marsh, the favourite to win, ahead on 36.6 per cent with just under 22 per cent of primary votes counted.
He was followed by community independent Tracee Hutchison on 23.8 per cent and One Nation's Darren Hercus on 21.2 per cent.
On a two-candidate preferred basis Mr Marsh was on 55.4 per cent and Ms Hutchison on 44.6 per cent.
Labor opted not to field a candidate in the by-election.
The ABC's elections analyst Casey Briggs said the Victorian Electoral Commission had chosen to count preferences between Mr Marsh and Ms Hutchison.
"That's just an indicative count for tonight, and they may or may not turn out to be the final two candidates in this race," he said in his by-election commentary.
The decision could make it difficult to project a result on Saturday evening if One Nation made it into the final two, Mr Briggs said.
Saturday's by-election was triggered by the sudden resignation of Victorian Liberal deputy leader Sam Groth, with the party holding the seat since its inception for all but four years when it fell to Labor.
Mr Marsh was forced to admit he didn't live in the electorate so couldn't vote for himself on Saturday, but said he had been mayor in the area three times over the past five years.
Mr Hercus told reporters it was a tight race with a mixed response from voters fed up with empty promises.
"We've had a few Liberal voters come over to us for frustrations they've been feeling with their own party," he said.
One Nation leader Pauline Hanson campaigned alongside Mr Hercus but was absent on polling day.
"People will vote for her rather than me," Mr Hercus said.
Ms Hutchison told reporters a lot of voters were looking for an alternative as the area had been overlooked and let down by the major parties for decades.
"I am putting myself forward as someone who is of this place, who cares deeply about this place," she said.
A strong One Nation showing could back up national polling and the South Australian election result, and bode poorly for the Liberals in the Victorian state election, senior politics lecturer at Monash University Benjamin Moffitt said.
"If they even do somewhat well in Nepean, that's a real problem for the Liberal Party," Dr Moffitt said.
Victoria's Opposition Leader Jess Wilson on Saturday acknowledged how important the by-election was for her prospects of becoming the next premier.
"We don't take anything for granted," she said.
"We're the only party down here who are running on a plan to deliver the Rosebud Hospital, to fix the roads, to put more cops on the beat down here."
Total voter enrolment at close of rolls was 50,910.
The electorate takes in the wealthy postcodes of Sorrento, Portsea and Flinders, and low socio-economic areas such as Capel Sound, formerly known as Rosebud West.
The Mornington Peninsula is officially part of metropolitan Melbourne, but does not have comparable services, with public transport and a planned revamp of Rosebud Hospital among key issues for locals.