The prime minister joined the chorus of condemnation of some 60 demonstrators linked to the Nationalist Socialist Network who assembled outside NSW parliament on Saturday.
They shouted Nazi slogans and unfurled a large banner with the slogan "Abolish the Jewish Lobby".
"Free speech is not about the right to vilify and to engage in anti-Semitic behaviour and to encourage hatred and division and violence, which is where this all leads," Mr Albanese told ABC Radio on Tuesday.
"This rise of people openly identifying as Nazis has no place in Australia."
Premier Chris Minns said he would aim to legislate new laws cracking down on hate speech in the coming weeks after attendees at the rally chanted "blood and honour", which is a Hitler youth slogan.
"We need to draw bigger, bolder, bright lines between hate speech and free speech so that those scenes are not repeated," he told reporters on Monday.
"It's illegal to have Nazi symbols in NSW but not the Nazi speeches or Nazi slogans."
Federal parliament in February mandated courts jail anyone displaying a Nazi symbol for at least a year and for up to five.
Mr Albanese also criticised the NSW protest permit system after police were given a week's notice about the neo-Nazi rally but decided against taking it to court.
"The fact that these Nazis in NSW notified the police and were essentially given permission to conduct such a hateful activity is completely unacceptable," the prime minister said.
Assistant NSW Police Commissioner Peter Thurtell reportedly apologised to newly installed Commissioner Mal Lanyon for not informing him of the protest, deeming it an oversight on his part.
"In hindsight, the nature of the 'White Australia' group is of such political interest and community interest to our organisation, that I should've advised the commissioner that this protest was taking place," he told the Daily Telegraph.
Police sought legal advice about a potential court challenge but were told it would likely fail.
The communication breakdown had been described by federal MP Allegra Spender and Liberal state MP Kellie Sloane, both targets of violent sexual threats on social media from neo-Nazis, as a "complete stuff-up".
The politicians' overlapping electorates in eastern Sydney include large Jewish populations.The same fringe group had rallied outside parliament in June when participants wore black uniforms and displayed a banner with the words "End Immigration".
18°C