An Iranian diplomat instead insisted any such visit would only come after a final deal.
The US and Iran offered contradictory remarks earlier about whether those sites would be inspected.
A week after the two countries signed an interim deal, leaders from the US and Iran have repeatedly disagreed in public about what that document actually means.
International Atomic Energy Agency head Rafael Mariano Grossi on Wednesday acknowledged the "war of words" over Iran's nuclear program, but the duelling narratives are playing out on a range of issues, including Israel's war with Iran-aligned Hezbollah militants in Lebanon and how Iran will spend billions of dollars once unfrozen.
After the signing of the memorandum of understanding, both sides agreed to a 60-day period to iron out these and other details, work that will take place in private.
Until those talks are completed, leaders from both countries will continue to negotiate in public, raising the risks of derailing the shaky ceasefire in the region.
Since Israel launched a 12-day war on Iran in 2025, the IAEA has been blocked by Iran from visiting enrichment sites where the Islamic Republic is believed to store enough highly enriched uranium to potentially build as many as 10 nuclear weapons, should it choose to rush for the bomb.
Iran long has maintained that its program is peaceful although it is the only country in the world to have uranium enriched up to 60 per cent purity without a weapons program.
"I can understand political statements, they are part of the reality, but the fundamental thing I would like to remind you and draw your attention to is that there has been a Memorandum of Understanding, signed by both presidents," he told reporters at a news conference at the tsunami-hit Fukushima Daiichi nuclear power plant.
The accord "says explicitly that the nuclear activities that are going to be carried out with regards to the nuclear material facilities will be supervised by the IAEA - in all letters," he said.
"Obviously, to do that, we will have to inspect," Grossi added.
"Whether this happens the day after tomorrow or in one week or in 10 days, it's important but not essential. This is going to happen."
Those inspections are key for the deal, which calls for Iran's stockpile of uranium to be "downblended" from highly enriched levels.
Kazem Gharibabadi, an Iranian deputy foreign minister, replied online to Grossi's remarks, saying Iran did not meet with him while in Switzerland.
"These issues will be reviewed and decided only within the framework of a final agreement and as a result of practical action by the other side to end all sanctions and other measures," Gharibabadi wrote on X.
He added: "You cannot advance the 'stir up and take over' policy with media hype".
The IAEA has been allowed to visit other nuclear sites in Iran since the 12-day war in 2025, such as the Bushehr nuclear power plant.
But without accessing the enrichment sites, the IAEA says it is unable to verify the status of Iran's stockpile or check the cascades of centrifuges used to enrich uranium.
Both Iran and the IAEA say Iranian authorities have not been enriching uranium but nonproliferation experts worry that Iran may be moving its stockpile to undeclared areas.
Grossi's remarks came as US Secretary of State Marco Rubio travelled to the United Arab Emirates and Kuwait, and was expected to later visit Bahrain.
