Olly Robbins' account deepens a row that threatens the British leader, who is facing calls to resign over the appointment of Labour veteran Mandelson despite his history and known ties to late US sex offender Jeffrey Epstein.
Starmer has said he was wrong to appoint Mandelson to the role and has expressed regret, but on Monday put the blame firmly on foreign ministry officials for failing to tell him that a security vetting body had advised against his appointment.
Robbins was the foreign ministry's former top official who was sacked on Thursday after Starmer and foreign minister Yvette Cooper said they had lost confidence in him.
"I walked into a situation in which there was already a very, very strong expectation ... that he needed to be in post and in America as quickly as humanly possible," Robbins told a parliamentary committee on Tuesday.
"I think throughout January (2025), honestly, my office, the foreign secretary's office, were under constant pressure," Robbins said.
"There was an atmosphere of constant chasing," he added, describing "very frequent phone calls" from the private office of Starmer's Number 10 Downing Street.
Robbins said that when he took office on January 20, 2025, Mandelson's appointment had already been announced, approval had been given by King Charles, it had been agreed by the US government and Mandelson was being granted access to highly classified briefings on a case-by-case basis.
He said it would have damaged relations with the US if the foreign office had blocked the appointment at that stage.
It was not clear whether Downing Street even wanted the foreign ministry to complete so-called developed vetting clearance - a status that allows individuals access to information regarded as top secret, Robbins said.
It was down to the foreign ministry to complete that process, he said, and officials had stuck to a system whereby reports from a unit called UK Security Vetting were never shared with ministers to protect the candidate's confidentiality.
But he did say the unit had advised the appointment was a borderline case and they were leaning against granting clearance - a message Starmer says his government never received.
"Whilst I think the department felt under pressure, we were proud of the fact we'd not bowed to that pressure," Robbins said.
Robbins said the concerns about Mandelson did not relate to his relationship with Epstein, but declined to say what they did relate to.
Even some in the government's top team of ministers have said they were concerned about Mandelson's appointment.
Asked what went through his mind over the decision to appoint him, energy minister Ed Miliband told Sky News: "That it could blow up, that it could go wrong."
"I had a conversation with David Lammy (then foreign minister and now deputy prime minister) about it before the appointment, and I said I was worried about it."
Robbins also revealed Starmer's office had been pushing for an ambassadorial job to be found for Matthew Doyle, the prime minister's former senior communications director who has been removed from the Labour Party after he campaigned for a man who was a convicted sex offender.
Doyle has apologised for his actions.
with AP