"I have taken the decision to step down as president of UMNO and chairman of Barisan Nasional (BN) immediately," he told a press conference, surrounded by senior party members.
The United Malays National Organisation (UMNO) is the main party in the coalition.
An opposition alliance headed by new Prime Minister Mahathir Mohamad, 92, inflicted a shock defeat on BN in Wednesday's election, ejecting the coalition from power after more than six decades of unbroken rule.
Mahathir, the world's oldest elected leader, had retired as premier in 2003 after over two decades in the top job but staged a surprising comeback in a bid to oust his ex-protege Najib.
He was spurred out of retirement by allegations that Najib oversaw the wide-scale looting of sovereign wealth fund 1MDB. Najib and the fund deny any wrongdoing.
"If the party failed in the general election, the leader has the moral obligation to step aside. So based on this principle I have decided to give up both positions," Najib said.
Ahmad Zahid Hamidi, the former deputy prime minister, would take over as UMNO president, he said.
"I urge all UMNO members to unite and support the new leadership."
Earlier, Najib was banned from leaving the country with his wife after speculation mounted that they were about to flee, in a possible bid to avoid prosecution over the 1MDB case.
He insisted he was only planning to take a short break and return to Malaysia next week.