Bashagha entered Tripoli overnight after two months of stalemate between Libya's rival administrations, but withdrew hours later as fighting rocked the capital, his office said.
The crisis risks plunging Libya back into prolonged fighting after two years of comparative peace or returning it to partition between the eastern-backed government of Bashagha and a Tripoli administration under Abdulhamid al-Dbeibah.
The stalemate has already led to a partial blockade of Libya's oil facilities, cutting its main source of foreign revenue by half. Diplomacy to resolve the crisis or lay the ground for new elections is making slow progress.
The sound of heavy weapons and automatic gunfire crashed across the capital on Tuesday morning, as schools were cancelled and the normally heavy rush hour traffic was sparse.
However, in central areas, away from the clashes on the northeast side of Tripoli, there was little evidence of military activity with the government of Bashagha's rival Dbeibah still in apparent control.
Bashagha had entered Tripoli overnight accompanied by allied fighters in the hope of taking over the government but was quickly met by opposition from forces aligned with Dbeibah, who was appointed through an UN-backed process last year.
Bashagha's office had released a statement saying his government would carry out its duties from inside the capital but later said he and accompanying ministers would depart to prevent civilian deaths.
The effort came after previous attempts by Bashagha to enter Tripoli ended peacefully when his convoy was turned back by groups allied to Dbeibah.
Battlefront
Libya has had little security since the 2011 Nato-backed uprising that ousted Mu'ammar Gaddafi and it split in 2014 between rival eastern and western factions before a 2020 truce that brought it under a fragile unity government.
A plan to hold an election in December collapsed amid arguments among major factions and prominent candidates over the rules and the parliament, which had sided with the east during the war, moved to appoint a new administration.
The unity government's prime minister, Dbeibah, rejected the parliament's moves saying his administration was still valid and he would only hand over power after an election.
Bashagha, a former interior minister who like Dbeibah comes from the powerful coastal city of Misrata, has repeatedly said he would enter Tripoli without violence. His previous attempts to do so ended with his convoy blocked by rival factions.
Last week, the parliament said Bashagha's government could work for now from Sirte, a central city near the frozen front line between eastern and western factions.