Heavy rain at the castle -- originally built in 1607 during Japan's feudal era -- caused a stone wall supporting the structure to crumble.
The spectacular collapse came amid a 20-year project to repair damage to the castle following a series of devastating quakes in April 2016.
The castle complex is surrounded by hundreds of cherry trees and was once a huge tourist draw, but had been closed to the public since the quakes.
The damaged "Mototaiko Yagura" tower was located close to the main gate of the castle, Tatsuo Nomoto, a municipal official at the castle, told AFP.
"Stories have it that it was used to beat drums when people came through the main gate," he said.
The spectacular collapse came amid a 20-year project to repair damage to the castle following a series of devastating quakes in April 2016.
A guard patrolling the castle complex found the tower in ruins late Wednesday, he said, adding that no one was injured in the incident.
The tower had already been severely damaged by the 2016 quakes, which killed 267 people and injured more than 2,800 others.
The castle was devastated during a civil war in the late 1800s, with the existing buildings rebuilt in subsequent decades.
Japan is prone to natural disasters such as earthquakes and heavy rains routinely cause landslides.
Most recently, Japan's second city of Osaka experienced a major quake that killed at least five while injuring 370 others.