Flick became the first and only Germany head coach to be sacked since the position was created in 1926.
Sporting director Rudi Voeller, who coached the side to a 2002 World Cup final against Brazil, will take over on an interim basis.
German FA boss Bernd Neuendorf said the sacking was "unavoidable" after the team's recent run of "disappointing" results.
In a statement, Neuendorf said "with a view towards the European championships in our own country, we need a spirit of optimism and confidence", saying the decision was one of his "most difficult so far."
Voeller is reportedly unlikely to lead Germany to Euro 2024.
Out-of-work manager Julian Nagelsmann, who previously replaced Flick as Bayern Munich manager, has been touted as a possible replacement ahead of the tournament.
Former Eintracht Frankfurt boss Oliver Glasner, out of work since May, has also been named as a possible successor.
The Germans were thoroughly outclassed by the Blue Samurai on home soil in the friendly in Wolfsburg, with only some acrobatic 'keeping from Marc-Andre ter Stegen preventing the score from eclipsing 2001's 5-1 loss to England in Munich.
Germany have now lost four of their six matches since their early exit from the World Cup in December.
Euro 2024 kicks off in Germany in June 2024.
The home fans booed the German team at half-time, with some chanting "Hansi out" at the end of the match.
Germany have been historically reluctant to make drastic changes to the coaching position, with none of Flick's predecessors sacked.
Appointed in May 2021 after Joachim Loew's 15-year tenure ended, Flick was Germany's 10th head coach since World War II.
Fresh off taking Bayern Munich to seven trophies in just 18 months in charge, Flick was victorious in his first eight matches as Germany boss.
Since then he has won just four of 17 matches and presided over a group stage exit at the Qatar World Cup.
Flick was bullish after Saturday's latest loss, telling reporters he "was convinced of what we are doing" and "wanted to continue" in the top job.
On Sunday at a training session following the Japan rout, Flick told fans "I'll keep fighting", but admitted to AFP subsidiary SID "many things are difficult to predict" in football.
Voeller said after the match emotions were running too high to make an immediate call on Flick's future.
"I suggest we collect ourselves and sleep on it" Voeller said, explaining "we are all a little in shock... a defeat like that hurts."
Taking over as interim boss in 2000, Voeller coached Germany to the final of the 2002 World Cup where they lost 2-0 to Brazil.
His first match as caretaker coach is on Tuesday, in a friendly in Dortmund against World Cup runners-up France.
