With the scores locked at 1-1 and the match looking destined for extra time, defender Gaus netted in the sixth minute of injury time to grab his team a famous victory.
"When a first-division team like Bayern comes to Saarbruecken and loses, then we did something wrong," veteran Thomas Mueller told ARD after the match.
Bayern were beaten by Holstein Kiel on penalties at the same stage three years ago, then thrashed 5-0 by Borussia Moenchengladbach in round two in 2021.
The record 20-time winners also lost in the quarter-finals at home to Freiburg last season.
"There could be 100 explanations or maybe none at all. We had injuries but some will say we should win anyway," Bayern coach Thomas Tuchel said.
An "incredibly proud" home coach Ruediger Ziehl said he had told his players that Thursday's training session was now "voluntary"
"If we don't celebrate today, then we can never celebrate."
Bayern, who travel to Bundesliga rivals Borussia Dortmund on Saturday, lost central defender Matthijs de Ligt to a potentially serious knee injury early in the match, leaving the side with only one fit centre-back.
Six-time Cup winner Mueller looked to have Bayern on the right track for a routine victory at their less-fancied opponents, before De Ligt injured his knee in a tussle soon after and signalled immediately to the bench.
The injury forced midfielder Joshua Kimmich into defence and appeared to put Bayern on the back foot against a resurgent Saarbruecken.
The hosts struck next, Patrick Sontheimer tapping in a pass from Lukas Boeder after Saarbruecken pressed midfielder Frans Kraetzig into a mistake.
Tuchel brought on the calvary, including Kingsley Coman, Jamal Musiala and Serge Gnabry, but star striker Harry Kane stayed on the bench ahead of Saturday's crunch match.
With the England captain warming up ahead of what looked like certain extra time, veteran defender Gaus popped up with a moment that will go down in Saarbruecken history.
Dortmund, Leverkusen into last 16
Earlier on Wednesday, a strike from Marco Reus gave Borussia Dortmund a 1-0 home win over Hoffenheim.
Teenagers Jamie Bynoe-Gittens and Youssoufa Moukoko were given rare starts as coach Edin Terzic shuffled his deck before Bayern's visit, but it was veteran forward Reus who gave Dortmund the lead late in the opening half.
Bynoe-Gittens dribbled down the left flank and shaped to shoot before finding Reus who looped in a one-touch finish to score his 165th goal for the club and give Dortmund a deserved half-time lead.
The hosts spurned chances but Hoffenheim posed little danger and were reduced to 10 men in injury time when former Liverpool defender Ozan Kabak saw red.
Dortmund sporting director Sebastian Kehl, who won the German Cup once during a 13-year career at the club, said: "Against a team which has won all of their away games, I thought we did very, very well."
Bundesliga leaders Bayer Leverkusen scored three goals in the final five minutes to win 5-2 at third-tier Sandhausen.
Argentinian World Cup winner Exequiel Palacios gave Leverkusen an early lead from the spot after Amine Adli was brought down in the box.
Sandhausen levelled twice early in the second half, Christoph Ehlich and Yassin Ben Balla scoring either side of a Jonathan Tah goal for the visitors.
Leverkusen forward Adam Hlozek scored in the 85th minute to give the visitors the lead, before Adli added two of his own to round out a nervous victory.
Elsewhere, Freiburg were beaten 3-1 at home by second-division Paderborn, meaning the 2022 finalists join Bayern, holders RB Leipzig, Champions League participants Union Berlin and five-time winners Schalke to be eliminated early.
Five-time winners Eintracht Frankfurt won at third-division Viktoria Cologne, while Hertha Berlin beat Mainz, Magdeburg edged out Holstein Kiel on penalties and four-time winners Nuremberg beat Hansa Rostock in extra time.
