Uruguay, who finished the game with ten men after Nahitan Nandez was sent off in the 74th minute, will face Colombia in Charlotte, North Carolina, in their semi-final on Wednesday.
Colombia beat Panama 5-0 in Saturday's other quarter-final earlier in Arizona.
The other semi-final, in New Jersey on Tuesday, will see world champions Argentina face surprise package Canada.
In a city better known for boxing than for the beautiful game, it was a bruising contest with little quality play and a tournament high 41 fouls.
A poor quality playing field hardly helped with both teams struggling to produce their best football on an uneven surface.
The first - and best - chance of a game of few opportunities came in the 35th minute when Uruguay striker Darwin Nunez had a clear header in front of goal but mis-timed his effort which flew wide off his shoulder.
Within moments, Brazil created an opening of their own with Raphinha breaking clear, but Uruguay keeper Sergio Rochet stayed tall and made a vital save.
Marcelo Bielsa's Uruguay were as tenacious as always, harrying Brazil in midfield and never afraid to interrupt their flow with a foul.
Brazil resorted too often to long balls forward but with only their 17-year-old talent Endrick, in for the suspended Vinicius Junior, as a central striker they lacked the physical presence to make that approach effective.
The game deteriorated the longer it went on with foul after foul, not deterred by lenient refereeing.
But Uruguay's hopes of wearing Brazil down were dealt a blow when Nandez hacked down Rodrygo with a dangerous slide into his ankle and after a VAR review the defender was sent off.
From then on it was clear that Uruguay were simply trying to make it to full-time and penalties and with no extra-time in Copa America, they were able to achieve their aim,
After Federico Valverde scored with the first spot kick, Eder Militao saw his effort saved by the diving Rochet.
When Douglas Luiz hit the post for Brazil, they trailed 3-1 and Jose Gimenez had the chance to clinch the game for Uruguay. His effort was superbly saved by Alisson Becker.
Substitute Gabriel Martinelli scored to keep Brazil alive but midfielder Manuel Ugarte kept his cool to drive home the decisive kick and send the 15-times Copa champions into the last four.
