Sophia Smith scored twice in the first half either side of a Mallory Swanson goal, and Lynn Williams came off the bench to seal the victory late on a sweltering night in Marseille as the USA secured a place in the quarter-finals.
"It was a mature performance. You have to suffer, of course you do, and they are a top team, but I thought we were devastating when we needed to be," Hayes said of her side's second win from as many matches in Group B.
"There are still things that irritate me about us, but that is for me to fix and I say that genuinely, purely because I think we could have controlled it even more."
The performance by the record four-time women's football gold medallists followed a 3-0 win against Zambia in their opening match in France.
A draw against Australia at the same Stade Velodrome in Marseille on Wednesday will secure top spot in the group, although their goal difference advantage is such that even a defeat may not prevent them finishing first.
Topping the group will take them to Paris for a last-eight tie on Saturday against the runners-up in Group C which contains heavily-fancied World Cup holders Spain as well as Japan, Brazil and Nigeria.
"I am not here to wow everyone for a game and not do it again. We have to build the momentum," said English former Chelsea manager Hayes, who has now been in charge for only six matches.
"We have to analyze things that are in our control. And we have to win against Australia, simple as that."
'Dynamic as hell'
Sunday's result against the 2016 Olympic gold medallists was without doubt the USA's most impressive performance in a major tournament since their triumph at the 2019 World Cup.
It came after Hayes named an unchanged line-up from the opening game against Zambia, spearheaded by an exciting front three with young stars Smith and Trinity Rodman alongside Swanson.
The latter missed last year's World Cup with injury but has three goals so far in two games at the Olympics.
"I think the front three in general were dynamic as hell, really fun to watch and most importantly they enjoyed themselves," said Hayes.
"This is probably only game six or seven of us playing up front together but I feel like we are clicking really well, really fast. This is only about 70 percent of what we can do," said Smith, the Portland Thorns attacker.
Smith, 23, revealed herself during last year's ill-fated World Cup campaign but is now maturing into a key player up front for the US. She was full of praise for the work being done by new coach Hayes.
"She is always teaching us so many things and we absolutely love Emma," Smith said.
