Boxing has been part of every Games since 1920, but faces an uncertain long-term future after its place at the 2028 Los Angeles Olympics was put "on hold" by the International Olympic Committee.
Garside, who won lightweight bronze in Tokyo and is bidding for gold in Paris, said it was more than a sport, with young people's dreams and aspirations at stake.
"All of our dreams started somewhere, and whether you're a boxer, whether you're in other sports, we all love the Olympic movement. That's why we're here," he said.
"To take that away from some young people around the world, it would be a crime, truly."
The International Boxing Association was effectively expelled from the Olympic movement last year following a bitter dispute with its Russian president, Umar Kremlev.
It followed concerns over the credibility of IBA-sanctioned tournaments as well as the boxing governing body's finances and governance.
No other boxing governing body has been recognised by the IOC, which is organising the tournament in Paris but has said it will not be in a position to do so in Los Angeles.
Garside, Australia's first boxing medallist in 33 years, said "politics is politics", but powerbrokers must think about the consequences.
"To say it may not be in the Olympic Games is obviously terrifying because I think boxing is historically a poor man's sport," he said.
"If you look around the world, it's the Third World countries that do the best. It's the countries that come from these really rough areas in the world. And they're the ones that come and win gold medals.
"Boxing deserves to be in the Olympics," he added. "It has saved many people's lives, and I've seen it save many people's lives. So I hope it stays."
