Patty Mills scored 26 points for Australia, including 14 in a row during a 20-0 run spanning the first and second quarters, but Serbia hung tough and gradually erased the deficit to pull off an epic comeback.
"I thought we threw everything at them, hell of a first half," said Mills. "In international basketball, once you lose that momentum, it's hard to get it back. Maybe not enough in the tank at the end of the day."
"It's heartbreaking to go out this way. We gave ourselves every chance," added Josh Giddey, who finished with 25 points for Australia.
Three-time NBA MVP Jokic tallied 21 points, 14 rebounds and eight assists and received plenty of help from his supporting cast. Bogdan Bogdanovic had 17 points as one of five other players to hit double figures for Serbia.
"They destroyed us at the start of the game. They played so well, they smelled blood. We didn't respond well. We had the right attitude but physically we were not ready to respond," said Serbia's Vasilije Micic.
"We had great players who led us in the last 25 minutes."
Australia made it through the group stage despite losing two of three games, whereas Serbia pipped South Sudan to second in their section behind Team USA.
Serbia, beaten 110-84 by the US in their tournament opener, could meet the reigning four-time champions again in the semi-finals if the Americans get past Brazil later on Tuesday.
Australia broke open a tight game early on, fuelled by the hot hand of Mills, the 35-year-old former NBA champion who played sparingly for Atlanta and Miami this season.
Mills drained back-to-back threes as part of a spree that fired Australia in a 44-20 lead early in the second period.
But Serbia chipped away at the deficit and slashed it in half by the interval, eventually nudging in front when Bogdanovic hit a short jump shot with 13 minutes to play.
Serbia outscored the Tokyo bronze medallists 25-11 in the third quarter to take back control, but Giddey ensured Australia didn't go away quietly and Mills sent the contest to overtime.
Mills sank a rainbow jumper over Jokic with 1.4 seconds to play to keep Australia alive, but it was the towering centre who had the final say as Serbia reeled off the final eight points in overtime.
Australia led 90-87 after a three from Giddey but Jokic produced a key block under his own basket and then connected with a hook shot and made a pair of free throws to put Serbia ahead for good.
