Zheng, the women's world number seven, fought back from a break down in the deciding set, sealing a 6-7 (4/7), 6-4, 7-6 (8/6) win on her fourth match point.
Only China's Li Na has previously played an Olympic singles semi-final in either men's or women's tennis, reaching the last four at the 2008 Beijing Games.
There were six breaks of serve in a topsy-turvy first set before 36-year-old Kerber, a silver medallist at the 2016 Games, won the tie-break.
Zheng battled back to level the match but was in deep trouble when trailing 4-1 in the decider.
However, she broke Kerber to love in the seventh game to rein in the German and get back on serve, with the set going to a tie-break.
Zheng, who hit 64 winners in total, took charge against her exhausted opponent in the breaker, establishing a 6-3 lead, but she squandered three match points.
However, as the drama mounted under the roof on Court Philippe Chatrier, Zheng made her fourth match point count as Kerber dumped a forehand into the net in the final shot of her tennis career.
The German, now ranked at 212 in the world, announced last week that she would retire at the end of the Games after returning from maternity leave at the start of the season.
"What can I say? I gave everything," she said. "I have loved playing tennis. Now I can leave in peace."
