La Rochelle set up Top 14 semi-final against Toulouse
Toulon, playing in the post-season for the first time since 2018, were indebted to full-back Melvyn Jaminet's kicking accuracy for all of their 15 points in the first half.
However, they were trailing 24-15 by that stage with fly-half Antoine Hastoy, winger Dillyn Leyds and centre Jules Favre all scoring tries as La Rochelle opened a nine-point lead.
After the interval, penalties and errors proved costly for Toulon and they quickly conceded a fourth try when flanker Oscar Jegou crossed.
Hastoy kicked his fourth conversion as La Rochelle stretched their advantage to 31-15.
La Rochelle forwards Will Skelton and Judicael Cancoriet were both yellow-carded in the last 10 minutes.
Toulon were awarded a penalty try before English hooker Jack Singleton cut the gap further with a try with the last move of the game.
On Sunday, Bordeaux-Begles face Racing 92 in the second play-off with the winners to tackle Stade Francais for a place in the final.