Barbora Krejcikova of the Czech Republic beat American Coco Gauff 7-5, 6-4, meaning the 23-year-old Pole cannot now progress from the Orange Group to Friday's semi-finals.
Earlier on Thursday, Swiatek had kept her hopes alive by beating Russian Daria Kasatkina 6-1, 6-0.
But despite winning two of her three matches in the group, Swiatek's defeat against world number three Gauff on Tuesday ultimately cost her.
The WTA Finals in Riyadh was Swiatek's first tournament since she was knocked out of the US Open quarter-finals by Jessica Pegula in September.
The five-time Grand Slam winner is expected to end her season by representing her country at the Billie Jean King Cup from November 13-20 in Malaga, Spain.
Wimbledon champion Krejcikova will take on China's Zheng Qinwen, while Gauff will face world number one Aryna Sabalenka in the semi-finals.
