After dropping the first set to the seventh seeded Paolini on Court One, Keys hit back to level the match at 3-6, 7-6 (8/6).
The 2017 US Open runner-up was on the verge of reaching a third Wimbledon quarter-final after breaking twice in the final set to build a 5-2 lead.
But Keys suffered a left leg injury in the eighth game of the decider, leaving her limping and in need of a 10-minute medical time-out when she was leading 5-4.
The 29-year-old returned with a heavy strapping on her leg and was visibly upset as she struggled to move and was immediately broken by Paolini.
Keys gamely tried to continue playing but, after failing to run for a Paolini drop shot, she finally decided to retire with the score level at 5-5 in the deciding set.
Paolini will face reigning US Open champion and second seed Coco Gauff or American 19th seed Emma Navarro for a place in the semi-finals.
"I'm very sorry for her. It's sad. Really good match. Really tough, lots of up and downs," said Paolini.
"I feel a bit happy but also sad for her because it's not easy to win like that."
Paolini, who finished as the French Open runner-up in June, had never been past the first round in her previous three appearances at Wimbledon.
She has reached a Grand Slam quarter-final for the second time, but her progress was overshadowed by the emotional exit of Keys, who was still weeping as she walked off court.
