"I will not enter the Eleona Domain today, because Israeli security forces entered with weapons, without prior French authorisation, without agreeing to leave today," Barrot said at the scene, calling the standoff "unacceptable".
"This violation of the integrity of a domain placed in the care of France is liable to weaken the ties that I came here to cultivate with Israel, at a time when all of us need to help the region advance on the path towards peace," he added.
The foreign ministry in Paris later said Israel's ambassador would be summoned over the incident.
The Sanctuary of the Eleona sits in a Jerusalem compound owned by France, and along with three other sites in Israel makes up the French national domain in the Holy Land.
The sites have been the focus of diplomatic incidents in the past.
The national domain was attributed to France before Israel's creation and is administered as a private property by the French consulate in Jerusalem.
The AFP journalist saw Israeli police surround the two French gendarmes before pushing one of them to the ground.
The gendarme identified himself and shouted "Don't touch me" several times, according to the journalist.
Both gendarmes were then led into police cars. They were later released.
It was not clear why the Israeli police had entered the site.
"The Eleona Domain... has not only belonged to France for more than 150 years, but France also ensures its security, maintains it," Barrot said.
"The integrity of the four domains that France is responsible for here in Jerusalem must be respected," he added.
