An AFP journalist saw mothers holding their children's hands and families piling onto donkey carts with their luggage as they made the journey.
They hoped to cross a military checkpoint on Al-Rashid road south of Gaza City, but the Israeli army told AFP that reports the route was open were "not true".
On the other side, desperate families waited for their loved ones in the rubble of the battered main city in the Palestinian territory.
Mahmoud Awdeh said he was waiting for his wife, who has been in the southern city of Khan Yunis since the start of the war on October 7.
"She told me over the phone that people are leaving the southern part and heading to the north," Awdeh said.
"She told me she's waiting at the checkpoint until the army agrees to let her head to the north," he said, hoping she would be able to cross safely.
During the day rumours also spread that the Israeli army was allowing women, children and men over 50 to go to the north, a claim denied by the army.
Since Israel's assault on Gaza following the October 7 Hamas attack, the army has besieged the territory, telling Gazans to leave some areas and preventing them from moving across the narrow strip.
More than 1.5 million Palestinians have taken refuge in the southern city Rafah, according to the United Nations.
Several Gazans said they came under attack on the route and AFP footage showed people ducking for cover.
The Palestinian official news agency Wafa said Israeli forces "bomb(ed) displaced Palestinians as they were trying to return to the north of Gaza Strip through Al Rasheed street."
Wafa shared a video on X which AFP has not verified showing people running away from a blast.
Nour, a displaced Gazan, told AFP: "When we arrived at the (Israeli) checkpoint, they would let women pass or stop them, but they shot at men so we had to return, we didn't want to die."
AFP has approached the Israeli military for comment.'Too little too late'
Elsewhere in Gaza the fighting continued on Sunday after Iran launched a huge drone and missile attack on Israel overnight.
Iran's first-ever direct assault on Israeli territory came in retaliation for a deadly strike on Tehran's consulate in the Syrian capital.
The strike that Iran blamed on Israel left seven Revolutionary Guards dead, including two generals.
But in Rafah on Sunday, Palestinians told AFP they were underwhelmed by Iran's attack on Israel.
"The Iranian response came so late, after 190 days of war," Khaled Al Nems told AFP. "You can see our suffering."
"Their response is too little too late," he added.
Walid Al Kurdi, a displaced Palestinian living in Rafah, said that "Iran's attack on Israel is not really our business".
"They only thing we care about is going back to our homes," he said.
"We are waiting for the coming 48 hours to see if (Israel) responds to Iran, or if they are playing with us and want to distract attention away from Rafah."
Israel has said it plans to send ground forces into Rafah to eradicate remaining Hamas militants there.
