Jones previously coached Japan for three years from 2012 and led them to their historic win over South Africa at the 2015 World Cup.
Speculation was rife that he would return after Australian media reported that he had interviewed for the job during the Wallabies' World Cup campaign in France.
The pugnacious 63-year-old repeatedly denied being linked with the job.
Japan Rugby Football Union CEO Kensuke Iwabuchi said Jones had fulfilled their wish to find a coach who both knew Japanese and international rugby.
"In terms of expressing what he wanted to do, identifying what the problems were and how he intended to tackle them, he was better than the other candidates," he said.
Iwabuchi said that the JRFU had not interviewed Jones for the job during the World Cup.
But he admitted that there had been contact through a third party shortly before the tournament to "gather information".
Iwabuchi said Jones had been chosen ahead of two other final candidates.
South African Frans Ludeke, who coached Kubota Spears to last season's Japanese club title, told reporters last week that he had been invited for a final interview.
Iwabuchi said there had been debate within the JRFU as to whether Jones had already hit his coaching peak.
Jones quit the Wallabies after just two wins from nine Tests since taking over in January, including a worst-ever World Cup performance where they failed to make it out of the pool phase.
"For us, the biggest point was can he lead us on the path to winning the World Cup?" said Iwabuchi.
Jones will officially take over on January 1.
He is set to hold a press conference in Tokyo on Thursday.
'Betrayed'
Despite being fired by England last December after their worst annual return in 14 years, he was hailed as the saviour of an underperforming Wallabies outfit.
But his insistence on fast-tracking rookies at the expense of veteran stars badly backfired in high-pressure games, as did a revolving door of unproven captains.
Being linked to the vacant Japan job didn't help his public persona either.
"Australia feels betrayed, embarrassed and humiliated, as much by Jones' dalliance with Japan as our nation's sorrowful World Cup effort," The Australian broadsheet declared.
Jones, whose wife is Japanese and mother is Japanese-American, has maintained links with the country since leaving the Japan job in 2015.
He continued to work as a consultant to club side Tokyo Sungoliath while coaching England, making regular visits.
He told The Australian in an interview that he had been "living apart from my wife because she lives in Japan".
"I want to stay married. I think at 63 I don't want to get divorced," he said.
Jones led Japan to unprecedented success in his previous spell in charge of the Brave Blossoms.
They stunned South Africa 34-32 at the 2015 World Cup in a match that became known as the "Miracle of Brighton".
That was only Japan's second win at a World Cup, and they also went on to beat Samoa and the USA that year before exiting at the pool stage.
Jones's successor Jamie Joseph built on that foundation, taking Japan to the 2019 World Cup quarter-finals on home soil.
Joseph announced before this year's World Cup that he would step down after the tournament.
Japan failed to make it past the pool stage after losing to Argentina in their final game in France.
