Prime Minister Shinzo Abe said his government would "start over from zero".
The original design, by British architect Zaha Hadid, had come under criticism as estimated building costs almost doubled, reaching $2bn (£1.3bn)
Mr Abe says the new stadium will still be completed in time for the games.
However, the delay means that the stadium will no longer be ready in time for the 2019 Rugby World Cup, which Japan is also hosting.
World Rugby said it was "extremely disappointed" and was "urgently seeking further detailed clarification".
Japanese officials say the contract with Zaha Hadid's architecture firm will be cancelled, and a new design chosen within six months.
Zaha Hadid Architects said that the stadium the firm had designed could be built cost-effectively.
"It is not the case that the recently reported cost increases are due to the design," the firm said in a statement.
The real challenges were "increases in construction costs in Tokyo and a fixed deadline", it said, adding that building costs in Tokyo were higher than many other places as the risk of earthquakes meant that strict safety standards were needed.
Under the original plans, Tokyo's stadium would have been bigger and more expensive than any of its recent predecessors.
It drew increasing criticism as estimated costs spiralled from $1bn to $2bn.
The futuristic design of the stadium also drew attention, with architects likening it to a turtle or a bicycle helmet.
Announcing the cancellation on Friday, Mr Abe said: "I have been listening to the voices of the people and the athletes for about a month now, thinking about the possibility of a review."
"We must go back to the drawing board," he added. "The cost has just ballooned too much."
He said that he had made the decision after being assured that it was still possible to complete construction of a new design in time for the Olympics.