“We reached an agreement on no spectators at venues in Tokyo,” Tamayo Marukawa said after talks involving local and national government officials, organisers, Olympic and Paralympic chiefs.
Most Olympic competitions will happen in Tokyo, but a few events will be held outside the Japanese capital.
The public has already been asked to stay away from the marathon, which is being held in the northern Hokkaido region to beat Tokyo’s summer heat.
And large parts of the torch relay traversing the country have also been held behind closed doors because of virus concerns.
The spectator decision comes after Japan’s government said on Thursday that Tokyo would be under a virus state of emergency from July 12 to August 22.