The dismissal is the latest in a string of scandals to hit the Games.
In March, creative chief Hiroshi Sasaki quit after suggesting that plus-size comedian Naomi Watanabe could appear as an “Olympig”. He later apologised.
And in February, Yoshiro Mori was forced to step down as the head of the organising committee after he made remarks about women that were criticised as “inappropriate”.
Mr Mori was quoted as saying women talked too much and that meetings with many female board directors would “take a lot of time”.
This latest scandal has seen former comedian Kobayashi strongly criticised for a sketch he performed 23 years ago, in which he and another comedian pretend to be children’s entertainers. Mr Kobayashi then turns to his colleague, referring to some paper dolls, saying they are “the ones from that time you said ‘let’s play the Holocaust’”, according to news agency AFP.
“Any person, no matter how creative, does not have the right to mock the victims of the Nazi genocide,” Rabbi Abraham Cooper, the associate dean and global social action director of the Simon Wiesenthal Center (SWC), said.
The scandals have done little to stop rising unease about the Games, which were postponed from last year due to the pandemic.
A recent poll found some 55% of people in Japan were opposed to holding the Games, amid fears it could become a super-spreader event, news agency Reuters reports.
An increase in cases among Japan’s population – only of a third of whom have been vaccinated – has also led to a state of emergency being declared for the duration of the Games.
The opening ceremony will officially kick off two weeks of competition. However, after the removal of Mr Kobayashi, organisers are now re-assessing how to hold Friday’s event – which will only be attended by 950 people, in order to minimise risks.