Tokyo governor Yuriko Koike has refused to join a meeting with Yoshiro Mori, president of the city’s Olympic organising committee, as pressure grows on the former prime minister to resign after making sexist remarks.

Koike said a planned meeting with Mori and the International Olympic Committee “would not send a positive message”, raising doubts about whether he can still lead Tokyo 2020.

The row is rapidly becoming a symbolic moment for gender equality in Japan. Older politicians have dismissed the remarks as a gaffe, while a younger generation pushes back and corporate sponsors are tested on their rhetoric about diversity.

Mori told a meeting of sports officials last week that women do not belong on committees because they talk too much. “It takes twice as long. Women have a strong sense of rivalry. If one raises her hand to speak then all the others feel they have to do the same. So it ends up with everybody talking,” he reportedly said.

The 83-year-old former prime minister, who has a long history of similar remarks, apologised the next day and then sought to ride out the criticism. Rather than dying down, however, the incident has dominated public and parliamentary debate with the Olympic Games due to start in just 195 days.

A recent poll by broadcaster TBS showed that 60 per cent of the public want Mori to go. Tokyo 2020 said it would hold a meeting on Friday to discuss his remarks.

Sponsors have also begun to speak out. In a rare statement issued at Toyota’s earnings on Wednesday, Akio Toyoda, chief executive, said: “We are disappointed by the recent comments from the President of TOCOG, which are contrary to the values that Toyota respects and supports.”

One sponsor held board-level discussions about whether to pull out of the Olympics following Mori’s remarks, but opted to continue because of the work already put in, according to a person familiar with the conversations.

Nomura, the investment bank, said: “Although he apologised, Mr Mori’s comments were regrettable.”

Panasonic, Fujitsu, Japan Airlines, Mizuho Bank, Meiji Holdings, Sumitomo Mitsui Banking Corporation and Asics all gave statements to the Financial Times highlighting their commitment to diversity or criticising Mori’s remarks.

Even as the pressure from sponsors grows, the incident has highlighted the extent of Mori’s enduring political clout and the expectation among his contemporaries that they can brush off such rows.

Responding to reports that Olympic volunteers were quitting over Mori’s remarks, Toshihiro Nikai, the ruling Liberal Democratic party’s 81-year-old secretary-general, said they would “change their minds once things calm down”. If they insisted on quitting, he added, “then we’ll just have to recruit more volunteers”.

Yoshihide Suga, Japan’s prime minister, said Mori’s remarks were “unacceptable” but his future was up to the Tokyo organising committee.

Mori, Japan’s prime minister from 2000 to 2001, was once described as having “the heart of a flea and the brain of a shark”. Although little-loved by his colleagues, he won election 14 times and held all of the top jobs in the LDP.

Mori, a rugby player in his youth, became chair of the Japan Rugby Football Union, and is still a feared and influential figure in sports administration and among Japanese politicians with an interest in sport.

That background made him the obvious choice for political leadership of Tokyo 2020. Takao Toshikawa, a veteran political commentator, said he was “a don in the sporting world” but was always flawed as a politician.

“He’s always been quick to say whatever he’s feeling. He just can’t control himself,” Toshikawa said.