John Bercow, the former speaker of the House of Commons and former Conservative MP has joined the Labour party, arguing that the party risks alienating voters in the south of England.

Just days after the Tories lost the safe seat of Chesham and Amersham to the Liberal Democrats in a by-election, Bercow told the Observer newspaper his former party was “reactionary, populist, nationalistic and sometimes even xenophobic”.

The former speaker also said many traditional voters particularly within the south expressed “considerable distrust” towards the Tory government prompted by its handling of issues such as the foreign aid cuts and the controversy surrounding catch-up funding for the education sector.

Bercow, who had been a member of the Conservative party for 29 years, was MP for Buckingham between 1997 and 2019 and held the position of speaker of the House of Commons from 2009 and 2019. When he became speaker, Bercow resigned his party membership, as is standard practice.

In his later years as Speaker, Bercow irked some within the Conservative party over his perceived anti-Brexit stance, while his tenure was dogged by bullying allegations — claims which he denied at the time.

Speaking to Sky News on Sunday, Bercow said that he no longer identified with the Conservative party and that his values were now “left of centre”, adding that he believed that the government was “unfit to govern”.

“The reality of the matter is that in the United Kingdom, the only credible vehicle for the removal of this government, is the election of a Labour government, committed to equality, social justice, internationalism and a fairer and more inclusive society. That is the Labour brand”, he said.

Responding to his comments, justice secretary Robert Buckland rejected Bercow’s claims that the party was xenophobic but said that as a private citizen the former speaker was entitled to his own political opinion.

“As a former speaker, he is somebody who even though he has left office does carry a degree of authority like his predecessors”, he said. “But I think him joining a political party actually has the effect of diminishing his voice in politics however strong he wants it to be.”

Following the shock loss of Chesham and Amersham, a previously safe Conservative seat in Buckinghamshire, Buckland on Sunday also pushed back on suggestions that the Conservative party had focused too heavily on voters in red wall seats in the north of England at the expense of voters in southern seats.

The Liberal Democrat candidate Sarah Green won Thursday’s by-election with a majority of 8,028 votes, overturning a Tory majority of around 16,000.

“I don’t accept that choice — the idea that somehow voters in the north or voters in the south are an entirely different species is frankly patronising and wrong,” Buckland said. “Clearly a message was sent, it was a strong message and it is one that we are listening to very carefully.”

Liberal Democrat leader Sir Ed Davey hailed the by-election as a “huge moment” for the party, adding that the win illustrated that Conservative prime minister Boris Johnson could be beaten.

Davey said the Lib Dems would be concentrating resources in areas where they were best placed to win, notably, in “blue wall” seats in the south of England.

“The Liberal Democrats under my leadership will be a campaigning party. We will go out there and campaign for every vote but we will use our resources wisely,” Davey added.