“Goodbye, high hopes and overconfidence,” John Betjeman lamented, in his Metro-land documentary about the expansion of London’s commuter buckle up to Amersham.

Maybe the same could be said of Boris Johnson, after the Conservatives’ shock by-election defeat in the same area? Certainly the result should temper the idea that the prime minister is an unbeatable political force.

In the 13 previous elections since the seat of Chesham and Amersham was created in 1974, the Conservatives had never received less than 50 per cent of the vote. On Thursday, despite the Johnson magic, they took just 36 per cent — and lost the constituency to the Liberal Democrats.

Like Betjeman, local residents didn’t like new housing developments intruding on this green corner of England. They also didn’t think much of Brexit (55 per cent voted Remain), or HS2, the high-speed rail line that will link London and the Midlands and the north, at the expense of the intervening suburbia. This was the revenge of the Metropolitan Line elite.

The result helps to put into perspective the Conservatives’ supposed triumph in elections in May. Yes, they won Hartlepool, but otherwise the picture was more mixed than the narrative suggested. And incumbents were flattered by a mood of crisis-cum-euphoria, with the successful vaccine rollout and the end of England’s third lockdown. Just six weeks later, politics is normalising.

The question is whether centre-left parties can assemble a majority built around university-educated and younger voters, even while the Conservatives make inroads among older, Brexity voters in the north.

Unhelpfully, Remain voters have been concentrated in cities like London and Manchester (and in Scotland). But as millennials look for family homes they can afford, and as working from home becomes a permanent shift, they are moving into the surrounding towns and countryside. That should balance the electoral geography a little.

Meanwhile, the Lib Dems have begun to return from the wilderness. They finished second in 91 seats in the last general election, up from just 38 in 2017.

Whether they have a coherent platform is unclear. Despite wooing Chesham and Amersham voters, the Lib Dems actually support HS2. They do well in affluent constituencies by opposing new houses. You can argue the Conservatives deserve to be punished by nimbyism. If I wanted to persuade local residents to accept housing developments, I would start by sacking a housing secretary who unlawfully approved one such development after sitting next to the billionaire behind it at a party fundraiser.

But progressives should not lose sight of the fact that homes need to be built somewhere. They should focus on the government’s other vulnerabilities. Johnson has a fragile coalition of big state and small state supporters. He has a levelling-up agenda that leaves the south cold; he sometimes seems happy not just to neglect people who have lived in London, but actively to insult them. It’s economically foolish, it may turn out to be politically foolish, too.

Any progressive revival relies on the Labour party. That is why recent Lib Dem by-election upsets — Richmond Park in 2016 and Brecon and Radnorshire in 2019 — had little lasting effect. At a by-election, voters decide whether to give the government a black eye. At a general election, they decide who they want the government to be. Right now, too few would choose Keir Starmer as prime minister.

Johnson has never been a hugely popular premier. He may even be, in the words of his former adviser Dominic Cummings, “a gaffe machine clueless about policy”. But for a sizeable chunk of the electorate, he just remains more personally palatable than the alternatives. Starmer’s job is to change that.