Europe’s traditional social democratic parties are on the back foot. In UK local elections and a by-election last week, Labour largely failed its first proper electoral test with Sir Keir Starmer as leader.

In Madrid, Spain’s Socialist party lost more than a third of its vote share while the centre-right regional president triumphantly doubled hers. In Germany, the centre-left SPD trails in distant third place, months before a general election.

It is the second paradoxical failure of European social democrats in as many decades. The global financial crisis seemed a golden opportunity for party traditions founded on reining in market excesses, but most of them failed to make themselves relevant. The pandemic has triggered a desire for more solidarity and has increased voters’ tolerance for a more interventionist state. But again social democrats seem unable to grasp the moment.

Their problems, unfortunately for them, are structural. Socio-economic change has made their traditional coalition of the industrial working class and urban middle-class liberals — often public sector employees — more unbalanced and disparate in terms of economic interest and social values.

The shift from an industrial economy to one based on knowledge services has shrunk the pool of factory workers and hurt the prospects of the groups they would previously have been recruited from. It has, meanwhile, worked out well for educated city dwellers. Straddling the two to fend off rivals — rightwing populism on one side and green-left liberalism on the other — has become ever harder.

Many of Europe’s social democratic parties are also burdened by past political choices. Some participated in post-financial crisis austerity drives that hollowed out public services, or at least tacitly endorsed the policy. In the decades before, the “third way” then in the ascendant often seemed indifferent to rising differences in the pre-tax incomes of individuals and whole regions, even when some — not all — increased redistribution.

They stood behind economic reforms that often aimed to boost work by making it more painful to be poor. No wonder that many of their former supporters ask what Europe’s social democratic parties are for.

There is reason to think this electoral dilemma could become more painful still. The transition to low carbon drives the split inside the old coalition even deeper: the traditional industrial working class fear for their jobs if fossil energy use is penalised, while urban liberals are the strongest supporters of the economy’s greening.

The left overall is still doing well in many countries, especially where electoral rules allow parties to cater to narrower groups. The problem for the old centre-left “people’s parties” — as indeed for their rivals on the right — is that such fragmentation forces coalition-building to take place between parties rather than within them.

It would be wrong, however, to proclaim the death of Europe’s social democratic parties. They have a natural constituency among precarious service workers, where the biggest challenge may be an apathy towards voting altogether, rather than rival parties. And they have an obvious role in formulating policies centred on green jobs — if they do not let other parties take ownership of such strategies before them.

Indeed there is one place where European-style social democracy increasingly seems to be thriving: in the engine room of Joe Biden’s White House, so far with the approval of many US voters. Socialism never quite made it to America. But Europe’s social democrats may find they can take inspiration from across the Atlantic.