The giant inflatable figure of Boris Johnson soaring above Hartlepool’s waterfront, following this month’s Tory by-election victory in the town, symbolised the Conservative advance into Labour’s once solid “red wall” of working class seats in northern England.

Yet on the sun-drenched seafront of Worthing, a comfortable town on the south coast, a less-told political story is unfolding. While the focus has been on the prime minister’s progress in the north, party figures are warning of the danger of leaving the back door open in the south.

Dan Humphreys, Tory leader of Worthing council, said there was “a clear correlation” between voters hearing about Johnson’s grand plans to “level up” the north and a feeling that the town, traditionally seen as the staid neighbour of raffish Brighton, has somehow been overlooked.

While Johnson continued to demolish Labour’s red wall in the May 6 local elections, Sir Keir Starmer’s party was bucking the trend in parts of the South East, including Worthing, where it made five net gains on the 37-member local council.

Humphreys noted how across the “blue wall” of Conservative seats in the party’s southern heartland, something is changing and Johnson ignores it at his peril. “Something different happened here to what happened in the rest of the country,” he said.

Labour made net gains in places such as West Sussex, Hampshire and Cambridgeshire, while the Liberal Democrats advanced in Oxfordshire, Wiltshire, St Albans, Surrey and Tunbridge Wells. The Greens, meanwhile, won seats on councils including East Sussex and Bristol.

To say that these opposition advances are causing Tory panic would be a major overstatement: Winchester is the only southern constituency among the party’s top 20 most marginal seats. But the stirrings of “southern discomfort” are real.

Box plot showing Labour’s gains in the South East bucked its overall dismal performance. Change in vote, by region (percentage points)

“We have got a problem,” said Tim Loughton, Tory MP for East Worthing and Shoreham. “All the focus has been on the red wall, winning and cementing gains up there, which is great. But levelling up can’t just be about the north.”

On a sunny day in Worthing’s thriving weekly market, set back from a shingle beach with a blue continuum of sea and sky beyond, it is hard to conceive how any dark clouds could blot the outlook for local Conservatives.

But demography is changing in the South East, as younger, college-educated Londoners move out of the capital in search of cheaper housing or a more spacious location from which to work from home.

Tory MPs report that while rural areas remain “solidly” Conservative, the May 6 elections showed a consistent pattern of towns in the South East shifting away from the Conservatives. “London is getting bigger,” said one.

Loughton said there was “quite an influx of younger and more left-minded people”. Many are detached from Johnson’s culturally conservative approach and some have not forgiven him for Brexit.

But one issue above all is troubling Conservative MPs: planning reform. Johnson’s drive to build 300,000 new homes a year is set to be the biggest political battle between the prime minister and his backbenchers in the new parliamentary session.

Theresa May, the former Tory prime minister, is among those southern MPs fearful that Johnson’s proposal to reform England’s sclerotic planning system, contained in this week’s Queen’s Speech, would see “the wrong homes in the wrong places”.

Voters living near an arc of proposed housing along a new “East-West Rail” linking the economic hotspots of Oxford and Cambridge seemed particularly willing to turn their back on the Tories on May 6.

Loughton also noted that while Johnson talks about new roads and railways in the north — the prime minister wants to build a bridge to Northern Ireland — improvements to local roads including the crucial A27 are badly needed. “Some will say, what’s in it for us?” he said.

Bim Afolami, Tory MP for Hitchin and Harpenden to the north of London, said: “We just have to bear in mind as a national party we govern for the whole nation and the whole range of constituents.” Anthony Browne, MP for South Cambridgeshire, said: “As we embrace new voters we should not lose sight of our traditional voters.”

Johnson appears largely relaxed about his southern flank for now, especially as Tory strategists consider it unlikely Starmer will pitch for disgruntled voters at the next election by promising less housebuilding or lower taxes than the Conservatives.

While opposition parties made gains in parts of the South East, Labour, the Lib Dems and Greens are all fighting for a share of the same “progressive” vote: without tactical anti-Tory voting they might all fail at the next general election.

Yet Peter Kyle, who surprisingly captured the Sussex seaside town of Hove for Labour in 2015, believes the party could advance in other areas such as neighbouring Worthing if Starmer relentlessly modernises the party.

“I think there’s an opportunity for Labour success across the south in a way that would be too startling for some people to believe,” Kyle said.

He argued that Johnson’s disruptive and robust style of politics might be popular in the north, but there was “an equal and opposite” reaction among moderate, liberal, internationalist southern voters.

“There’s something closed and inward-looking about his approach — the showing a bit of leg on nationalism — that doesn’t work with One Nation Tories,” he said.

Kyle said that like former Labour prime minister Tony Blair — who made a point of respecting the police, judiciary and armed forces — Starmer had to reassure wavering Tory supporters that the party stood for decency and British values.

But crafting a message which fans the modest Labour revival in the south while reversing Johnson’s advance in the north is likely to be a massive challenge.