Tilbury, June 1948. I know this story so well. Pathé newsreel footage shows black men and women dressed in their Sunday best (though it was a misty Tuesday morning), elegantly disembarking the SS Empire Windrush after 22 days at sea. They had finally arrived at the “motherland” – a Great Britain they were told they belonged to, the piece of geography with all that history they had studied in Caribbean classrooms. It was a place some of them had fought for, and helped build with their toil under exploitative conditions. They did all this because, despite the inequality and contradictory evidence that surrounded them, they believed in a particular story about Britain: that it was a fair, just nation, a land of modesty, balance, equality and opportunity, the story most Britons like to believe about their country.

I don’t have Caribbean heritage, and I wasn’t born and raised in London, but to this day, little evokes the notion of black Britain to me as much as Trinidadian calypsonian Lord Kitchener (named after the general and politician of the same name) singing the celebratory ballad “London is the place for me” at Tilbury Docks: “To live in London you are really comfortable / Because the English people are very much sociable / They take you here and they take you there / And they make you feel like a millionaire / London that’s the place for me.”

There was already a long history of immigration to Britain, of course, and a black presence was nothing new when the Windrush docked. But 1948 was the moment that, finally, seemed to inscribe blackness into the soul of British mythology. It was a moment in history that I, as a black Sheffielder, could anchor myself to during moments of cultural unease, and when I was faced with British racism, because it held all kinds of blackness to its bosom. The way those men and women walked, how they spoke, what they wore, expressed a coherent and confident plurality. They didn’t look exactly like the picture Britain had painted of itself in the first half of the 20th century, yet they were undeniably British. They wrote a new chapter in the story of the country, one that spoke of somewhere that would never be able to look at itself the same way.

Tilbury, June 2019. I know this story well, too. It’s two o’clock on an overcast and drizzly midweek afternoon in an unloved place. Polish labourers, covered in splatterings of paint and smudges of dust enjoy a stress-relieving smoke outside a Romanian bakery called Nehoiu. A Turkish man preps the evening’s doner meat in Stallions kebab shop, and I buy my chocolate drink from Tilbury Wines, run by an Indian man, before taking some photographs of a white man smoking outside a Chinese fish and chips shop called Rainbow. For a while the rain pours down and I seek refuge under a bus stop, from where I can see the Thurrock Irish association club and God Saviour, an African hair salon, through a lattice of graffiti etchings and cigarette burn marks on the plastic window. Behind me is a telephone booth, also scarred. It isn’t an iconic, listed red telephone booth – this area isn’t important enough for one of those – but rather a boxy KX100, a relic of Thatcher’s privatisation of British Telecom in the 1980s and now, along with the council estate, an emblem of urban decay. And yet, there is a sense of a down-to-earth friendliness you won’t find to the west along the river in central London. It makes me think of a passage in Samuel Selvon’s The Lonely Londoners (1956), perhaps the definitive novel of the black experience in postwar London, where Selvon’s protagonist Moses Aloetta says of the city: “It divide up in little worlds, and you stay in the world you belong to and you don’t know anything about what happening in the other ones except what you read in the papers.”

Those words still ring true, especially in the age of social media, because despite Tilbury’s evident street-level conviviality, the Essex district of Thurrock, which Tilbury belongs to, was one of the five districts in the UK with the highest percentage of Brexit supporters. Not all votes to leave had racist connotations; indeed, a sizeable number of black people voted to exit the EU. When Britain joined the single market in 1992 there was a lot of anxiety among black communities because most businesses within them were small, and so – the argument went – most likely to be negatively affected by new competition from the continent; community leaders such as Bernie Grant were vehemently opposed to the idea.

I believe, however, that as white Europe appears to be splintering off into insular nationalisms and parochial identities, black communities in Europe must forge stronger trans-local networks of support than ever before. The fact remains that, across the UK, if you are xenophobic and racist, then if you voted, you voted to leave. In the area of Sheffield where I grew up, Firth Park (which reminds me of Tilbury, with its faded industrial heritage, working-class multicultural community, and which also voted overwhelmingly to leave the EU), I’ve seen people I once called friends stirred from inertia to xenophobia by powerful narratives this country began telling itself in the aftermath of the global financial crash and the lead-up to Brexit. Some of my schoolmates joined the English Defence League; another is now a staunch supporter of Nigel Farage, who has done as much as anyone to create the toxic atmosphere surrounding “the other” that people now feel so clearly in Britain.

During the New Labour years, when Britain became more European than ever before, I noticed how multiculturalism was detached from working-class culture, and connected to globalisation: seen on billboards and in elite international spaces – the hotel lobby, the airport lounge, the global business network or what Marc Augé famously called ‘nonplaces’. The neoliberal project that was failing people co-opted multiculturalism as it undermined it; a global elite, an underpaid poor, more immigration to fill the labour market, working classes left to their own devices to forge goodwill with newcomers, while the right-leaning press vilified not business owners, but the new proletariat with different accents.

This works so well because of the crumbling of two big stories in the imagination of Britain: empire and capitalism. Racism is so virulent in British society because the non-white presence is a visible reminder of the death of an empire that once ruled over much of the world. After all, we are often here because Britain had been over “there”. When people say they want their country back what they really mean is that they want other people’s countries back. In the same way, multiculturalism is a charged reminder of the failures of neoliberal capitalism and globalisation. Those Polish workers are here because a business owner – somebody who wants cheap labour to build homes perhaps – is exploiting them and trying to drive down wages for low-skilled workers. The vilification of the eastern European worker down the road, or the imam at the local mosque, works because it replaces a more terrible, banal truth: that there are people in power – politicians and business leaders – who don’t care about the majority of people living in the country the politicans claim to represent, whether they are black or white.

But unlike the story of the Empire Windrush, which was built on hope, plurality, working-class connection and cross-cultural goodwill, the stories you can find passing through places such as Tilbury now are those built on anger, intolerance and resentment. Which is why the undermining of the Windrush generation, in the hostile environment imposed by the Conservative government, is such a gross act of violence against the cultural fabric of this country. It’s not just about people who have been here for years contributing to society, working for the NHS, bringing culture and labour, but about the very idea of them being part of the British story: this is what is under attack. We are the stories we tell ourselves. Who are “we”? Why are we here? What are we trying to achieve? These are questions we need constructive answers to in a 21st century where we’ve all been left adrift and divided by late capitalism.

As I left Tilbury to get a boat to Gravesend, a group of young men and women disembarked. They were students on their commute home from school. Most of them were black, and they were more British than those men and women who disembarked the Empire Windrush 71 years ago could ever have hoped to be – they were born and raised here after all. But they were also more British because Britain is, more than ever, like them. I heard it in the language of some of the white kids in the group – an urban concoction of words that had London’s history of multiculturalism embedded in every intonation. Yet they were still disembarking on to a land coming to grips with the presence of brown-skinned citizens. A land where the story of what Britain is is still being fought over.

• Johny Pitts’s Afropean: Notes from Black Europe is published by Allen Lane (£20). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99.