In February 1965, the civil rights activist Malcolm X visited Smethwick, in the Midlands, where the Conservative-run council was planning to buy up houses to stop the area becoming a “coloured ghetto”. He wanted to see how bad the racism was and he experienced it first-hand when refused service in a pub. White residents thought his visit a “disgusting” intrusion. Nine days later, back in the US, he was shot dead.

Britain in the mid-60s is associated with Carnaby Street, the Beatles and miniskirts, not racism. But what Black people endured, from a colour bar on property to the abuse of Leeds United winger Albert Johanneson as a “golliwog”, is no less a period marker than Harold Wilson’s pipe, Twiggy’s haircut, Mary Whitehouse’s clean-up TV campaign and Bob Dylan’s use of an electric guitar. David Kynaston doesn’t underplay the excitements of the period. But he rightly emphasises that the rotten old 50s – patriarchal, homophobic, class-divided and bigoted – wouldn’t just fade away.

His mammoth project, a history of Britain from 1945‑79, is now past halfway. He originally expected this volume to continue to 1967. In the event it covers just 18 months, from February 1965 to July 1966, and (with notes) runs to nearly 700 pages. The challenge is to balance an in-depth account of politics, economics, industry, education and culture with personal stories of what it was like to be there: the food you ate, the clothes you wore, the films you saw. The sources he draws on range from statistical surveys and sociological reports to political biographies, personal diaries and the grouchy letters of Philip Larkin. No living historian is as comprehensive.

With much to accommodate, he relies on the semicolon for day-by-day narration: “At a Monday morning cabinet session…;” “later that afternoon …;” “on the box the same evening …;” “next morning in Manchester … ;” and so on. The sentences are long but full of humour and never dull. The high point comes with a “freewheelin’, mid-60s play list” (duly abbreviated here) that takes up five pages:

Multi-flavoured Golden Wonder crisps displacing Smith’s plain; one-third of new marriages now civil marriages; pubs reluctantly installing fruit machines; Jimmy Hill’s Sky-Blue revolution at Coventry City; AA patrols no longer required to salute; picture rails, seen as dust traps, removed en masse from Victorian and Edwardian houses; Boots from 1965 at last consenting to sell condoms; arrival of open-plan offices; arrival, too, of the hot drinks vending machine; no car more iconic of modernity than the Mini Cooper …

As well as giving us snippets from daily life and reminding us how nasty the 60s could be (Ian Brady, Myra Hindley and the Kray brothers), Kynaston devotes chapters to larger-scale cultural shifts, chief among them in education. Harold Wilson dreamed of “a Britain in which anyone can get to the top on merit and ability, not on social or family connections”, and the shift to comprehensive schools was integral to that. Grammar schools resisted, public schools clung on to their independence, and refusenik local authorities caricatured the change as a Nazi-like conspiracy designed to “tyrannise the minds of the young” and create “animal farms” run by state commissars. But Anthony Crosland pushed hard for the reform and it did Labour no harm in the March 1966 general election, which kept them in power with a majority of 97 seats.

Polytechnics arrived too – 30 of them over the following eight years, along with the Open University. This wasn’t quite the age of social mobility it’s cracked up to be. Most children still left school at 15. Whereas 50% of high-school graduates in the US went on to university, it was still less than 6% in the UK. But expansion was under way. And many of the era’s famous names – David Bailey, Albert Finney, Michael Caine, Jean Shrimpton, Harold Evans, Terence Stamp, Cilla Black and Lulu – had working-class roots.

Kynaston also explores the rebuilding of cities: slum clearances, tower blocks and (in the words of the song by pop band Unit 4 Plus 2), concrete and clay. It was a peak time for high rises, with whole streets destroyed (as if wartime bombing had resumed) and millions of people displaced to new estates. Some were grateful to escape the damp and squalor of the houses they’d been in; many were not: “I’d rather have a tent on the moors.” Kynaston detects “a certain residual arrogance” in the architects and planners. The pendulum would gradually swing against a “dim-witted adulation of Le Corbusier”, but until then the vogue was for streets in the sky.

Happily, there was also a vogue for liberalisation: of birth control, abortion law, consensual sex between men, and greater opportunities for women. All were much needed: homosexuality was widely vilified; the Factories Act forbade women to do night shifts and only 80 of 1,707 candidates at the election were female; family planning clinics wouldn’t see young, unmarried women, one of whom asked agony aunt Marjorie Proops: “If a girl has intercourse, and then has nothing to do with boys for a year, can she have become a virgin again?” More surprising is to find that half the workforce in 1966 was already in the service industry and that the City of London was a hub of global power long before the 1980s.

The book’s likeliest readers will be those alive at the time, whose memories will be triggered by what was new (the Post Office Tower, microwaves, Laker Airways), what needed phasing out (bottom-pinching, outdoor loos, “the brain-damaging heaviness” of leather footballs) and what songs, plays, films and television programmes had to say about contemporary life. Kynaston quotes generously from reviews, many of them more robust than you find today (“pretentious bunkum”, “inane, asinine, witless”); the BBC’s brilliant Wednesday Play series infuriated audiences and even the Beatles were denounced when awarded MBEs: “All they do is exploit a basic jungle rhythm for money.” A younger audience may be struck by the parallels between then and now: a Labour prime minister from the north, economic strife, the US in a pointless war, hostility to immigrants. We’re not facing devaluation of the pound, as Harold Wilson was. Still, the continuities stretch forward as well as back.

The last section of the book is a tour de force, interspersing Kenneth Wolstenholme’s commentary on the World Cup final (“Four minutes now left to play and the crowd biting fingernails down to the knuckles”) with the reactions of those watching on television, among them young Laurence Marks, the finest of the diarists quoted in the book, who stood to lose £2/12 shillings on a bet if England won, and Kynaston himself, on his 15th birthday: “I retreat upstairs to my bedroom and play a side of a Nancy Sinatra album, reckoning it will see both me and England through.” It did. The match was a triumph and so is the book. But spare a thought for the 6,000-strong crowd in Devon who’d paid to see the Kinks perform that night. The band stayed in London till the match was over and, arriving two hours late at 11.45pm, were allowed to play for just 10 minutes.

• Deep into the Sixties: Britain 1965-66 by David Kynaston is published by Bloomsbury (£35). To support the Guardian, order your copy at guardianbookshop.com. Delivery charges may apply.