‘You look knackered,” says David Bailey, greeting me at his studio. It’s up a small mews and sprawls so casually across two floors that it still feels like the 60s inside. “Look at you,” he says. “Your buttons aren’t even done up right.” I look down at my jacket: that bit is true. But I tell him: “I’m not tired!”

“I was watching you walking along the street,” he says. “I thought, ‘That must be the journalist, she looks knackered.’” The combination of acuity (he must be right: he is, after all, the one who makes a living with his eyes) and demonic overfamiliarity (by this point, we are holding hands; I have no idea who started it) is disarming. If this is his shtick, it’s working on me, totally and overwhelmingly. Or maybe he has a tailored shtick for everyone he meets.

We’re here at his London studio to discuss David Bailey, as his massive new book is titled. It’s part of the Taschen Sumo series of huge tomes costing thousands that you have to wear white gloves to look through. Ordered chronologically, it draws you through his most iconic images: Jean Shrimpton, Michael Caine and the Krays in the 60s; then Andy Warhol, Jimi Hendrix and Jack Nicholson in the 70s; skipping through to Kate Moss and Oasis in the 90s, not forgetting Tony Blair and a beautiful, giggly portrait of the Queen.

There is nothing around to date these pictures: no background, no props. “It’s common sense to me,” he says. “Take everything out. I don’t see why I need a palm tree with Kate Moss. Kate’s enough. That’s why I like white backgrounds. It’s not because I’m lazy – it’s because you take everything out till you’ve just got the person’s personality. I talk to people much more than I photograph them.”

Flicking through them all at once makes you realise the extraordinary extent to which Bailey created the touchpoints of what our culture looks like. Image by image, they are unfailingly the portrait by which you best remember that particular person. It feels as if I’m holding Britain’s collective memory in my hands.

In the 60s, Bailey became the poster boy for a seismic cultural shift. “It was really a working-class revolution,” he says. “It was the first time people like me and Michael Caine had a voice. Before that, if you came from East Ham, you stayed in East Ham.” He has never really fallen out of fashion since. I ask if he’s ever foundered, ever had a moment’s boredom or self-doubt. “Girls have left me,” he says. “But you get over things like that.”

That’s not your career, I point out. “Well, they’re often linked,” he says, and he has a point, when you consider that Jean Shrimpton, the model and actor, was also his lover. He continues musing about women who have left him. “Like Catherine Deneuve. Actually, with Deneuve, it wasn’t sad. We just drifted apart. She was in Paris, I was in London. She phoned me once and said, ‘Oh, Bailey, it’s great.’ ‘What’s great?’ ‘We got divorced today.’ I said, ‘Is that great?’ She said, ‘Yes, it means now we can be lovers.’”

It’s not a new story, but it’s so evocative, so winning, you can see why a person would never stop telling it. Well, we may as well talk about #MeToo. Fashion remains the unmapped territory of the movement, full of dark casting-couch secrets that everybody knows but doesn’t say. “I never tried that,” he says. “It wouldn’t work for me. If you wanted someone for the cover of Vogue, you couldn’t do it just because you’d slept with her. She had to be good. Not at fucking. She had to be good at modelling. It didn’t come into my life at all.”

It’s not the most watertight argument, since “good at fucking, good at modelling” would have a sizeable overlap. But let the man finish. “I think it’d be difficult to get taken seriously if you can’t look after yourself. Like Deneuve. She always said, ‘If someone makes a pass at me, I’ll tell them to fuck off. Otherwise, it makes women look weak.’ She’s always bollocking me for being late.” Being late? That’s hardly the same thing. He shrugs, as if to say it sort of is. And he was objectified like crazy himself: Shrimpton raved about his beauty, as did others.Suddenly we shift to politics and he tells me Darren Rodwell, the Labour leader of Barking and Dagenham council, should be mayor of London. I ask what Rodwell’s politics are. “He’s like me really. The left think I’m right and the right think I’m wrong.” Yeah, about that – what did you vote leave for? He looks rueful. “I know. Maybe I made a mistake. I don’t know. I just didn’t want to be German. You know the Germans run the common market?

“I suppose it’s my background really. I lived through the blitz. We weren’t evacuated. In the East End, nobody was. Spent most of it down the coal cellar. Everyone my age” – he’s 81 and fit as a flea – “I think we all feel like that. Now, I love the Germans.” Wait, what? “Taschen’s one of my favourite people.” Ah, he’s remembered that his publisher, Benedikt Taschen, is German. He turns suddenly serious and says: “Really, everyone’s fine. Nobody’s nasty. The Japanese aren’t nasty. Afterwards, you’re thinking, ‘Why did I mind them?’”

Bailey describes a childhood of almost Victorian poverty, two families living in two rooms. “My father was never there. My mother’s brother, Artie, was gay and I shared a room with him, and my father was really uptight about it. I mean, he was ignorant. He was just an East End guy. Women and drink. Whole life devoted to it.” This isn’t why Bailey doesn’t drink, though. He gave it up in his 30s, when he was shooting commercials and needed all the energy he could muster.

“I made more money out of commercials than I ever made from photography. And I won an Emmy! For an advert! I couldn’t believe it. I said, ‘Are you going to give one to the manicurist as well?’” He credits his social ease to his upbringing. “It’s defence. When you’re surrounded by villains, make them laugh.” And he definitely brought something out in the Kray twins, taking incredible portraits of them on numerous occasions. He even lived with them for a fortnight, though he didn’t actually sleep there. These shots were always controversial, given the sheer audacity of making something beautiful out of people so corrupt.

“You can’t be moral,” he says. “I liked Reg. He was great. He was just a regular guy. I’d seen him kill people, but he wished he could have done it my way, legit.” Wait, you saw him kill people? Bailey sort of waves and it’s hard to tell whether this means “We all did, back in the day” or “Stop being so literal”.

Bailey was called up for national service in 1956. “I tried to get out of it by making out I was gay. But I think everybody tried that. They say, ‘What’s your favourite sport?’ And you say, ‘Ping-pong.’ But it didn’t work because every fucker tried it. I feel sorry for the ones that were gay, because nobody believed anybody.”

He was demobbed in 1958, slogged for two years, then got taken on by Vogue. Six decades of unbroken renown followed. He has always managed to straddle both the establishment and the counterculture: in the 70s, Bailey was the photographer chosen to go with prime minister James Callaghan to China. “Oh, he was great. He carried my camera case. He didn’t have general knowledge. He’d pick up a piece of jade and go, ‘Look at this malachite.’”

These days, he says, “they’re a bit like actors, prime ministers. You’re not really getting the real them. What’s his name, that other one on the left?” Tony Blair? “Not him, the other one.” Gordon Brown? “Everybody liked him. My assistants came out going, ‘He was great.’ And I said, ‘He was nice to us because he wanted a nice picture, but it meant I couldn’t see him. He was playing the game. I said to my assistants, ‘We’ve just been fucked and you haven’t seen it. I just got fucked by Gordon Brown.’”

Bailey’s description of portrait photography is almost mystical. He says it’s more interesting than painting “because that’s all interpretation – whereas with photography, you’ve got something real there”. But what’s real is the person’s spirit: you coax it out and catch it. Sometimes he’s stalking something ineffable, though, something that can’t be caught. “It’s like Andy Warhol. You couldn’t photograph him. It was like photographing smoke. Suddenly he wasn’t there. I knew Andy quite well. He wasn’t there, really. He really liked watching, he didn’t like being.”

Actually, it’s not true that he couldn’t photograph Warhol. His first shoot with the artist, which is in the book, is amazing. There’s a picture of himself and Warhol, taken from above. “He wouldn’t do it unless I got into bed with him,” says Bailey. “That was my #MeToo moment!”

His description of Warhol, always watching, never being, has been the traditional role of the photographer, living in the no man’s land between the starers and the stared at. Yet Bailey is viscerally present in the moment, buoyantly alive, relentlessly curious, devoid of regret. His only beef, these days, is with mortality. “Life’s tragic really,” he says. “You have a great life and then you get old. Most people get diseased. I’ve got so many mates who have walking sticks now. This is how it ends. Life’s sad. It’s tragic.”

It’s all true, of course, but it doesn’t have that truthy ring. All the way through these complaints, he is smiling.

• David Bailey is published in April by Taschen, £2,250, in an edition of 3,000.