Small Man In A Book, Rob Brydon's memoir, stops rather abruptly the moment he becomes successful, which is just one of a number of endearing things about it. It was the end of the 90s and Brydon was making a living in radio, but not what you'd call a handsome one. He'd written Marion And Geoff, and he knew it would succeed; he'd also started collaborating with Julia Davis on Human Remains, and felt certain that would work as well. "But I remember saying to someone, 'I'm writing this thing with Julia Davis' and she said, 'Has it been commissioned?' I said, 'No, but it will be.' And she looked at me like I was simple. Like, 'Ah, bless him, he's got his hopes up.' But I knew by then I was on a wave."

Another appealing thing about the book is that it reads like a compendium of all the times he realised how fantastically talented other people were. The first time he saw Catherine Tate, "standing at the back of the room watching her and thinking, 'Is it just me, or is she actually very good indeed?'" Or when James Corden floated the idea that he wanted to write a sitcom (Brydon told him to get on with it, and of course it soon became Gavin And Stacey). And spliced in with all this admiration of others are his own fondly remembered disappointments: the doomed near-stardom on a shopping channel, the time he lost his solitary line in a film – effectively bumped down from bit part to extra. He's a perfectionist in his self-deprecation, and just will not let the matter slide until he's conveyed exactly how inadequate he was, how much too short for the part, how much too shy to get laid, how naive and hapless.

It's an interesting trajectory for showbusiness. His seminal comedy turn – the small man in a box, where he does a tiny voice and it sounds like a small man trapped in a box – is slapstick, surrealist, studenty. But I would contend that his real comic persona, until he was 35, was that he was a total loser. All his most fascinating comic creations involved building a labyrinth around a total loser, to see where the loser would go next. His most enduring comic partnership – with Steve Coogan in A Cock And Bull Story, Michael Winterbottom's riff on Tristram Shandy, then, later, in The Trip – involves him playing loser-sidekick to Coogan's arrogant tosser. It makes you wonder how long he can keep it up, the unassuming nice guy, when he's clearly a winner, one of those Midas comedians, now, who can make something funny just by looking at it.

Well. Maybe he is. But he won't be arrogant, or even nasty about anybody. It's just too much against his nature or learned behaviour: like trying to get a guide dog to jump on the sofa. When I charge him with the fact that there isn't one mean remark in his entire book, he throws me this: "I do mention when, around that time [Human Remains was being written], I was in doing one of those Saturday night voiceovers"… pause, while he does me a Saturday night voiceover – "Pierce Brosnan has a recipe for death… There's Casualty at eight – 'He's gonna mainline!' – and with Match Of The Day at 10, that's Saturday night, on BBC1…" and the guy I was doing it with was very bitter. There's a level of achievement in TV that is OK, but when they hear that someone has scrambled out of the trench, to get to the level above, they're going, 'What? How?'"

That's the best/worst he's got. A nameless guy who said, "What? How?" in not a very nice way. Later, though, Brydon does allow himself this remark about Coogan: "On one level, I adore him. But he's never been shy of saying how I can annoy him. So I'll take this opportunity now to say he can be annoying… he can be an arse as well. Quite recently, I've read a few interviews he's given for The Trip In America, and I've thought, all right, let's not air all our dirty laundry here. Because if anybody can be an irritant, sir…" I feel like I'm finally getting somewhere. For one intoxicating moment, I think I'm about to find out what Coogan's like when he's ripped to the tits on cocaine, on the pull in LA… "But that's 10% of the relationship and 90% of it is… when we're creating stuff together, it's a joy…"

"Wow, you're really going for the jugular."

"Look, whenever I've said stuff in the past, I've always regretted it. The bottom line with Steve is that I think the man's wonderful, but given that he's recently said that I annoy him, I sort of think, oh, all right then. I'll take one glove off and say that you can be a bit of a turnip as well. But wonderful to work with."

One has to concede that there's something alchemical about the creative relationship between them. Coogan, clearly, is hilarious whatever he does, but there seems to be something reserved about him. Yet the competition of another comedian triggers some kind of turbo-Coogan, and he'll suddenly say anything, throw reserve and caution and discretion to the four winds, just to be the funniest. Brydon, meanwhile, needs someone obnoxious to rough him up a bit, otherwise he's just this incredibly sweet 46-year-old man who can do funny voices and has a beautiful baritone. Against Coogan he becomes something more profound, a real-life study in nuance to set against Coogan's more try-hard complexity. There's a brilliant exchange in The Trip where Coogan is essaying the view that people with a rock'n'roll lifestyle are automatically more interesting than people without, having more demons, being more easily bored. The look on Rob Brydon's face is a perfect balance between snorting scepticism and the fear that maybe he's right, maybe the universe really is that unfair, and the people who get the bad behaviour also get to have the hidden depths. Brydon famously, by the way, can't drink beer because he has a yeast intolerance, and giggles like a girl after one glass of wine.

It's an ongoing theme in Brydon's career, that his memorable partnerships are with people who seem to be a lot more diabolical than he is. Julia Davis famously has an imagination that would stop most people sleeping, which – if you look at Human Remains, set against her solo show, Nighty Night – Brydon seemed to take the edge off. And he agrees with this: "Often with Julia I would feel she'd gone too far. But nine times out of 10 she was right." Nevertheless, he is beginning to balk at all the nice-guy stuff. "The obvious answer is, you'd say Julia has the dark side. I have all that, but I don't put it to the fore. You know, I was married with three children and that marriage ended, I have plenty of angst, believe me, I have plenty of torment, but I don't really shine a light on it."

Ah, the marriage: this is another glaring omission from the memoir. Aside from the rank failure to slag anybody off, there is no mention of how that relationship worked and what sunk it, despite the fact that it produced three children – now nearly 17, nearly 15 and just turned 12. He explains, reasonably enough, that he can't talk about it because his ex-wife has no love of the spotlight. His current wife (with whom he has two more children, three years and three months old, respectively) is no great fan of publicity either, so it's almost impossible to get anything out of him except the bare branches of his family tree. All he'll say is that the end of his first marriage left him with a crushing sense of failure, which paradoxically coincided with his first taste of professional success.

Before then, Brydon's career had ticked along since he left the Royal Welsh College of Music and Drama at 20 to join Radio Wales. He'd had a few TV roles in forgettable dramas, small film parts in First Knight and Lock, Stock And Two Smoking Barrels, and a radio career doing voiceovers and sketches on shows such as The Treatment, on Radio 5 Live. It was there that I first met him, just before his career took off (I was one of the journalists who did news round-up stuff). He was always in an incredibly bad mood because he thought the sketches were so rubbish. Stuart Maconie said he liked having him around for the "crackle of open hostility", but he may have been being sarcastic. He was obviously just very frustrated, having ideas of his own that hadn't yet come to anything; but I never gave it much thought because his default setting was so polite and good-natured.

Anyway, we are talking about someone else who used to do the show and he says, "He's a big shagger. I don't say that in the book." Which is, again, quite endearing, because there is nothing a big shagger likes more than to appear in someone else's book as a big shagger. "He used to say to me, 'You could so easily do it, you just don't have that confidence.'"

"But you were married already?" (I'm perplexed: I don't mean to be naive… I get that people cheat on their spouses. But wouldn't that be because they were inescapably driven to, not because someone else showed them how to do it?)

"Oh," Brydon explains, a bit vaguely, "he was just talking about attitude, about having more confidence in myself."

"Why do you think you were so reserved?"

"I was very nice. People say that about me now."

I blame his parents. There's possibly nothing that comes across more clearly in his book than the persistent kindness and support of his progenitors, hard-working, church-going people who made Christmases in Port Talbot magical and went for a run in the car on a Sunday afternoon. Scrabbling for conflict, I ask him who was nicer, his mum or his dad? "They both were. There was no horrible nastiness, there was no roughness. I was stoical, eager to please, polite."

And that is pretty well how he's remained. "I've had my moments. But it'll usually come from insecurity. I've had photo shoots where I'm feeling insecure about my skin, and I'm in the wrong clothes. And I've been short with people. But I don't think I've shouted."

In short, he doesn't conform to the standard modern image of a comedian: he doesn't have tantrums, he doesn't seem to have an ego, beyond what a person needs to stay buoyant. Talking about comedians generally, he lights upon Sacha Baron Cohen as a prime example of the attitude one might expect. "He has this incredible self-belief. I want to say superiority, but it sounds rude. It's a sense of entitlement. I don't have that; I have a quiet, dogged kind of determination. That's more my thing."

It's in that spirit that he laughs off my suggestion that he might have sold out by doing the advert for Crunchy Nut Cornflakes. "What's to sell out? I'm a comedian!"