Within a few minutes of meeting David Nicholls, I feel as though I've gone a little awry: we have fallen, almost instantly, into talking about the novelist Edward St Aubyn, whose Melrose series Nicholls has recently been adapting into scripts for television. Isn't this bad form, to kick off an interview with an enormously successful writer, who's about to publish his first novel for five years, by chatting about how brilliant another one is?

A bit later, I realise why that's not necessarily so. We have wound round to Nicholls's novels, and to Us, a picaresque and poignant tale of a family under strain that was longlisted – to the surprise of many – for this year's Man Booker Prize, and which followed One Day, the romantic comedy so popular (selling to the tune of 5 million copies) that it qualifies as a genuine publishing phenomenon. And One Day was not a one off: Nicholls is also an accomplished and prolific screenwriter, contributing scripts to the television ensemble comedy Cold Feet back in 2000, reimagining his own novels for the big screen, as well as adapting classics such as Great Expectations and Far from the Madding Crowd, and writing original screenplays, the most recent of which was BBC1's The 7.39, starring David Morrissey and Sheridan Smith.

Yet despite these triumphs, he is evidently not one of life's natural self-boosters. It would be unfair to say that he is evasive, but he is certainly self-effacing: he hesitates frequently, reframes his thoughts, worries aloud about whether what he's said might come across badly. He is entirely friendly but, I notice, not one for prolonged eye contact; throughout our conversation in a Clerkenwell coffee shop, he seems to need something to play with, a coin, a spoon. A man further from One Day's exuberantly brash Dexter it is hard to imagine.

And yet none of this should be taken for a lack of belief in his work. Even though he has been busy with other projects, he explains, the five-year gap between One Day and Us is predominantly the result of a desire to get it right, a determination "not to write the disappointing followup", not to come up with something "maybe a little pretentious or a little slight".

Us could not be described as either of these things. It is the story of Douglas Petersen, at first sight a solid, somewhat unimaginative biochemist, who is thrown into crisis on the book's opening pages when his wife, the more free-spirited Connie, announces that their long marriage has run its course and should probably end; this is complicated further by the couple's plan to spend the summer undertaking a modern, train-based version of the classical Grand Tour in order to educate – and civilise – their bolshie teenage son, Albie. An awful lot happens in Us: missed connections, both literal and metaphorical; a narrative that flits between present-day drama and a more slowly unspooling family back‑story; and a shifting tone that encompasses comic bathos and heartfelt sadness. So what were the doubts about?

"It wasn't so much the act of writing that was difficult," Nicholls replies. "I always put words on the page. It was just coming up with an idea that was strong enough to sustain not just the writing period, but the period afterwards, talking about it. I just didn't want to go into conversations like this with an apologetic air because I'd sort of knocked off a sub-par One Day 2. I didn't want to write a dodgy disappointment."

One Day was Nicholls's third novel and came after a highly creditable beginning. In 2003, when he was in his late 30s, he published Starter for Ten, a coming-of-age comedy based on a teenage student's obsession with University Challenge, which caught the attention of the then extremely powerful Richard and Judy Book Club. Two years later came The Understudy, which drew on his early career as an actor (on this, he is very funny: "I had no capacity, I had no charisma, I had no ability. I saw enough good actors to know that you couldn't take your eyes off them. I was one of those actors that you could very happily take your eyes off. And people did").

He couldn't have known that One Day – which traced the on-and-mostly-off love affair between Emma and Dexter from studenthood to middle age, revisiting them each year on St Swithin's Day – would take off quite as stratospherically as it did, but he definitely knew it had something: "I thought that idea of these 20 snapshots telling a bigger story was a good idea. And I really loved writing it. I would have been disappointed if it hadn't done OK. If it had disappeared, I'd have probably stopped, I imagine. If it had just slipped out and gone away, I would have found it very hard to write another one." (Of the not entirely well received 2011 film version, for which he wrote the script, he says he wishes "we were doing it now, with a little bit of air", although he points out that a lot of people loved it.)

Nicholls enjoyed the things that One Day brought – much increased financial security, obviously, but also the streams of letters from readers recounting their resolutions to get in touch with long-lost friends or their St Swithin's Day pilgrimages to Arthur's Seat in Edinburgh, where the novel begins – but maintains that it didn't greatly change his day-to-day life. "I mostly just sort of went back to my desk," he smiles.

Once there, he waited for another idea to come, which proved harder than he might have expected. "There were a couple of false starts. But the hardest thing, I suppose, was that One Day had a very strong central idea, which was structural. Terrible word, but it was very pitchable: you could say, it's a story of a relationship over 20 years told on the same day. There you are. And so for a long time, you sit there and you think, well, maybe it's  a series of New Year's Eves, or maybe it takes place at two minutes to midnight on Valentine's Day, or maybe it all takes place during a single car journey – all of these, for want of a better word, gimmicks that provide the structure that I could then fill. But the filling has to be more important than the structure."

He knew the book was going to be about family life, but he didn't know its shape, or its characters, well enough. What came out, he explains, was a bit mean, a bit sour, altogether too farcical. At one point he had 35,000 words, which featured only father and son, "and they weren't even on the train". He had to abandon them: "I knew that page by page it was fine. I knew that the writing was OK. But I also knew that it didn't really have any heart, and that it was a bit chilly, and a bit spiteful."

He did all the usual things, and more besides: changed offices – he never works at home in north London, where he lives with his partner Hannah and their two young children, preferring libraries or rented rooms – and experimented with different pens, different notebooks, different software. He even removed all the copies of One Day from view. "But," he says, "you can't write an idea you don't have." Neither does he subscribe to the notion that it's all in there somewhere and you just have to write it out: "I need to be able to synopsise it to myself, which doesn't mean that I need to stick to that synopsis, but I don't believe you can go into a kind of trance and summon it up."

But then, all of a sudden, it was there. He chucked those 35,000 dud words in the bin last February, girded his loins and let the words flow. "When the idea was right and the voice was right, I wrote it very quickly, and I didn't think for one second about One Day. Once the idea was something that I was passionate and enthusiastic about, One Day never crossed my mind."

In July of this year, Nicholls and his publishers received an unexpected fillip: inclusion on the 13-strong Man Booker longlist, this year open for the first time to American novelists and so arguably even more competitive than usual. How did it feel?

"It was great while it lasted!" he laughs. "At that rather nerve-racking stage of the book's life, before publication, before many people had read it, it was brilliant. I wasn't at all blase about it. I never expected as a novelist to find myself in that company, and those are brilliant writers – Ali Smith, Howard Jacobson, David Mitchell."

Did he mind that Us fell by the wayside at the next stage? "I knew I was never the top runner. Of course I would have liked to have been on the shortlist. I never really thought it could win. And, in a way, even if it had, I don't even necessarily think it should win. I think people like Ali Smith have been writing brilliant, challenging but accessible books for many years longer than me, so I was just happy to be in that company. I'm trying to give an answer that isn't too insipid, but I was extremely flattered by it, and in the same way I was a bit surprised, I'm sure lots of other commentators were surprised."

There's no point in beating around the bush: many were. The Booker is not known to favour books that might be described as romantic comedies, or anything approaching genre fiction, or authors whose books sell in shedloads. And he certainly didn't, he insists, sit down and think, "right, I'm going to slightly up the literary qualities, because I think as soon as you start thinking about that, it's a disaster". There is a degree of snobbery about popular, humorous, easy-to‑read books of the kind for which he's known. When I ask him if that grates, he is tactful: "It would bother me if someone just dismissed it as a serious book without reading the book, purely on what I've written before, or on a perception. I don't mind if people read it and say, I don't think this is Booker-worthy. But to not read it seems a bit unfair."

But he also, quite rightly, pulls me up, insisting that Us is hardly a romantic comedy. "It's as much about fathers and sons, it's about husbands and wives, it's about art and science, and Europe, and the British in Europe. I think there are lots of other things in there. There's no reason why that subject matter can't be worthy of serious study, especially if you look at American fiction, where love and marriage and family are frequently dealt with by critically revered writers."

Nicholls is often mentioned in the same breath as Nick Hornby, whose approving quote emblazons the paperback of One Day; and his main character Douglas Petersen, I suggest, also shares a bit of DNA with the embattled heroes of Jonathan Coe. Nicholls also cites the work of David Nobbs and novels such as Billy Liar as examples of the tradition of English comic writing that he could be seen as belonging to. But he was also reading a lot of John Cheever during the writing of Us, in particular the short stories, which he would turn to in the small hours during his habitual insomniac spells, enjoying the portraits of suburban men "who get on the same train every morning, but who are bubbling away with unexpressed passion. So I hope it draws on that tradition as well."

The sleepless nights are another story. He is clearly something of a worrier, perhaps particularly in his life as a screenwriter. Write "it's raining" in a novel, he argues, and you've expended a bit of ink; write it in a script, "then suddenly they're hiring rain machines for £10,000 a night. And it doesn't have to rain." I point out to him that some writers would see this as encouragement to go wild rather than reason to exercise caution: never mind rain, bring on the explosions and volcanoes. Does he think he has a highly (perhaps even overly) developed sense of responsibility? He admits to a certain level of anxiety, but maintains that that's no bad thing – "I think you should worry about your work" – and says that he's been pretty much like this since he was at school, where he was "a bit of a swot". Is he a swotty author?

"I don't like missing deadlines. And I want things to be the best they can be." He particularly dislikes it when he's reading in public and realises there's a better word he could have used. "I find that really maddening. But I think that's the minimum responsibility of an author, to care about your work before it goes into print." There is an element, too, of the fear of exposure here: he says that he finds negative reviews "unbelievably hurtful". On a personal level? "Absolutely. Because you're trying to do it as well as you can."

I mention that, when we were talking about Edward St Aubyn, Nicholls recalled that it had been daunting to meet a "you know, proper novelist". Does he not think of himself as a proper novelist? Well, he responds, he sometimes finds himself reading "really fine prose, really beautifully expressed, clear prose, and feel, well, I should probably give up now". But he also believes that you should read "writers you aspire to, and you should read writers who write beautifully". He expands that to include commercial writers, those who have mastered the art of story or dialogue; of his own output, he says: "I'm working hard to get better, to try to scratch out the obvious joke, to avoid the glib ending."

For now, he's going to continue with the Melrose scripts, which are currently waiting for the green light, but which he hopes will make it to the small screen; he'd also like to focus on some more original scriptwriting. But, he says, he doesn't think it will be five years before the next novel arrives, however many nights' sleep that might cost him. "It's a big deal, and I count myself very lucky, so I don't do any of it lightly. That means there's a certain amount of anxiety and stress. And that's fine. That's fine."

• David Nicholls will discuss his new novel at Guardian Live events in Manchester and Edinburgh on 1 & 2 October. Manchester: cornerhouse.org. Edinburgh: membership.theguardian.com/events.