Ed Victor is on the telephone to an acquaintance in Paris, explaining that he hasn't been to New York since 11 September. 'I'm going soon,' he drawls, his American overlaid by 40 years of Englishness; rich and mellow, as if the vocal cords have been rubbed with rosin. 'I'm thinking of talking to Tina Brown about whether she can get me a pass to Ground Zero. I'd like to see it.'

Now, to be fair, Victor did have an acquaintance who died in the Twin Towers. But the conjunction of Tina Brown with Ground Zero, with sightseeing and passes, suggests, at the very least, a strange take on the world. It is as if acts of international terrorism offer yet one more test of exclusivity. It is as if, once you have the ear of those who matter, you can get into places and never mind that the rescue workers and demolition experts have asked you not to go.

Victor, literary agent to some of the most glamorous, and profitable, authors around (Nigella Lawson, Jack Higgins, Kathy Lette, Frederick Forsyth, Erica Jong, Josephine Hart) has written his own first book. It is called The Obvious Diet and on the first page he explains that a couple of years ago, he and his wife were number two on a Tatler list of the most invited people in London. He evidently feels this achievement - 'only Elton John was ahead of me!' - qualifies him to advise us how to control our appetites: not only is he subject to all those lavish publishing lunches, and a nightly assault by canapés, but all the socialising has provided him with a battery of food-avoiding friends.

There is, for example, 'my good friend, the American actress Gayle Hunnicutt', who used to suffer from chronic fatigue after eating, but cured it by cutting out wheat, yeast, sugar and vinegar. Or 'my friend Melvyn Bragg', who used to swear off alcohol for the first week of every month. Other celebrities have provided whole new perspectives on eating. His attitude to the humble egg has changed because of Freddie Forsyth and his wife, 'who live on a farm in Herefordshire and regularly send us the best, freshest eggs, which are irresistible!' The dinner party has been transformed: '"Keep it simple" is great advice for giving dinner to friends. And no one does it better than our friends, Ruth and Richard Rogers. Dinner at their Chelsea home is always simple and always a pleasure.'

This diet-book-in-the-style-of-Jennifer's-Diary is punctuated by original contributions from many of Ed's famous friends, whom he badgered to supply their own dieting tips. This proved to be a useful exercise, because The Obvious Diet is rather, well, obvious. Essentially, it boils down to figuring out what works best for you, constructing your own programme out of fragments of failed diets and sticking to it. This makes the giving of bossy advice, which is the staple of most diet books, rather tricky: 'There are no must-eat food combinations, no complex points systems, no demonised foods... unless you have decided that this is what works for you.'

So it's a boon to have on hand Andrew Lloyd Webber, 'the Puccini of our times', or Anne Robinson, who recommends swimming 50 lengths of your pool a day. Or there's Rose Gray and Ruth Rogers of the River Café who announce: 'We both start the day with an interesting breakfast. Sometimes mangoes or melon and freshly squeezed juices, sometimes a light bruschetta, perhaps with olive oil, fresh tomatoes, or salted anchovies.' And if this still isn't Pseud's Cornerish enough for you, there's Koo Stark, who advises pink- and cream-coloured foods in winter and green in summer.

Victor remarks that 'the adjective that has most often accompanied my name in the press is "flamboyant"'. But this is not how he comes across this afternoon in the book-lined room of a Georgian house in Bloomsbury that houses his literary agency. Sure, his hair is long enough to curl and flick over his collar, and his windowpane check suit is elegant in a stated manner, but he seems to me extraordinarily controlled. It's partly the voice, which is cloned from William Hurt's - modulated, almost lazy, huskily on the edge of being caressing. It's the same voice whether he's on the phone, exchanging pleasantries or furious.

The voice would make him suave even if he were short and dumpy. But he's 6ft 4in and panther-like. He occasionally gives the impression he isn't fully there; then he'll suddenly lean forward, ready to pounce and you realise he is. I search the walls of photos that line the office in vain for a podgier Ed, and when I complain that there's no 'before', he says: 'I always had good tailoring.'

Perhaps, though, I am too much influenced by an incident that takes place before we really start talking, when he's having his photograph taken. His secretary comes in to tell him that a Don Epstein has been on the telephone. Without changing timbre or pace, without screwing up his face, Victor is visibly furious. 'You want me to get him back?' she offers. 'More than life itself,' he says, which comes out sounding deadly. 'If you knew how hard it was to get hold of him, you wouldn't have let him go.' When she comes back later, he picks up where he left off: 'That's the second time today I haven't had a really important call,' he says softly, and the emollience makes the words cutting, vicious.

Vicious, of course, is what authors want their literary agents to be. 'They want a killer agent, they want a shark in the water. Not a guy with an M Litt.' His own M Litt involved a thesis on the artist as hero, with reference to the work of Henry James, James Joyce, George Gissing and George Moore.

Ed Victor was born in Queen's, New York, to Russian-Jewish immigrant parents (his father owned a photographic equipment store) and grew up being 'a good son, a good student'. From the local high school, he went to the Ivy League Dartmouth College and thence to Cambridge on a Marshall Scholarship in 1961. When he left the university two years later, at the age of 23, he married an Englishwoman, Micheline Samuels, and decided to stay in Britain. 'When I came here, it was an austere, postwar society, a black-and-white movie. And then - boom - everything changed. This place became so vibrant, so sexy, so full of life - pop music, photographers - I mean, it became really fun.'

He got a job with a small outfit called the Oborne Press, part of the Daily Express, then moved swiftly to Weidenfeld and Nicolson. He was working on coffee-table books, but: 'I looked around and saw Mary McCarthy, Saul Bellow, Vladimir Nabokov on the list and I thought, "Hmm, why am I making these coffee-table books about the great houses of Ireland?" So I cornered George Weidenfeld in the men's room and insisted he move me to general books.'

And he said yes, just like that? I ask, trying to envisage the callow graduate bullying the august publisher. 'Yeah,' he answers, as if to say, 'well, obviously'. 'And, you know, he and I are still really close pals. It's nice. I went to work for Weidenfeld and Nicolson as this kind of editor and tea boy in 1963, and 10 years ago, George and I co-hosted a dinner to celebrate Nigel Nicolson's 75th birthday, because they're both such good friends.' Victor, I suspect, is the opposite of Groucho Marx: he can't see a club without having to make sure it will let him join.

By 1970, his marriage, which had produced two sons, was falling apart and he was 'tired of being good'. He left Weidenfeld and Nicolson to join Felix Dennis and Richard Neville, the founders of Oz, in order to start up a newspaper called Ink. Unfortunately, they failed to agree in advance exactly what sort of newspaper it was meant to be. 'I thought it was going to be a British Village Voice, an alternative to The Observer and the Sunday Times. My colleagues saw it as an underground newspaper.' Ink lasted only 'eight or 10 issues' and Victor returned to America.

He describes the paper as his 'first major failure. And there was no one to blame but me. Usually I run on pretty high-octane fuel, but there was a period then when I was really flat. I was lethargic and depressed.' He joined Knopf as a senior editor and 'did my job, although not in a particularly inspired way. But in that year I did fall in love with the woman I am still in love with.'

She was Carol Ryan, a beautiful American lawyer, 'and we decided to celebrate falling in love by travelling round the world'. They took a year off, during which Victor took two decisions: they would make their home in England, to be close to his sons, Adam and Ivan, who were then nine and seven. 'And I decided to make some money, because I'd always been very fond of saying I didn't make money because I chose not to, and if I chose to, I could.'

So he became a literary agent, and promptly got a $1.5 million deal for the book and film rights to a novel called The Four Hundred, by Stephen Shephard, an unknown author who remained unknown other than for kick-starting Victor's career. 'You know,' he says blandly, 'I was just doing my job.'

He won't be drawn on how many clients he has now - 'I always say seven, because that was how many I had when I started. Lots of authors come after me, but I take on very few. Every now and then I chase a particular client, because there's a big feeding frenzy. One that I was really keen to get was Nick Leeson, and I did get him. And, of course, if I see... I ran into Michael Caine last night at a party, and I said, "Michael, if you ever do a book, I want to be your agent."'

He has three tests of whether a project is worth the effort: 'Is the person fabulous? Is the work good? And is there a lot of money in it?' He declines to say how big an advance he got for The Obvious Diet. 'A reasonable advance, a fair advance... no, I'm not going to tell you. Let's put it this way: it's helping to pay for some of the new suits I need for my new body.' He lost 40 pounds between January and September this year. His tailors (Anderson and Sheppard in Savile Row, also name-checked in the book) are making some suits now, so he's trying not to lose any more. And he employed an agent, Jonathan Lloyd, the managing director of Curtis Brown. 'Obviously, I could have chosen any number of people, but I chose Jonathan because we've known each other a long time. He's a very senior citizen.'

It is a puzzle why he wrote the book. He lacks the proselytising zeal of the true diet missionary. He is more enthusiastic, and more engaging, when he is talking about eating than not eating. His own initial claim is that 'I wanted to see what it was like to write a book', although you'd think he would have seen enough of it by now. He possibly comes closer to the truth when he says: 'I would like the book to sell a lot of copies. I didn't write it for the money, although it's always nice to make money. But I would like it to sell a lot of copies.'

There is, though, a bit of a problem with this. Ed Victor writes from the celebrity stratosphere: in order, for example, to make a point about sticking out for what you want in restaurants, he launches into a long description of a visit to the Ivy. 'When Mitchell Everard, the head waiter, came to take our order, I said, "Mitchell, all I feel like eating tonight is a big bowl of salad". People were flooding in - Tony Blair arrived with friends; John Cleese came in just after him - and waiters were scurrying around tables of the great and the good of London, all clamouring for their meals. So, I called Mitchell over and asked him not to bother about my special order. "Nonsense, Ed!" he said, "You have no idea what a pleasure it is."'

Normal people can't even get a table at the Ivy, let alone on to first-name terms with the head waiter. When I object that the book doesn't read as if it's pitched at mortals, he says: 'It's not written for the average person on the Tube, no. I wrote it for people like me, who have a lot going on in their lives. I think there are a hell of a lot of people like me out there.' Well, maybe. The Obvious Diet also requires you to engage the services of a nutritionist and embark on an exercise programme. Ed's own is such a complicated arrangement of steps and squats and lunges and press-ups, not to mention weights and stretches, that I would need several months with a personal trainer to learn the moves.

He surely can't need the money. He already drives a Bentley, and has a home in Regent's Park and another on Long Island, where he does a lot of entertaining. 'This summer, for example, I became very close to Mel Brooks. I did a book with him and we became very close - his wife and my wife, we all got along pretty well - so I spent the summer with Mel. We have houses pretty close to each other on Long Island and he's an extraordinary guy and we had a terrific time.'

His own house in Bridgehampton was constructed from two English barns, which he had dismantled and shipped over. Not so long ago, it was featured in House and Garden. He gets a copy of the magazine to show me. The house is wonderful: all bleached wooden floors, gigantic sofas, and pergolas dripping wisteria. He flicks over the page, and there's a piece on Richard and Ruth Rogers's home in Chelsea. 'It's funny we're in the same magazine,' he says. 'I was in this house today. My wife and I had lunch there. Richard and Ruthie are among our closest friends.'

The Obvious Diet is published by Vermilion, £9.99. To order a copy with free UK p&p, call the Observer Books Service on 0870 066 7989