It is December 1998 and I'm sitting in the living room of one of Britain's most important novelists, drinking champagne. But we're not celebrating. AS Byatt drinks champagne because other alcoholic drinks give her migraines. We've never met before and I am here to ask if she'll let me make a film about her. She is warm and friendly and very talkative. I'm amazed when she says she likes the idea of my project and wants to take part. Neither of us imagine that we are about to embark on a relationship that will continue for the next four years.
Scribbling is a series of documentaries, each one following a well-known novelist through the entire process of writing a book - from initial concept right through to publication. The idea was brave not only because we were attempting to film the unfilmable - that elusive and painful thing called creativity - but because we were going to stick with it however long it took: a year or even 18 months, a filming period unthinkable and usually unaffordable on today's programme budgets. The project was only possible because I have a staff job at an independent production company, Wall to Wall, and would be making other programmes at the same time.
I was delighted but also surprised that AS Byatt wanted to be the subject of one of these films. Many other writers turned me down. And although Antonia is more than happy to appear on Start the Week and write pieces for the broadsheets, she hates talking about her personal life and is fiercely protective of her time. I was asking to be there with my camera through all the ups and downs as she wrote her next novel. So why did she say yes?
Antonia was about to start writing A Whistling Woman, the final book in a quartet featuring her heroine Frederica Potter. She'd begun work on the first Frederica novel, The Virgin in the Garden, in 1961, and had never imagined she'd still be writing the series nearly 40 years later. She liked the fact that my film was a serious venture, and she also thought it would force her to write her book and write it quickly. I rigged a diary camera to her desk for her to talk to daily about her progress. The pressure of being watched by this tiny camera would make her finally complete her quartet.
I don't think Antonia suffers from writer's block - ever. But she had a strange ambivalence about starting A Whistling Woman. She'd lived with Frederica and her world for so long that it had less appeal than hundreds of newer ideas for stories and novels. Yet she knew she had to write the book. Readers were waiting for it. And she couldn't get on with all the other books in her head until it was done.
In our first filmed interview I learned that Antonia had begun A Whistling Woman two years earlier and then stopped. When I asked why, her answer put the fear of god into me. The book had been killed by a series of long and exhausting publicity tours, which had broken her concentration and made her depressed. What Antonia needs in order to write is long periods of total isolation, and what unsettles her most is doing publicity and talking to journalists. I suddenly realised that although Scribbling could help give her the impetus to write her book, my need to film with her regularly was exactly the sort of thing that could stop her too.
It didn't take long for Antonia to realise this. In the summer of 1999 she began to write and I began filming in earnest. I accompanied her to a house in the south of France where for three months each year she locks herself away from the world to work. The location of this place is a closely guarded secret. The fact that I was going to be allowed to film there was a privilege she'd never offered a film crew before. But my frequent trips that summer were too much. I sensed it on the diary tapes she sent to me each week: she talked often about the horror of an upcoming book tour in Paris, but that horror was as much in anticipation of my next filming visit.
It was hard for me to understand how my presence in the village and a few hours of filming over two or three days could have such a bad effect. But it did. Things reached a head on the platform of Gare du Nord station in Paris. She'd just arrived for her week-long publicity tour, and when I suggested filming that afternoon at a photo-shoot, she exploded in a way that seemed completely unreasonable. An apology soon followed, but I now realised that Antonia was close to pulling out of the project. She told me later that she'd felt that taking part in my film had been a terrible mistake. Perhaps at this early stage in the writing, A Whistling Woman was still so fragile in her mind that my film could have destroyed it.
After that summer I stepped back a bit, asking less for filming visits and relying on Antonia's frequent and fascinating video-diary entries to keep on top of her progress. Sometimes I worried that I wasn't making a film at all. Usually, programme-making is incredibly hard work, but apart from the odd email we barely had any contact. And then A Whistling Woman began to flow. Although it was meticulously planned in notebooks before she started to write, Antonia says that the beginning was so hard because there were still many possible directions the book could take. Once the whole story was clear in her mind, she began to race ahead and we were both back on track.
In 2000 we met in Yorkshire to look at cottages which could help her to invent the home of one of her characters. I filmed her consulting with her friend the geneticist Professor Steve Jones about the sex lives of snails. In interviews we talked about how scientific ideas feed into her writing, and how whenever she's away from the book she holds it in her head as a visual image - currently a beautiful, coloured glass cube. Although I was anxious that I still wasn't filming enough, we were getting on incredibly well.
Antonia was now confident she would finish writing by spring 2001, and A Whistling Woman was scheduled for publication in the autumn of that year.
But then disaster struck. She woke in the middle of the night with an agonising stomach ache and was rushed to hospital to be diagnosed with a twisted bowel - the surgeon said she was four hours from death. After an emergency operation that took weeks to recover from, the book was stopped in its tracks. Antonia had told me countless times that she had to limit the time she spent with her family and friends because she had so little time left and so many books still to write. Now I realised what she meant.
Finishing the book became even more urgent. In the summer of 2001, Antonia cancelled everything and shut herself away in the house in France. She even persuaded her husband to abandon his holiday there to leave her to get on with it alone. I arrived in August, a few days after the first draft of A Whistling Woman was complete. Antonia seemed dazed, having finally reached the end of a project that had been in her head for more than four decades.
Following A Whistling Woman all the way to publication meant I filmed for three and a half years. In that time I had two children, made three other films and executive produced about 15 more. I can't imagine ever again being asked to make a programme over such a long period.
Four years after our first meeting, Antonia and I drank champagne again, this time to celebrate the end of our respective projects. It has been an incredible privilege to spend so much time in the company of one of Britain's finest writers. But there was one, final question. What did Antonia think of the film?
The answer is: I don't know. She's never watched it. Antonia doesn't like to see herself on television because, she says, it makes her feel unreal. It's the final irony that this most honest of writers, who let her inner world be subjected to the relentless glare of the television camera, doesn't want to see herself revealed on screen.
· Scribbling, tonight (Robert Newman) and tomorrow (AS Byatt), 11.20pm, BBC2
