Julian Barnes has been nominated for the Man Booker prize four times and won it once; he is an essayist and memoirist par excellence. Now, however, he is trying his hand at yet another discipline: art curating. He is in the early stages of co-organising an exhibition of works by the late 19th-century Swiss painter Félix Vallotton for the Royal Academy of Arts (RA) in London, he told an audience at the Edinburgh international book festival.

The artist – associated with post-impressionist painters such as Édouard Vuillard and Pierre Bonnard – has never been the subject of a major exhibition in the UK before, with the exception of a show of his woodcuts in 1976. For the exhibition, expected to open in 2019, Barnes and the RA will bring together about 70 works never before seen in the UK. With the exception of a single work belonging to the Tate, there are no paintings by him in British public collections, though his work is well represented in Swiss and French museums.

Barnes spoke of his love of the artist, the subject of an essay originally published in the Guardian and now anthologised in his new book, Keeping an Eye Open. He was a painter who, Barnes said, depicted “the paradoxes, the hidden violence, and the ambiguities of the emotional life”, in works that often both seem to offer, and simultaneously to withhold, a narrative. The paintings have an “affectless style that at times prefigures Hopper”, he writes in his essay.

Barnes told the Edinburgh audience about encountering, at the Baltimore Museum of Art, Vallotton’s painting The Lie, in which a couple are locked in an awkward embrace on a sofa – the woman, swathed in a red dress, seems to be whispering in the ear of the man. Barnes and his students at Johns Hopkins university in Baltimore offered completely different versions of the implied narrative of the picture; he had been drawn to its enigmatic, irresolvable nature.

Not everyone has been so keen: Gertrude Stein called Vallotton “a Manet for the impecunious”, and even Barnes acknowledges that his work “ranges from high quality to fierce awfulness”, among them nudes he described as “hulking zeppelins of female flesh”. He will, he said, choose only the very best quality works for the exhibition.

Airing his views on the state of visual art more widely, Barnes spoke expansively of his love of British painter Howard Hodgkin.

But he said artists were today expected to explain and write about their work far too much: Matisse had offered good advice to young practitioners “when he said that ‘artists should have their tongues cut out’, because it has increasingly become the case that from a very young age artists have to have a narrative about what it is they are actually doing. You sometimes feel that the narrative is almost floating free from the art; it’s part of the publicity that they have to do. You feel that instead of gradually discovering what it is they are doing they seem to have to have a thesis to begin with.”

By way of example, he offered a text written by American artist Jeff Koons to accompany his work Puppy, a vast sculpture formed from flowering plants belonging to the Guggenheim Bilbao in northern Spain. Reading aloud from Koons’ text, he told the Edinburgh audience that Puppy “helps you have a dialogue about the organic and the inorganic. It’s really about the issue of the baroque, where everything is negotiated. The different aspects of the eternal through biology. Whether you want to serve or be served, love or be loved, all these types of polarities come into play because Puppy sets them up.”

Barnes added: “To use the technical term of art criticism, it’s bollocks. I know it’s like shooting fish in a barrel but sometimes fish need to be shot.”

He defended his statement in the book that “Andy Warhol is to art what Fergie is to royalty”. “He’s just not an interesting artist to me,” he said of the pop artist. “One of the tests of art is when you go back and see it after 10 years, and you have changed, has the work changed? Or do you have exactly the same reaction to it? … [With Warhol] you either have exactly the same reaction to it as you did before, or perhaps a lesser one. That’s OK – there’s room for a Fergie in the royal family. I don’t think art is a temple – I think it’s a refugee camp.”