In Gerald Murnane’s 1985 story “Precious Bane”, an unpublished writer imagines a man at home, browsing his bookshelves, several decades into the future: the year 2020, in fact. Years before, the daydream goes, the man bought one of the narrator’s as yet unwritten novels, but he no longer remembers a single detail from it. “I know,” the narrator’s future self notes sadly, “that no one now remembers anything of my writing.”

For much of Murnane’s career, which began in 1974 (he is 81), this pitiful fantasy seemed more like prophecy: he was a marginal figure in the Australian literary scene, let alone around the world. He had been publishing for 15 years before a UK publisher took note – Faber released an edition of his fifth book, Inland, in 1989 – and 29 more years passed before And Other Stories brought out his second UK publication, last year’s Border Districts (his final novel, he says). In the last few years, however, new paths have been cut to Murnane’s formerly remote territory. And Other Stories has also published his first novel, Tamarisk Row (1974), and now they are giving UK readers a chance to explore his short fiction and essays.

The title of Invisible Yet Enduring Lilacs, which collects essays published between 1984 and 2003, is taken from a biography of Proust. Given that Murnane’s work is entirely obsessed with memory, it makes sense that Proust would be supremely important to him. Well, almost: above Proust there’s horse racing. “Someone has written that all art aspires to the condition of music,” he writes. “My experience is that all art, including all music, aspires to the condition of horse racing.”

In a foreword to the book, Murnane claims he “should never have tried to write fiction or nonfiction or even anything in-between. I should have left it to discerning editors to publish all my pieces of writing as essays.” This bleed between forms is illustrated by the fact that two of the pieces in Enduring Lilacs appeared in a 2018 US collection of Murnane’s short fiction. The new editions patch up the fence between essays and stories, but Murnane jumps it so frequently, with such agility, that the distinction becomes largely meaningless. Whomever the authorial voice in any given piece of Murnane’s writing supposedly belongs to – narrator, character or the author himself – his work always enacts, in the words of the Australian literary critic Geordie Williamson, a “circling of the plughole of his own subjectivity”.

While Murnane’s project is an enormously self-obsessed one, it doesn’t feel self-indulgent. Somehow the totality of focus on the contents of his mind passes through the other side of egocentrism into a strange kind of universality. As with Proust, the specificities of the images he pursues and catalogues provide their own pleasure. In one story he describes the mental picture he saw as a child whenever summer storms raged: “When I thought of the beginnings of a storm, I saw a dark cloud rising from the earth in the way that the evil genie rose from the jar where he had been imprisoned for hundreds of years, in one of the illustrations that I often stared at in the pages of The Arabian Nights Entertainment.” But the effect of his writing is less about the images themselves, and more about the way thought works in the human mind.

That sounds abstract, but as Murnane states in his essay “The Breathing Author”, “I believe I may be unable to think abstract thoughts” (he subsequently announces he has never been on an aeroplane, gone swimming, or worn sunglasses). In fact, his fiction is full of concrete imagery and often simple in setting and incident, at least to begin with. Consider the opening of “Pink Lining”: “The image that caused me to begin writing this story is an image of a single cloud in a sky filled with heaps or layers of clouds.” Many of his stories proceed in this building-block manner: the clouds are printed on a postcard, the postcard is a piece of Catholic memorabilia linked to indulgences, his aunt once explained indulgences to him, and so on we go, one step at a time, until the inciting cloud is far behind us.

From these simple beginnings Murnane’s most impressive pieces ramify into works of great intricacy, and sometimes beauty. If you are a newcomer to his work and start with the stories (which I’d recommend), read “Stone Quarry”, “Emerald Blue” and “In Far Fields”. If you go with the essays, read “Stream System” and the title essay, in which Murnane builds a series of unlikely connections between his first reading of Proust, Kangaroo Island, the grasslands of southern Africa and northwest Victoria, and a rock in the Southern Ocean into which his father, as a child, carved his name. This extraordinary piece of writing quivers on the same line between inspiration and mania that many of the stories do, and delivers a strain of exhilaration I have only encountered through the work of Gerald Murnane. He is, as I suppose his work keeps telling us, singular.

• Chris Power’s Mothers is published by Faber. Collected Short Fiction and Invisible Yet Enduring Lilacs are published by And Other Stories (RRP £12.99, £11.99). To order copies go to guardianbookshop.com. Free UK p&p over £15.