“It’s a weird thing,” said Gwendoline Riley after the publication of First Love, shortlisted for the Women’s prize in 2017. “I don’t feel as if I’m building a body of work; each book is saying what I want to say, but better.” Over two decades and six novels now, she has refined her material; a writer/academic, called Aislinn or Carmel or Natalie or Neve, kicks away from the claustrophobia and emotional incontinence of her childhood in the north of England in search of a self-contained, autonomous existence in Manchester or Glasgow, America or London. Often she recounts vignettes from her early years; her mother’s flight from her father, weekend access visits to be suffered through with a sibling. She fixes her parents on the page with darkly comic precision, mercilessly attendant to their tics and repetitions: a monstrous bully of a father who, in Opposed Positions, says after his daughter’s novel is published, “Oh dear! Oof! Posing! Er, what?”; a mother who bleats in book after book, “Well, it’s a long time ago now, isn’t it?” “Well, it was just what you did.”

Other themes drift through the novels – art, friends, cities, alcohol, lovers and in First Love a toxic husband – but these parent figures repeatedly haunt them. In My Phantoms, Bridget, a fortysomething academic, faces down her ghosts. There is a fascinating tension between Riley’s concision – the books are slim, and her honed sentences can encapsulate a character in a few short words – and this expansive reworking of her subject area. Repetition is key to damaged psychology, of course; the worst thing about dealing with screwed-up people is that each interaction is exactly like the ones before. In First Love, Neve suggested therapy to her habitually unhappy mother. “I mean, if you feel disappointed, or stuck.” In My Phantoms, Bridget does the same: “That’s something you can do if you feel stuck or frightened.” As Riley might write, does the mother ever try therapy? She does not.

One solution to the endless roundabout of toxic relationships is withdrawal: “I don’t do ‘family’ these days,” writes Bridget’s aunt merrily, informing her of her estranged father’s death. Bridget has simply waited her father out, as she waited out those ghastly weekend outings as a teenager. “I’m not sure I even thought of him as a person, really. Energised bother, in short. And yes, legally mandated.”

But it’s easier to outrun a bad father than a bad mother, and the figure of Helen Grant here is more complicated, nuanced and interesting – to Bridget as well as to the reader – than her awful dad. Like all Riley’s narrators, Bridget has worked hard to resist her parents’ incursions into her adult self. Her boundaries are a fortress. But where her father would gleefully trample those boundaries – pulling down her trousers, snatching her book – her mother endlessly contests them, negotiating the fault lines between parent and child, appealing to the court of normality. “‘Bridge?’ she said. ‘Bridge? Why aren’t I allowed to meet John?’” “Everyone meets their children’s boyfriends”, she points out; “it’s so embarrassing when people ask”. “I wanted to say, What bloody people? But that would have been cruel, wouldn’t it? So she had me there.”

This is a brilliant portrait of a mother-daughter relationship in which every encounter is a battle because both sides want something more, or different, than the other will give. “I loved being in our flat. I loved closing the door behind me,” Bridget writes. And then in a deliciously uncomfortable standoff, Helen threatens to piss in her doorway if Bridget won’t let her in. “‘I’ll just go here then shall I?’ she said. ‘I’ll just do it here.’” But Helen also resists and refuses Bridget in her turn: her questions, her advice, any revelation of her authentic, uncontainable self. The mothers in Riley’s books are always baring their teeth to their daughters, like angry dogs or frightened chimps (Helen is more than once compared to a dog: abject, unknowing, animal). They bring intimacy, and its trailing twin, shame; in First Love, when the child Neve kisses her mother’s bare foot, she recoils. “‘That’s like what a … boyfriend would do,’ she said. ‘Not your daughter. No.’”

There is a vigorous outpouring of revulsion towards the mother figure in First Love, and Neve is a stylish, aphoristic declarer of her truth. My Phantoms is a more sober, understated, subtle reckoning with an unfixable problem. Bridget has engineered the relationship to a manageable once-a-year birthday dinner (which, typically enough, falls on Friday 13th; “‘I feel an extra resentment when I have to come out in the dark and the cold,” Bridget lets slip; “Pardon me for … being born,” her mother replies). But as Helen ages, suffering a fall and illness, she is inevitably drawn closer. The previously unthinkable happens, because it has to – a stay in Helen’s flat, sorting through clutter together (“I’d grown up surrounded by shit, and I always enjoyed getting rid of it”). Emergency phone calls. Attempts at hugs.

All Riley’s novels are about authenticity – the difficulty of being true to yourself, when you have been raised in bad faith. In Riley’s debut, Cold Water, Carmel states, “What I don’t like is when these people need to make others complicit in their big lie. When they need an audience to bore or someone to push around.” She’s talking about customers in the bar where she works, but it’s also an excellent description of toxic parents. How to avoid remaining complicit in your family’s big lies? Therapy is the usual means of escape; Bridget deploys its soothing rhythms when talking to her mother – “Of course. I’m sorry. That’s a shame, isn’t it, that he’s like that” – but she is also uncomfortably aware of her own superior perspective and “usual mean reasonableness” stoking the power imbalance between them.

Bridget’s partner, John, is an analyst, which allows for a professional’s evaluation of Helen – life for her, he opines, is a performance that is “desperately committed but gratingly false”. (Yes, she is “allowed” to meet him at last, and the encounter is as awful as you might expect.) This exterior perspective doesn’t feel necessary: Helen is starkly illuminated through every exchange with Bridget, every feint and failure of communication. She condemns herself, as we all do, out of her own mouth. And yet the forensic quality of Bridget’s attention is fuelled by imaginative sympathy as well as distance and disgust. As the book goes on, in all its horrible, funny, uncomfortable truthfulness, it feels increasingly like a complicated act of love.

• My Phantoms by Gwendoline Riley is published by Granta (£12.99). To order a copy go to guardianbookshop.com. Delivery charges may apply.