There are images in barrister Andrew Boe’s new book that are indelible, sketched only in a few words. Like when Boe had represented a man acquitted of an extremely violent assault, and was almost shirtfronted by a friend of the complainant outside the courtroom who was “shaking with rage”. “How do you people sleep at night?” she challenged him.

Or the family who came to see him after their son was charged with child sexual offences. They appeared catatonic, the son unable to explain why he had acted so abhorrently.

Or the image Boe paints of himself as an exhausted trial lawyer, falling asleep and waking to find himself resting his head on his client’s shoulder. The client was backpacker killer Ivan Milat, who was “holding me like a baby”.

It was during his three years acting for Milat, who was found guilty of murdering seven people, that the Brisbane lawyer became widely known outside legal circles. His work would gain further public attention through Chloe Hooper’s vivid portrayal of him in The Tall Man (2008), her extraordinary book about a death in custody on Palm Island.

In The Tall Man, Boe is a relentless, campaigning lawyer – an “elegant monk-bald figure with glasses and a tattoo on his biceps in Burmese that meant ‘freedom from fear’ … [who] did a hard line in moral earnestness”.

Meeting him now at his chambers in central Sydney, Boe, 55, is nattily dressed in a black flat cap and knotted scarf. His chambers are decorated with paintings from his daughter, Mia, work by the Indigenous artists Vernon Ah Kee and Richard Bell, and gifts from his clients, including a First Nations talking stick from Canada.

Fifteen years on from Palm Island there’s not so much evidence of Boe doing a “hard line in moral earnestness” as there is in his having a clear-eyed, unsentimental view of the legal system and human nature.

“We expect so much from the law because the law says it’s so important,” says Boe. “The framework within which these disputes are litigated are put into these grand spaces of public theatre. There’s genuflecting, robing, all of that, but the human agents in this system often do not live up to this ideal.”

Boe’s stunning new memoir, The Truth Hurts, reflects this perspective. There’s a section about his early years, from arriving in Australia aged four, the child of Burmese immigrants escaping a military junta, to his father’s mysterious stint in jail, and his first job at a law firm: an errand boy in a Salvation Army suit.

But this book is not the story of Boe’s life. Once his legal career takes off – first as a solicitor, then as a barrister – the cases he takes on become the focus of the book, and provide a fascinating glimpse into a world many of us will never see.

It becomes more than that, too. A story emerges of life on Australia’s margins, of how ill-suited our legal system, imported from our colonial rulers, is to provide “perfect justice”.

There’s the man accidentally killed in a ritual spearing, and the utter incomprehension an urban Perth jury would have in deciding the fate of the accused, an Indigenous man who was acting in line with conduct expected of him by his community.

There’s the Indigenous woman sentenced to life for fatally stabbing her partner with a kitchen knife, and how only on appeal was it revealed that she had suffered years of rape, bashings and mistreatment, including three days tied to a bed.

There’s the slight, quiet schoolteacher accused of murdering her husband after years of horrendous physical, emotional and sexual abuse. Boe was her lawyer on appeal.

Boe calls lawyers “actors with brains” but his job in all these cases is to find space or accommodation within the legal system for his clients to make it through the other side without too much damage, deprivation of liberty or injustice.

Boe is aware he’s operating within an imperfect system. But throughout the book there is commitment to some of the legal system’s higher principles, such as the presumption of innocence and the cab-rank rule that obliges barristers to accept the clients sent their way. These higher principles allow Boe to advocate for people such as Milat and those accused of other heinous crimes.

Does Boe ever worry that spending decades at the coalface of depravity had a coarsening effect on him? Or has it deepened his reserves of compassion?

“Some of the cases can be very corrosive. I think I’ve had both experiences happen to me,” says Boe.

“When I did do horrific cases where there were photos, I’d never look at them. If it’s going to be shown to the jury I’ll see it but I’m not going to pore through volumes of photographs about cutting open a person to see where the bullet went. All that stuff, I didn’t need to have in my head. You have to protect yourself a little bit.”

But something that Boe does see, particularly among young criminal lawyers, is anxiety.

“A lot of the alcohol abuse that happens from criminal barristers, it’s anxiety of not having done something properly, having missed something. And only you know if you’ve read every page of that brief,” he said.

“More and more I just decided not to do those cases unless I thought I honestly had the resources to apply to them. If you are doing back-to-back murder trials funded by legal aid at a tuppence, not only are there issues but the consequences of any mistakes or oversights that you make can be very corrosive.”

That anxiety about getting it wrong is mostly gone at 55, as is the feeling of “always looking over your shoulder”. He’s lived in Sydney for 10 years now, with his partner, the filmmaker Samantha Lang. And while the briefs are slow to come along during the pandemic, he’s looking forward to getting stuck into work that’s either “a really complex forensic explanation of something that will surprise me, or matters where there’s a bit turning on it.”

Matters involving race still fire him up.

“Will we have a watershed moment about race in Australia? I think no. First and foremost the fact that it took a transatlantic death (George Floyd) to excite the sort of levels of emotion we’ve had in Australia says a lot about Australia. It’s mimicry,” says Boe.

Symbolism – the walk across the Sydney Harbour Bridge or hashtag activism – doesn’t meet Boe’s criteria for enacting meaningful change.

He remembers the funeral of Mulrunji Doomadgee on Palm Island – how the whole community turned up – and laments the lack of those community connections in mainstream Australia. “We live in these little boxes – $3m, $4m, $5m boxes – in which we have all the games and tricks in there; where we protect our children from the world but expose them to all the crap on social media. But they don’t actually understand where they are in this country,” he says.

“I don’t think the conversation will change until more of us say we break bread with Indigenous people. To me, that’s not happening right now,” he says. “Australians have always had a special racism towards Indigenous people and that comes from the guilt of having stolen from them and not [being] willing to take responsibility for that.”

He’s hopeful that in a generation or more “when that’s done, it will be more than just the marching, we will become more culturally rich. We will have a greater sense of time and space – and kinship.”

• The Truth Hurts by Andrew Boe is out now through Hachette