“It can be harder than aid work,” a colleague cautioned me when I first applied to work in the remote Northern Territory. I’d just returned from six months with Médecins Sans Frontières, volunteering in a corner of Africa that was recovering from decades of civil war, so I’d assumed that I’d be ready for anything.

“It’s harder,” my colleague explained, herself an experienced aid worker, “because you don’t expect it to be. And because it shouldn’t be.”

Still, I thought I was prepared. But I wasn’t at all.

The distances are an obvious challenge. The 20-bed hospital I arrived at was far from Alice Springs, and served a region the size of Norway, with an overall population of about 8,000 people. Healthcare needs were high and it was a long way from help.

One of the first patients I met was Billy, a thirtysomething Aboriginal man, who walked in with severe shortness of breath. He was dressed neatly in jeans and shirt, a well-worn cowboy hat and dusty boots, and was suffering from heart and kidney failure. What he needed was an intensive care unit and a team of specialists; what he got instead was me, in that little hospital. He declined to be transferred to a bigger hospital because he was tired of all the tests, he said, and his family were there, and it was his country, so he’d take his chances with us. Fortunately, he did OK.

A few days later a woman arrived having a life-threatening heart attack. We gave her a clot-busting drug and stabilised her, and the Royal Flying Doctor Service promptly came, but the nearest centre that could manage her treatment was 1,000km away – like going from London to Berlin for urgent care.

The rates of chronic disease were higher than anything I’d encountered. Chatting with a teenaged patient, I noticed a constant, soft, ticking sound, not unlike an old clock – his metallic heart valve, inserted in Melbourne years before. This was from rheumatic heart disease due to repeated streptococcal infections, driven by overcrowded housing and poverty. Rates in this region were among the highest in the world.

Kidney disease is rampant, too; a dialysis unit there runs six days a week and has 16 machines – the highest number in the world, per capita, as far as I can tell. Diabetes is three times more common among Aboriginal patients in remote communities compared with other Australians, and heart disease more than twice as common. Access to affordable, healthy food is a huge contributor to this. I’ve paid $10 for a piece of rubbery broccoli in the only supermarket in town, making chicken wings, chips and a Coke far better value on a budget.

And then there are the social issues, shaped in large part by historical traumas and ongoing inequality. A man walked in one night, unsteadily, wanting pain relief, and it took me a moment to notice the bloodstain on his shirt. He casually showed me a knife handle in the middle of it – the blade still embedded. We gave him far more than just pain relief: he received a blood transfusion, a tube into his chest, antibiotics and an urgent flight.

Mental health crises were a common presentation, too, and I saw more psychological trauma than anywhere I’d worked. Of course, this was biased – in an emergency department, you see all the problems. Those who are doing OK don’t come in. That said, it was striking. And, after that first trip, I didn’t think I’d return.

That was almost 15 years ago. I’ve since changed paths and specialised in rural and remote medicine. I even moved to that little NT town for a year, and now continue to do fly-in/fly-out work in various communities. Why the shift? Adventure and a challenge were a part of it but then I got to know people, and it became hard to close the door and walk away.

The work gets under your skin. It’s also immensely enjoyable. Most people are warm, open and welcoming. A local artist told me she was declining a free trip to Paris for an exhibition of her art; she’d been before, she said, and it was cold, crowded and people weren’t friendly. “And we got barra season here,” she said.

I’ve worked with the Royal Flying Doctor Service for years as well, doing community clinics and rescue flights in far north Queensland, and I’ve spent time working in Arnhem Land and the Kimberley. It’s infinitely interesting but at times I’ve wondered how much longer I’ll do it. The vicarious trauma adds up, as does the moral injury of not always being able to do the best thing for people. The social problems grind me down. But I can always leave – the patients I work with can’t. And I’m acutely aware of being the outsider; I only ever pass through these worlds but others have to endure them.

One thing that’s become obvious to me, everywhere I work, is the impact of the social determinants of health – the upstream, systemic factors that are outside the influence of most individuals, and the clinic. Things like poor housing; nutrient-poor, highly processed food; limited employment opportunities; distance from services; historical trauma; substance abuse; and more. Much of what I do as a clinician is reactive. Of the 17 Closing the Gap targets, few are on track to be met. Some are outright worsening – a public health crisis.

My colleague was right: working out there is harder because you don’t expect it to be. And because it shouldn’t be. Not in the middle of a wealthy country.

• Bush Doctor by Dr Damien Brown is out on 28 April through Allen & Unwin