In 2017, I published my fourth novel, Exit West, and bought a small notebook to jot down ideas for the next one. I thought it would be about technology. I came across an article by Simon DeDeo, an assistant professor at Carnegie Mellon University, discussing an experiment he and his colleague John Miller had conducted in that same year. They simulated cooperation and competition by machines over many generations, building these machines as computer models and setting them playing a game together. An interesting pattern emerged. Rather than constant trading for mutual benefit among equals, or never-ending fights to the death among foes, instead a particular type of machine became dominant, one that recognised and favoured copies of itself, and enormous prosperity ensued, built on ever-growing levels of cooperation. But eventually the minute differences that naturally occurred (or were, in the experiment, designed to occur) in the copying process, as they do in organisms when genes are passed on, became intolerable, and war among the machines resulted in near-complete devastation and a new beginning, after which the cycle repeated, over and over.

I remember being struck by this article. Not because I fully understood what the simulation was or even how it worked. No, I was struck by its similarity to a narrative I had already been feeling drawn to myself: that the rise and fall of human society is not merely something that has happened but also something that will continue to happen, that moments of peak cooperation contain within them the tendency for differences to become utterly intolerable, and that the transition from one societal epoch to the next is rarely a series of gently eliding waves, each a bit higher than the previous one – to the contrary, humanity’s trajectory on the way down is often far more steep than it was on the way up.

These points might be rather obvious. But they were not always obvious to me, not at an emotional level. I long believed that things would probably keep getting better for our species. We humans might mess up devastatingly, but we were more likely to find a way. A way to avoid nuclear war and mitigate climate change and expand equality and diminish poverty. This had been my mental model’s base case, as it were. Now I was wondering whether humanity’s true base case was far more grim, and whether my own expectations represented instead an optimistic but improbable best case scenario.

To be clear, I never doubted that there would be horrifying wars and mass displacements and economic dislocations in the years ahead. But I somehow imagined that these would be downward zags in our species’ upward-zigging trendline. I am not sure what exactly my faith rested upon. My relative youth, maybe. The overall shifts in human life expectancy and per capita income during my lifetime, perhaps. But also something else. Something from the realm of the spirit, of feeling: a sense of techno-optimism ungrounded in any profound understanding of technology. And now here was this blast of techno-pessimism. DeDeo’s machines had spoken, and they had not said what they were supposed to say. No, you fools, they had proclaimed, the end is quite possibly nigh.

Truthfully, I had been reeling since September 11, 2001, and my move in 2009 to what seemed to be a democratising Pakistan, with expanding freedom of expression and rule of law, had not worked out quite as swimmingly as I had hoped, and then in 2016 my former home Britain voted for Brexit, and my former former home America voted for Trump, and I was admittedly in a bit of a funk. It was as if I had been goading the universe: grant me pessimism, but not yet; and the universe had finally slapped me across the face and responded: now, child, it is time.

***

The American Empire is waning. This might well be a good thing, for Americans and non-Americans alike. But the early signs are not promising. Like DeDeo’s machines in the moments immediately after peak hegemonic cooperation, we are finding that our differences are becoming intolerable. When empires that span diverse populations disintegrate, history suggests the potential for conflict is high. The British empire in India gave birth to violent sectarianisms; the Ottoman and Austro-Hungarian and Soviet empires did much the same. The American empire was far more powerful and wide-ranging than any that came before. As it diminishes, aggressive sectarian impulses are seemingly in the ascendant almost everywhere. Name a country today, and it likely is either run by a strongman who claims to speak for the true people, the folk, or otherwise has a strongman-in-waiting manoeuvring to take control. (There are strongwomen too, but far fewer of them.)

I imagine the empire-building machines of DeDeo’s simulation as doing three things increasingly (and exquisitely) well: identifying differences that permit sorting into categories of like-me and not-like-me, cooperating with those in the like-me category, and destroying those in the not-like-me category. Our species has, similarly, become incredibly skilled at cooperating (thousands of people in dozens of different countries collectively manufactured the computer on which I am writing this). We too have developed the capacity to kill along an entire spectrum from wholesale (nuclear, chemical, biological weapons) to bespoke (quietly disappearing those who trouble us) – which is obviously a pressing, indeed existential, concern. But our ability to kill has not changed as rapidly, in recent years, as our sorting mechanism appears to have. It is our impulse to sort, and the importance that we place on sorting, that has truly gone haywire.

The end of the American empire is coinciding with the age of the cyborg. I spent much of the 1970s as a child in Silicon Valley. My father was a graduate student, my mother had an entry-level job at a technology company (they made a cutting edge storage medium known as the “audio cassette”), and I ran barefoot up bleached blond dry foothills and watched a black and white television set in which I was convinced I saw colours. The science fiction I loved seemed to suggest that the future would contain people just like us playing around with transporter beams and hyperdrives and photon torpedoes. None of that has really come to pass: ours is still a world of automobiles and not land speeders, rifles and not laser guns. But a transformation worthy of science fiction did occur regardless. We became attached to our screens, merged with the machine culture behind those screens, and changed far more than some child might in the process of merely becoming a Jedi.

The machine world is a binary world, and it strikes me that we have learned to apply those zeroes and ones to our thinking, intensifying our impulse to sort one another into like-me and not-like-me at what might well be, historically speaking, the worst possible moment: a moment when, as empire recedes, we are already predisposed to sort excessively and to fetishise tests of purity.

The result, as we can see all around us, is a disastrous confluence of polarisation, militarism, democratic dysfunction, and environmental disregard. In the same way that the most deadly aspect of Covid, before vaccines and pharmaceutical treatments became available, was an overreaction of our immune systems to the virus – the dreaded cytokine storm – destroying healthy lung tissue in an overzealous attempt to fight disease, the challenge we face now is an overreaction of our societal immune systems to one another. It is ruinous for us to be ones rampantly identifying as zeros those with whom we have any significant level of disagreement. It underestimates the human capacity for messy and unexpected plasticity. Other approaches to seeking a better, more inclusive, and more equitable future urgently need to be found.

***

Literature, fortunately, is a profoundly weird creature, capable of being zero and one at the same time. This characteristic is of vital importance. Profound weirdness is quite possibly what our age of the cyborg most demands.

A strange thing happens when we read novels. We are simultaneously ourselves and not ourselves. We are ourselves because we are alone, in solitude, without anyone else present. Who else could we be? And yet we are not ourselves because we contain within us the consciousness of another person, the writer. This consciousness is transmitted in the form of words, words that we readers animate into people and emotions and images and events. Reading a novel is to experience two consciousnesses present in one body, reader and writer co-creating their novel as it is read, a novel unique to each reader-writer pairing, because it has been imagined into being jointly. The self while reading is uncanny, a plurally conscious peculiarity: transgressive, fertile, and very much at play.

A strange thing happens when one writes novels too, at least in my experience. When I am writing there is an aspect of desiring to communicate what it is to be me: the events of my life, perhaps, but also my way of seeing things, my way of expressing myself, my sense of how language works, of what is interesting or funny, of what matters. But there is also an aspect of desiring not to be me, of sitting down at my desk and opening a portal, of leaving myself behind, of being elsewhere, other places, of inhabiting, from the inside, other imagined people. As a writer, I wish to share myself, to be less alone by doing so, but I also wish to be free of myself, free of my habits and my ways and my constant performances, the rituals of enacting me that endure out of habit, sometimes even in the absence of true faith that the person I am playing and the person I am are one and the same.

Authenticity and transgression. There is such a wonderful, unresolved tension between the two: one we often shy away from, but one which remains at the heart of reading, of writing, and, it should go without saying, of our individual attempts at being human.

***

In my new novel, The Last White Man, Anders wakes up one morning to find that his skin is dark. When he went to bed it was not. He reveals his secret to his lover, Oona, and together Anders, Oona, his father and her mother navigate a world where Anders’s predicament starts to spread. It is a world of turmoil and transformation.

I have been asked where the book comes from, why I felt the need to write it, and while it is never fully clear to me where my books come from, just as it is less than clear to me where I come from, my strong sense is that this novel spent two decades in gestation, and that its beginnings can be traced back to September 11, 2001. After that day, many things changed for me. I no longer breezed through airport immigration, but was instead held and questioned for many hours. I was invariably selected for additional security checks before flights. Visas became more difficult to come by. Fellow passengers on buses and trains appeared less comfortable at the sight of me boarding with my backpack; occasionally they got up and switched seats. People I did not know seemed more inclined to dislike me, even to fear me.

I was 30 then and had lived 18 of my years in the west, mostly in America, and very recently in Britain, my new home. I had always been a brown man with a Muslim name. That had not changed. And yet something had changed. I had lost something profound. I was saddened and angered and confused by my loss. But it took some time for me to understand what it was: I had lost my whiteness. Not that I had truly been white. But I had been white enough – as a relatively well-paid, university-educated inhabitant of cosmopolitan cities – to partake in many of the benefits of whiteness. And now my partial membership was being revoked. It hurt, both the loss itself and, later and perhaps more hauntingly, the recognition of my own complicity in an unjust system that had benefited me.

I had been offered a reminder that race is a construct. It is brought into existence by our imaginations, and from there it is deployed with real consequences. There are darker and lighter skin colours, of course, but these skin colours are in and of themselves no more meaningful than blood types. It is we who invent race and its terrible meanings. But, though potent, our inventions are not stable. They sprang into existence not so very long ago, they are constantly changing, and one day they will be gone.

I wrote this novel to explore what it has been to be myself, and also to explore what it is to be other selves. I intend it as a means for readers to do the same. We risk being trapped in a dangerous and decadent tyranny of binaries. Perhaps fiction can help us investigate the space between the ones and zeroes, the space that presently seems empty, impossible, but then, when entered, when occupied, continues to expand and expand, bending and stretching and eventually, possibly, revealing its unexpected capacity for encompassing us all.

• The Last White Man by Mohsin Hamid is published by Hamish Hamilton (£12.99) in the UK, and Penguin Random House ($32.99) in Australia. To support the Guardian and Observer, order your copy at guardianbookshop.com. Delivery charges may apply.

• Join Mohsin Hamid for a Guardian Live online event on 10 August, when he will discuss The Last White Man with Today in Focus host Nosheen Iqbal. Book here