I first encountered Shirley Jackson’s The Lottery in fourth grade at an elementary school in a small north-eastern American town not unlike the one where the tale is set. Our teacher, Miss Halloran, liked to give us challenging reading material, so I can only assume she knew what she was doing when she had us read Jackson’s story aloud, each student taking a turn as the tale progressed from its folksy beginning to its shattering conclusion. Glancing up from the page, I saw the expressions of my classmates change, from mild boredom to confusion to shock to outright horror when someone read the tale’s final lines: “‘It isn’t fair, it isn’t right,’ Mrs Hutchinson screamed, and then they were upon her.”

Published in the New Yorker in June 1948, The Lottery depicts the residents of an “ordinary” small American town indulging in an annual ritual orgy of violence. Each year, one of the villagers, chosen by lot, is stoned to death by neighbours and family. Our class read it in 1967, the year of the Summer of Love, and also of cultural and social upheavals we were barely aware of. Was Miss Halloran trying to teach us about moral culpability, the dangers of conformity, the threat of violence that was soon to erupt in riots across the country?

I have no clue. All I knew was that I wanted more Shirley Jackson. I read The Haunting of Hill House next, and then The Bird’s Nest and We Have Always Lived in the Castle, and some years later Hangsaman and The Sundial, along with her short fiction. I’ve reread her work over the decades, continually amazed by Jackson’s ability to unsettle, provoke and elicit sympathy for characters who are often unabashedly unpleasant, in prose spare and sharp as a boning knife.

She grew up in California, the only daughter of conservative country club parents. Throughout her life, Jackson’s mother never stopped harping on about her daughter’s appearance – her clothing, her hair, her weight. Jackson attended university in Syracuse, where she met her future husband, the critic and college professor Stanley Edgar Hyman. They married and moved to North Bennington, Vermont, the picturesque, insular New England village that inspired the settings of some of her most enduring work.

Hyman taught at Bennington College, then a haven for privileged young women; Jackson spent much of her adult life on its outskirts. There she raised four children, immortalised in her bestselling nonfiction collections Life Among the Savages and Raising Demons, precursors to the domestic humour popularised by Erma Bombeck, Jean Kerr, and Nora Ephron and numerous contemporary blogs and influencers.

During her too-short life – she died in 1965 at the age of 48 – Jackson produced six novels, a collection of short fiction, three book-length works of nonfiction and several children’s books, along with myriad stories for women’s magazines. Her most memorable work shows women in the process of disintegration, often through the lens of trauma, emphasising how their desires – for love, sexual union, knowledge, creative freedom – curdle into despair and terror under the relentless, venomous drip of sexism and misogyny.

In The Lottery, that disintegration is literal. More often Jackson details a psychological rupture. Natalie, the 17-year-old protagonist of her second novel, Hangsaman (published three years after The Lottery), is raped on the eve of her departure for a nearby college (an obvious stand-in for Bennington). The novel traps the reader in Natalie’s subsequent dissociation from campus life, culminating in a nightmarish visit to an abandoned amusement park, where she contemplates suicide. In one extraordinary scene, she envisions herself as a giantess attacking her housemates, dismantling the college dorm “while the mannikins inside run screaming … I shall eat the room in one mouthful, chewing ruthlessly on the boards and small sweet bones.”

Jackson’s most famous novel, The Haunting of Hill House, puts a nihilist, supernatural spin on similar material. Its protagonist, the socially awkward 32-year-old Eleanor Vance, has spent 11 years caring for her spiteful invalid mother. As a girl, Eleanor caused an outbreak of poltergeist activity – for three days, stones fell on her family’s house as crowds gathered to witness the destruction. After her mother dies, Eleanor is invited to take part in a paranormal investigation, headed by Dr John Montague. He summons her and three others to Hill House, an adventure she undertakes with delighted anticipation.

But it soon becomes clear that, with Hill House, she has traded one grimly dysfunctional relationship for a far more dangerous one. In a reversal of house-sized Natalie devouring her dormitory and its residents, Hill House proceeds to engulf Eleanor’s psyche. The book can be read as a record of mental illness as much as a ghost story, especially in the terrifying climactic scene where Eleanor drives her car wildly down the mansion’s driveway, her final thoughts an aria of desperate disbelief: “Why am I doing this? Why am I doing this? Why don’t they stop me?”.

Houses aren’t safe spaces in Jackson’s fiction. They are stages for disruption, mirroring both their protagonists’ vulnerability and their worst impulses. In Jackson’s last and greatest novel, We Have Always Lived in the Castle, the two Blackwood sisters have barricaded themselves in their decaying mansion, feared and reviled by those in the neighbouring village (North Bennington again). Six years earlier the younger sister, Merricat, poisoned the other members of their family, apparently for being sent to her room without supper. Her older sister, Constance, took the rap but was acquitted. Now 18, Merricat lives with her and their doddering Uncle Julian, sole survivor of the mass murder.

Preternaturally serene, Constance compulsively cleans and cooks, stores preserves and clucks over Uncle Julian. Meanwhile, Merricat endures the taunts of village children whenever she ventures out to get groceries. Accompanied by her cat, she spends her days setting protective spells around the overgrown estate, burying coins and nailing books to trees. Occasionally she has savage fits, smashing dishes and glassware that Constance calmly cleans up.

The arrival of their cousin Charles threatens this peculiar idyll. Constance finds herself tempted to leave their mansion, but Merricat correctly sees Charles as an opportunist intent on the Blackwood fortune. Rather than surrender her sister and their home to the outside world, she sets a fire that destroys the upper floors. What happens to the Blackwood home next, at the hands of the fire department and villagers, magnifies the compressed violence of The Lottery.

Like Hill House, Castle was a commercial and critical success, nominated for a National Book award. Today, more than a half century later, her fiction is having a resurgence. The recent Netflix adaptation of The Haunting of Hill House was widely acclaimed and there are more screen adaptations in the works, including a film of The Lottery. You can see Jackson’s influence on writers such as Silvia Moreno Garcia and Carmen Maria Machado, and in films including The Babadook and Midsommar.

Jackson’s work has a particularly disturbing resonance today, when our experience of the pandemic and reports of rioters rampaging through the US Capitol and erecting gallows outside are still fresh in the memory. Small–town meetings are routinely interrupted by people spewing hate speech, and rightwing politicians work tirelessly to dismantle women’s rights, setting the clock back 75 years, to when The Lottery first appeared. Yet although Jackson acknowledges issues of abuse and female disempowerment, the ostensible victims in her tales often in the end gain the upper hand, or at least a new mode of survival.

Throughout her career, Jackson fixed a dispassionate, basilisk gaze on men and women alike, shattering the carapace of respectability that hides our darkest drives. She exposed the fragility of so-called civilised life, how swiftly ordinary people can brutally turn against their own family, friends and neighbours. Pierce the skin of any one of us, she suggests, and what oozes out isn’t blood, but venom.

• A Haunting on the Hill by Elizabeth Hand (Sphere) is the first authorised sequel to The Haunting of Hill House. To support the Guardian and Observer, order your copy at guardianbookshop.com. Delivery charges may apply.

• This article was amended on 27 October 2023. An earlier version said Shirley Jackson was an only child – in fact she had a brother – and that she attended Syracuse University 3,000 miles away, though it was much closer than that. Also, Hangsaman was published three years after The Lottery, not in the same year.