More than a fifth of the authors shortlisted for British literary prizes last year were black, a turnaround for an industry in which no black authors were shortlisted in four of the years between 1996 and 2009.

In 1996, there was not a single black author shortlisted in prize shortlists analysed by the Guardian, but that figure jumped to 21% in 2020.

The gradual improvement in diversity of shortlists follows years of anger and frustration with Britain’s publishing industry, which has been criticised for failing to address racial inequalities.

The backlash against the lack of diversity in UK publishing intensified during last summer’s Black Lives Matter protests, which were the largest anti-racist mobilisations in British history.

The Guardian looked at the racial diversity of the shortlist of eight leading literary prizes between 1996 and 2020: the Booker prize, Women’s prize for fiction, Folio prize, Orwell book prize, Baillie Gifford, International Dylan Thomas prize, Carnegie medal and Costa book awards (encompassing Costa first novel award, Costa novel award, Costa biography award, Costa poetry award and Costa children’s award).

Overall, black authors made up 6% of shortlisted authors in the UK’s top literary prizes in the past 25 years. Over the same 25-year period, black Britons made up 3.1% of shortlisted nominees.

The percentage of black and minority ethnic (BAME) authors increased from 4.65% in 1996 to 34.25% in 2020.

Between 1996 and 2020, there were 1,357 entries, of which 82 were black authors (7.1%). In the years 1996, 2001, 2002 and 2009 there were no black authors shortlisted across any of the prizes. The 2009 shortlists did not include the Dylan Thomas Prize and the Folio Prize.

The analysis demonstrates that disparities remain between prizes, with some awards announcing more diverse shortlists than others.

The most diverse prize in terms of black entries was the Dylan Thomas Prize, the leading prize for young writers, with black authors representing 15.28% of the shortlist. The prize was also the most diverse for BAME authors, making up 29.17% of shortlisted entries.

The least diverse prize for BAME entries was the Carnegie medal, where just 6% of those shortlisted were BAME authors.

The Carnegie medal, the UK’s oldest children’s book award, was criticised for its lack of diversity in 2018.

Between 1996 and 2017, Malorie Blackman was the only black author shortlisted for the prize out of 150 authors. Between 2018 and 2021, having made an effort to increase racial diversity, black authors made up 37.5% of shortlisted nominees.

In 2015, the novelists Sunny Singh and Nikesh Shukla launched the Jhalak prize, to be awarded annually to a writer of colour with the book of the year, after backlash against the lack of diversity in prize nominations.

Singh advised people to be cautious about the findings. She called for a breakdown on how many of the shortlisted BAME authors were British or longstanding residents, and for further analysis on how many went on to win the award.

She said: “The Carnegie medal to this day, 84 years down the line, has still not managed to be awarded to a black British writer or a British writer of colour. What’s going on?

“I think the problem is that as long as the publishing industry and the literary establishment sees inclusion as a problem, sees equity as a problem, on whichever axis, not just race, gender, sexuality, class, and disability … we will not move past the superficial thing of a kind of tokenism where it’s the boxes are ticked, and then they can move on.”

She added, however, that the past year had felt like a “watershed moment” for tackling racial inequalities in the industry. “It just feels like there’s a lot more of us speaking up and speaking out, just pushing in the same direction and doing so publicly than they were even five years ago.”

After the protests in June 2020, Bernardine Evaristo and Reni Eddo-Lodge became the first black British women to top the UK’s fiction and nonfiction paperback charts. The then newly formed Black Writers’ Guild (BWG) wrote an open letter airing concerns that British publishers were “raising awareness of racial inequality without significantly addressing their own”.

In that same year, the Booker prize announced its most racially diverse shortlist to date. Evaristo became the first black British writer to win the award, which she shared with Margaret Atwood.

Methodology

The Guardian researched every individual listed, looking at pictures and references of how each author identifies. In cases where a person’s ethnicity was unclear, further checks were carried out to determine how the author said they identified, which may include speaking to the author in question.

The overall proportion of black authors – 6.2% – is based on the number of nominees meaning the same author could be counted more than once. The equivalent figure for the number of individual black authors is 6.5%.

Between 1996 and 2020, some of the individual years within this period did not feature every prize listed, as some did not run in certain years.

Books that were double-authored were counted as one author, as none of the books that featured double authorship had authors from different ethnicities.

• The subheading of this article was amended on 1 October 2021. An earlier version said no black authors were shortlisted “between 1996 and 2009”; as the article made clear, this was the case in four of the years during that period.