When he first visited the Soviet Union in 1954, Eric Hobsbawm discovered that the theory of a workers' state and the practice of a Moscow still bleeding from Stalin's last purge did not quite gel. "It was an interesting but also a dispiriting trip for foreign communist intellectuals," he recalled in his autobiography, "for we met hardly anyone there like ourselves."

Wrong part of Russia, Eric. If he had gone to Siberia, alongside the corpses of "anti-Soviet" Ukrainians, Lithuanians, Estonians, Latvians, Chechens, Tartars and Poles, of tsarists, kulaks, Mensheviks and social revolutionaries and of merely unlucky citizens who had been denounced by malicious neighbours, or rounded up by the secret police to meet an arrest quota, Hobsbawm would have found the bodies of communist intellectuals – just like him.

No one killed as many communists as the communists did. If Hobsbawm had followed the logic of his convictions and moved from Nazi Germany to seek a home in the Soviet Union rather than Britain, his chances of surviving would have been slim. Either the party would have shot him in the great purge for being foreigner and a Jew to boot, or he would have been forced to denounce innocent comrades to save his skin. After concluding the Nazi-Soviet pact, Stalin handed German communists over to Hitler as a gesture of goodwill. If the purge of 1936-38 had not killed Hobsbawm, the pact of 1939 probably would have done for him instead.

Perhaps sensing that there may be uncomfortable reasons why intellectuals were not in evidence in Moscow, Hobsbawm retreated into the comfort of socialist utopianism. What foreign intellectuals "thought about the October revolution was not the same as what our guides from the Leningrad branch of the Academy of Sciences thought," he concluded. They had to live with the murderous practice. He could return to London and enjoy the glorious theory.

Hobsbawm is now 94, and although I have no wish to usher the old boy from the room, I can see his obituaries now. Conservative and liberal writers will say that his loyalty to totalitarianism disfigured his writing, most notably, in his whitewashing of Soviet atrocities in The Age of Extremes, his history of the 20th century. Leftists will say he was the greatest Marxist historian of our times, whose sweeping accounts of the world from the Enlightenment to the present have shaped the way we think. Neither will acknowledge that both have a case, as Hobsbawm's new book, How to Change the World, suggests.

It has a misleading title. This is not a guide to protest in the 21st century or even a coherent history of the socialist movement of the 20th, but a series of essays dating back to the 1960s on Marx and Marxists. Several are for connoisseurs of the far left only. If readers do not know who Earl Browder was and why Stalinism denounced "Browderism" as yet another heresy, Hobsbawm will not tell them, and a passage in his account of the tensions within the communist movement of the 1940s will make little sense.

As always when he writes about the Soviet Union, Hobsbawm offers critics an embarrassment of targets. He opines that "the most difficult part of Marx's legacy for his successors [has been that] all actual attempts to realise socialism along Marxian lines so far have found themselves strengthening an independent state apparatus." As bashful euphemisms go, "strengthening an independent state apparatus" ranks as one of the coyest descriptions yet of the merciless terror of socialism in its Soviet, Chinese, Korean and Cambodian forms. He offers a perceptive description of the anti-fascism of the 1930s, the moment of his political awakening and "the only part of their political past on which the survivors of that time look back with unqualified satisfaction." But he cannot bring himself to acknowledge that it ended with Hobsbawm and everyone else who stayed with the Communist party allying with Hitler on Stalin's orders and declaring that Britain was the left's real enemy as the blitz began. (All Hobsbawm can say of the Hitler-Stalin pact was that it was a "temporary reversal", which again deserves some kind of prize for euphemism.)

When he stops trying to persuade us to avert our eyes from some of the worst crimes in human history, his merits peek through the fog of obfuscatory prose. To stay with the Communist party and ignore the mountains of corpses requires a certain cold-bloodedness; a willingness to put everyday sentiment to one side and try to see the sweep of history as Hobsbawm does magnificently in works such as The Age of Revolution, The Age of Capital and The Age of Empire.

Because he remained a Marxist, Hobsbawm did not fall for the idea that the end of the Soviet empire brought with it the end of history. The best writing in How to Change the World comes in the opening and closing essays on the future of leftwing politics, where he drily reminds us that so ingrained has the Washington consensus become, "no leader of a party of the European left in the past 25 years has declared capitalism as such to be unacceptable as a system. The only public figure to do so unhesitatingly was Pope John Paul II." He is surely right to say that the crash of 2008 destroyed the faith in rational markets, and that if we are to understand capitalism's tendency towards crisis, re-reading Karl Marx is not as odd an idea as it once seemed. Even at these lucid moments, however, Hobsbawm cannot keep his yearning for tyranny in check. "Political liberalism" is as unable to provide solutions to the problems of the 21st century as economic liberalism, he concludes. Whether he is still recommending dictators and secret policemen is a question he coquettishly leaves unanswered.

I normally say of Hobsbawm "he's a great man but…" But you have to take his communism as part of the package. But you must understand how he was shaped by the 1930s before you can appreciate him. I cannot be so even-handed with the airbrushing of atrocity in How to Change the World. The "but" is too big. It overshadows all the good in him.