The only public execution of a British head of state occurred 371 years ago outside the Banqueting House in Whitehall on 30 January 1649. It was a radical, unintended act, born of failed negotiations and it entirely disregarded the people of Scotland and Ireland. The surviving details are piquant and shaming: Charles I in two shirts so as not to betray shivered fear on a cold day, a masked executioner in a wig and false beard, a collective groan from the crowd.

This used to be the moment when the curtain dropped. After an interval, or interregnum, in which Oliver Cromwell assumed power and killed all joy, the Stuarts returned for the second act, Charles II: the merry monarch. His reign was backdated to his father’s death and an Act of Oblivion drew a line under the horrid business of republic and civil war. As Paul Lay demonstrates in his immensely stimulating study of Cromwell’s Protectorate, this simply will not do. This was one of the most extraordinary, exhilarating, innovative and anxiety inducing periods in British history. There is no better time to be looking at it.

The providence of Lay’s title is the lodestone Protestant belief that God in his mystery had a hand in all things. Nothing could happen, not salvation, nor a sneeze, without divine direction. This made the hotter sort of Protestants, the Puritans, off-balance and twitchy for approval, like social media addicts, Lay observes. None was more hooked on providence than Cromwell, a yeoman farmer from Cambridgeshire (and kinsman of Henry VIII’s minister), who had risen through the ranks of the army to take effective charge of Whitehall and Westminster.

Providence is also a volcanic island in the Caribbean and it is here that the book opens, on Christmas Eve 1629, with idealistic Puritans hoping to settle a godly colony. They failed, largely because they refused to trade with Spaniards, but the vision endured, as did its backers, including key players in the Long Parliament that went to war with the king. At the battles of Naseby, Marston Moor and the “crowing mercy” of Worcester, God blew and the royalists were scattered. Providence justified the abolition of the monarchy and the House of Lords, just as it explained Cromwell’s clean sheet of victories. After his seemingly miraculous defeat of the Scots at Dunbar in 1650, he was seen laughing excessively “as if he had been drunk; his eyes sparkled with spirits”.

Others thought he was high on hubris. In December 1653, having forcibly dissolved two parliaments, Cromwell accepted the title of Lord Protector, which was written into a constitution known as the Instrument of Government. It was resisted by the first protectorate parliament and so, in a constitutional contortion that might make even today’s politicians blush, Cromwell dissolved parliament by counting its minimum period in lunar months, rather than the conventional, longer, calendar months. “I speak for God and not for men,” he told MPs. He was beginning to sound a lot like Charles I.

Cromwell’s fundamental goal was liberty of conscience, but his challenge was to keep his army standing, deliver strong and stable government (“healing and settling”) and shape the people into a nation worthy of Christ’s imminent return. In that great gross line, he described the English as “under circumcision, but raw”. Above all, he strained to hear God’s voice once the guns had fallen silent.

The answer, he decided, was global. He would take God’s fight back to the Caribbean, seize the island of Hispaniola, avenge the loss of Providence and return with enough bullion to pay the army. The expedition commander was so buoyed by the venture that it doubled as his honeymoon. He returned to Portsmouth “almost a skeleton” and was dispatched to the Tower of London. The “Western Design” was an abject failure. The subsequent conquest of Jamaica was a face-saving afterthought and, it would turn out, the inversion of liberty.

In that crucial year, 1655, Cromwell also suppressed an ineffectual royalist rebellion. Spurred by John Lambert, the hero of Dunbar, he surmised that God would be placated by a clampdown on security and sin. England was divided into military districts. Former royalists had to register their movements and pay for the privilege in the form of a decimation tax. Vagrants were apprehended. Meetings at racecourses and bear-baiting rings, where conspiracies might fester, were dispersed. War was waged on fornication, drunkenness and gambling: “wars unwinnable”, Lay notes. In Altham, Lancashire, 200 unlicensed alehouses were closed within three months of the arrival of Major-General Worsley, a fundamentalist who drove himself to death in his maniacal pursuit of vice. The major generals lacked the resources to effect a moral reformation and were soon rejected by the electorate. It was, nevertheless, a dystopian time and not forgotten: “No man in uniform has ruled Britain since Cromwell,” Lay points out.

Providence Lost is a learned, lucid, wry and compelling narrative of the 1650s as well as a sensitive portrayal of a man unravelled by providence. Lay’s Cromwell is dithering and ambiguous, too slippery to pin down, but almost likable, almost true. He welcomed Jews back to England, even if he aimed to convert them to Christianity (Sigmund Freud named his son Oliver). He regretted, on a personal and procedural level, the Commons’ retrospective judgment of James Nayler, a priapic, lank haired Quaker with messianic pretensions, who was branded for blasphemy and whipped until his back was a sheet of raw flesh. (One MP even mooted the obscure Roman punishment of stitching him into a sack with a dog, a cock, a viper and an ape, and tossing him into the river.)

As Cromwell settled into his reign – an accurate enough word – he displayed shades of hypocrisy. “Lascivious” songs and stage plays were forbidden, but the elite could enjoy dancing and the first English opera. Cromwell dipped into Charles I’s art collection and had a room at Hampton Court draped in sinful tapestries. And yet: when, in the wake of a royalist-leveller gunpowder plot in 1657, the issue of the succession led to parliament offering Cromwell the crown, he turned it down: “I would not seek to set up that which providence hath destroyed.”

On Cromwell’s death in 1658, the Protectorate passed to his son Richard, who had that rare gift in politics of realising when he was not up to the job. He did not resist when army radicals ousted him in May 1659. The following year, having lost any semblance of order or legitimacy, they too were gone. The Protectorate was too out of touch, too reliant on the sword and far too expensive to be missed. Its failure ensured that whichever randy prince might come along, monarchy would endure.

Two contemporary works capture the contrasting tempers of the time. The Anglican Izaak Walton’s The Compleat Angler is a fishing manual that rejoices in the beauty and bounty of the land. It reflects still waters and a contented patience beyond the ken of Puritanism. Lay doesn’t discuss John Milton’s Paradise Lost, but the title and its themes of tyranny and failed revolution shadow every page of his book. Milton gave his life to the Puritan cause and was crushed by its betrayal, but even when he went blind, he did not lose sight of liberty, or God. Paradise Lost closes with Adam and Eve wiping their tears and venturing out of Eden with “providence their guide”. It is an epic, eloquent scream into the cataclysm and a testament to the intellectual ferment of the republic, but it is the quiet little fishing book that has been reprinted more often.

• Providence Lost: The Rise and Fall of Oliver Cromwell’s Protectorate is published by Head of Zeus (RRP £30). To order a copy go to guardianbookshop.com. Free UK p&p over £15.