No wonder scholars love 18th-century British gardens. Their affluent owners did not just pay men such as Capability Brown and Humphry Repton to landscape them, they filled the space with every kind of fantastic “eye-catcher” (as garden buildings were called): temples and shrines, grottos and fake ruins. Plants were the least of it. Proprietors seem to have indulged a kind of semiotic frenzy, so historians can discover in their garden whimsies the pleasures and fantasies of rich Georgians.

Kate Felus has written an enjoyable anecdotal account of what Georgians did in their gardens: walking, sailing (if you had a decent lake), fishing, reading, enjoying amorous assignations, playing with telescopes and camera obscuras, practising archery, singing, dressing up or, if your grounds were spacious enough, just circling around in your carriage. Really, this is a book about the history of pleasure. Its arrangement is not, therefore, conventionally chronological. Instead it is divided into times of day (Morning, Afternoon, Evening, Night-Time) and the pleasures that belonged to these.

In their gardens – or those of rich friends – the British found themselves liberated to play new parts. Gardens became a “modern” kind of space, claimed Horace Walpole, when garden designer William Kent saw the potential of the ha-ha, a sunken wall that made the border between garden and countryside invisible: “He leaped the fence, and saw that all nature was a garden.” The owners of country houses pulled country activities into their prospects. The rural ritual of haymaking became particularly attractive, so much so that some genteel ladies began to participate in it. In an attempt to allure the young and still unmarried George III, Lady Sarah Lennox, dressed in suitable rustic garb, used to toss the hay in a field next to the road where he took his daily ride. Eighteenth-century paintings show the nouveau riche angling en famille in their carefully stocked lakes and ponds, or simply posing, rod in hand, in group paintings known as conversation pieces. Really wealthy landowners built pavilions over the water, from which they and their guests could fish.

Gardens were for showing off. Some aristocrats staged naumachia – mock naval battles – on their rural lakes, a fashionable fad in an age of British naval victories. Others set up gun batteries and forts around theirs, re-enacting the triumphs of British arms over the French in the seven years war. Some royals led fashion. George II’s wife, Queen Caroline, hired Kent to design buildings for her garden at Richmond. First there was the Hermitage, a kind of artificial cave for contemplation (though with a fireplace), then there was Merlin’s Cave, a gothic confection with steep thatched roofs where the queen kept her books and sometimes slept.

Much of the evidence that Felus uses was designed to celebrate garden pleasures en plein air. The book is cleverly illustrated with some of the paintings and drawings that depict Georgian ladies and gentlemen relishing the landscaped English outdoors. Of course, most of these pictures were paid for by the garden owners and were supposed to celebrate the pleasures they sponsored. In some catty extracts from Walpole’s letters we hear that some of the alfresco entertainments laid on by his aristocratic friends at Stowe were endured rather than enjoyed by guests past the first flush of youth, wrapped in cloaks and greatcoats against the evening chill.

Felus would like to find that louche garden lovers of the period imitated Mr B in Samuel Richardson’s 1740 novel Pamela, when he draws his teenage maidservant into a well-furnished summerhouse in order to seduce her. Lord Cobham’s Temple of Venus at Stowe, another Kent creation, had a concealed entrance and was furnished with “luxurious couches” and decorated with murals depicting lubricious episodes from Edmund Spenser’s The Faerie Queene. Something must have gone on there. Less speculatively, Felus offers a couple of late 18th-century divorce proceedings in which servants testified to the trysts enjoyed by their employers in summerhouses.

She promises that we will learn about the genteel classes as well as the aristocracy, but many of the garden pleasures that went in and out of fashion in the period were evidently for the wealthy only. In the 1720s, Dr George Cheyne – eventually to be Britain’s leading expert on the treatment of “hypochondria” among the well-to-do – prescribed regular immersion in cold plunge pools. Those with large estates and plenty of cash built such pools, fed by springs. The banker Henry Hoare described a summer’s day “souse” in his cold pool in a purpose-built grotto at Stourhead as “Asiatick luxury”. We meet plenty of aristocrats consuming venison pasties in specially constructed cottages or rustic halls on their estates, but few bourgeois picnickers.

A few created their gardens on a limited budget. The naturalist Gilbert White at Selborne in Hampshire installed “statues” in the fields adjoining his limited grounds that were made of painted board. He had a Heath Robinson Hermitage made of woven branches on the top of a nearby hill. There he drank tea or sang with friends, and occasionally had his brother dress up as a hermit, with his guests garbed as shepherds and shepherdesses. The poet William Shenstone made his Shropshire gardens famous with modest expenditure, partly via his poetry. He became skilled in the building of “root houses”, garden buildings made out of materials that grew in the garden (latterday versions are hawked at the Chelsea Flower Show to this day). The idea that a garden was the best place to read and write took hold. In the late 18th century, William Cowper had a tiny summerhouse in the garden of his Buckinghamshire home where he wrote his poetry.

The age of Cowper seems a long way from that of Alexander Pope, penning his verse about the landscaped idylls of his aristocratic friends as recompense for staying in their houses. Felus has paid a price for her generous historical sweep, where “Georgian” ranges from the accession of George I in 1714 to the death of William IV in 1837: we sometimes lose a sense of what changed during more than a century. Yet one theme is consistent: nowadays the landscaped Georgian garden looks like an elegant and decorous arrangement, but originally it was the theatre for plenty of eccentric behaviour. Their gardens were where the British could go a little mad.

• The Secret Life of the Georgian Garden is published by IB Tauris. To order a copy for £14.75 (RRP £17.99) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.