On the way to The Lost Lending Library, a new immersive family theatre show, I describe Punchdrunk’s The Drowned Man to my eight-year-old, Hilda, in a bid to explain the “immersive” bit. One floor of a huge building was covered with sand, I say, and at one point I had to wash down a stranger with an old rag. “Ew!” she grimaces.

This show by Punchdrunk Enrichment, the company’s charitable offshoot that specialises in educational and community work, is altogether more serene. A dozen or so kids gather in an oval room at Coventry’s recently refurbished Drapers’ Hall, across the road from the fantastic Herbert Art Gallery, which has lent some props to inspire our ideas.

With its pair of curved doors and a gong in front of the fireplace, this is the first of three characterful rooms used in a show that combines the history of the building with flights of fancy. It’s site-specific but could perhaps be a tad more city-specific; although presented for Coventry’s UK city of culture year it lacks a strong local flavour.

Alice Devlin plays Petra, our guide, who confides in and conspires with the children as they quickly get into the spirit of it all. What is a drapers’ hall, she asks. “A school for dragons,” comes one quick reply. Petra fills our heads with real accounts of the building’s past as a court of law, a site for cloth sales and its role as an air raid shelter – all facets that will be explored later.

The plan is to sneak from this room into a secretive library and Petra calls on Hilda to poke her head out of the door and check the coast is clear. We take two books with us: a battered copy of Roald Dahl’s The Witches (I’m still not sure why) and a locked, dusty tome that helps the kids gain access to a world behind a bookcase.

Then comes what you could call the Punchdrunk effect. We enter a treasure trove for book-lovers, a room filled with typewriters and maps, seats built from stacks of hardbacks and walls lined with an A-Z of authors including local hero George Eliot. We sit beneath Maya Angelou and meet our new host, librarian Peabody (played by Barry Fitzgerald). Like Devlin, he has a mischievous air and deftly handles some leftfield questions from these curious kids.

The show is designed for kids aged six to 11 but I confess I lose the plot when Peabody tells us a story (so many distractions!). Still, Hilda is rapt. We’re sent out to gather around a stately table to come up with new stories to fill Peabody’s empty shelf. There’s a knack to handling these sorts of group collaborations and Devlin carefully involves everyone to dream up a tale about a magic wand stolen by a queen (Hilda’s called upon to provide her name: Wanda).

This session, like the oval-room prologue, is casually freeform but it peters out with a bit of confusion about what happens next rather than clearly signposting The End. An object-driven show feels like it needs us to create something tangible to return to Peabody’s shelf. When it turns out we write a story at home and share it online with a hashtag it’s a letdown; the 50 minutes are up and we’d probably all stay with the show for longer. But the kids clutch the membership cards they’ve been given – “Can I use this in a real library?” queries one – and stroll out into the daylight, primed to turn the city into their own story.

• At Drapers’ Hall, Coventry, until 6 November