Traffic is fierce. Pause, turn to examine the wayside attractions, and you risk a major shunt: the benefactor-branded culture malls of the National Gallery. Trophy art racked like quotations made familiar by repetition; improved memories of themselves, prompts for postcards. Years ago, as a film student, living in London for the first time, I haunted this place; soft-footed, respectful of its traditions, grateful to be let in. At different times, certain paintings - Seurat's Une Baignade, Asnières, The Miraculous Draught of Fishes by Rubens, Saint Jerome in a Rocky Landscape (ascribed to Joachim Patenier) - have proved my markers. Window-shaped panels in the noise and movement of London by which I could orientate myself, stop the world.

The ascent, from the business of the town, by steps, through pillars, to the colour-coded labyrinth of the interior, is less of a transition than it once was. Now the galleries are themselves boulevards, loud, active, policed. But it's still possible to wonder at the courage and determination required of the 10-year-old Leon Kossoff to find his way here from Hackney in 1936. To begin a lifelong engagement with high art and image-making. The journey, for someone with no experience of original paintings, is a kind of pilgrimage, an initiation. Looking up, in every sense, at the works of Rembrandt, Veronese, Velázquez, was an equivalent risk to that of confronting a building like Nicholas Hawksmoor's Christ Church, Spitalfields, for the first time. Colin Wiggins, curator of the forthcoming National Gallery exhibition Leon Kossoff: Drawing from Painting, speaks of the artist, investigating form and function by making serial versions, as "working against time".

But trying to get close to the current Velázquez holdings is trying indeed: a booming explainer is instructing an adult party in the points of interest in Venus and Cupid, how the luscious curve of that shaded bottom is reprised in the plump crease of Cupid's knee. The panorama of the royal boar hunt is obscured by the easel of a sanctioned copyist who is making an accurate but pointless duplicate of Don Adrián Pulido Pareja by Juan del Mazo. The elements are there, without the spirit; this permitted violation is all respect and no theft. This low-level reproduction is not what Kossoff does. Or ever did. Particular paintings, they could have been here for ever, possessed him: he had to reconstruct them, carry them home.

On a damp February morning, I returned to the National Gallery for a preview of the drawings Kossoff has made from works that belong in the great collections. I was to have met the artist, to undertake a short walk, in his company, down to the Embankment. Kossoff, so Colin Wiggins tells me, is a London scholar, immersed in necessary texts, a lover of Blake. But the painter has to cancel, he's unwell, confined to bed. The exposure of this National Gallery exhibition, selected from many thousands of drawings kept in his studio, is an anxious moment, subject to many revisions and hesitations. The difficulty, of showing private things up against paintings that have challenged his practice and absorbed his concentration for so many years, is extreme.

"He comes here at seven in the morning," Wiggins reported. "What time that entails getting up, heaven knows. He travels on the tube. He'll be left alone with the pictures, having specified the rooms to which he wants access." Carrying a large-scale sketchbook, a drawing board, the artist enters into a peculiar intimacy with the source work. His charcoal drawings, now framed, set against the wall, are marvels of intensity, internal weather. Reimagined, they appear to anticipate the originals to which they pay homage. A theatre of shapes, naked human bodies, arcane rituals, defining the space that contains them.

Kossoff's expeditions to the National Gallery, across London, up those steps, began with the memory of that childhood pilgrimage and continued into his student days: he was a constant visitor, a supplicant with an agenda. In the days when the painter, unrecognised, worked within public opening hours, his enthusiasm sometimes alarmed the security staff. "On more than one occasion," Wiggins said, "the porters intervened. When they see a man with a metal spike - Leon's dry-point tool - bearing down on a Rubens, they are trained to get hold of him. And charcoal makes a mess. We don't allow people to draw with charcoal in the galleries."

The drawings gathered for this new show demonstrate not only unsuspected affinities between artists, romantics and formalists, but the long history of how Kossoff educated himself in his affections. In the background of works by Poussin, such as The Rape of the Sabines and The Destruction and Sack of the Temple of Jerusalem, are pillared public buildings - which become metaphors for the architecture of Kossoff's London. Precursors of the formal grandeur of the National Gallery (which the boy from Hackney can penetrate) and Hawksmoor's Christ Church (which remains a forbidding presence, never to be entered).

As I wait in the room where the drawings have been brought, before they are carried upstairs to the exhibition space, the big Christ Church canvas is unwrapped. Protective sheets, like the coverings that flap against buildings under construction, are cut away to reveal a late masterwork, an epic occasion of paint and factored light and mass. Christ Church is a pivot, an overwhelming argument of elements rearing above the viewer. The work is made from on-site drawings, because every time Kossoff contemplates Hawksmoor's church is the first time. He is undertaking a form of parallel construction, rebuilding in pigment, against the unstoppable tide of development, the encroachment of capital, the icy facades of Bishopsgate and the City. Aestheticised money. But Kossoff refuses to be limited by the rhetoric of protest and sentimentalised heritage: of place or racial allegiance. He is a Jew painting a Christian temple, but he does not see himself as being, primarily, a painter of exile and loss. He celebrates. Christ Church, that monster of ambition and achieved conceit, is democratised in this seeing: a thing that is itself and nothing else.

It has been said, and Kossoff is happy to confirm it, that his subject is the city, London. London in a period of revision and redevelopment, more furious and impatient than at any time since the beginnings of the railways. Building sites, excavations, war damage: Kossoff's close attention confirms the reproduction of landscape as a form of portraiture. He grew up in Bethnal Green, Whitechapel and Hackney - his family worked as bakers - in territory where movement, tenement room to terraced house, to semi-detached suburban villa, was mandatory. Many of the largely Jewish population felt themselves more closely aligned, as the writer Emanuel Litvinoff has remarked, with eastern Europe than with the West End of London.

London consists of areas of recession (life behind closed doors, lace curtains) and portals that draw the eye into the shadows of forbidden buildings. When Gilbert & George are photographed for a newspaper portrait by Herbie Knott, they are seated on the roof terrace of their Fournier Street house, alongside two nudging chimney pots. The tower of Christ Church is an accidental backdrop. Public and private have become so confused that there is no real distinction between them. Whispers of undisclosed selves are a Jekyll and Hyde game of secret corridors, covert entrances. Profiles, commissioned to promote major exhibitions, stress the astonishing fact that Gilbert & George have a life in the city. They walk, they ride on buses, they visit tailors. This is as important as the artefacts they contrive. And all the time their own collections grow, the objects in cupboards, the files, the archive. London is revealed through its signs and symbols: taxonomies of bird shit, piss puddles, graffiti (before the graffiti became the art). The art of Gilbert & George, scavenged from the performance of an accidental biography, aspires to the condition of the stained-glass window. Christ Church might feature, in one of the panels, but only as a confirmation of the beatitude of the men in matching suits, self-ordained saints of the city.

Kossoff's return to Spitalfields was of another order. He had lived here. His family had worked in these streets. The church was part of the topography of childhood, an anchor to memory. He begins a series of paintings that have been compared, in stature and achievement, with Monet's versions of Rouen cathedral. The favoured low-angle viewpoint, it has been suggested, represents a child's-eye view, a kind of arthritic nostalgia. It's nothing of the sort. It is the awe an artist feels, the recognition, of a structure that appears, illusion stacked on illusion, to have defeated time. The pyramidical steeple is foreshortened. The elegance of the head-on prospect, so beloved of developers that it is retained, as something glimpsed through a roof panel in the Milanese mall that replaces the old Spitalfields market, is never part of Kossoff's remit. The wonder of Christ Church is how so many contrary elements are held in balance. The way this vertiginous mass, with its curves, its sharp Egyptian angles, floats. The architectural writer Pierre de la Ruffinière Du Prey has a convincing explanation of the effect. Christ Church is a miracle of stacking: from the paganism of the Tuscan columns, there is a chronological ascent, through historical periods and systems of belief, to the steeple with its flamelike crockets.

Kossoff puts his church under enormous pressure; it folds back into discriminations of sticky grey and pearl-white. Paint alchemised from pigeon droppings, egg shells, lost libraries and London afternoons when the sun never breaks through the cloud mantle. The mass of this amputated form, with its Portland stone blocks, is a sanctified space that Kossoff chooses not to enter. The child viewer, still alive in the elderly man, stands on the pavement: acknowledging the power of place through the scuttling presence of lowlife characters, vagrants, alcoholics. The invisibles who choose to remain within the force field of a structure that resists the blandishments of present time.

The National Gallery is another forbidding edifice. But now the painter, status assured, is permitted, even encouraged, to explore its interior spaces. The breathtaking accumulation of treasures. So begins the "slow emergence" of what Wiggins calls a "long and deep familiarity". The process of copying, re-making, of drawing from originals, is an important aspect of Kossoff's practice. The thirst for intimacy with masterworks, bunched together, curated, offered to a discriminating public, is almost erotic. The versions can be carried home, as warped prompts, to form a secondary exhibition: as they are arranged in the painter's studio. He can observe, without pressure, how these new-old things react, one against the next. Everything is in flux, with the drawings as a record, a significant catalogue of influences. Everything begins with the memory of that first confrontation, in the National Gallery, with Rembrandt's A Woman Bathing in a Stream

Status assured, honoured at the Venice Biennale, Kossoff is given access to the galleries at times when the general public are excluded. This, I feel, is one of the privileges of the city. It is part of the mythology of a figure such as Lucian Freud, that he is able to stalk the National Gallery, a skeleton security staff for company; the distant hum of floor-waxing machines and vacuum cleaners. Critics, promoters and sponsors get their turn, mingling with their own kind, at allocated hours - before the ticket-purchasers are let in. But it is only the established artists of the moment who are allowed, as it suits them, to make their pact with images that challenge and inspire every atom of their vision.

But Kossoff's reimagined Spanish dwarfs are not the dwarfs of Velázquez; the Spanishness has been leached out, the chill of pride, the naked confrontation, the cruelty. Kossoff's figures bulk the frame, dispense with landscape. The respect with which the London artist makes his approach is activated by his ability to recognise what can be adapted, subverted. Supposed classicists such as Poussin, with his immaculate apparatus of design and reference, can be granted the physical weight and incipient violence of early Cézanne. By observation, close study, Kossoff rescues certain paintings from the fix of time: they are made new, in a furious thatch of disputing marks and lines. In a culture of duplicates, calendars, mugs, digitised reproductions, Kossoff's drawings are always fresh and inspirational.

During the second world war, many of the paintings from the National Gallery were removed for safekeeping to the slate quarries of Blaenau Ffestiniog. But some of the greatest treasures, one at a time, were returned to London. To take their chances like the rest of us. A single masterwork could stand for all the others and sustain a wearied and battered population. Kossoff's Christ Church, in this exhibition, is of that order. A large truth in a mendacious era. When Kossoff saw the work, placed in the gallery, his knees buckled. His earlier hesitations about the show were set aside. "It's all right," he said.

· Leon Kossoff: Drawing from Painting is at the National Gallery, London WC2, from Wednesday until July 1. Details: 020-7747 2885