Opinion: in response to Joseph Rykwert's Royal Gold Medal lecture this week, where the critic stressed that all design has political implications, Kieran Long rejects Zaha Hadid's assertion that architects have "nothing to do with the workers" who die on construction sites.

"Did I ever tell you about the time I met Walter Gropius?" Joseph Rykwert is leaning across a table in Daquise, the Polish restaurant in South Kensington he has been frequenting for nearly 50 years, a sparkle in his eye. "I was at the Royal Academy, and on the landing halfway up the stairs I saw Jane Drew, who I knew quite well, and Gropius, who I knew from photographs," he says. "I walked up to them, and Jane Drew said: 'Professor Gropius, this is Joseph Rykwert. Joseph, go and find Professor Gropius a taxi.'"

Rykwert gave me this finely turned anecdote on Saturday, at dinner after a symposium about his work at the Victoria & Albert Museum. This week in London has been a festival dedicated to the venerable architectural historian, focussed around the Mardi Gras of Tuesday's award of the Royal Gold Medal by the RIBA.

Rykwert for me, although I have met him many times and was his editor at the Architects' Journal, is a legend, a different order of individual to most other writers and certainly to myself. The Gropius story shows how he links us with the first generation of modernist masters, but he is also someone whose work (and that of his prominent students like David Leatherbarrow, Mohsen Mostafavi and Alberto Perez-Gomez, not to mention legion younger protégés) has tangible and I think growing influence in contemporary architecture. He is one of the few historians whose works are routinely assigned by teachers in architecture schools and all of us, surely, aspire to his literate, balanced prose.

The Gropius story also shows Rykwert's appetite for gossip, and for the almost implausibly perfect story. Despite some claims to the contrary that I've heard in the last few days, I think Rykwert would like Dezeen and the writing found around the web, and I'm positive he'd be writing in these forums if he were beginning his career today.

The thing about giving the gold Medal to any critic, and especially one as widely read and respected as Rykwert, is that his opinions are unmistakably available to the rest of the world. It's not really a question of convenient interpretations allowing generic and polite appreciation. You either agree with Rykwert's words, or you don't. You either believe, for instance, the idea that the plan of a Roman city had mythic origins giving each citizen a sense of their place in the cosmos, or you think he's wrong and it was all about troop movements. And you can either deal with the implications of that insight, or ignore them.

Norman Foster could plausibly say of an architect and fellow Gold Medal winner like Alvaro Siza that he has the deepest respect for his work etc, without really having to face the question of their diametrically opposed views of what architecture is and how human beings find a place in the world. Perhaps all buildings are themselves ambiguous enough that we can elide even fundamental differences (with the possible exception of work by the progeny of the Prince of Wales school of architecture, or narcissistic and vocal numbskulls like Wolf Prix). Flattening difference in architecture probably results from a profession keen to avoid conflict within: the idea that one shouldn't criticise a fellow professional.

In the big, if-not-exactly-happy-then-mutually-uncritical family of the profession that the Royal Institute of British Architects tries to bring together, it would have been interesting to know, for instance, what Dominique Perrault might have thought of Rykwert winning the prize (Rykwert says of the Bibliotheque Nationale de France in The Seduction of Place: "Demoralised by disaffection and labour problems, by inept book stack towers and disgraced by sterile, unhappy public spaces – both within and without – it seems a perfect candidate for a revised edition of Peter Hall's Great Planning Disasters."), or, say, any developer or architect involved in London's Docklands (which Rykwert calls "socially confused" amongst other things in the book already cited).

A true critic is the unwelcome guest at a party of architects. Rykwert himself found that out the hard way when he began teaching architectural history at the University of Essex in 1967. What Eric Parry in the Gold Medal citation called the "architectural authorities" (in fact the RIBA itself) tried to close down this course because it stood outside their approved version of how architectural history should be taught and who might be permitted to learn it.

This is why Rykwert's Gold Medal lecture on Monday night, which some found tortuous, was so important. He began by describing his work as a designer, and followed that with accounts of the three architects who turned down the RIBA Royal Gold Medal (Richard Norman Shaw, John Ruskin, who turned it down twice, and William Richard Lethaby) embroidering it with anecdotes so detailed that at one point I thought Rykwert was about to say no to it himself. His theme was the age-old problem of whether architecture was an art or a profession.

For Rykwert, this is a non problem - a false dichotomy. Some audience members at Monday's lecture tried to ask him about where politics stands in relation to architecture, but Rykwert's work stands for the idea that every act of design, or writing, is political. Design is a set of ethical commitments or reticences. We cannot absolve ourselves of responsibility for the full range of implications of a project - as Zaha Hadid tried to week - any more than we can choose just to breathe the oxygen in the atmosphere, but not the nitrogen and carbon dioxide. Rykwert has earned a place alongside Ruskin and others because of the clarity of his commitment to an adulterated but rich and meaningful view of architecture.

Kieran Long is senior curator of contemporary architecture, design and digital at the Victoria & Albert Museum. He presents Restoration Home and the series The £100,000 House for the BBC, and is currently the architecture critic for the Evening Standard newspaper.