It would be better to pull down some of London's Georgian terraces than demolish Amin Taha's stone-framed housing block, says Sean Griffiths.

Amin Taha's controversial building at 15 Clerkenwell Close is still under threat of demolition, despite the planning documentation apparently now being in order. According to the council, it is "rough, ugly and detrimental to the conservation area and listed buildings", so therefore in contravention of its planning permission.

As an admirer of the building's brave exterior, I have been rooting for it to escape intact from its confrontation with the Heritage Taliban. So I was disappointed to read a recent interview with Taha in which he undermined the design by indulging in the sentimental adage that architects are story-tellers.

Having confronted us with a compelling vision of modernity, in the form of a building whose power surely lies in the saturated presence of its object-hood, he retreats to a cosy world of "story-telling". This nod to folksy comfort belies the aesthetic impact his building so successfully projects. I feel compelled to ask, what stories does this building wish to tell?

Its tormented frame conjures not a narrative, but a geometry of fragmented masonry of the sort one might find in a JG Ballard novel. Or perhaps it evokes the precarious equilibrium that marks the pocked and twisted ruins of bombed cities.

On the other hand, the semi-exhumed fossils that encrust the frame index a non-human entropy, manifested as an instant in time that is geological, not human in scale. Perhaps most powerfully, the building seems to be encoded with the immanent premonition of the very destruction that its enemies seek. None of this imagery has much to do with comforting, anthropocentric chronologies comprising neat beginnings, middles and ends.

Buildings are not stories any more than novels are sculptures

If architectural stories belong to a withered and bankrupt postmodernism, disengaged from objective reality and floundering pathetically in our "post-truth" world, 15 Clerkenwell Close's greatest strength surely, lies in its offering a resistance to that world, stepping back from the fashionable obsession with "narrative" and towards a neo-realism.

Buildings are not stories any more than novels are sculptures. In fact, it is precisely because they are ill-suited to the role of truth-telling, that the concept of architectural story-telling is so lacking. For while architects indulge in bouts of pseudo-poetic mystification, the real histories from which buildings emerge are too often disguised by elegant aesthetics and arcane obfuscation.

This is not necessarily the case with all cultural artefacts, as is demonstrated by a number of examples that, like Taha's creation, are threatened with destruction.

For instance, no injunction to the poetic is likely to derail campaigns seeking to remove monuments of infamous imperialists like Cecil Rhodes, whose likenesses in stone and bronze adorn colleges in Oxford and the university of Cape Town. In the United States, violence resulting in the murder of one protester, by a white supremacist during a demonstration in Virginia, centred around the attempts of anti-racism campaigners to have a statue of the confederate war hero Robert E Lee removed from a park in Charlottesville.

Heritage reveals itself in such cases to be not so unequivocally charming. And yet, ironically, it is in the name of the architectural cousin of just this sort of heritage that the proposed destruction of Clerkenwell Close is justified.

Heritage is now a curse rather than a blessing for many places

Contra the abominable statues, no one demands the destruction of the Georgian terraces of British cities whose very streets are named after the slave owners who built them, using the ill-gotten gains of their barbarous trade. I hear no calls for the pulling down of country houses erected on lands attained through the violent eviction of those for whom it was their only means of sustenance. And any threat to the Victorian inheritance begat on the backs of exploited labour and colonial subjects is given short shrift.

The white-washing practiced by the conservation industry has been one facilitator of its enormous success, which has resulted in heritage infecting every aspect of our lives. It festers in the midst of the Brexit debate. It is now a curse rather than a blessing for many places whose inhabitants' lives are overrun by a mass tourism, the perpetrators of which seek an authenticity that they, themselves undermine.

For architects, it increasingly means having to negotiate, not so much history's mystical palimpsests, but an onerous legislative patchwork of conservations areas. The latter protect buildings of often-dubious merit on the false premise that they "tell our story" and for reasons that often have more to do with the maintenance of property values than the protection of buildings of architectural worth. They do so at the cost of new and urgent architectural ideas.

Readers will not need reminding that we are experiencing immense technological and social change. Our emergencies include climate change and a housing shortage – problems that call for imaginative thinking about our building stock.

Yet talented young architects find even their modest laboratories – the publicly invisible rear extensions to 19th century terraces, buildings ill-suited to the needs of modern living – stalked by conservation officers acting in the interests of curtain-twitching nimbys. So afraid of the future are we, that we cannot imagine allowing behind the polite facades of established streets, the creation of a veritable zoo of architectural experiments reflecting new forms of habitation, made from unfamiliar renewables and rendered in as yet unimagined aesthetics.

The Georgians cared little for the countryside

God forbid that we suggest Georgian houses, with their dubious provenance and leaky, jerry-built envelopes, be extended upwards to accommodate expanding populations or radically retrofitted to meets today's meteorological needs. And as for knocking a few of them down to make way for the new, such is our pathetic obsession with nostalgic and biased visions of our collective history, the mere suggestion of such is likely to have one transported to an island colony for the criminally insane.

And yet it was not always thus.

The Georgians cared little for the countryside or the largely mediaeval city they ruthlessly bricked over to create the urban realm we fetishise today. The Victorians were not averse to a touch of prometheanism, as they carved up cities with sewers and railways. Haussmann's Paris would not exist without a bit of the enthusiasm for tabula rasa that also informed the architecture and planning of the 1960s – perhaps the last time we wholeheartedly embraced the future in architecture. The finest examples of the latter period are now better appreciated and some that are lost are already lamented.

But it was in reaction to the alleged excesses of that period that the conservation lobby first got up its head of steam. And if the scientific notion that any action necessarily provokes an equal and opposite reaction can also be applied to history, then it is my contention that, if the comprehensive redevelopment of the 1960s did indeed go too far, it is the conservation lobby whose wings are now in need of an urgent manicure.

Perhaps, if we are to sate the appetite the conservationists seem to show for a touch of destruction when it suits them, we might requisition the words of one of their own, the poet John Betjemen, who famously called for "friendly bombs" to fall on Slough. But instead of pointing the explosives in the direction of Amin Taha's admirable edifice in Clerkenwell, how about we knock down some old stuff instead?

Perhaps we could start with some of the elegant Georgian terraces of London's Bloomsbury. There is, after all, rather a lot of them. Do we really need them all? Apparently not. Or at least, they didn't think so in the 1960s when they demolished a block of Bloomsbury and built, in its place, a miniature city of the future in the form of a magnificent concrete vessel designed by Patrick Hodgkinson. Today the Brunswick Centre is thriving and Grade-II listed to boot.

So why not do it again? In the spirit, if not the letter, of Betjeman's snooty poem, let's have some "gentle bombs on boring Bloomsbury". It has a ring to it that the late poet laureate might even have admired.