Opinion: when the Tour Montparnasse was built in Paris, its residents were so appalled they banned all high-rises. Londoners should do something similar in the wake of Rafael Viñoly's walkie talkie, says Owen Hatherley.

What do the Tour Montparnasse in Paris, designed and built in the early 1970s by a large, anonymous committee headed by Eugene Beaudouin, and 20 Fenchurch Street – also known as the Walkie Talkie – by (and very much by) Rafael Viñoly have in common?

The most obvious answer would be "they're both massive, horrible and unavoidable", but they're also cousins in causing people to wish for their destruction almost immediately after construction. The Tour Montparnasse's moody glass shaft was the spur for a mid-1970s ban on anything taller than seven storeys in Paris; this month, a competition for renovation was announced which made clear the aim was to make the building look as little like itself as possible. 20 Fenchurch Street, meanwhile, has just had the honour of receiving the annual Carbuncle Cup for the worst British building, and is so widely loathed that many must surely be hoping it leads to legislation banning London skyscrapers named after domestic objects.

What both of them make perfectly clear is just how dangerous it is to take lightly the task of designing something almost every dweller in a city is going to look at, every day. Both designs were obviously half-hearted and careless, in their very different ways. Some will always hate anything big and modern, but these two have provoked intensities of loathing that go way beyond normal grumbling. So what to do with them?

In its rather dull, corporate way, the Tour Montparnasse is not a completely awful building. There are worse towers in Paris, but they're put into the just-out-of-"Paris" skyscraper zone in La Defense. It's smooth, sleek, coherent, uncompromising, and by now has acquired a certain patina, a whiff of Alphaville nostalgia, standing as an emblem of paranoid, Gaullist technocratic Modernism.

But it was dated already when it was finished and opened in 1973 – it looks as if it's from at least 10 years earlier, a product of a sober corporate CIAM functionalism that had long since been superseded and denounced by Brutalism, incipient High-Tech and Postmodernism. The real reason why people hate it, of course, is where it is. Although there's a little cluster of smaller high-rises on the Seine, it's still by far the tallest building within the Paris peripherique, and stands out from its surroundings with extreme aloofness. Every possible view of it is of a single smoked-glass erection rising out of the limestone city, an irrelevant intrusion into its skyline of domes, towers and mansard roofs. It's unlike everything around it in its materials, size, aesthetic, shape.

A building can stand out this much if it does a lot of work to make itself worthwhile; think here of the bizarre, biomorphic Zizkov Tower in Prague, or indeed the Eiffel Tower itself. Defenders of crap tall buildings like to point out how loathed the Tour Eiffel was for its first two decades, as if that exonerates others that have also been hated; but after two decades, that tower was passionately loved by Paris's Futurists, Cubists and tourist guides. The Tour Montparnasse, on the other hand, hasn't acquired many defenders in its 40-year existence.

The story of 20 Fenchurch Street is a very different tale of woe. In terms of urban manners, the building should have been less controversial. At least since the fateful decision was made in the early 1960s to allow high-rises to complement St Paul's Cathedral on the City of London's skyline, immediately obliterating the play of parish church spires that Christopher Wren had assembled around it, the City has been a skyscraper district. A "viewing corridor" means they can't get too close to the St Paul's dome (the grimly sprawling "ground scrapers" around it are the trade-off), but that's about it.

Most of the resultant towers are surprisingly intelligent, from the residential Brutalism of the Barbican to Richard Seifert's metallic NatWest Tower, to Foster and Rogers' more convincing essays in domestic-object-skyscrapers, the Gherkin and Cheesegrater. So it is odd that just another tower caused so much fuss. It is just outside the City's main high-rise cluster, but not as far away as Renzo Piano's Shard, which, although taller, has caused less disgust.

The answer lies with the curious architectural talents of Rafael Viñoly, and his ability to use massive budgets and advanced engineering to create overbearing, whimsical nonsenses. Opponents of the building have often pointed out the way that in views from, say, the Design Museum, the tower's top-heavy bulk, resembling the silhouette of a drunk bending over to vomit, sits precisely between the two neo-Gothic pylons of Tower Bridge. It would be surprising, given London's fixation on "viewing corridors", if Viñoly was not aware of this possibility. He probably really thought he was improving it.

So what to do about these unfortunate towers? Worryingly, the competition for altering the Tour Montparnasse seems to be deliberately encouraging something "whimsical" and "eye-catching", although you'd have hoped that Viñoly's work at the other end of the Eurostar was great evidence of the dangers of such an approach.

20 Fenchurch Street, on the other hand, has already had to be substantially altered, because the glass on its overhanging cap was creating a "death-ray" effect, credited with melting some unfortunate stockbroker's Jaguar. The semi-public "sky garden", which was a condition of planning permission, is apparently to be redesigned after being widely lambasted as an utterly pathetic collection of shrubbery. But the awfulness of the building is completely inherent and unavoidable – while you can probably hang all sorts of things from the Tour Montparnasse's subtly curved frame, should you want to, the frame of Viñoly's tower is itself the problem, the totally arbitrary, goofy and shoulder-padded bulk of the thing.

Apart from being reopened as a Museum of Finance Capitalism, where animatronic brokers could yell at each other through concealed microphones and trading floors could be repurposed to exhibit the suffering of foreclosed home owners and "decanted" council tenants, it's hard to imagine any possible fate for it. Londoners will just have to put up with 20 Fenchurch Street, and they'll have to create some sort of instrument – legal or otherwise – to stop it from happening again.

Owen Hatherley is a critic and author, focusing on architecture, politics and culture. His books include Militant Modernism (2009), A Guide to the New Ruins of Great Britain (2010), and A New Kind of Bleak: Journeys Through urban Britain (2012).