Photo essay: London-based photographer John Short was determined to depict the death of New York's creative scene, but his images of its once-legendary arts venues proved that the spaces are far from "lost" (+ slideshow).

Short set about capturing the forgotten creative spaces in New York City, which he believed to have all but disappeared due to the city's rising rents.

In this essay written for Dezeen, he describes his discovery that many of the city's lofts, bars, studios and clubs had not been lost at all – just renamed or repurposed within the field of arts.

Short has presented his photographs of these spaces, alongside archival images from their heyday, in a book titled Unforgotten New York – Legendary Spaces of the Twentieth-Century Avant-Garde. The volume was designed by graphics studio Praline and is published by Prestel.

Unforgotten New York – Legendary Spaces of the Twentieth-Century Avant-Garde came about from a collaboration between myself, David Brun-Lambert – a writer specialising in the arts and culture – and David Tanguy, a respected graphic designer and founder of Praline.

Brun-Lambert was producing a radio documentary for Swiss National Radio around the idea that New York had lost its cultural importance on the world stage since its economic renaissance and that much of its cultural history was being forgotten. He thought there could be a book in it and I must have agreed, as I was on a plane to rendezvous with him at JFK five days after our first introduction.

So, there we were in the Lower East Side with only the loosest of ideas and less than a week to come up with enough content to sell the concept to a publisher. That first night over a few beers we came up with a stronger idea of what we wanted to say. That changed of course, but first we went about trying to evidence our theory that the New York that had been the hotbed of the arts from the 1950s to the late 1980s was dead. And the reason it was dead was that the city had become too prosperous.

We knew it wasn't a case of everyone in New York was too comfortable that they had nothing to express, but we did think it was a case of there not being enough affordable space left for people to create in. Whether that be to rehearse, create fine art, perform or just congregate.

We wanted to draw attention to the spaces that we thought had allowed New York to flourish through the latter half of the 20th century. By showing them as they were today, we'd show that the city was devoid of places that allowed people the space necessary to evolve.

David had done his research and had a list of 10 or so places we had to access. We just knocked on people's doors and told them what we were doing and asked if we could shoot the inside of their properties. We were slightly astonished that most said: "Sure, come back tomorrow and you can shoot what you like".

That trip we managed to shoot eight iconic spaces and we knew we were onto something special. We'd accessed Jean-Michel Basquiat's last loft, Warhol's Factory II, Jack Kerouc's apartment where he'd written On The Road, Paradise Garage, and the Project in the Bronx where Kool Herc held his Block Parties where he played his first break beats – now considered the birthplace of hip hop.

When we arrived back in London, we went about researching more and contacting some of the more corporate spaces like Studio 54 and CBGBs to arrange shoots for our next trip. As we delved deeper into the research we realised that quite a few of the spaces were actually still in operation – Village Vanguard, El Quijote, Stonewall Inn, St Mark's Church and Judson Hall. Others had changed in name only and were still being used for a similar purpose – Merce Cunningham Dance Company Studio had become the Martha Graham Center of Contemporary Dance, Studio Rivbea was now a fringe theatre and performance space and the Leo Castelli Gallery had become the Michael Werner Gallery.

So, our initial concept was transforming from a book about the forgotten New York to the unforgotten legendary spaces of New York's avant-garde. We also realised that our initial plan of covering spaces in all of New York's five boroughs was maybe too ambitious, so we decided to only focus on Manhattan. And most of what we shot was in the lower half of the island as that's where the cheap rents were, and hence where the artists took spaces, worked, showed and hung out. It was also where later the city's first artist co-operatives were formed and where many older artists still live and work.

The second and third trips went well but with a few more rejections. We interviewed some great people who talked about the New York they'd lived in 20, 40 and even 60 years earlier, and the people who had been around then but were no longer with us. There were some great stories, but what came out of most of the conversations was that the cheap rents allowed them to be experimental in a way younger artists now in the city couldn't, and the nature of the city back then made them all live for the moment a little bit more.

They acknowledged that the avant-garde was not dead in the city, but it was now to be found out in Brooklyn. The images in the book highlight this. Lots of the artist's lofts have become private residences, bars are still bars, theatres still theatres, but the larger clubs have been redeveloped into galleries, supermarkets and branded stores.

In terms of the photography, I shot everything on film on my old Hasselblad for both an aesthetic reason – I like the grain – and because it's much quicker. You commit when shooting film, with digital you're always trying to make things better. This project was never about perfection, it was about capturing a sense of the space in the short window given (often 30 to 60 minutes), and trying to make a visually interesting image in the process.

Everything was as is, I didn't move things around. I treated each shot as an honest portrait of a space as it was. All the spaces were shot empty and I tried to capture a sense of what they were being used for. Some were photogenic and I knew exactly what to shoot as soon as I walked into them. Others where less obvious, but we felt they were just as important in the story of what had happened to the city.

When it came to the design of the book, by David Tanguy and his team at Praline, we wanted to add another level and show what the spaces had been like before without detracting from my imagery. We didn't want to go for a traditional "before and after" shot so we decided on a single archive image of each space being inhabited. We found some great archival shots, which you can see in the book.