Royal College of Art graduate Zhaodi Feng has created an interactive toolkit named Promptac, which allows users to input prompts to AI image generators using hand movements rather than text.

The kit includes a fingertip controller that recognises colours and materials from the real world and small tools that users twist, pinch, press and bend to shape changes in AI-generated digital designs.

Feng created Promptac to bridge the gap between the digital and physical world, aiming to provide a more intuitive creative process when designing with AI. Usually, generative AI models only accept prompts in the form of digital text or images.

Made from foam clay and silicone, the tools in the kit act as a "translator" between the user's actions and AI.

Feng tested the prototype products using AI tools Midjourney and InstructPix2Pix, which she used to generate a basic image of an object by entering a text prompt, such as a vase, mug or plant pot.

This image can then be modified by interacting with the Promptac toolkit.

The fingertip controller is used to single out colours and materials from the real world to apply to the AI-generated object.

The controller contains a colour sensor and a radio frequency identification (RFID) sensor that recognises a palette of RFID-tagged materials, which generates a text prompt to the AI and changes the appearance of the object in the image.

Physical gestures are translated into AI-generated design changes

Similarly with the other Promptac tools, which Feng calls soft sensors, users apply physical gestures that are translated into text prompts to alter the shape of a 3D model representing the AI-generated image.

Feng designed the soft sensors to allow users to modify AI images in ways that imitate crafting things by hand, saying it's "just like playing with clay".

"The twist sensor creates a spiral shape, the pinch sensor forms an irregular change of shape and makes it longer or narrower, the press sensor flattens the shape, and the bend sensor forms a curved or angled shape," she told Dezeen.

Most AI image generation tools require precisely written text describing materials, textures, colours and shapes to generate the desired image.

Feng created Promptac with non-designers in mind, hoping the tools make it easier to communicate design changes without the precise language needed.

"This kit is designed to reduce the reliance on explicit texts in modifying prompts, especially for people who rely more on sensory experiences and hand-object interaction to perceive and express their ideas," she said.

"There is a potential for improving co-design communication efficiency with this kit because it can enable non-designers and clients to naturally communicate desired materials, colours and shapes to designers without a specific vocabulary of design terms."

"I also think Promtac has the potential to benefit people with difficulties or disabilities in text communication and may help promote inclusiveness of technology in the potential future scenario of widespread usage of generative AI tools," Feng added.

Feng began developing Promptac when researching traditional crafts. She was informed by the evolution of design tools and the ways that technological advances change how people create.

She believes that although there are benefits in changing technologies, there has been a detrimental disconnect between our relationship with the physical world.

"Though having the benefits of improved efficiency, we are gradually disconnected from the physical world, distancing ourselves from making and somehow experiencing deskilling," the designer said.

"There might be an opportunity to bring traditional ways of making and the physical sensation of making back to our creative process, and mediate our relationship with technology."

Promptac is currently a prototype, but Feng hopes to develop the product for different types of designs and at larger scales with the help of AI researchers.

While the toolkit currently works by translating movements to text prompts that the AI models can recognise, Feng believes it is possible to create a new AI model that reads the movements directly.

"It's feasible to train a new model that directly translates hand manipulations or other tangible interactions to images," she said.

"The influence of AI on design and creativity depends on how it is designed to be used," Feng added. "We need a better touchpoint to interact with AI and let technology help creativity flow."

Other projects designed with AI include a foldable sofa by Space10 that recognised AI as a "creative collaborator" and a project that used AI to turn images of crumpled paper into architectural models in the style of famous architects.

The photography and videography is by Zhaodi Feng.

AItopia

This article is part of Dezeen's AItopia series, which explores the impact of artificial intelligence (AI) on design, architecture and humanity, both now and in the future.