How Pinterest’s Assistant brings Visual AI to Shopping

Visual discovery platforms are increasingly deploying AI systems that interpret user behaviour to generate product recommendations, moving beyond keyword-based search towards conversational interfaces that process both text and images.

Pinterest, the San Francisco-based visual discovery platform, has introduced Pinterest Assistant, a tool that uses visual language models to generate product recommendations based on individual user preferences and browsing behaviour. 

The system analyses content saved to user boards, collages and historical activity, alongside data from users with comparable preferences, to respond to voice or text queries.

Bill Ready, CEO of Pinterest, says the platform appeals to younger users through its ability to understand personal style: “People, especially Gen Z, say that the magic of Pinterest is that it ‘just gets me’, whether that’s finding the perfect outfit or knowing your distinct style,” he says.

“With Pinterest Assistant, we’re supercharging that magic by leveraging AI to help our users discover and shop like they would with that person who knows them best.”

How visual language models combine text and image processing

Visual language models process text prompts against visual information by training on relationships between language and images. 

These models contain two components: a vision encoder that extracts visual elements including colour, shape and texture from images and a language encoder that maps semantic and contextual meaning of words for natural language processing.

Pinterest has built its assistant on this architecture to interpret queries that combine visual and textual elements. 

The system accesses what the company terms its Taste-graph, a proprietary data structure that maps individual user preferences across the platform’s catalogue. 

This allows the system to track how preferences shift over time and adjust recommendations accordingly.

Matt Madrigal, Chief Technology Officer at Pinterest, says the approach differs from keyword-based search systems: “Unlike traditional search, which relies on keywords and scrolling through results, Pinterest Assistant is designed for open-ended exploration,” he says.

“You simply start a conversation – ask about a vibe, style, colour or even show an image. 

“It draws from your boards, saves and Pinterest’s vast catalogue to deliver recommendations that fit your unique taste in real time.”

Why Pinterest claims 30% improvement in recommendation relevance

The company states that its multimodal system produces shopping recommendations with relevance scores that exceed traditional models by more than 30%. 

Pinterest attributes this performance to the integration of user-specific data through the Taste-graph, which tracks preferences and adjusts recommendations as user behaviour changes.

The assistant accepts queries in conversational formats, processing combinations of text, voice and images to interpret user intent. 

Matt says the technology represents a development in how users interact with visual search platforms.

“What sets Pinterest Assistant apart is our multi-modal AI and proprietary taste-graph, built for smarter, more intuitive AI,” he says.

“It can understand complex, conversational queries that blend text, visuals and even voice, making it possible to capture intent in natural, human ways.”

The company has integrated the assistant into its core product offering as part of its strategy to combine social media functionality with commerce capabilities. 

Pinterest generates revenue through advertising and partnerships with retailers whose products appear in search results and recommendations.

Matt says the assistant represents an initial phase in the company’s work on visual reasoning systems.

“This new assistant is just the beginning of our journey toward a future of AI-powered, visual reasoning,” he says.