Millions are now crafting their own AI companions, yet most lack understanding of how these creations will behave. MIT Media Lab’s Assistant Professor Pat Pataranutaporn and graduate students Anthony Baez and Sheer Karny have introduced “neural transparency,” a tool allowing users to see inside an AI’s neural network before interaction. This research is being presented at the ACM Conference on Intelligent User Interfaces. In an interview, Pataranutaporn discusses their findings, the underestimated risks by users, and the potential of truly transparent AI in the future.
Pat Pataranutaporn explains that people are creating personalized AI chatbots using large language models, transforming them into various roles through simple prompts. However, users often do not understand how these prompts affect the AI’s behavior until they interact with it. “Neural transparency” is designed to change this by revealing internal patterns within the AI’s neural network, offering insights into potential behavior before the AI speaks. The research combines human-AI interaction and mechanistic interpretability to make these patterns accessible to users.
The method involves selecting behaviors of interest, such as empathy or honesty, and comparing the model’s internal activations for those traits. This comparison forms a “behavior direction” within the model. Users can then see a visualization, like a sunburst diagram, that previews the AI’s likely traits. The focus is on the design phase to prevent issues before they occur, moving from reactive correction to anticipatory design.
The study found that people often misjudge their AI’s behavior, overestimating positive traits and underestimating harmful ones like sycophancy. This highlights a blind spot in designing AI companions. Despite believing they understand their chatbot’s personality, users mispredicted it on several traits. This underscores the need for tools that help users understand AI before use, as some seemingly helpful behaviors may be unhealthy over time.
Although transparency increased user trust, it did not change how they designed their chatbots. Upcoming research is examining how a model’s internal neural representation shifts during conversations. Initial findings suggest that visualizing these changes helps users better anticipate AI behavior and reduces overconfidence. As AI companions become more integrated into daily life, tools like these may become as common as nutrition labels, helping people understand AI’s influence on thoughts and emotions.
Original Source: news.mit.edu
