The Maya virtual guide, first developed in the frame of the CHROME project at the University of Naples Federico II

Machines talking about art: language versus communication models

The Timeless Language of Visual Art: Communicating Through Centuries

Works of visual art are among the first attempts of human beings to express themselves and probably to leave a long-lasting message, so that artistic expression has been considered as an inherent need. In its long history, human kind as always used art as a communication medium with its peculiar aspects: on one hand easy to understand because it is visual, on the other hand not based on a unique code of interpretation. Moreover, art requires a skillful use of techniques (it’s probably useful to remind that the Latin word ars means, above all, technique), a talent but also knowledge and experience to express a message, to stimulate emotions and feelings.  Works of art are therefore fascinating human beings since ever, so that we are continuously looking for a deeper understanding, a better knowledge of their principles.

Even this short introduction leads us to imagine the richness of possible interpretations of a single work, what then when we face many of them in the same place and we understand they they can talk to each other so creating a net of relationships that seems to widen and multiply the message?

Museums as Complex Communication Systems: Enhancing Visitor Engagement

That’s what usually happens in historical buildings or in museums, where visual stimuli become protagonists of a complex communication. The definition provided by the International Council of Museums states that “A museum is a not-for-profit, permanent institution in the service of society that researches, collects, conserves, interprets and exhibits tangible and intangible heritage. Open to the public, accessible and inclusive, museums foster diversity, sustainability and citizen curation. They operate and communicate ethically, professionally and with the participation of communities, offering varied experiences for education, enjoyment, reflection and knowledge sharing.”.

Furthermore, if we refer to the wider idea of heritage, the concept of “education, enjoyment, reflection and knowledge sharing” remains fundamental. Ideally, technology should be employed to support cultural environments in achieving the mission stated before. According to the museum studies literature, the definition of museum as a Communications System refers to every kind of museum and includes every aspect: exhibits, their selection, the set-up, the lighting, labels, educational programs, videos, and whatever contributes to communicate a message related to the museum itself. Even considering such an articulate system, some museums show peculiar aspects that make them complex, i.e. when beyond the museum function of form and content, they are monuments in themselves. The nature of monument implies, on one side, historical aspects, a notable architecture, with decoration and often even remarkable works of art stratified over times. Usually, all these aspects are kept together by centuries of history that contributed to merge all of them by multiplying and enhancing the sense. In other words, the whole environment, in its complexity, plays a role in the process of creation of sense.

The Intersection of AI and Cultural Heritage: Redefining Virtual Experiences

Modern technology can intervene on these aspects like never before, as the modern era has seen dramatic improvements in the field of Natural Language Processing and Conversational Artificial Intelligence, with Large Language Models (LLMs) attracting a great deal of interest. As powerful as this new technology is, however, it is necessary to correctly frame it in the context of Artificial Intelligence and understand its capabilities and limitations to be able to correctly deploy them. Humanities studies play a significant role both in understanding what conversational architectures based solely on generative approaches can do and what is simply beyond their reach. From a linguistic point of view, for example, it can be argued that LLMs model surface aspects of communication but fail to capture the deeper motivations and intentions for communicative acts. In other words, a Language Model, capturing complex regularities in words use, is not a Communication model, capturing the reasons why words have been produced. These aspects find correspondence in the AI field, particularly when considering the Ladder of Causation from Judea Pearl, where the difference between observation and prediction capabilities are considered inferior with respect to intervention capabilities. For a full discussion about this topic, we refer to the Conversational AI series found on the URBAN/ECO Research Center (University of Naples Federico II) blog.

The Ladder of Causation (Judea Pearl, The Book of Why [1])

Advancing AI in Cultural Heritage

Museum studies have extensively investigated the strategies for exhibit design and museum space usage, also considering the difficulties coming from some museums also being historical buildings. Moreover, visitor studies have investigated multiple aspects of people’s behaviour when they are found in cultural environments. While these models have been used, in the past, to motivate technological interventions in museum environments, they have rarely been used to drive the design of the communicative intent of exhibits. This is, of course, understandable on multiple aspects: first, a public institution cannot delegate its communication design to technological artifacts of the current generation. Also, the same artifact may serve different roles in the design of different exhibits depending on characteristics that may very well be mostly visual rather than textual or catalographic. Moreover, balancing communication needs with budget, space and lighting constraints further complicates the task, leading to a situation where specific competence coming from domain experts is needed to understand how to deploy currently available AI tools.

A particularly interesting case is represented by virtual spaces, where Conversational AI technology paired with the use of Virtual Humans can surpass the limitations of physical spaces and real-world constraints, providing an insight into the world of history and art that may fuel user motivation into visiting cultural sites. When photography became widely adopted, the availability of the images of cultural artifact was first seen as a threat for museums, as people may have chosen not to visit them because they could see the reproduction more easily. This turned out to be the opposite: people were motivated to go see the original because they appreciated the reproduction. Current technology may serve a similar purpose, today, as multiple objects may be presented as part of a coherent communicative intent that can only be described in a computational model by collaborating with cultural heritage experts. At the same time, the use of cultural artifacts reproductions and natural language aimed at producing positive changes in the society calls for new AI paradigms.

The Limitations of LLMs: Understanding Intent Beyond Language Generation

While LLMs, in fact, have shown exceptional capabilities to produce fluent language, they fail in understanding both goals and consequences of what they are doing. Being very powerful Natural Language Generators, in this sense, they need other AI modules, deciding why to say something, to instruct them in what to say. These aspects can only be modelled based on a deep understanding of cultural communication models, which is a topic humanities researchers have studied for decades. These communication models, then, must be interpreted in a technological framework general enough to avoid being tailored for a specific task, since the goal is to build machines that communicate, in general, rather than industrial tools. Embodied communication, in this sense, is particularly important as both speech and body language are relevant for the case of cultural heritage presentation in virtual tours. For the specific case of Virtual Humans, for example, deciding what to say, how to say it and how to complement the provided information with relevant movements, like pointing movements, is critical for a successful presentation. Previous work has shown that Virtual Humans used to provide introductory presentations to visually rich cultural sites help people identify important details autonomously. Moreover, a linguistically motivated use of different kinds of pauses in synthetic speech is important to help people memorise information.

The Maya virtual guide, first developed in the frame of the CHROME project at the University of Naples Federico II

This research is currently being conducted at the URBAN/ECO Research Center, at the University of Naples Federico II, and builds upon the experience of the Cultural Heritage Resources Orienting Multimodal Experiences (CHROME) research project, funded by the Italian Ministry of University and Research. The technological framework used for this research is the Framework for Advanced Natural Tools and Applications with Social Interactive Agents (FANTASIA), which architecture is shown in the following Figure. FANTASIA allows to inspect deep aspects of decision making using multiple AI tools. This allows to keep the AI models explainable, so that every decision they take can be inspected by the developers. It also allows to develop and test communication theories using humanities studies, which hold great significance in the development of human-like intelligence going much beyond statistical learning. Our experience with communication tasks based on the use of Natural Language tells us that, in an era where human-like intelligence is more sought than ever in machines, humanities research has great value in defining the guiding principles to help machines better understand what it really means to speak, going beyond merely generating words.

The FANTASIA architecture
Don't miss out on the latest news!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

People also read

3D Technologies and AI

3D technologies and spatial AI applications in archaeology

This article explores the transformative impact of 3D technologies on archaeological research and documentation. It outlines three primary sensing tools used in modern archaeology: photogrammetry, Lidar scanning, and 3D cameras. Each method's strengths and applications are discussed, highlighting their roles in creating accurate digital representations of artifacts, sites, and landscapes. The integration of artificial intelligence in processing 3D models is examined, including element classification and the conversion of outdated maps into vector polygons.
noam bar
Noam Bar David
October 22, 2024
7
min read
Generative AI
Archaeology
notre dame assassin's creed rebuild

Preserving history through modern technology

Digital technologies are transforming cultural heritage preservation, making history more accessible and engaging. From virtual reality experiences to AI-powered restorations, innovative tools are breaking down barriers and allowing deeper connections with historical narratives. Discover the projects and technologies leading this digital revolution in heritage preservation.
author
Tatiana But
October 16, 2024
7
min read
History
ai driven maps

The World Needs Gen-AI World Maps: How the Global South Uses Knowledge Tools That the West Takes For Granted

This article explores the potential of conversational AI to revolutionize how we interact with large data sets. Such tools could improve research, learning, and global development by allowing users to create personalized maps and data visualizations.
Michael Mroz
Michael Mroz
October 1, 2024
7
min read
AI Bias
Digital Humanities
Generative AI

Contribute to Historica's blog!

Learn guidelines, requirements, and join our history-loving community.

Become an author

FAQs

How can I contribute to or collaborate with the Historica project?
If you're interested in contributing to or collaborating with Historica, you can use the contact form on the Historica website to express your interest and detail how you would like to be involved. The Historica team will then be able to guide you through the process.
What role does Historica play in the promotion of culture?
Historica acts as a platform for promoting cultural objects and events by local communities. It presents these in great detail, from previously inaccessible perspectives, and in fresh contexts.
How does Historica support educational endeavors?
Historica serves as a powerful tool for research and education. It can be used in school curricula, scientific projects, educational software development, and the organization of educational events.
What benefits does Historica offer to local cultural entities and events?
Historica provides a global platform for local communities and cultural events to display their cultural artifacts and historical events. It offers detailed presentations from unique perspectives and in fresh contexts.
Can you give a brief overview of Historica?
Historica is an initiative that uses artificial intelligence to build a digital map of human history. It combines different data types to portray the progression of civilization from its inception to the present day.
What is the meaning of Historica's principles?
The principles of Historica represent its methodological, organizational, and technological foundations: Methodological principle of interdisciplinarity: This principle involves integrating knowledge from various fields to provide a comprehensive and scientifically grounded view of history. Organizational principle of decentralization: This principle encourages open collaboration from a global community, allowing everyone to contribute to the digital depiction of human history. Technological principle of reliance on AI: This principle focuses on extensively using AI to handle large data sets, reconcile different scientific domains, and continuously enrich the historical model.
Who are the intended users of Historica?
Historica is beneficial to a diverse range of users. In academia, it's valuable for educators, students, and policymakers. Culturally, it aids workers in museums, heritage conservation, tourism, and cultural event organization. For recreational purposes, it serves gamers, history enthusiasts, authors, and participants in historical reenactments.
How does Historica use artificial intelligence?
Historica uses AI to process and manage vast amounts of data from various scientific fields. This technology allows for the constant addition of new facts to the historical model and aids in resolving disagreements and contradictions in interpretation across different scientific fields.
Can anyone participate in the Historica project?
Yes, Historica encourages wide-ranging collaboration. Scholars, researchers, AI specialists, bloggers and all history enthusiasts are all welcome to contribute to the project.