Multimodal AI for Social Work Research: Analyzing Image, Audio, and Video Data Using Large Language Models

Social work research increasingly involves data that extend beyond text and now include images, audio recordings, and video. Qualitative approaches to analyzing interview recordings, visual documentation from community-based research, and multimedia case materials are common in social work practice and research; however, computational methods for analyzing these data types are still evolving and remain underused. Recent advances in multimodal artificial intelligence (AI) models, which can simultaneously process and analyze texts, images, audio, and video, are opening new possibilities for social work researchers to leverage large, complex, multi-source datasets to generate granular insights into human behavior and social interactions.

This workshop introduces participants to multimodal AI analysis using open-source models available through Hugging Face. This session focuses on demonstrating AI multimodal capabilities, including analyzing images, processing audio, and extracting information from video data. Participants will learn how to use open-source multimodal models such as Gemma through LMStudio or Ollama for these tasks. The session is divided into the following three parts:

Part 1: Foundations of Multimodal AI. This segment introduces the foundations of multimodal AI models and how they differ from text-only language models. Participants will learn how these models process different data types (image, audio, video), what types of research tasks they can support, and what their current limitations are. The session will cover key concepts including modality-specific encoding, prompt construction for multimodal inputs, and output interpretation.

Part 2: Technical Implementation. Participants will work through Python-based tutorials in a local environment. The session will demonstrate how to run open-source multimodal models (Gemma) through Ollama or LM Studio. Step-by-step demonstrations will cover local environment setup, sending multimodal prompts, processing model outputs for research-relevant tasks, and streaming multiple files.

Part 3: Practical Considerations. The final segment addresses responsible use of multimodal AI in social work research. Topics include data privacy when transmitting sensitive image, audio, or video data to cloud-based APIs, informed consent considerations for multimedia research data, algorithmic bias in multimodal models (e.g., disparities in speech recognition accuracy across accents and languages), and current limitations in model reliability across different data types. The session will discuss when local open-source models may be preferable to cloud-based APIs for protecting participant confidentiality.

This workshop is designed for social work researchers and practitioners with no prior experience in multimodal AI. Basic familiarity with Python is helpful but not required; all codes will be provided in ready-to-run Jupyter notebooks. By the end of this workshop session, participants will be able to use open-source multimodal AI tools to analyze image, audio, and video data for their own research projects.