Final Year Project - Speak Up!

Abstract

This study presents the development and evaluation of a Virtual Reality (VR) public speaking training application enhanced with Generative AI-driven multi-agent interactions. The purpose of the study is to provide learners with an immersive platform where they can practice presentation skills in realistic scenarios. The proposed system integrates multi-agent non-player characters (NPCs) powered by generative AI, capable of simulating social interactions, providing supportive feedback, and adapting to user performance.

The VR application was developed using Unity and integrated with technologies such as Ready Player Me avatars, Whisper speech-to-text, text-to-speech synthesis, and ChatGPT-based feedback mechanisms. A structured test case framework was designed to evaluate NPC realism across five levels: surface realism, spatial awareness, interactive realism, social multi-agent realism, and cognitive personality realism.

To assess user experience, a System Usability Scale (SUS) and Igroup Presence Questionnaire (IPQ) were administered to a sample of four participants. The results indicate that users found the system engaging, immersive, and effective for public speaking practice, with positive feedback on the generative AI feedback system and NPC responsiveness. However, limitations were identified in terms of NPC social realism, where behaviors such as group dynamics, emergent reactions, and long-term role consistency were not fully achieved.

Future work will focus on enhancing multi-agent social awareness, improving NPC collaboration, and scaling the system for larger sample testing. The findings highlight the potential of combining VR with generative AI and multi-agent systems to advance training in communication and presentation skills.

Project Highlights

• Developed in Unity with Whisper, Text-to-Speech, and ChatGPT integration.
• Implements multi-agent NPC interactions powered by Generative AI.
• Supports Ready Player Me avatars with dynamic personalities.
• Includes structured evaluation of NPC realism and user presence.
• Tested using SUS and IPQ to assess usability and immersion.

What I learnt

• How to integrates avatars, animations, and lip sync into immersive VR characters.
• The practical use of Whisper API for speech-to-text and its challenges in real-time contexts.
• Building reactive NPCs with awareness of range, FOV, and player actions.
• The complexity of AI-driven dialogue systems, requiring prompt engineering and structured logic to create believable interactions.
• The design of a custom feedback system that blends automated AI feedback with pedagogical goals.
• Implementing multi-agent NPC systems for group conversations and emergent behavior.
• Balancing realism and performance when combining TTS with lip-sync.
• Managing a large, multi-scene VR project by prioritizing tasks and controlling scope.

System Demonstration