VR Chatbots and interactable NPCs

The latest Modbox update includes the start of the NPC system – allowing Modbox creators to add and edit human characters.

One idea I wanted to try was having the user speak to the NPCs and have them respond – I already added voice commands to edit mode (and found speaking in VR to feel pretty natural), so it seemed pretty easy to just add a few voice commands you could give to NPCs, then have them respond with some preset responses. Then I decided rather than specific commands I would just let the user talk freely, using voice dictation APIs rather than voice commands, and rather than preset replies I could just hook up the NPC AI to a chatbot service and use text to speech for responses.

Surprisingly the hardest part of this was the speech to text. Unity has a Dictation Recognizer, which uses the Windows speech tools, but due to a Windows update that has been broken for half a year on 64 bit, and based on the Unity forums they apparently aren’t going to fix it. So I had to have Modbox create a new 32 bit process, then use the clipboard to send the text data back to the 64 bit application (hackiest thing I’ve done in Modbox, couldnt figure out how to inter process communication working with Unity)
For the text to speech I was expecting to just use the old Windows tools – the horrible robot voice every played with 15 years ago. I ended up trying out Amazon Polly – and while getting the API to work with Unity was a giant timesink the results were amazing. I am really hopeful these voice APIs will expand with more options like emotion selection. Then to make the lip sync work I used Oculus’ lip sync tools – I just needed to manually set what phonemes resulted in which mouth blendshapes on the Morph3d models.

For multiple characters I then added a start up command, where you say ‘Hello ‘Name” to the character to have them start listening. This is shown in the Modbox update video

I have no idea how I’ll end up using this for the future of Modbox, if at all, but it was a fun experiment to try. Problem with how it currently uses chatbots is it has zero connection to gameplay – the current API just gives a random response based on what other people have said to it (I just used Cleverbot, which can give pretty hilarious responses). I imagine there are chatbot API’s out now that I could program to understand the users intent (mostly used now by horrible AI help systems on websites, and messaging startups), but that’d be a lot more effort than I was willing to put into this for now – plus I didn’t have gameplay in mind yet for what I would do with these guys (just giving them commands to pick up objects and interact with the world would be great, but that’ll have to come after I give the NPC AI a lot better understanding of the environment).
Another easier way to do this, that I hope future VR games do, is just show the user a few voice command options. So rather than the traditional dialog selection system in RPG games have the user say the commands. Recognizing preset voice commands is a lot easier and less error prone than doing full speech to text.