You don’t need to use Whisper, I got some names mixed up. I was thinking of wyoming-faster-whisper which uses the FOSS speech to text system faster-whisper, but there are others that can be used.

Edited my original comment to fix that.

Wyoming is a protocol for voice assistants.

It ties together:

• speech recognition services (faster-whisper, vosk, whisper.cpp, OpenAI’s Whisper API)
• text to speech services (piper)
• wake word detection services (openWakeWord, snowboy, porcupine1)
• intent handling services
• intent recognition services

Home Assistant can interact with that protocol. I think the addons run servers for various components used by the wyoming protocol server that the integration can use, but I run it separate from Home Assistant, so idk.

Not sure what futo is capable of, but you can use anything that can communicate with a wyoming server. I’m willing to wager you can, but idk.

OpenAI’s ChatGPT API and LLM models are orthogonal to this, but probably could be used as an intent or as the fallback when no other intent was recognized. So I’m pretty sure you could link up getting a response from OpenAI or any other LLM API, but I haven’t tried setting that up for myself yet. wyoming-handle-external lets you pipe the input text to the stdin of whatever program you give it and responds with the program’s stdout, so you could definitely use this to pass it to OpenAI or Ollama.

Checkout OpenWhisper Faster Whisper and Wyoming

edit: corrected name and added links

Lotta words to say GPL and offline-first