Euan's Blog

Speech To Text With JavaScript

I've recently been playing with some options to automatically convert speech into text. My original plan was to do it on the back-end in the ASP.NET core server, but my experiments with the System.Speech.Recognition.SpeechRecognitionEngine turned up a few key issues:

I therefore went looking for alternative options and stumbled across the Web Speech API. This API provides support for both speech recognition (speech to text) and speech synthesis (text to speech) in the web browser. My thinking was that I could convert the speech to text on the client side, write it into a hidden input and post that back to the back-end.

I'm most interested in the SpeechRecognition class in this post, which has quite an easy to use API that supports custom grammars, specifying the language to use for recognition, interim results and final results including degrees of certainty.

It's probably best to demonstrate with a quick code sample. You can preview it on CodePen too. Unfortunately I can't embed the CodePen here as the API is blocked when embedded in an iframe. This code sample uses React for state and event handling:

import React from "https://esm.sh/react";
import ReactDOM from "https://esm.sh/react-dom";

const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition || null;

(function(document, undefined) {
   const targetElement = document.getElementById("root");

   if (!targetElement) {
      console.error('failed to find root');

      return;
   }

   const root = ReactDOM.createRoot(targetElement);

   root.render(
      <React.StrictMode>
         <SpeechRecognizerDemo />
      </React.StrictMode>
   );
})(document);

function SpeechRecognizerDemo({}) {
   const [recognizeSpeech, setRecognizeSpeech] = React.useState(false);
   const [recognizedSpeech, setRecognizedSpeech] = React.useState('');

   React.useEffect(() => {
      let speechRecognizer = null;

      if (recognizeSpeech) {
         setRecognizedSpeech('');

         speechRecognizer = new SpeechRecognition();

         speechRecognizer.continuous = true;
         speechRecognizer.interimResults = false;
         speechRecognizer.maxAlternatives = 1;

         speechRecognizer.onerror = (err) => {
           console.error(err);
         };

         speechRecognizer.onresult = (event) => {
            setRecognizedSpeech('');

            for (const result of event.results) {
               if (result.isFinal) {
                  setRecognizedSpeech(oldValue => (oldValue + ' ' + result[0].transcript).trim());
               }
            }

            setRecognizedSpeech(oldValue => {
               return oldValue.replace(
                  /\S/,
                  c => c.toUpperCase()
               );
            })
         };

         speechRecognizer.start();
      }

      return () => {
         if (speechRecognizer) {
            speechRecognizer.stop();
         }
      };
   }, [recognizeSpeech]);

   if (!SpeechRecognition) {
      return <div className="alert alert-danger">
         Your web browser doesn't support speech recognition. Please try in a Chromium based browser or Safari.
      </div>
   }

   return <div className="container-fluid">
      <div className="row">
         <div className="col">
            <button type="button"
               className={`btn btn-sm btn-${recognizeSpeech ? 'danger' : 'success'}`}
               onClick={evt => {
                  evt.preventDefault();

                  setRecognizeSpeech(oldValue => !oldValue);

                  return false;
               }}>
               {recognizeSpeech ? 'Stop' : 'Start'}
            </button>
         </div>
      </div>

      <div className="row mt-2">
         <div className="col">
            <textarea className="form-control"
               readonly
               value={recognizedSpeech}></textarea>
         </div>
      </div>
   </div>
}

My initial testing looked pretty positive, but there are some (fairly serious) downsides:

I'm hoping that Firefox will add support fairly soon. For now I'm looking at providing this as an option hidden behind a setting labeled as a preview so that end-users can opt-in if they wish to.

#JavaScript