How to connect wavesurfer with Tone.js

[battistar]

Hi,

I want to use wavesurfer player with some Tone.js effects (e.g. pitch shift). Can someone explain me how to connect them together and show me some examples?

Thanks.

[corysimmons]

I'd love to know this as well.

Update: This works https://gist.github.com/corysimmons/00e0e458b253d415a7537e44c002fd9b

[omerman]

Thank you so much I'll totally try that!

[katspaugh]

Check out this example: https://wavesurfer.xyz/examples/?webaudio.js

It shows how to get a Web Audio media element source node from wavesurfer. You can connect that node to other Web Audio nodes (such as Tone.js effects).

[katspaugh]

Try this one: https://wavesurfer.xyz/examples/?4436ec40a2ab943243755e659ae32196

const wavesurfer = WaveSurfer.create({
  container: document.body,
  url: '/examples/audio/audio.wav'
})

// Create Web Audio context
const audioContext = new AudioContext()

// Create a filter
const filter = audioContext.createBiquadFilter()
filter.type = 'lowpass'
filter.frequency.value = 300
filter.connect(audioContext.destination)

// Create a MediaElementSourceNode from the audio element
const mediaNode = audioContext.createMediaElementSource(wavesurfer.getMediaElement())
mediaNode.connect(filter)

[corysimmons]

I appreciate the attempt to help me, but I still can't get it working with Tone.

I figured out the native AudioContext thing and using Tone's wrapper of it.

But I keep getting stuck around the filter + mediaNode part.

I'm using Next.js because it's actually what I'm using and the HTML/CSS/JS StackBlitz kept throwing saying I needed a browser window(?!).

Here is a demo: https://stackblitz.com/edit/stackblitz-starters-gasfqb?file=app%2Fpage.tsx

For posterity (in case the demo goes down), here is the code:

'use client';
import React, { useRef, useState, useEffect } from 'react';
import { useWavesurfer } from '@wavesurfer/react';
import * as Tone from 'tone';

export default function Page() {
  const containerRef = useRef<HTMLDivElement>(null);
  const audioRef = useRef<HTMLAudioElement | null>(null);
  const [playing, setPlaying] = useState(false);

  const { wavesurfer } = useWavesurfer({
    container: containerRef,
    height: 100,
    dragToSeek: true,
    media: audioRef.current!,
  });

  useEffect(() => {
    if (!audioRef.current) {
      audioRef.current = new Audio('/piano-loop.mp3');
    }
  }, []);

  useEffect(() => {
    let audioContext: Tone.Context | null = null;
    if (wavesurfer) {
      wavesurfer.on('ready', () => {
        audioContext = new Tone.Context();
        Tone.setContext(audioContext);
        const filter = new Tone.Filter(300, 'lowpass');
        filter.connect(audioContext.destination)
        const mediaNode = audioContext.createMediaElementSource(wavesurfer.getMediaElement())
        mediaNode.connect(filter) // This filter type errors here
      });

      wavesurfer.on('play', () => setPlaying(true));
      wavesurfer.on('pause', () => setPlaying(false));
      wavesurfer.on('finish', () => setPlaying(false));
    }

    return () => {
      if (wavesurfer) {
        wavesurfer.destroy();
      }
      if (audioContext) {
        audioContext.dispose();
      }
    };
  }, [wavesurfer]);

  const handlePlayPause = () => {
    if (wavesurfer) {
      Tone.start();
      wavesurfer.playPause();
    }
  };

  return (
    <div>
      <div ref={containerRef}></div>
      <button onClick={handlePlayPause}>
        {playing ? 'Pause' : 'Play'}
      </button>
    </div>
  );
}

Sorry to keep bugging you.

[corysimmons]

cc @marcelblum @jgentes

Tonejs/Tone.js#612

[corysimmons]

The problem was the mediaNode in mediaNode.connect(filter) is an instance of the native implementation and I needed to use Tone.connect instead. 😬 🤦

I'm currently working on a slightly larger example that will allow you to scrub around a timeline and have effects applied/removed in realtime. Once that's ready I'll post it somewhere for posterity.

Update: Here ya go. This combines Wavesurfer and Tone so you can add whatever effects you want and scrub the timeline to have those effects applied in realtime.

https://gist.github.com/corysimmons/00e0e458b253d415a7537e44c002fd9b

Thank you for your help @katspaugh and @marcelblum ! <3

[marcelblum]

Trying out your test page at stackblitz and there are multiple CORS errors due to the way the environment is set up so it's tough to debug as is... on FF the mp3 won't load while on Chrome it says "MediaElementAudioSource outputs zeroes due to CORS access restrictions for <URL>" (never seen that one before, hmm...!). Also - not sure if this is the problem but you might need to wait until you do Tone.start() before you can createMediaElementSource().

[corysimmons]

Oh, super strange. It definitely works locally. I am pretty sure it was working on StackBlitz but now it's not. I went ahead and got rid of the hotlinking and I'm just pointing to /public/piano-loop.mp3 now, but it still doesn't play anymore. 🤷

I tried adding Tone.start() but it doesn't seem to do anything or fix the type error.