Suggestions

close search

Add Messaging, Voice, and Authentication to your apps with Vonage Communications APIs

Visit the Vonage API Developer Portal

Adjusting audio and video — Web

You can make audio and video adjustments to published and subscribed streams:

Publishing audio or video only

When you create a Publisher object, you can specify whether to initially publish only audio or only video. For example, the following code creates an audio-only Publisher: 

var pubOptions = {publishAudio:true, publishVideo:false};

// Replace replacementElementId with the ID of the DOM element to replace:
publisher = OT.initPublisher(replacementElementId, pubOptions);

By default, a Subscriber object is initialized to subscribe to audio and video, if they are available.

Once you have created a Publisher object, you can toggle audio and video on or off, by calling the publishAudio() and publishVideo() methods (passing in a Boolean value). For example, the following code turns audio off: 

publisher.publishAudio(false);

When toggling video on or off with the publishVideo() method, you can pass in an optional completion handler that will execute after video starts or stops publishing. For example, the following code could be used to make UI changes after publishing video: 

  publisher.publishVideo(true, (error) => {
      if (!error) {
        // You may want to adjust the user interface to let users know
        // their video is visible to others
      }
    });

Note: This is a beta feature.

When you toggle audio or video on or off, the Session object in each connected client dispatches a streamPropertyChanged event. For more information, see StreamPropertyChangedEvent.

Setting the resolution and frame rate for a video

You can set the frame rate and resolution for a publisher's stream by setting a resolution property of the options you pass into the OT.initPublisher() method. See Setting the video resolution of a stream and Setting the frame rate of a stream.

You can set the frame rate and resolution for a subscriber's stream by setting the preferredFrameRate and preferredResolution properties of the options you pass into the session.subscribe() method. We recommend setting the preferreResolution option to "auto" to optimize the CPU and network usage. See Setting the preferred frame rate and resolution.

Switching the camera used by a Publisher

You can switch the video input device (camera) used as the video source for a Publisher by calling the Publisher.cycleVideo() or Publisher.setVideoSource() method:

Publisher.cycleVideo()

The Publisher.cycleVideo() method lets you cycle through the available video sources (cameras). For example, on a mobile device you can switch between the front and back camera. The method returns a promise that resolves when the operation completes successfully. The promise resolves with an object that has a deviceId property set to the device ID of the camera used: 

publisher.cycleVideo().then(console.log);
// Output: {deviceId: "967a86e52..."}

If there is an error, the promise is rejected. This can occur in the following conditions:

Note that this method only works for a publisher that is using a camera video source.

Publisher.setVideoSource()

The Publisher.setVideoSource() method lets you pass in the device ID of the new video source (camera). The method returns a promise that resolves when the operation completes: 

publisher.setVideoSource(deviceId)
  .then(() => console.log('video source set'))
  .catch((error) => console.error(error.name));

The following will result in errors:

You can use the OT.getDevices() method to enumerate the available video input devices (and get their device IDs).

You can use the Publisher.getVideoSource() method to get the current video source and its device ID (if it is a video input device).

Mirroring the local display of a Publisher's video

You can set the mirror property of the options passed into the OT.initPublisher() method to have the publisher's locally rendered video mirrored (true) or not (false). By default, video is mirrored for a publisher that has a camera video source, and not mirrored for a screen-sharing video.

This setting only affects the rendered video in the publisher's client application. It has no effect on the video in subscribing clients.

Switching the audio source used by a Publisher

You can switch the microphone or MediaStreamTrack object used as the audio source for a Publisher by calling the setAudioSource() method of the Publisher object.

Pass a device ID for a microphone or an audio MediaStreamTrack object into the Publisher.setAudioSource() method. The method returns a Promise that is rejected on error (see the reference documentation for setAudioSource()).

For example, the following code shows you how to implement a cycleMicrophone() function that cycles through the microphones available: 

// Setting an audio source to a new MediaStreamTrack
const stream = await OT.getUserMedia({
  videoSource: null
});

const [audioSource] = stream.getAudioTracks();
publisher.setAudioSource(audioSource).then(() => console.log('Audio source updated'));

// Cycling through microphone inputs
let audioInputs;
let currentIndex = 0;
OT.getDevices((err, devices) => {
  audioInputs = devices.filter((device) => device.kind === 'audioInput');
  // Find the right starting index for cycleMicrophone
  audioInputs.forEach((device, idx) => {
    if (device.label === publisher.getAudioSource().label) {
      currentIndex = idx;
    }
  });
});

const cycleMicrophone = () => {
  currentIndex += 1;
  let deviceId = audioInputs[currentIndex % audioInputs.length].deviceId;
  publisher.setAudioSource(deviceId);
};

The Publisher.setAudioSource() method only works for a publisher that has an audio source. If you set audioSource to null (or false) when calling OT.initPublisher(), you cannot later add an audio source to the publisher.

The Publisher.getAudioSource() method returns the MediaStreamTrack object used as the current audio input source for the publisher.

The OT.getDevices() method enumerates the audio and video input devices available to the browser.

Switching the audio output used

You can switch the audio output device (a speaker or headphones) used to play audio from all publishers and subscribers (in all OpenTok sessions in the browser).

The OT.getAudioOutputDevices() method enumerates the audio and video input devices available to the browser.

The OT.getActiveAudioOutputDevice() method identifies the currently active audio output device.

Use the OT.setAudioOutputDevice() method to set the audio output device.

For example, the following code shows you how to implement a cycleAudioOutput() function that cycles through the available audio output devices: 

// Cycling through audio output devices
let currentIndex = 0;
const audioOutputs = await OT.getAudioOutputDevices();
const currentOutputDevice = await OT.getActiveAudioOutputDevice();
audioOutputs.forEach((device, index) => {
  if (device.label === currentOutputDevice.label) {
    currentIndex = index;
  }
});

const cycleAudioOutput = async () => {
  currentIndex += 1;
  let deviceId = audioOutputs[currentIndex % audioOutputs.length].deviceId;
  await OT.setAudioOutputDevice(deviceId);
};

Detecting when a publisher switches audio input devices

By default, the SDK automatically handles audio input device switching for any Publisher object when an audio input device is added or removed, unless audio input device management has been disabled. See Disabling default audio input device management if you wish to disable automatic audio input device switching.

The Publisher object dispatches an audioInputDeviceChanged event when the SDK automatically changes the audio input. This event will be dispatched when a new audio input device is added, the current audio input device is removed, or when the Publisher's operating system switches audio input devices. The dispatched event has a device property that contains information about the audio input device. You may want to let your users know their microphone has changed: 

publisher.on('audioInputDeviceChanged', (event) => {
  console.log('audio input device', event.device);
  console.log(`changing device to: ${event.device.label}`);
});

Note: This is a beta feature.

Publishing in a voice-only session

To set up a voice-only session, set the videoSource property to null or false when you create each Publisher object in the session. For example, the following code creates a Publisher for a voice-only session: 

var pubOptions = {videoSource: null};

// Replace replacementElementId with the ID of the DOM element to replace:
publisher = OT.initPublisher(replacementElementId, pubOptions);

When you set the videoSource property to null, the publishing client does not request access to the camera, and no video is published.

Subscribing to audio or video only

When you subscribe to a stream, you can specify whether to initially subscribe to audio or video (if they are available). For example, the following code subscribes to the audio stream only: 

var options = {subscribeToAudio:true, subscribeToVideo:false};

// Replace stream and replacementElementId with your own values:
subscriber = session.subscribe(stream,
                             replacementElementId,
                             options);

After you create a Subscriber object, you toggle audio on or off by calling the subscribeToAudio() method of the Subscriber object: 

subscriber.subscribeToAudio(false); // audio off
subscriber.subscribeToAudio(true); // audio on

You toggle video on or off by calling the subscribeToVideo() method of the Subscriber object: 

subscriber.subscribeToVideo(false); // video off
subscriber.subscribeToVideo(true); // video on

Note however that you can only subscribe to audio or video if the client publishing the stream includes audio or video. For example, calling subscribeToVideo(false) will have no effect if the client publishing the stream is publishing audio only.

Changing the audio level of a subscriber

When you subscribe to a stream, you can set the initial volume of the subscriber when you call the subscribe() method of the Session object: 

// Set a value between 0 (silent) and 100 (full volume):
var subOptions = {audioVolume = 10};

// Replace stream and replacementElementId with your own values:
subscriber = session.subscribe(stream,
                             replacementElementId,
                             subOptions);

After you create a Subscriber object, you can set its volume by calling its setAudioVolume() method, passing in a value from 0 (silent) to 100 (full volume): 

subscriber.setAudioVolume(0); (silent)

Note that the user can also mute the subscriber via user interface controls in the subscriber.

Detecting whether a stream has audio or video

By default, a Subscriber object plays back both audio and video (if they are available). You can check if a stream has audio or video (if the publisher of the stream is streaming audio or video) by checking the hasAudio and hasVideo properties of the Stream object: 

if (!stream.hasAudio) {
	// You may want to adjust the user interface
}
if (!stream.hasVideo) {
	// You may want to adjust the user interface
}

For example, when you subscribe to a stream, you may want to adjust the user interface based on whether the stream has audio or video. For example, you may want to indicate to the user whether a stream has audio or not; or you may not want to hide a subscriber if a stream does not have video.

Detecting when a stream adds or removes audio or video

The Session object dispatches a streamPropertyChanged event when a stream toggles audio or video on or off. The streamPropertyChanged event is defined by the StreamPropertyChangedEvent class. The event object has a changedProperty property (identifying the Stream property that changed) and a newValue property (the new value of the Stream property). For example, the following code listens for changes in a audio and video in a Stream: 

session.on("streamPropertyChanged", function (event) {
    var subscribers = session.getSubscribersForStream(event.stream);
    for (var i = 0; i < subscribers.length; i++) {
        // You may want to display some UI text for each
        // subscriber, or make some other UI change,
        // based on event.changedProperty and
        // event.newValue
    }
}

Note that a subscriber's video can be disabled or enabled for reasons other than the publisher disabling or enabling it. A Subscriber object dispatches videoDisabled and videoEnabled events in all conditions that cause the subscriber's stream to be disabled or enabled. For details, see the documentation for the Subscriber videoDisabled and Subscriber videoEnabled events.

Optimizations in voice-only sessions

There are a number of user interface optimizations that you can make in a voice-only session. See the Voice tutorial.

Tuning audio quality

The OT.initPublisher() method includes options for tuning audio quality. This lets you publish streams that use high-quality (or lower quality) audio:

audioBitrate (Number) — The desired bitrate for the published audio, in bits per second. The supported range of values is 6,000 - 510,000. (Invalid values are ignored.) Set this value to enable high-quality audio (or to reduce bandwidth usage with lower-quality audio).

The following are recommended settings:

If you do not set this option, OpenTok.js automatically sets the audio bitrate for the stream.

Currently, this setting is not supported in streams published in Firefox.

autoGainControl (Boolean) — Whether to enable automatic gain control for the published audio. You may want to set this to false when publishing high-quality audio (by setting the audioBitrate property of the OT.initPublisher() options). The default value is true. This setting is ignored if you set disableAudioProcessing to true (which disables echo cancellation, automatic gain control, and noise suppression for the published stream).

disableAudioProcessing (Boolean) — Whether to disable echo cancellation, automatic gain control, and noise suppression for the published audio. You may want to set this to true when publishing high-quality audio (by setting the audioBitrate property of the OT.initPublisher() options). The default value is false.

echoCancellation (Boolean) — Whether to enable echo cancellation for the published audio. You may want to set this to false when publishing high-quality audio (by setting the audioBitrate property of the OT.initPublisher() options). The default value is true. This setting is ignored if you set disableAudioProcessing to true (which disables echo cancellation, automatic gain control, and noise suppression for the published stream).

Note: Some browsers (such as Chome 73+) do not support echo cancellation for stereo audio (see this Chrome issue report).

enableStereo (Boolean) — Whether to publish stereo audio. The default value is false.

noiseSuppression (Boolean) — Whether to enable noise suppression for the published audio. You may want to set this to false when publishing high-quality audio (by setting the audioBitrate property of the OT.initPublisher() options). The default value is true. This setting is ignored if you set disableAudioProcessing to true (which disables echo cancellation, automatic gain control, and noise suppression for the published stream). You can also utilize the advancedNoiseSuppression audio filter, which uses the Vonage Media library (see this topic).

Reducing audio bandwidth with Opus DTX

Opus DTX (Discontinuous Transmission) is an audio codec that can reduce the bandwidth usage when a participant is not speaking. This can be useful in large sessions with many audio participants.

You enable Opus DTX by setting the enableDtx property of the options object you pass into the OT.initPublisher() method (when initializing a Publisher).

For more information, see this Vonage Video API knowledge base article.

Applying filters and effects to audio and video

You can apply filters and effects to published audio and video in the following ways:

You can apply filters and effects to subscribed audio and video by using the Vonage Media Processor library to apply transformations.

These are described in the sections that follow.

Publishing a MediaStreamTrack object and applying effects to the object

You can use an audio MediaStream track or a video MediaStream track as the source audio or video for a published stream. Using this feature, you can apply filters and effects, such as background blur or background replacement, to the published audio or video.

You can use the OT.getUserMedia() method to get a reference to a MediaStream that uses the camera selected by the user. You can then use the video MediaStreamTrack obtained from that MediaStream object as the source for a Video element. You can then add that Video element to an HTML Canvas element, apply filters or effects to the canvas, and use the filtered video MediaStreamTrack object obtained from the canvas as the video source for a published stream. For an example, see the Stream-Filter sample opentok-web-samples repo on GitHub.

You can use the OT.getUserMedia() method to get a reference to a MediaStream that uses the microphone selected by the user. You can then use the audio MediaStreamTrack obtained from that MediaStream object as the as the audioSource when calling OT.initPublisher(). You can then create an AudioContext object and call its createMediaStreamSource() object, passing in the MediaStream object to to create a MediaStreamAudioSourceNode object. You can then apply audio filters to the MediaStreamAudioSourceNode object, which will result in the filters being applied to the published stream.

Applying a background blur or background replacement filter

Use the Publisher.applyVideoFilter() method to apply a video filter for the publisher. You can apply a background blur filter or a background image filter. A publisher can have only one (or zero) filters applied at one time. When you set a new filter, a previously set filter is removed.

The following code applies a background blur filter to a publisher's video: 

publisher.applyVideoFilter({
  type: 'backgroundBlur',
  blurStrength: 'low',
});

The following code applies a background replacement filter to a publisher's video: 

publisher.applyVideoFilter({
  type: 'backgroundReplacement',
  backgroundImgUrl: 'http://example.com/image.jpg',
});

You can also apply a video filter by setting the videoFilter option when calling the OT.initPublisher() method: 

const publisher = OT.initPublisher(null, {
  videoFilter: {
    type: 'backgroundReplacement',
    backgroundImgUrl: 'http://example.com/image.jpg',
  },
  mirror: false,
});

Important:

  • Video filters require adequate processor support. Even in supported browsers, transformers may not be stable when background processes limit available processing resources. See Client requirements.
  • The background image is resized to match the dimensions of the publisher's video. For best results, use an image that has the same dimensions (or at least the same aspect ratio) as the published video. Set the mirror option to false when calling the OT.initPublisher() method to have the background image appear in the correct orientation (not mirrored) in the publisher's page.

For details, see the reference documentation for Publisher.applyVideoFilter().

This method is incompatible with the Publisher.setVideoMediaProcessorConnector() method. Calling this method after Publisher.setVideoMediaProcessorConnector(connector) returns a promise that is rejected with an error, with the name property of the error set to 'OT_NOT_SUPPORTED'. You can remove the connector by calling Publisher.setVideoMediaProcessorConnector(null).

You can also set the initial video filter by setting a videoFilter option when calling OT.initPublisher() method.

To clear the video filter, call the Publisher.clearVideoFilter() method.

Notes:

  • Video filters are only supported in recent versions of Chrome, Electron, Opera, and Edge. They are not supported in other (non-Chromium-based) browsers or on iOS. You can check if the client supports this feature by calling the OT.hasMediaProcessorSupport() method.
  • The Vonage Media Library includes transformers that provide more options than are provided with the background blur and background image filters provided by the Publisher.applyVideoFilter() method. Also, you can use the Vonage Media Library to write your own custom audio and video transformers. See the Using the Vonage Media Processor developer guide.

Applying the advanced noise suppression filter

Advanced noise suppression is an audio filter that greatly reduces the background noise around the publisher. This can be useful when a participant is in a crowded or noisy environment, like in a cafe, or if white noise is present, like from an HVAC system. It can work separately or in conjunction with the noiseSuppression property set when calling the OT.initPublisher method.

Use the Publisher.applyAudioFilter() method to apply an audio filter for the publisher. You can apply an advanced noise suppression filter. A publisher can have only one (or zero) filters applied at one time. When you set a new filter, any previously set filter is removed.

The following code applies an advanced noise suppression audio filter to a publisher's audio: 

publisher.applyAudioFilter({
  type: 'advancedNoiseSuppression',
});

You can also apply the advanced noise suppression audio filter by setting the audioFilter option when calling the OT.initPublisher() method: 

const publisher = OT.initPublisher(null, {
  audioFilter: {
    type: 'advancedNoiseSuppression'
  },
});

Important:

  • Audio filters require adequate processor support. Even in supported browsers, transformers may not be stable when background processes limit available processing resources. See Client requirements.

For details, see the reference documentation for Publisher.applyAudioFilter().

This method is incompatible with the Publisher.setAudioMediaProcessorConnector() method. Calling this method after Publisher.setAudioMediaProcessorConnector(connector) returns a promise that is rejected with an error, with the name property of the error set to 'OT_NOT_SUPPORTED'. You can remove the connector by calling Publisher.setAudioMediaProcessorConnector(null).

To clear the advanced noise suppression filter, call the Publisher.clearAudioFilter() method.

Notes:

  • Audio filters are only supported in recent versions of Chrome, Electron, Opera, and Edge. They are not supported in other (non-Chromium-based) browsers or on iOS. You can check if the client supports this feature by calling the OT.hasMediaProcessorSupport() method.
  • When using Publisher.applyAudioFilter(), assets are downloaded from Vonage servers. If you prefer to self-host these assets, please refer to this blog post for information on hosting, and for using the Publisher.setAudioMediaProcessorConnector() method to apply the advanced noise suppression filter with self-hosted assets.

Using the Vonage Media Processor library to apply custom transformations

See the Using the Vonage Media Processor developer guide.

Publishing audio only for Publishers with low bandwidth

Publishers can be configured to disable video, keeping audio enabled, in low bandwidth situations. Video publishing will resume when the Publisher's bandwidth improves. For more information, see the Audio Fallback developer guide.