Suggestions

close search

Searching ...

Use the Vonage Media Processor library to apply custom transformations to audio and video.

You can use the Publisher.setAudioMediaProcessorConnector(), Publisher.setVideoMediaProcessorConnector(), Subscriber.setAudioMediaProcessorConnector(), and Subscriber.setVideoMediaProcessorConnector(), methods to apply custom audio and video transformers that use the Vonage Media Processor library to audio and video. The Publisher methods will affect the subscribers' videos whereas the Subscriber methods will only apply to that single Subscriber.

OpenTok.js includes three ways to implement transformers:

• Basic: Use the Publisher.applyVideoFilter() method to easily apply the pre-defined background blur or background replacement filter. This method implements these filters using the Vonage Media Library, requiring only a few configuration settings:

  await publisher.applyVideoFilter({
    type: 'backgroundBlur',
    blurStrength: 'high',
  });

  See Applying a background blur or background replacement filter.

• Moderate: You can use transformers from theVonage ML transformers (@vonage/ml-transformers) library.

  const config = {
    transformerType: 'VirtualBackground',
    backgroundAssetUri: "https://MY_IMAGE_URL"
  };
  const processor = await createVonageMediaProcessor(config);
  publisher.setVideoMediaProcessorConnector(processor.getConnector());
  

  See the background image replacement and background blur sample apps.

• Advanced: You can write your own transformer class that implements the Transformer class:

  class OverlayTextTransformer  implements  Transformer {
    // transform function must be implemented.
    transform(frame:any, controller:TransformStreamDefaultController) {
      // This will be more complicated in a real-world transformer...
      this.startCtx_.drawImage(frame, 0, 0, .., ..,)
      this.startCtx_.font = "30px Arial";
      this.startCtx_.fillText(this.message_, 50, 150);
      frame.close();
      controller.enqueue(new VideoFrame(this.startCanvas_, {timestamp, alpha: 'discard'}));
    }
  }
  
  const transformer = new OverlayTextTransformer('Hello World');
  const mediaProcessor = new MediaProcessor();
  mediaProcessor.setTransformers(transformers));
  const mediaProcessorConnector = new MediaProcessorConnector(mediaProcessor);
  publisher.setVideoMediaProcessorConnector(mediaProcessorConnector);
  

  See Applying a video transformer from the Vonage Media Library.

Important: Media transformations are not supported on all devices. See Client requirements.

The Vonage Media Processor library, available at npm, facilitates the use of the Insertable Streams API to perform transform operations on video and audio tracks.

Applying a background blur or background replacement filter

See this documentation.

Applying a video transformer from the Vonage Media Library

The library includes two key classes:

• MediaProcessor — Controls transformers and includes a setTransformers() method that lets you set transformers.
• MediaProcessorConnector — Accepts a MediaProcessor object in its constructor and handles input and output tracks. The OpenTok.js implementation handles the tracks. You only have to provide the MediaProcessorConnector instance.

To use a MediaProcessorConnector to connect a transformer to a publisher's audio or video, use the Publisher.setAudioMediaProcessorConnector() and Publisher.setVideoMediaProcessorConnector() methods.

Publisher.setVideoMediaProcessorConnector() method

The Publisher.setVideoMediaProcessorConnector() method applies a video transformer to a published stream:

import { MediaProcessor, MediaProcessorConnector } from '@vonage/media-processor';
import OT from '@opentok/client';
import MyTransformer from './path/to/my-transformer';

const mediaProcessor = new MediaProcessor();
const transformers = [
  new MyTransformer(),
];

mediaProcessor.setTransformers(transformers);
const connector = new MediaProcessorConnector(mediaProcessor);
const publisher = OT.initPublisher('targetDiv', () => {
  publisher.setVideoMediaProcessorConnector(connector);
});

Note: The Publisher.setVideoMediaProcessorConnector() method is incompatible with the Publisher.applyVideoFilter() method.

Publisher.setAudioMediaProcessorConnector() method

The Publisher.setAudioMediaProcessorConnector() applies an audio transformer to a published stream:

import { MediaProcessor, MediaProcessorConnector } from '@vonage/media-processor';
import OT from '@opentok/client';
import MyTransformer from './path/to/my-transformer';

const mediaProcessor = new MediaProcessor();
const transformers = [
  new MyTransformer(),
];

mediaProcessor.setTransformers(transformers);
const connector = new MediaProcessorConnector(mediaProcessor);
const publisher = OT.initPublisher('targetDiv', () => {
  publisher.setAudioMediaProcessorConnector(connector);
});

Subscriber.setVideoMediaProcessorConnector() method

The Subscriber.setVideoMediaProcessorConnector() method applies a video transformer to a subscribed stream:

import { MediaProcessor, MediaProcessorConnector } from '@vonage/media-processor';
import OT from '@opentok/client';
import MyTransformer from './path/to/my-transformer';

const mediaProcessor = new MediaProcessor();
const transformers = [
  new MyTransformer(),
];

mediaProcessor.setTransformers(transformers);
const connector = new MediaProcessorConnector(mediaProcessor);
await subscriber.setVideoMediaProcessorConnector(connector);

Subscriber.setAudioMediaProcessorConnector() method

The Subscriber.setAudioMediaProcessorConnector() applies an audio transformer to a subscribed stream:

import { MediaProcessor, MediaProcessorConnector } from '@vonage/media-processor';
import OT from '@opentok/client';
import MyTransformer from './path/to/my-transformer';

const mediaProcessor = new MediaProcessor();
const transformers = [
  new MyTransformer(),
];

mediaProcessor.setTransformers(transformers);
const connector = new MediaProcessorConnector(mediaProcessor);
await subscriber.setAudioMediaProcessorConnector(connector);

Client requirements

The Insertable Streams API used by the Publisher.setVideoMediaProcessorConnector(), Publisher.setAudioMediaProcessorConnector(), Subscriber.setVideoMediaProcessorConnector(), Publisher.applyVideoFilter(), and Subscriber.setAudioMediaProcessorConnector() methods is only supported in recent versions of Chrome, Electron, Opera, and Edge. It is not supported in other (non-Chromium-based) browsers or on iOS. You can check if the client supports this feature by calling the OT.hasMediaProcessorSupport() method.

Transformers and video filters require adequate processor support. Even in supported browsers, transformers may not be stable when background processes limit available processing resources. Use Chrome (desktop or Android) for optimal results.

For the best user experience, ensure laptop/desktop computers meet these minimum requirements:

• CPU — Intel Core i5 (with 4 cores or higher) or Apple Silicon M1 (or higher)
• 8G RAM (or more)
• Dedicated GPU (for example, NVIDIA)

Some operating systems may offer multiple battery settings or modes to conserve energy (for example, to extend laptop battery life) by throttling CPU performance. This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend setting your operating system to use best performance mode (or to to not use low-power mode) in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of whether the transformation is a basic, moderate, or advanced implementation.

More about transformers and sample code

For a more information, see the Vonage Media Processor library documentation.

The opentok-web-samples repo includes the following sample apps using the Vonage Media Processor library to apply transformations to streams published in OpenTok.js:

• Basic-Audio-Transformer — Shows how to apply an audio transformer using a web worker and a separate thread for the web worker.
• Basic-Video-Transformer — Shows how to apply a video transformer using a canvas, a web worker, and a separate thread for the web worker.
• Basic-Video-Zoom — Shows how to use a basic transformer to zoom the publisher.
• Moderate-Background-Blur — Shows how to apply a video transformer from the @vonage/ml-transformers module.

There are additional samples that use the Vonage Media Transformers library here.

Other user interface customization options

For other available options for customizing the user interface, see the Customizing the UI guide.