Suggestions

close search

Add Messaging, Voice, and Authentication to your apps with Vonage Communications APIs

Visit the Vonage API Developer Portal

Using the Vonage Media Processor library and applying media transformations

You can use pre-built transformers in the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video.

You can use the PublisherKit.setAudioTransformers() and PublisherKit.setVideoTransformers() methods to apply audio and video transformers to a stream.

Important:

Media transformations are not supported on all devices. See Client requirements.

The Vonage Video Android SDK includes two ways to implement transformers:

Client requirements

The transformers from the Vonage Media Library are supported on Android API Level of 23 or above, on the following devices:

Test on other devices to check for support.

Transformers require adequate processor support. Even on supported devices, transformers may not be stable when background processes limit available processing resources. The same limitations may apply with custom media transformers in addition to transformers from the Vonage Media Library.

Android may throttle CPU performance to conserve energy (for example, to extend battery life) by throttling CPU performance. This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend disabling battery saver or low-power mode, if it is an option, in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of the transformation.

Vonage Media Library integration

Due to significant increased size when integrating Vonage Media Library into SDK, from OpenTok SDK v2.27.2 the Media Transformers are available via the opt-in Vonage Media Library. This library needs to explicitly be added to the project.

The Vonage Media Library was initially embedded in OpenTok SDK. If your OpenTok SDK version is older than 2.27.2, move directly to Applying a video transformer from the Vonage Media Library and Applying an audio transformer from the Vonage Media Library.

A Maven version is available at https://central.sonatype.com/artifact/com.vonage/client-sdk-video-transformers. The artifact ID is "client-sdk-video-transformers".

Modify the app's build.gradle file and add the following code snippet to the dependencies section:

implementation 'com.vonage:client-sdk-video-transformers:2.27.2'

If a call to PublisherKit.VideoTransformer(String name, String properties) or PublisherKit.AudioTransformer(String name, String properties) is made without loading the library, the transformer returned will be null. An exception will be raised with the following error code 0x0A000006 - OTC_MEDIA_TRANSFORMER_OPENTOK_TRANSFORMERS_LIBRARY_NOT_LOADED.

Applying a video transformer from the Vonage Media Library

Use the PublisherKit.VideoTransformer(String name, String properties) constructor to create a video transformer that uses a named transformer from the Vonage Media Library.

Two transformers are supported:

Applying an audio transformer from the Vonage Media Library

Note: This is a beta feature.

Use the PublisherKit.AudioTransformer(String name, String properties) constructor to create an audio transformer that uses a named transformer from the Vonage Media Library.

One transformer is supported:

Creating a custom video transformer

Create a class that implements the PublisherKit.CustomVideoTransformer interface. Implement the PublisherKit.CustomVideoTransformer.onTransform​(BaseVideoRenderer.Frame frame) method. The PublisherKit.CustomVideoTransformer.onTransform​(BaseVideoRenderer.Frame frame) method is triggered for each video frame. In the implementation of the method, apply a transformation to the frame object passed into the method:

public class MyCustomTransformer implements PublisherKit.CustomVideoTransformer {
    @Override
    public void onTransform(BaseVideoRenderer.Frame frame) {
        // Replace this with code to transform the frame:
        frame.convertInPlace(frame.getYplane(), frame.getVplane(), frame.getUplane(), frame.getYstride(), frame.getUvStride());
    }
}

Then pass the object that implements the PublisherKit.CustomVideoTransformer interface into the PublisherKit.setVideoTransformers() method:

  Publisher publisher = new Publisher.Builder(MainActivity.this).build();
  MyCustomTransformer transformer = new MyCustomTransformer();
  PublisherKit.VideoTransformer myCustomTransformer = publisher.new VideoTransformer("myTransformer", transformer);
  ArrayList<PublisherKit.VideoTransformer> videoTransformers = new ArrayList<>();
  videoTransformers.add(myCustomTransformer);
  publisher.setVideoTransformers(videoTransformers);

You can combine the Vonage Media library transformer (see the previous section) with custom transformers or apply multiple custom transformers by adding multiple PublisherKit.VideoTransformer objects to the ArrayList passed into the PublisherKit.setVideoTransformers() method.

adding multiple OTPublisherKit.VideoTransformer objects to the array used for the OTPublisherKit.videoTransformers property.

Creating a custom audio transformer

Create a class that implements the PublisherKit.CustomAudioTransformer interface. Implement the PublisherKit.CustomAudioTransformer.onTransform​(AudioData audioData) method. The PublisherKit.CustomAudioTransformer.onTransform​(AudioData audioData) method is triggered for each audio frame. In the implementation of the method, apply a transformation to the frame object passed into the method. The following example applies a simple amplitude limiter on the audio:

public class MyCustomAudioTransformer implements PublisherKit.CustomAudioTransformer {
    private short CROP_LIMIT = (short)(32767 * 0.05);
    @Override
    public void onTransform(AudioData audioData) {
        int samplesPerChannel = (int)audioData.getNumberOfSamples() * (int)audioData.getNumberOfChannels();
        ShortBuffer samples = audioData.getSampleBuffer().asShortBuffer();
        for (int s = 0; s < samplesPerChannel; ++s) {
            short sample = samples.get(s);
            if (sample > CROP_LIMIT)
                samples.put(s, CROP_LIMIT);
            else if (sample < -CROP_LIMIT)
                samples.put(s, (short)-CROP_LIMIT);
        }
    }
}

Then pass the object that implements the PublisherKit.CustomAudioTransformer interface into the PublisherKit.setAudioTransformers() method:

  Publisher publisher = new Publisher.Builder(MainActivity.this).build();
  MyCustomAudioTransformer transformer = new MyCustomAudioTransformer();
  PublisherKit.AudioTransformer myCustomTransformer = publisher.new AudioTransformer("myTransformer", transformer);
  ArrayList<PublisherKit.VideoTransformer> audioTransformers = new ArrayList<>();
  audioTransformers.add(myCustomTransformer);
  publisher.setAudioTransformers(audioTransformers);

You can apply multiple custom transformers by adding multiple PublisherKit.AudioTransformer objects to the ArrayList passed into the PublisherKit.setAudioTransformers() method.

Clearing video transformers for a publisher

To clear video transformers for a publisher, pass an empty ArrayList into into the PublisherKit.setVideoTransformers() method.

videoTransformers.clear();
mPublisher.setVideoTransformers(videoTransformers);

Clearing audio transformers for a publisher

To clear audio transformers for a publisher, pass an empty ArrayList into into the PublisherKit.setAudioTransformers() method.

audioTransformers.clear();
mPublisher.setAudioTransformers(videoTransformers);

Sample apps

See this Java-based sample and this Kotlin-based sample at the opentok-android-sdk-samples repo on GitHub.