Suggestions

close search

Add Messaging, Voice, and Authentication to your apps with Vonage Communications APIs

Visit the Vonage API Developer Portal

Using the Vonage Media Processor library and applying media transformations

You can use pre-built transformers in the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video.

You can use the PublisherKit.setAudioTransformers() and PublisherKit.setVideoTransformers() methods to apply audio and video transformers to a stream.

Important:

Media transformations are not supported on all devices. See Client requirements.

The Vonage Video Android SDK includes two ways to implement transformers:

Client requirements

The transformers from the Vonage Media Library are supported on Android API Level of 23 or above, on the following devices:

Test on other devices to check for support.

Transformers require adequate processor support. Even on supported devices, transformers may not be stable when background processes limit available processing resources. The same limitations may apply with custom media transformers in addition to transformers from the Vonage Media Library.

Android may throttle CPU performance to conserve energy (for example, to extend battery life) by throttling CPU performance. This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend disabling battery saver or low-power mode, if it is an option, in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of the transformation.

Applying a video transformer from the Vonage Media Library

Use the PublisherKit.VideoTransformer(String name, String properties) constructor to create a video transformer that uses a named transformer from the Vonage Media Library.

Two transformers are supported:

Creating a custom video transformer

Create a class that implements the PublisherKit.CustomVideoTransformer interface. Implement the PublisherKit.CustomVideoTransformer.onTransform​(BaseVideoRenderer.Frame frame) method. The PublisherKit.CustomVideoTransformer.onTransform​(BaseVideoRenderer.Frame frame) method is triggered for each video frame. In the implementation of the method, apply a transformation to the frame object passed into the method:

public class MyCustomTransformer implements PublisherKit.CustomVideoTransformer {
    @Override
    public void onTransform(BaseVideoRenderer.Frame frame) {
        // Replace this with code to transform the frame:
        frame.convertInPlace(frame.getYplane(), frame.getVplane(), frame.getUplane(), frame.getYstride(), frame.getUvStride());
    }
}

Then pass the object that implements the PublisherKit.CustomVideoTransformer interface into the PublisherKit.setVideoTransformers() method:

  Publisher publisher = new Publisher.Builder(MainActivity.this).build();
  MyCustomTransformer transformer = new MyCustomTransformer();
  PublisherKit.VideoTransformer myCustomTransformer = publisher.new VideoTransformer("myTransformer", transformer);
  ArrayList<PublisherKit.VideoTransformer> videoTransformers = new ArrayList<>();
  videoTransformers.add(myCustomTransformer);
  publisher.setVideoTransformers(videoTransformers);

You can combine the Vonage Media library transformer (see the previous section) with custom transformers or apply multiple custom transformers by adding multiple PublisherKit.VideoTransformer objects to the ArrayList passed into the PublisherKit.setVideoTransformers() method.

adding multiple OTPublisherKit.VideoTransformer objects to the array used for the OTPublisherKit.videoTransformers property.

Creating a custom audio transformer

Create a class that implements the PublisherKit.CustomAudioTransformer interface. Implement the PublisherKit.CustomAudioTransformer.onTransform​(AudioData audioData) method. The PublisherKit.CustomAudioTransformer.onTransform​(AudioData audioData) method is triggered for each audio frame. In the implementation of the method, apply a transformation to the frame object passed into the method. The following example applies a simple amplitude limiter on the audio:

public class MyCustomAudioTransformer implements PublisherKit.CustomAudioTransformer {
    private short CROP_LIMIT = (short)(32767 * 0.05);
    @Override
    public void onTransform(AudioData audioData) {
        int samplesPerChannel = (int)audioData.getNumberOfSamples() * (int)audioData.getNumberOfChannels();
        ShortBuffer samples = audioData.getSampleBuffer().asShortBuffer();
        for (int s = 0; s < samplesPerChannel; ++s) {
            short sample = samples.get(s);
            if (sample > CROP_LIMIT)
                samples.put(s, CROP_LIMIT);
            else if (sample < -CROP_LIMIT)
                samples.put(s, (short)-CROP_LIMIT);
        }
    }
}

Then pass the object that implements the PublisherKit.CustomAudioTransformer interface into the PublisherKit.setAudioTransformers() method:

  Publisher publisher = new Publisher.Builder(MainActivity.this).build();
  MyCustomAudioTransformer transformer = new MyCustomAudioTransformer();
  PublisherKit.AudioTransformer myCustomTransformer = publisher.new AudioTransformer("myTransformer", transformer);
  ArrayList<PublisherKit.VideoTransformer> audioTransformers = new ArrayList<>();
  audioTransformers.add(myCustomTransformer);
  publisher.setAudioTransformers(audioTransformers);

You can apply multiple custom transformers by adding multiple PublisherKit.AudioTransformer objects to the ArrayList passed into the PublisherKit.setAudioTransformers() method.

Clearing video transformers for a publisher

To clear video transformers for a publisher, pass an empty ArrayList into into the PublisherKit.setVideoTransformers() method.

videoTransformers.clear();
mPublisher.setVideoTransformers(videoTransformers);

Clearing audio transformers for a publisher

To clear audio transformers for a publisher, pass an empty ArrayList into into the PublisherKit.setAudioTransformers() method.

audioTransformers.clear();
mPublisher.setAudioTransformers(videoTransformers);

Sample apps

See this Java-based sample and this Kotlin-based sample at the opentok-android-sdk-samples repo on GitHub.