Suggestions

close search

Add Messaging, Voice, and Authentication to your apps with Vonage Communications APIs

Visit the Vonage API Developer Portal

Using the Vonage Media Processor library and applying media transformations

You can use pre-built transformers in the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video.

You can use the Publisher.AudioTransformers and Publisher.VideoTransformers properties to apply audio and video transformers to a stream.

Important:

  • The Vonage Media Processor API is a beta feature.
  • Media transformations are not supported on all devices. See System requirements for more information.

The Vonage Video Windows SDK includes two ways to implement transformers:

System requirements

NVIDIA GPUs are recommended for optimal performance when using Vonage Media Library transformers.

Test on other configurations to check for support.

Transformers require adequate processor support. Even on supported devices, transformers may not be stable when background processes limit available processing resources. The same limitations may apply with custom media transformers in addition to transformers from the Vonage Media Library.

Windows may throttle CPU performance to conserve energy (for example, to extend battery life). This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend setting your operating system to use best performance mode (or to to not use low-power mode) in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of the transformation.

Applying a video transformer from the Vonage Media Library

Use the VideoTransformer(string name, string properties) method to create a video transformer that uses a named transformer from the Vonage Media Library.

Two transformers are supported:

Creating a custom video transformer

Create a class that implements the ICustomVideoTransformer interface. Implement the ICustomVideoTransformer.Transform() method, applying a transformation to the VideoFrame object passed into the method. The ICustomVideoTransformer.Transform method is triggered for each video frame:

public class MyCustomTransformer : IVideoTransformer
{
    public void Transform(VideoFrame frame)
    {
        // transformer implementation
        PixelFormat pixelFormat = frame.PixelFormat;
        int numberOfPlanes = frame.NumberOfPlanes;

        //invert U and V planes
        IntPtr[] planes = new IntPtr[] { frame.GetPlane(0), frame.GetPlane(2), frame.GetPlane(1) };
        int[] strides = new int[] { frame.GetPlaneStride(0), frame.GetPlaneStride(2), frame.GetPlaneStride(1) };

        frame.ConvertInPlace(pixelFormat, planes, strides);
    }
}

Then set the PublisherKit.VideoTransformers property to an array that includes the object that implements the OTCustomVideoTransformer interface:

MyCustomTransformer myCustomTransformer = new();
List<VideoTransformer> transformers = new ArrayList<VideoTransformer> 
{
  myCustomTransformer
};
publisher.VideoTransformers = transformers;

You can combine the Vonage Media library transformer (see the previous section) with custom transformers or apply multiple custom transformers by adding multiple VideoTransformer objects to the ArrayList used for the PublisherKit.VideoTransformers property.

Creating a custom audio transformer

Create a class that implements the ICustomAudioTransformer interface. Implement the ICustomAudioTransformer.Transform() method, applying a transformation to the AudioData object passed into the method. The CustomAudioTransformer.Transform() method is triggered for each audio frame. The following example applies a simple amplitude limiter on the audio:

public class MyCustomTransformer : IAudioTransformer
{
    private const short CROP_LIMIT = (short)(short.MaxValue * 0.5);

    public unsafe void Transform(AudioData frame)
    {
        // transformer implementation
        short* data = (short*)frame.SampleBuffer;
        int samplesXChannels = (int)frame.NumberOfSamples * (int)frame.NumberOfChannels;
        for (int s = 0; s < samplesXChannels; ++s)
        {
            if (data[s] > CROP_LIMIT)
                data[s] = CROP_LIMIT;
            else if (data[s] < -CROP_LIMIT)
                data[s] = -CROP_LIMIT;
        }
    }
}

Then set the OTPublisherKit.AudioTransformers property to an array that includes the object that implements the CustomAudioTransformer interface:

MyCustomTransformer myCustomTransformer = new();
List<VideoTransformer> transformers = new ArrayList<VideoTransformer> 
{
  myCustomTransformer
};
publisher.AudioTransformers = transformers;

You can apply multiple custom transformers by adding multiple OTPublisherKit.AudioTransformer objects to the ArrayList used for the OTPublisherKit.AudioTransformers property.

Clearing video transformers for a publisher

To clear video transformers for a publisher, set the Publisher.VideoTransformers property to an empty array.

publisher.VideoTransformers =  new ArrayList<VideoTransformer> {};

Clearing audio transformers for a publisher

To clear audio transformers for a publisher, set the Publisher.AudioTransformers property to an empty array.

publisher.AudioTransformers =  new ArrayList<AudioTransformer> {};

Sample app

See this sample at the opentok-windows-sdk-samples repo on GitHub.