Suggestions

close search

Add Messaging, Voice, and Authentication to your apps with Vonage Communications APIs

Visit the Vonage API Developer Portal

Using the Vonage Media Processor library and applying media transformations

You can use pre-built transformers in the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video.

You can use the otc_publisher_set_video_transformers() and otc_publisher_set_audio_transformers() functions to apply audio and video transformers to a published stream.

Important: Currently, only Apple silicon Macs are supported. See System requirements for more information.

The Vonage Video macOS SDK includes two ways to implement transformers:

System requirements

Currently, Vonage Media Library transformers are only supported on Apple silicon Macs.

Transformers require adequate processor support. Even on supported systems, transformers may not be stable when background processes limit available processing resources. The same limitations may apply with custom media transformers in addition to transformers from the Vonage Media Library.

macOS may throttle CPU performance to conserve energy (for example, to extend laptop battery life). This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend setting your system to not use low-power mode in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of the transformation.

Vonage Media Library integration

Due to significant increased size when integrating Vonage Media Library into SDK, from OpenTok SDK v2.27.3 the Media Transformers are available via the opt-in Vonage Media Library. This library needs to explicitly be added to the project.

The Vonage Media Library was initially embedded in OpenTok SDK. If your OpenTok SDK version is older than 2.27.3, move directly to Applying a video transformer from the Vonage Media Library and Applying an audio transformer from the Vonage Media Library.

The Vonage Media Library is available as the Pod "VonageClientSDKVideoMacOSTransformers", for use with CocoaPods.

If a call to otc_video_transformer_create() or otc_audio_transformer_create() is made without loading the library, the transformer returned will be null. You should use errno, a global variable that is set by system calls in the event of an error to indicate what went wrong. It's defined in the "" header. The error code for missing library is 0x0A000006.

Applying a video transformer from the Vonage Media Library

Use the otc_video_transformer_create() function to create a video transformer that uses a named transformer from the Vonage Media Library.

Two transformers are supported:

After you create the transformer, you can apply it to a publisher using the otc_publisher_set_video_transformers() function:

// Array of video transformers
otc_video_transformer *video_transformers[] = {
  background_blur
};

otc_publisher_set_video_transformers(publisher, video_transformers, 1);

The last parameter of otc_publisher_set_video_transformers() is the size of the transformers array. In this example we are applying one video transformer to the publisher. You can apply multiple transformers by adding multiple otc_video_transformer objects to the transformers array passed into otc_publisher_set_video_transformers().

Applying an audio transformer from the Vonage Media Library

Note: This is a beta feature.

Use the otc_audio_transformer_create() function to create an audio transformer that uses a named transformer from the Vonage Media Library.

One transformer is supported:

After you create the transformer, you can apply it to a publisher using the otc_publisher_set_audio_transformers() function:

// Array of audio transformers
otc_audio_transformer *audio_transformers[] = {
  noise_suppression
};

otc_publisher_set_audio_transformers(publisher, audio_transformers, 1);

The last parameter of otc_publisher_set_audio_transformers() is the size of the transformers array. In this example we are applying one audio transformer to the publisher. You can apply multiple transformers by adding multiple otc_audio_transformer objects to the transformers array passed into otc_publisher_set_audio_transformers().

Creating a custom video transformer

Use the otc_video_transformer_create() function to create a video transformer.

Here is a basic example:

void on_transform_black_white(void* user_data, struct otc_video_frame* frame)
{
    // implement transformer on the otc_video_frame data
}

otc_video_transformer *black_white_border = otc_video_transformer_create(
  OTC_MEDIA_TRANSFORMER_TYPE_CUSTOM,
  "blacknwhite",
  NULL,
  on_transform_black_white,
  NULL
);

After you create the transformer, you can apply it to a publisher using the otc_publisher_set_video_transformers() function:

// Array of video transformers
otc_video_transformer *video_transformers[] = {
  black_white_border
};

otc_publisher_set_video_transformers(publisher, video_transformers, 1);

Creating a custom audio transformer

Use the otc_audio_transformer_create() function to create an audio transformer.

Here is a basic example:

void on_transform_audio(void* user_data, struct otc_video_frame* frame)
{
    // implement transformer on the otc_audio_data audio data
}

otc_audio_transformer *lowpass_filter = otc_video_transformer_create(
  OTC_MEDIA_TRANSFORMER_TYPE_CUSTOM,
  "lowpassFilter",
  NULL,
  on_transform_audio,
  NULL
);

After you create the transformer, you can apply it to a publisher using the otc_publisher_set_audio_transformers() function:

// Array of audio transformers
otc_audio_transformer *audio_transformers[] = {
  lowpass_filter
};

otc_publisher_set_audio_transformers(publisher, audio_transformers, 1);

The last parameter of otc_publisher_set_audio_transformers() is the size of the transformers array. In this example we are applying one audio transformer to the publisher. You can apply multiple transformers by adding multiple otc_audio_transformer objects to the transformers array passed into otc_publisher_set_audio_transformers().

Clearing video transformers for a publisher

To clear video transformers for a publisher, pass an empty array into the otc_publisher_set_video_transformers() function.

otc_audio_transformer empty_array[] = {};
otc_publisher_set_video_transformers(publisher, empty_array, 0);

Use the otc_video_transformer_delete() function to delete an otc_audio_transformer instance:

otc_video_transformer_delete(lowpass_filter);

Clearing audio transformers for a publisher

To clear audio transformers for a publisher, pass an empty array into the otc_publisher_set_audio_transformers() function.

otc_video_transformer empty_array[] = {};
otc_publisher_set_audio_transformers(publisher, empty_array, 0);

Use the otc_audio_transformer_delete() function to delete an otc_audio_transformer instance:

otc_audio_transformer_delete(black_white_border);

Sample app

See this sample at the opentok-macos-sdk-samples repo on GitHub.