Using the Vonage Media Processor library and applying media transformations

You can use pre-built transformers in the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video.

You can use the OTPublisherKit.audioTransformers and OTPublisherKit.videoTransformers properties to apply audio and video transformers to a stream.

Important: Media transformations are not supported on all devices. See Client requirements.

The Vonage Video iOS SDK includes two ways to implement transformers:

Moderate — For video, you can apply the background blur and background replacement video transformers included in the Vonage Media Library. See Applying a video transformer from the Vonage Media Library. For audio, you can apply the noise suppression audio transformer included in the Vonage Media Library. See Applying an audio transformer from the Vonage Media Library.
Advanced — You can create your own custom video transformers and custom audio transformers.

Client requirements

The transformers from the Vonage Media Library are supported on Apple A11 Bionic chipset and above, on the following device-operating system combinations:

Device	Supported operating system
iPhone 11 and above	iOS 13 and above
iPad Air 4 and above	iPadOS 14.1 and above
iPad mini 5th generation and above	iPadOS 14 and above
iPad Pro 5th generation or higher	iPadOS 14.5 or above
iPhone SE 2nd generation and above	iOS 13.4 or above

Test on other devices to check for support.

Transformers require adequate processor support. Even on supported devices, transformers may not be stable when background processes limit available processing resources. The same limitations may apply with custom media transformers in addition to transformers from the Vonage Media Library.

iOS may throttle CPU performance to conserve energy (for example, to extend battery life). This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend disabling low-power mode in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of the transformation.

Vonage Media Library integration

Due to significant increased size when integrating Vonage Media Library into SDK, from OpenTok SDK v2.27.4 the Media Transformers are available via the opt-in Vonage Media library. This library needs to explicitly be added to the project.

The Vonage Media Library was initially embedded in OpenTok SDK. If your OpenTok SDK version is older than 2.27.4, move directly to Applying a video transformer from the Vonage Media Library and Applying an audio transformer from the Vonage Media Library.

The Vonage Media Library is available as the Pod "VonageClientSDKVideoTransformers", for use with CocoaPods.

If a call to [OTVideoTransformer initWithName:properties:] or [OTAudioTransformer initWithName:properties:] is made without loading the library, the transformer returned will be null. An exception will be raised with the following error code 0x0A000006 - OTC_MEDIA_TRANSFORMER_OPENTOK_TRANSFORMERS_LIBRARY_NOT_LOADED.

Applying a video transformer from the Vonage Media Library

Use the [OTVideoTransformer initWithName:properties:] method to create a video transformer that uses a named transformer from the Vonage Media Library.

Two transformers are supported:

Background blur. For this filter, set the name parameter to "BackgroundBlur". And set a properties parameter to a JSON string. The format of the JSON is "{"radius":"None"}". Valid values for the radius property are "None", "High", and "Low". If you set the radius property to "Custom", add a custom_radius property to the JSON string: "{"radius":"Custom","custom_radius":"value"}" (where custom_radius is a positive integer defining the blur radius).
```
var backgroundBlur = new OTVideoTransformer()
backgroundBlur.init(name: "BackgroundBlur", properties:"{\"radius\":\"High\"}")
var myVideoTransformers = [backgroundBlur]
publisher.videoTransformers = myVideoTransformers
```
Background replacement. For this filter, set the name parameter to "BackgroundReplacement". And set a properties parameter to a JSON string. The format of the JSON is "{"image_file_path":"path/to/image"}", where image_file_path is the absolute file path of a local image to use as virtual background. Supported image formats are PNG and JPEG.
```
var backgroundReplacement = new OTVideoTransformer()
backgroundReplacement.init(name: "BackgroundReplacement", properties:"{\"image_file_path\":\"path-to-image\"}")
var myVideoTransformers = [backgroundReplacement]
publisher.videoTransformers = myVideoTransformers
```

Applying an audio transformer from the Vonage Media Library

Note: This is a beta feature.

Use the [OTAudioTransformer initWithName:properties:] method to create a video transformer that uses a named transformer from the Vonage Media Library.

One transformer is supported:

Noise Suppression. For this filter, set the name parameter to "NoiseSuppression". Set the properties parameter to a JSON string defining properties for the transformer. For the noise suppression transformer, this JSON does not include any property at the moment. Set it to an empty string "".
```
var noiseSuppression = new OTAudioTransformer()
noiseSuppression.init(name: "NoiseSuppression", properties:"")
var myAudioTransformers = [noiseSuppression]
publisher.audioTransformers = myAudioTransformers
```

Creating a custom video transformer

Create a class that implements the OTCustomVideoTransformer protocol. Implement the OTCustomVideoTransformer.transform() method, applying a transformation to the OTVideoFrame object passed into the method. The OTCustomVideoTransformer.transform() method is triggered for each video frame.

Then set the OTPublisherKit.videoTransformers property to an array that includes the object that implements the OTCustomVideoTransformer interface.

You can combine the Vonage Media library transformer (see the previous section) with custom transformers or apply multiple custom transformers by adding multiple OTPublisherKit.VideoTransformer objects to the array used for the OTPublisherKit.videoTransformers property.

Creating a custom audio transformer

Create a class that implements the OTCustomAudioTransformer protocol. Implement the OTCustomAudioTransformer.transform() method, applying a transformation to the OTAudioData object passed into the method. The OTCustomAudioTransformer.transform() method is triggered for each audio frame.

Then set the OTPublisherKit.audioTransformers property to an array that includes the object that implements the CustomAudioTransformer interface:

You can apply multiple custom transformers by adding multiple PublisherKit.AudioTransformer objects to the array used for the OTPublisherKit.audioTransformers property.

Clearing video transformers for a publisher

To clear video transformers for a publisher, set the OTPublisherKit.videoTransformers property to an empty array.

publisher.videoTransformers = [];

Clearing audio transformers for a publisher

To clear audio transformers for a publisher, set the OTPublisherKit.audioTransformers property to an empty array.

publisher.audioTransformers = [];

Sample app

See this sample at the opentok-ios-sdk-samples-swift repo on GitHub.

Could we improve this page? Let us know what's missing.

Feedback

Suggestions