Suggestions

close search

Add Messaging, Voice, and Authentication to your apps with Vonage Communications APIs

Visit the Vonage API Developer Portal

Using the Vonage Media Processor library and applying media transformations

You can use pre-built transformers in the Vonage Media Processor library or create your own custom audio or video transformer to apply to published video.

You can use the OTPublisherKit.audioTransformers and OTPublisherKit.videoTransformers properties to apply audio and video transformers to a stream.

Important: Media transformations are not supported on all devices. See Client requirements.

The Vonage Video iOS SDK includes two ways to implement transformers:

Client requirements

The transformers from the Vonage Media Library are supported on Apple A11 Bionic chipset and above, on the following device-operating system combinations:

Device Supported operating system
iPhone 11 and above iOS 13 and above
iPad Air 4 and above iPadOS 14.1 and above
iPad mini 5th generation and above iPadOS 14 and above
iPad Pro 5th generation or higher iPadOS 14.5 or above
iPhone SE 2nd generation and above iOS 13.4 or above

Test on other devices to check for support.

Transformers require adequate processor support. Even on supported devices, transformers may not be stable when background processes limit available processing resources. The same limitations may apply with custom media transformers in addition to transformers from the Vonage Media Library.

iOS may throttle CPU performance to conserve energy (for example, to extend battery life). This may result in suboptimal transformer performance and introduce unwanted audio or video artifacts. We recommend disabling low-power mode in such cases.

Many video transformations (such as background blur) use segmentation to separate the speaker from the background. For best results, use proper lighting and a plain background. Insufficient lighting or complex backgrounds may cause video artifacts (for example, the speaker or a hat the speaker is wearing may get blurred along with the background).

You should perform benchmark tests on as many supported devices as possible, regardless of the transformation.

Applying a video transformer from the Vonage Media Library

Use the [OTVideoTransformer initWithName:properties:] method to create a video transformer that uses a named transformer from the Vonage Media Library.

Two transformers are supported:

Creating a custom video transformer

Create a class that implements the OTCustomVideoTransformer protocol. Implement the [OTCustomVideoTransformer transform:] method, applying a transformation to the OTVideoFrame object passed into the method. The [OTCustomVideoTransformer transform:] method is triggered for each video frame:

@interface CustomTransformer : NSObject <OTCustomVideoTransformer>

@end


@implementation CustomTransformer
- (void)transform:(nonnull OTVideoFrame *)videoFrame {
    OTPixelFormat pixelFormat = videoFrame.format.pixelFormat;
    int strides[] =
    {
        [videoFrame getPlaneStride:0],
        [videoFrame getPlaneStride:2],
        [videoFrame getPlaneStride:1]
    };
    uint8_t* planes[] =
    {
        [videoFrame getPlaneBinaryData:0],
        [videoFrame getPlaneBinaryData:2],
        [videoFrame getPlaneBinaryData:1]
    };
    [videoFrame convertInPlace:pixelFormat planes:planes strides:strides];
}
@end

Then set the OTPublisherKit.videoTransformers property to an array that includes the object that implements the OTCustomVideoTransformer interface:

CustomTransformer* myCustomTransformer;
myCustomTransformer = [CustomTransformer alloc];
OTVideoTransformer *invertColors = [[OTVideoTransformer alloc] initWithName:@"invertColors" transformer:border];

NSMutableArray * myVideoTransformers = [[NSMutableArray alloc] init];

[myVideoTransformers addObject:invertColors];

_publisher.videoTransformers = [[NSArray alloc] initWithArray:myVideoTransformers];

You can combine the Vonage Media library transformer (see the previous section) with custom transformers or apply multiple custom transformers by adding multiple PublisherKit.VideoTransformer objects to the ArrayList used for the OTPublisherKit.videoTransformers property.

Creating a custom audio transformer

Create a class that implements the OTCustomAudioTransformer protocol. Implement the [OTCustomAudioTransformer transform:] method, applying a transformation to the OTAudioData object passed into the method. The [OTCustomAudioTransformer transform:] method is triggered for each audio frame. The following example applies a simple amplitude limiter on the audio:

@interface CustomAudioTransformer : NSObject <OTCustomAudioTransformer>

@end

@implementation CustomAudioTransformer

- (void)transform:(nonnull OTAudioData *)audioData {
    int CROP_LIMIT = 32767 * 0.05;

    int samplesPerChannel = (int)audioData.numberOfSamples * (int)audioData.numberOfChannels;
    int16_t* buffer = audioData.sampleBuffer;

    for (int s = 0; s < samplesPerChannel; ++s) {
        int sample = buffer[s];
        NSLog(@"CustomAudioTransformer. Samples %d", sample);
        if (sample > CROP_LIMIT)
            buffer[s] = CROP_LIMIT;
        else if (sample < -CROP_LIMIT)
            buffer[s] = -CROP_LIMIT;
    }
}

@end

Then set the OTPublisherKit.audioTransformers property to an array that includes the object that implements the CustomAudioTransformer interface:

CustomAudioTransformer* customAudioTransformer;
ArrayList<PublisherKit.AudioTransformer> audioTransformers = new ArrayList<>();
customAudioTransformer = [CustomAudioTransformer alloc];
OTAudioTransformer *audiotrans = [[OTAudioTransformer alloc] initWithName:@"audio"
                                                              transformer:customAudioTransformer];

NSMutableArray * myAudioTransformers = [[NSMutableArray alloc] init];
[myAudioTransformers addObject:audiotrans];
_publisher.audioTransformers = [[NSArray alloc] initWithArray:myAudioTransformers];

You can apply multiple custom transformers by adding multiple OTPublisherKit.AudioTransformer objects to the ArrayList used for the OTPublisherKit.audioTransformers property.

Clearing video transformers for a publisher

To clear video transformers for a publisher, set the OTPublisherKit.videoTransformers property to an empty array.

_publisher.videoTransformers = [[NSArray alloc] init];

Clearing audio transformers for a publisher

To clear audio transformers for a publisher, set the OTPublisherKit.audioTransformers property to an empty array.

_publisher.audioTransformers = [[NSArray alloc] init];

Sample app

See this sample at the opentok-ios-sdk-samples repo on GitHub.