Deep dive into the ‘Video Driver’

MOBILECBLOGOpenTok Mobile SDKs, Revision 2: The Video Driver

In the latest versions of the OpenTok SDKs for iOS and Android, everything is new. We found an opportunity to learn from the lessons of the past two years, and seized it to conduct an overhaul of the architecture of the client. The 2.2.0 release of the iOS and Android SDKs marks the second major revision of the implementation of the OpenTok Mobile SDKs. This post highlights one of the many new features of the 2.2.0 SDKs, about which we are feeling particularly excited: the “Video Driver”. Although the feature exists with parity in both platforms, today we’ll focus on the iOS-variant of the new API.

As you may know, every mobile platform has a unique interface for allowing application developers access to device hardware, specifically the camera. Similarly, each platform offers unique and powerful APIs for managing the view hierarchy that the end user sees. The “Video Driver” refers simply to a set of simple interfaces in the OpenTok SDK that allow for complete customization of both input and output of video to the client SDK, and therefore the whole OpenTok ecosystem.

The inspiration for this change came largely as a result of what we have been seeing our talented partners working on and asking for. There have been requests for HD streaming over a local network, to briefly pause the video and snap a photo, to add filter effects implemented on the GPU, and many others. We decided that attempting to simultaneously support all of these features in one closed-source SDK would not be as easy as giving the necessary tools to the developers asking for them, in order to implement their use case without waiting for us.

The Details

Diving into the details with an iOS-specific point of view, we introduce two new properties on the newly refactored OTPublisherKit and OTSubscriberKit, videoCapture and videoRender. These properties are instance references to any Objective-C class that implements the OTVideoCapture and/or OTVideoRender protocols. By setting these fields to classes that you implement, the code for capturing, pre- and post-processing, and rendering video is now available for you to modify.

Under the hood, the OpenTok iOS and Android SDKs are built on top of the WebRTC reference implementation, generously provided by our friends at Google. We use a virtualization of the video capture and rendering interfaces used by the WebRTC engine, and proxy all of the relevant function calls in and out of the Objective-COTVideoCapture and OTVideoRender instances. From here, you can define your own video sources, manipulate received video prior to rendering, or implement any other customization idea you might have.

An Example

The Video Drivers make a lofty promise that you can put nearly any video source into the SDK and be ready to consume it on all other platforms supported by OpenTok. To support this claim, I have written a sample app that uses the latest version of the OpenTok iOS SDK to publish video from a quicktime .mov file, encoded in H.264. Using the iOS AVFoundation API, the video capturer is able to decode the movie at a reasonable frame rate, and push that decoded video into the OpenTok SDK, for publishing. You will notice that any of the other demo apps for OpenTok are able to subscribe to this stream without modification. For the single cost of implementing a custom video source in iOS, we are able to consume this new video source in all other platforms supported by OpenTok.

Next Steps

Although this feature is currently only focused on the video interface, we are investigating the feasibility of a similar set of APIs to support customization of the audio input/output system as well. We hope to have something to share later this summer.

In addition to the application described above, we have also released some examples under the Apache 2.0 License that should be enough to easily get you started. Clone and tinker! If you are able, consider sharing the results of your hacking with us; we are very curious to see what our partners will do next with this new feature.