A flexible layout container for multiparty video chat

Flexible container for multiparty video chat webrtc

When anyone builds a multiparty video chat application they pretty quickly run into the issue of how to lay out the many different participants. You want everyone to be visible to everyone else and you also want video to take up as much space as possible, without wasted white-space. In the web you have the additional complexity of lots of different display sizes. Participants can be on a mobile device or tablet, or even just resizing their browser window so that they can see something else beside the video chat. For this reason you want a layout algorithm that is responsive.

We have been using a layout container we wrote internally for our applications (eg. meet.tokbox.com). We’ve been gradually iterating on it, adding more and more features and different ways to lay out participants. You can find the layout container at https://github.com/aullman/opentok-layout-js along with instructions for how to easily drop it into any OpenTok application.

You can also see this layout algorithm in action on meet.tokbox.com.

Fixed Aspect Ratio

Our first attempt at solving this problem was to build a layout container that makes all participants the same size. It just assumes that every participant is the same aspect ratio and then it calculates what the optimum layout is.

 

This algorithm looks at the aspect ratio of the first participant, in this case 16:9 and then makes every other participant that dimension. It then uses a brute force method to determine the best layout. It does this by trying different combinations of rows and columns and finds which one takes up the most space. So for example if there are 4 participants it will try 1 column and 4 rows, 2 columns and 2 rows, 3 columns and 2 rows or 4 columns and 1 row. It will then figure out which of these arrangements takes up the most space inside its container. This way it will actually work no matter what the dimensions of the container is.

A tall container results in 1 column and 4 rows.

Vertical layout container for multiparty video chat

A wide container results in 4 columns and 1 row.

Horizontal layout container for multiparty video chat

Most other sizes result in 2 columns and 2 rows.

2 x 2 layout container for multiparty video chat

You can get this layout by using the layout container and passing in fixedRatio: true like so:

var initLayoutContainer = require('opentok-layout-js');
 var layout = initLayoutContainer(document.getElementById("layout"), {

fixedRatio: true

});
 layout.layout();

Cropping

You probably noticed that you are still often left with some white-space around the videos. We can eliminate some of that white-space by cropping the videos. In a normal face-to-face conversation you don’t need the sides of the video. So if we crop those off we can have more of our layout be the video of our participants faces.

Respecting lots of aspect ratios

Sometimes not all participants have the same aspect ratio. With OpenTok now supporting Safari on iOS there are more and more participants joining from mobile devices. Also with screensharing there are any number of different aspect ratios possible.

There are also times when you don’t want to crop any of the video. For example, when sharing a screen or when there are multiple participants all in front of one camera and you want to be able to see all of them. In these situations you don’t want to crop any of the video because you could lose some important information.

To solve this let’s modify our layout algorithm to respect multiple different aspect ratios.

multiparty video chat flexible layout example

In this example you can see that there are several different aspect ratios being displayed in the container and all of them are being respected. In order to make this algorithm work we looked for some inspiration from photo layout algorithms like the Flickr justified layout. We couldn’t use the Flickr layout directly because it does not work quite the same way. It assumes you are going to be scrolling through a list of photos and so it doesn’t constrain the height. But we can borrow some of the concepts.

Again this is just a variation on the existing layout algorithms. It uses the same algorithm to determine how many rows and columns to use. Then when placing the elements, it just makes sure to respect their native aspect ratios. Then (borrowing from Flickr) if the whole height of the container isn’t taken up it does another pass through of each row to see whether they could be made taller to take up more of the width. This means that some rows can now be taller than others but it also means that we either take up the whole height or the whole width of the allocated space.

Putting it all together

We have put all of these layout algorithms together in meet.tokbox.com with different buttons allowing you to switch between different layouts. In the future it would be interesting to automatically detect when it makes sense to respect aspect ratios and when not to. We could detect faces in the video and if there were more than one make sure that you respect the aspect ratio so you don’t crop participants out.

If you want to get started building a multiparty video chat app with this layout algorithm, sign up here for a trial account with $10 of free credit.