What to do with all these (get)Stats?

tokbox-inc_markMany of our partners eventually find themselves asking how to tell whether their users tend to experience good quality while using the OpenTok Platform. As time has taught us, this can be a difficult question to answer. The most common source of complaints stem from underwhelming audio/video (A/V) quality between endpoints. These complaints are nearly always rooted in issues with performance of the endpoint network. The correlation between network performance and A/V quality has been accepted as an industry standard. In fact, we have built tools to expose network performance data, as a proxy indicator of subjective quality. While objective data about a network may be easy to collect, it is much more difficult to assign a number to represent the quality of experience that a user subjectively experiences.

Nonetheless, one such number is the mean opinion score (MOS). MOS is the basis for representing the opinion of an end-user consuming audio and video content. By gathering opinions from real users consuming content in a controlled environment, MOS tries to represent subjective quality in a numerical form. OpenTok, however, is a WebRTC platform supporting a large and diverse population of endpoints; we do not qualify as a “controlled environment”, nor do we provide means of gathering actual user scores. Still, we can capture some of the spirit of the score with analysis of network performance data.

Developers seeking more visibility into the statistics for their WebRTC sessions have many tools at their disposal. At TokBox, we have thus far yielded to the recommendations outlined in our network test. The network test exposes a subset of WebRTC’s internal statistics API. WebRTC getStats offers a trove of interesting data. Yet, the original question remains: partners asking us about A/V quality get an answer about network performance. “That’s not what I asked!”, they might say. They’re right. What’s missing is a definition of the relationship between these two. How does one take information about network health and surmise how a user will feel about the quality of their experience?

To answer this question, we are experimenting with a supplemental component that works alongside our Client SDKs, the OpenTok MOS estimator. In this module, we have borrowed from an approach offered by the ITU-T, with modifications appropriate for our own environment. The module works by analyzing the statistics exposed via the subscriber network stats handler. This analysis produces an on-demand estimate of the score an average user would have given the stream. The estimate combines heuristics using network metrics of both audio and video channels. Similar to the Session Inspector’s ‘session quality’ estimate, this data is available in real-time, generated on the endpoint, rather than an offline data process on the server.

The MOS estimator module is an experimental project seeking feedback from our developers. Give it a try and let us know what you think! Future iteration of the project will be guided by the feedback we receive in this early stage.