2017-11-12

Still No Time To Wait

The second edition of No Time To Wait was a success. It's a conference where archivists meet the developers of the software and formats they use or might use.

Since last year a lot has changed. We were advocating people to use Matroska and FFV1 because they meet their needs in a very good manner. This year we heard many stories of people who actually did the move and are happy about it. Reto Kromer even made a presentation explaining he actually does the conversation on the fly when transferring between tapes.

One presentation particularly caught my attention: the look for the perfect player by Aghate Jarczyk (University Of The Arts, Bern). Working daily on improving VLC that's certainly something we want to do and make every user happy, even in a professional way not just for casual file playback. It turns out many of the issues mentioned, preventing the switch from QuickTime Pro 7, are already solved. Here's the list:
  • display metadata from the file. It's there with Ctrl+i (Cmd+i on macOS I suppose) in the metadata tab. It's not at the MediaInfo level but useful nonetheless. It's also refreshed during playback so if you switch between formats midstream you can see it there. It won't tell you if the data come from the codec or the container, it's aggregated by the player. If you really need that feature file an issue in Trac
  • the list of codecs used for playback. It's also available in a tab when you do Ctrl+m (Cmd+m) and can be refreshed during playback (for example with streams that have mixed interlacing). It's probably more an issue with QuickTime Pro where there might be plug-ins in the system you're not aware of. It's much less likely with VLC. It doesn't load modules compiled for an older version and usually doesn't have extra modules coming from third parties.
  • added black borders when opening a video. This is surprising as that's not the behavior on Windows or the Qt interface in general. It may be a mac version specific behavior or an option to use the "fit screen" aspect ratio. A reset of the preferences should fix that.
  • Can we display timecodes? It's technically possible, we decode them but they are not frame accurate because of our internal clock design. To be accurate it needs a redesign that we are going to do for VLC 4.0. And that version will take less time to be done that it took to do 3.0.
  • To go back one frame at a time: it's possible to use a LUA script to do that, see: https://forum.videolan.org/viewtopic.php?p=462937#p462937
  • Émilie Magnin who hosted the Format Implementation panel also mentioned the possibility of launching the player more than once at a time. It is an option that's possible on Windows and Linux but apparently it takes a little more work on macOS. You'll need an external AppleScript to do that.

There were a lot of talks about open source in general as well. Everyone is pretty much sold on the idea now and how crucial it is for archivists that they can rely on code that can be reused and tweak for decades. A guaranteed no proper software can offer. An interesting twist is that sometimes the software to play the content has to be archived as well. Usually when using proprietary solutions that might (will) die over time. Another good reason not to use that in the first place.

Some people are still not using Matroska. One of the reasons, which make sense in their context, is that it's not (yet) a standard. That is endorsed by a standards body you trust. As pointed out by Ethan Gates that level of trust may vary and totally arbitrary. For example some still use AVI even though the specifications has never gone through any of the common standards bodies (AFAIK) . This is on us, and particularly me, to make the standardization of Matroska happen and finish the work that is already on the way. The main issue being that we all do that on our free time, so we may look for funding to be done sooner rather than later. A crowdfunding was mentioned. We're going to discuss how we can make this happen (suggestions welcome). That would be a first for Matroska as we never received money for the project (apart from around 200$ of PayPal donations over 15 years).

A big thanks to all the organizers and especially Dave Rice and Jerome Martinez and to Michael Loebenstein of the Austrian Film Museum for a great venue.
My apologies to Kieran O'Leary, I promised I'd bring the VLC hat on the second day and then I forgot.

2017-11-05

Matroska versus fragmented MP4

In an earlier post I was worried that Matroska might have lost its edge compared to MP4 when it comes to overhead size. So I dug a little deeper with some real life samples from no other than Apple to see if what we could improve. It turns out that Matroska is still the best when it comes to overhead (and just about everything else).

Here are some comparison from the Apple adaptive streaming sample page. I don't how they compare to real life files, maybe they are improperly muxed but the results are always in the favor of Matroska even when large padding and tags are left in the file.

Advanced Stream

The lowest bitrate video is 530kbps according to the manifest and 369 kbps according to MediaInfo. Then I remuxed it with mkvmerge. Then go through mkclean and here are the results:
  • 27 672 619 original fMP4 with H264
  • 27 449 794 mkvmerge with default options (we win already)
  • 27 447 068 mkclean with default options
  • 27 439 197 mkclean with --live
  • 27 357 090 mkclean with --remux --optimize
  • 27 349 220 mkclean with --remux --optimize --live
The normal usage when preparing with for streaming would be mkclean with --remux --optimize and that gives a 1.1% size advantage that could be better used for the codec. That stream even includes checksums, tags and is fully seekable.

Advanced Stream HEVC

Here Matroska doesn't have the advantage of using Header Compression as with H264, which saves 3 bytes per frame as they are always the same. The 145 kbps is also closer to the limit of everyday files.
  • 11 492 052 original fMP4 with HEVC
  • 11 410 786 mkvmerge with default options (we win already)
  • 11 407 257 mkclean with default options
  • 11 371 266 mkclean with --remux --optimize
But we're still 1.1% percent smaller than the same content in fragmented MP4.

Advanced Stream H264

This is the same as above but in H264 format, so we get to use header compression.
  • 10 663 861 original fMP4 with H264
  • 10 558 115 mkvmerge with default options (we win already)
  • 10 554 002 mkclean with default options
  • 10 498 886 mkclean with --remux --optimize

Conclusion

So Matroska is still the best when it comes to overhead and still keeps all its advantages. Only very very small fine tuned files might actually go in favor of fMP4. I'd really like to have such real life samples if you have some.

2017-10-25

FOMS and Demuxed

On October 3rd and 4th I attended the FOMS workshop in San Francisco then Demuxed on the 5th. There were a lot discussions about video, mostly distribution and playback via web browsers. It was interesting as it’s a different take from my daily work on VLC. Vendors developed very specific techniques targeted at their particular use case, often to get around bogus past decisions or competing solutions.

As Matroska (used by WebM) was primarily designed for playback over network connections (that were slow at the time of design) it was interesting to see if we can cover all these use cases in an optimal way. It is especially important to remain relevant as the AV1 codec is coming soon. It seems to be getting huge traction already and might end up being the main codec everyone uses in the years to come, especially for web videos. Even though it’s targeted at high quality it seems people want to use it ASAP for very low bitrates. I suppose the quality gain for the same bitrate is even more significant there.
FOMS 2017

Two subjects particularly caught my attention in terms of challenges for the container.

Extremely low latency

It seems a lot of companies are looking at reducing the time between the moment something happens and the time it’s displayed on your screen. In the age of Twitter it sucks to see a goal or other (e)sport event happening on your feed before you actually get to see it. In games it also means the people streaming the game can interact in real time with what people are seeing.

Due to the nature of video encoding you can hardly get lower than one frame delay (17 ms in 60fps) and the transmission latency (10 ms if you have an incredible ping). But right now the target is more around a few second or a single second. One of the issue here is how adaptive streaming is currently used. It encodes a bunch of frames and then tell the user it’s available (in various bitrates). That’s because the container needs to know all the frames it contains before it can actually be used. So they wrap about 1s of video to have a minimum latency.

Matroska and EBML have a mode called live streaming. It allows writing frames as they come in and never rewriting the beginning of the file to tell how much data it contains or where the data actually are. So you can start reading the file even while it’s being written. Many years ago GStreamer was used to stream conferences that way (without even an actual file being written) and that’s how VLC 3.0 sends videos to the Chromecast. This is also how most Matroska/WebM muxers work. They write in “live streaming” mode by default: they write a special “unknown” value in the length field and when the size is known this value is overwritten. So a streamer can create files on the fly that people could start reading. And when the file is done write the proper values so that the next people reading from that file actually get proper values they can use to seek.

I hope the web people get a look at this as it would allow to go way below the 1s latency target they currently have. It would also work for adaptive streaming as you still get Clusters that you can cut in many parts on a CDN as currently done for WebM. This solution is already compatible with most Matroska/WebM readers. It’s been in our basic tests suite for at least 7 years.

CMAF

I learned the existence of a new MP4 variant called CMAF (Common Media Application Format). It’s an ISOBMFF profile based on Fragmented MP4 (fMP4). It was developed by Microsoft and Apple. The goal was to use a similar format between DASH and HLS to reduce the cost of storage on CDNs and get better caching. In the end it might not be of much use because the different vendors don’t support the same DRM systems and so at least 2 variants of the same content will still be needed.

This is an interesting challenge for Matroska as with AV1 coming there will be a battle for what container to use to distribute videos. It’s not the main adoption issue anymore though. For example Apple only supported HLS with MPEG TS until iOS10 so many Javascript frameworks remux the incoming fragmented fMP4 to TS on the fly and feed that to iOS.

Regular MP4 files were not meant to be good for progressive downloading, nor fragmented playback needed for adaptive streaming as the index was needed for playback and so needed to be loaded beforehand and not necessarily at the front of the file. The overhead (the amount of data the container adds on top of the actual codec data) wasn’t not great either. So far it was a key advantage towards Matroska/WebM as these were two of the main criteria when the format was designed 15 years ago. There were cases where MP4 could be smaller by at the price of using compressed headers. The situation changes with fMP4 and CMAF. In fact the overhead is slightly lower than Matroska/WebM. And that’s pretty much the only advantage it has over Matroska.

On a 25 MB file of 44 kbps (where overhead is really hurting) the difference between the fMP4 file and one passed through mkclean is 77 KB or 0.3%. It may seem peanuts, especially for such a small bitrate, but I think Matroska should do better.

Looking at the fMP4 file, it seems the frames are all packed in a blob and the boundaries between each frame in a separate blob (‘trun’ box). And that’s about it. It must only work with fixed frame rates and probably allows no frame drop. But that’s efficient for the use case of web video over CDNs that were encoded and muxed for that special purpose. There’s hardly any overhead apart from the regular track header.

One way Matroska could be improved for such a case would be to allow frame lacing for video. It is already used heavily for audio to reduce the overhead and since audio doesn’t need a timestamp for each block, the sampling rate is enough (except when there are drops during recording, in which case lacing is not used). We could allow lacing video frames as long as the default duration for the track is set (similar to a frame rate) and that each frame has the same characteristics in the Matroska Block, especially the keyframe flag. So keyframes would stand alone and many other video frames could be laced to reduce the overhead, the same way it’s done for audio. With such a small bitrate it could make a significant difference. On higher bitrates not really, but the overhead difference between fMP4 and Matroska is probably small if not at the advantage of Matroska in this case (thanks to header compression).

I will submit the proposal to the CELLAR workgroup of the IETF (Internet Engineering Task Force), a group that is currently working on specifying properly EBML, Matroska but also FFv1 and FLAC. This is not a big change, it’s just something that we didn’t allow before. And because it’s already in use for audio in just about every Matroska/WebM file that exists, the parsing already exists in current players and may work out of the box with video frame lacing. It doesn’t add any new element.

The advantages of Matroska over MP4 remain the same for fMP4.
Demuxed 2017

TL;DR

Matroska has a lot to offer to web distribution, like one frame latency at scale not possible with ISOBMFF formats, doesn’t require new designs for current and future use cases and is the most open and free solution.

2014-07-08

Android Versions by The Millions

Last month I blogged about the missing market share progress graph that Google used to publish. And also provided some extra graphs based on the collected data, with much interesting facts to extract from them. I updated these data/graphs with the latest Play Store stats. But there was something significant missing and what really matters for app developers: how many actual people are on the versions, in millions.

To get this information, we need to know how many active users there are. We never had this information until Google I/O 2014 where Sundar Pichai announced that there are 1 billion users active on the Play Store at that time. The other regular piece of information I could find was the quarterly worldwide shipment of Android devices since 2010 up to April 2014. Using this data and a lifetime of 18 months for each shipped device, I managed to reconstruct the progression in millions of active devices and get to the 1 billion number we have now. The math may not be all sound, but in the end the growth is pretty linear from the beginning and the milestones from each IO keynote seem to coincide (number of activation vs active device). All these data are added to the original spreadsheet in the page "Active Users".

Given these grossly accurate data I could build the graph of each version progression in millions of users, not just in market share.



The road to 1 billion has been pretty linear. The last quarter global shipment are unknown yet. And they also take in account an explosive growth in China where the Play Store is not available.

Another interesting graph, and the real information I was looking for is how many users are currently using each API.

To compare with the original one



You can see the story is very different.
  • KitKat was the fastest growing platform in recent Android history and it's showing even more by million of users. If the growth continue like that it may reach 300 million users in the next 3/4 months. Before Android L comes out.
  • Although API v16 has been slowly declining for a while, the platform was still growing and so the number of users was still growing, the market share alone is not a good indicator. The number of users are in free fall though, despite still being the dominant API.
  • API v10 has still 140 millions of active users, these are not Chinese users.
  • There were never more Ice Cream Sandwich users than Gingerbread users. It topped 200 millions, compared to 300 millions for Gingerbread.
  • The growth of API v17 is more significant when taking in account the amount of users, it's still growing well.
  • On the other hand, API v18 is still not very meaningful in the number of users.
  • There are still 7 million users using v8.
The good news is that supporting v17 and up gives you a very good amount of users. But failing to support v10 or v15 gives 250 million more potential users to your competitors.

We can assume that in the next 3 or 4 months v10 and v15 will drop below 100 million users each. And v19 should reach 250 to 300 million users and might have more users than v17.

2014-06-14

Android Version Distribution

A long time ago, Google used to provide a graph of the evolution of version distribution with their monthly update of the Android Dashboards. I missed this graph ever since because it was giving a good indication of where we're at and what to expect in the coming months. This is especially important when planning a new project, to know what most users will have when your product ships.

After digging in the Internet Archive Wayback Machine, I reconstituted all the data published by the Play Store since December 2009 up to now (June 2014). The result is this bare spreadsheet table with a link to the source I used for each line.


Reconstructed Play Store Statistics
With the data in hand, it's now easy to create the graph that Google used to publish. But as soon as you see it (see in the Google Doc spreadsheet, after the numbers), you realize it makes more sens to group the Android versions by their codename (Froyo, Gingerbread, Honeycomb, Ice Cream Sandwich, Jellybean and Kit-Kat). This gives this chart:



This graph looks familiar, just with more versions in it. Each major version seems to have the same life cycle and you can see that Jellybean is currently very dominant, that you get from the official monthly pie chart. What you don't get is the idea of how things are probable to move in the next 3 to 6 months. For example Gingerbread and lower currently represent 15% of the active Play Store users. When will it reach 10% ? Judging by previous Android versions, it took 4 months to get from 15% to 10%. So that would be in October of 2014.


This area view is nice, but there are other ways to represent the evolution by simply plotting the numbers in a line graph, as follows:


This graph, IMO, gives a much better idea of how things went for each version and the real importance of each version. Here are some noteworthy points:
  • Each major version has a similar life cycle. A rapid growth and then a slow "logarithmic" decline.
  • The decline starts 4 months after the next main Android version is released (4 for Froyo, 5 for Gingerbread, 4 for ICS, 4 for Jellybean)
  • Honeycomb never had much of an impact
  • Ice Cream Sandwich was never more popular than Gingerbread
  • During Google I/O 2013, the most used Android version was Gingerbread (so much for minSdkVersion=14)
  • Around April 2013 Froyo and Gingerbread lost a lot of market share at the benefit of Jellybean (harder to see in the area graph)

There's still another way to plot the data, from the time the version was introduced and counting how many months it was in use. That gives the following graph:


There's plenty of extra information that can be found from this graph.
  • Since Gingerbread, the evolution during the first months of each version is very similar
  • Older versions of Android were growing more rapidly to their peak
  • After a slow start Kit-Kat has caught up with the growth of Jellybean
  • We can project that Gingerbread will be below 10% in 4 months, ICS will be below than 10% in 2 months 

With Google I/O 2014 on the way and Android 5.0 likely to be revealed there, we can get an idea of the fate of Kit-Kat. It will be similar to the one of Ice Cream Sandwich. Of course, Jellybean is a bit cheating here, since it had 3 major versions. Here is a version of the graph, not grouped by main versions and starting at API v7.




More points can be found from this graph:
  • Obviously the older versions had less versions to share with so had more market share.
  • In the recent years API v16 is the one with the most market share by itself.
  • Kit-Kat (v19) is growing faster than all the versions since Gingerbread (except for API v18)
  • API v15 has reached its peak 15 months ago
  • API v16 has reached its peak 6 months ago
  • API v17 is growing very slowly but still growing
  • All popular recent versions reached their peak around 15 months of existence
  • Versions like v14 or v9 almost never existed

Conclusion / TL;DR: Know Android Users Through Fragmentation Graphs

2013-03-17

Control

A while back I created a company in France on the side of my day job in hope to make a little cash on the side. I created GAWST. My first idea was to make some programs for iOS and OS X as people to pay more for softwares there. I had various projects lined up. The more advanced one was a replacement for the crap Finder, something more in tune with Windows Explorer. Then Apple introduced the App Store and plenty of rules and restrictions. It not possible anymore to sell a file explorer there. The App Store being the easiest and most obvious way to sell content there, I decided it wasn't worth the efforts (working on OS X was never my cup of tea).

So I switch to Android. I wanted something like Ghostery for Android, a way to see the kind of traffic goes to sites I don't want and a way to block it. So I started working on a proxy written from scratch in java. Something lightweight and fast. It took me a while to get it right but now it's pretty stable. I can see and block all the traffic just like I wanted. And I can do a lot more with this little tool. But then last week Google removed all the ad blockers from the Play Store. I planned to sell a restriction unlocker via the Play Store (while basic functionalities would be available to everyone). And once again all the time I spent on this project is wasted because a large company wants to have more control with what we can do with our machine.

I don't know yet what I will do with the code of gawsttp (my proxy). I use it every day on all my devices and it's useful, even a debug tool (or to test Twitter's API blocking). But what is pretty clear is that I won't try again to make a commercial project of my own anytime soon. I'm tired of all the rules that are only good for the big players, despite real user demand.

2012-03-24

RĂ©sistance

When I read this discussion on Mozilla Dev thinking about supporting H.264 I felt like I had to say something. Not just as the creator of Matroska (on which WebM is based) but also as a general net citizen.

Ever since I started working on video technology I realized how vast and encompassing the subject is. In this particular case it's about letting patents dominate a fundamental web technology. It's about who is controlling the technological stack when you go on the Internet (aka market shares). It's about a technical decision that is actually changing the whole principle of the web: the technologies it uses have to be Royalty Free, be it on the user side, on the server side or on the creation tools side.

So far Mozilla was opposed to using H.264, not for technological reasons but on the principle that it didn't fit the Royalty Free web. They were also saying it was “not good for free softwares”, that if used “you [couldn't] have a completely free software Web client stack”, that would be “honoring the letter while violating the spirit” or that it “is not a game [they] are interested in playing” and eventually “it pushes the software freedom issues from the browser (where [they] have leverage to possibly change the codec situation) to the platform (where there is no such leverage)”. None of what was said then has changed. So what has changed must be Mozilla.

When BootToGecko was announced my first reaction was whether they would support H.264 when it's hardcoded in the phone chipset. Because answering this question has a lot of repercussions. The amount of reaction to their answer is proof that the issue is deep. In fact it goes further than just H.264 which is hardcoded and paid for on the hardware (I suppose a lawyer should double check on that to be sure). It also means supporting the MP4 file format, which is also patented but not hardcoded in chipsets, unless they plan to only support H.264 inside Matroska (they already have the WebM code to do it easily), but I doubt that. At some point they will likely support MPEG DASH which made the W3C uneasy because it's not yet a Royalty Free technology. And there may be other patented technologies in the future.

The W3C will never endorse the use of the H.264 codec, it doesn't fit its rules. Vendors using it for the <video> tag are just using a backdoor to put patented technologies on the web, where it should have never been in the first place. Now the message is sent loud and clear, there are no reason big companies shouldn't try to impose their technological dominance on the web (Skype and Facebook come to mind). Everyone will surrender in the end. The bigger you are, the biggest momentum you have, the more chance you have to succeed. And patents are just as virus like as the GPL license: once you touch them you can't escape from them...

Maybe Mozilla is just acknowledging the fact that the web is going away from its core roots and what made it such a global success. But it still seems shortsighted to me. The timing of their announcement comes just weeks/months before new devices with VP8 hardware decoding are widely released. If they wanted to emphasis their continued support/favor for WebM they could at least have first released B2G on a device that has VP8 capabilities. Instead the message sent to those wondering if they should support WebM is: don't bother anymore, in the end we'll have H.264 everywhere. Such a decision to kill WebM should have been taken collectively. Especially since it's just to satisfy the need of a project that currently has 0 users and will have to compete against Apple, Google and Microsoft on the smartphone/tablet market. Billion dollar companies that heavily depend on this market for their future. Whereas Mozilla's future depends on product placement from the same companies. But that may change if I read this correctly: “We will not require anyone to pay for Firefox. We will not burden our downstream source redistributors with royalty fees.“ Payment not required, but possible ?

To be fair, Google deserves a big share of the blame as well. Even though they provided VP8 to WebM, they didn't put all their weight in the battle. They have done too little too late. And there is no sign of that changing. They even failed to deliver on their promise of abandoning H.264 in Chrome to favor the technology they built and is perfect for the web. If there ever was a strategy for WebM I have never heard of it.

Good reads on the subject: