One of the more interesting aspects of Ogg Video is that it allows an essentially unlimited number of subtitle tracks to be included. This is especially useful for free-culture videos, since they are generally released globally, and there are often contributed subtitles. In fact, for "Sintel", I was able to find 44 subtitle files. I will be including them all as Ogg Kate streams in my prototype "Lib-Ray" version of "Sintel", and in this column I will demonstrate the use of several command line utilities useful for this, especially the kateenc tool for creating the streams.

After writing my recent piece on my Blu-Ray Blues with finding a way to distribute high definition video, I began to experiment with free-licensed, non-patent-encumbered formats. The natural choice was, of course, to start with the Xiph.org formats and work my way along until I had achieved the goal. Today I'm going to create the subtitle streams.

If you've been following this series, you'll recognize this as the same site I got the PNG frames and audio soundtracks from

The mean source for the "Sintel" material is a download directory on the Xiph.org website. If you've been following this series, you'll recognize this as the same site I got the PNG frames and audio soundtracks from. This site also has the original .srt format subtitle files for the nine languages that are included on the DVD version of Sintel.

Since then, however, 36 additional .srt files have been provided by the community, for a total of 45 different subtitle tracks. These are collected at a different site. I'll be using all of these.

Subtitles

There are also Ogg streams to carry subtitles. The most popular, and the one I'm going to be using, is Kate. Like the Theora, FLAC, and fhe command line tool for manipulating this format (kateenc) is included in the Debian archive (as part of the libkate-tools package).

There are also Ogg streams to carry subtitles. The most popular, and the one I'm going to be using, is Kate

The first problem I encounter with the .srt files from Sintel is that they do not use consistent encodings (I didn't realize this until kateenc choked on some of the files -- it expects UTF-8 encoded files!). Using the file command, I can see this right away:

$ file *.srt

sintel_afr.srt:     ISO-8859 text, with CRLF line terminators

sintel_ar.srt:      ISO-8859 text, with CRLF line terminators

sintel_bg.srt:      ISO-8859 text, with CRLF line terminators

sintel_bn.srt:      UTF-8 Unicode text

sintel_chs.srt:     ISO-8859 text, with CRLF line terminators

sintel_cn.srt:      Little-endian UTF-16 Unicode text, with CRLF, CR line terminators

sintel_cz.srt:      UTF-8 Unicode text

sintel_da.srt:      ISO-8859 English text, with CRLF line terminators

sintel_eo.srt:      UTF-8 Unicode (with BOM) text, with CRLF line terminators

    [...]

I started by collecting these into directories for each of the major encodings, but unfortunately, this is a little hard to untangle. In particular, the ISO-8859 encoded files use different code pages according to language, so you have to recognize the language codes in the file names (or look them up in an ISO-639 language code table) and figure out the correct page. For example, ar is the code for Arabic, and this means that we should decode from ISO-8859-6. Or so I thought -- actually attempting this results in an error. Opening the file up in Iceweasel, I noticed some weird characters that didn't make sense. It provides some other options for encoding, and with "Windows-1256" it actually looked like Arabic. So, that's what I'll use (in fact, it turned out that the majority of these files that file identified as ISO-8859 were actually in one of various Windows encodings -- they were probably submitted in the default encoding of the user who contributed the translations).

This can be converted with the iconv command line tool:

$ cd ISO-8859

$ iconv -f WINDOWS-1256 -t UTF-8 sintel_ar.srt -o ../UTF-8/sintel_ar.srt

I didn't figure out any way to automate this, so I just went through the files one-by-one to convert the encodings appropriately (fortunately, they were not all this hard). I'm not going to go through this in detail, but in the end, I had all 44 of my subtitle files in one directory with UTF-8 encodings.

At this point, a tcsh loop is handy for processing the files in bulk to get my Ogg Kate streams:

$ cd All/UTF-8/

$ tcsh

> foreach lang ( af ar bg bn cz da de en eo es fi_ep fi_FI fi_ps fr gl gr he hr hu id it jp ko ku la lv mk ml nb_NO nl pa pl pt ro ru sk sr th tr uk vi zh_CN zh zh_TW )

foreach? echo $lang

foreach? kateenc -t srt -c SUB -l $lang -o ../../OGG/sintel_$lang.ogg sintel_$lang.srt

foreach? end

  [...]

> 

Note that by providing this code to kateenc via the -l option, it will identify the subtitle track correctly by language. This will allow the player to identify the tracks correctly by language (I tested this in VLC, and it provides the full-name of each language in that language on the subtitles pull-down menu for the user to select from).

Fixing bugs in the SRT files

At this point I must be honest -- many of these .srt files had bugs. As a result, I got quite a number of syntax warnings from kateenc. I had to go back and fix a lot of these files in order to get them to work smoothly.

I had to go back and fix a lot of these files in order to get them to work smoothly

When everything is working smoothly, the code above will simply list the language/country code extensions. If something goes wrong, though, the echo line will tell which language files was being processed, so you can check it out.

Some of the errors I found on inspection:

• Two subtitle blocks run together without an intervening blank line (kateenc skips the second one and then complains about "non-consecutive ids"). Fix by adding the blank line.
• Extra blank lines inside a subtitle block, resulting in a syntax error.
• Incorrect time codes (typos?) resulting in nonsensical time intervals (reversed, or of zero-length)
• Non-standard annotations, such as author information. These can be converted into normal subtitles and placed at the end (they'll appear near the end titles), thus preserving the correct attribution.
• Unicode byte-order marks at the beginning seem to confuse kateenc in a couple of cases, so I just removed them.

At this point we have 44 Ogg files with Kate streams in them. Combined with the audio and video streams from before, we'll be ready to assembled them into a single multimedia file, which will be the goal of my next column.