System.Diagnostics.Process proc = new System.Diagnostics.Process();
proc.StartInfo.UseShellExecute = false;
proc.EnableRaisingEvents = true;
proc.StartInfo.CreateNoWindow = true;
proc.StartInfo.RedirectStandardOutput = true;
proc.StartInfo.RedirectStandardError = true;
proc.OutputDataReceived +=
new DataReceivedEventHandler(EncodeOnOutputDataReceived);
proc.ErrorDataReceived +=
new DataReceivedEventHandler(EncodeOnErrorDataReceived);
proc.Exited +=
new EventHandler(EncodeOnProcessExited);
proc.StartInfo.FileName = “\”" + directoryLocation + “x264.exe\”";
proc.StartInfo.Arguments = parameters;
proc.Start();
proc.BeginOutputReadLine();
proc.WaitForExit();

The Section  with:

proc.OutputDataReceived +=
new DataReceivedEventHandler(EncodeOnOutputDataReceived);
proc.ErrorDataReceived +=
new DataReceivedEventHandler(EncodeOnErrorDataReceived);
proc.Exited +=
new EventHandler(EncodeOnProcessExited);

points to local functions which I use to receive the output from the process I’m calling, so I can report back cool percentage updates in a progress bar. This works pretty well for most applications, but when trying to execute x264 using this method, the process would appear to hang indefinitely, without any answer or messages, so after looking on the Internet a lot I found this excellent article.

How to use System.Diagnostics.Process correctly

The advise provided is really good, you should follow it to the letter, or you can just include the provided ProcessRunner class and get done with it, but I decided to just take a look at the code and see what I was missing. After some work, that particular piece of code now looks like this:

System.Diagnostics.Process proc= new System.Diagnostics.Process();
proc.StartInfo.UseShellExecute = false;
proc.EnableRaisingEvents = true;
proc.StartInfo.CreateNoWindow = true;
proc.StartInfo.RedirectStandardOutput = true;
proc.StartInfo.RedirectStandardError = true;
proc.StartInfo.RedirectStandardInput = true;
proc.StartInfo.ErrorDialog = false;

proc.OutputDataReceived +=
new DataReceivedEventHandler(EncodeOnOutputDataReceived);
proc.ErrorDataReceived +=
new DataReceivedEventHandler(EncodeOnErrorDataReceived);
proc.Exited +=
new EventHandler(EncodeOnProcessExited);
proc.StartInfo.FileName = “\”" + directoryLocation + “x264.exe\”";
proc.StartInfo.Arguments = parameters;

proc.Start();
proc.StandardInput.Close();
proc.BeginOutputReadLine();
proc.BeginErrorReadLine();
proc.WaitForExit();

I added all the recommendations, but the one that really made the difference for me was:

proc.BeginErrorReadLine();

Turns out that x264 outputs all of its messages to stderr (the standard error), and not to stdout (the standard output) so in my case the process would hang until the stderr stream was read, weird, those x264 programmers should really correct it.

There’s and Standard version which is free and a PRO version which is $999 (ouch!), the differences between standard and pro are mainly on the colorspaces and resolutions that PRO can handle. The standard version cannot handle HD resolutions (1080p), my encodes where all from a high resolution source so I had to rely on a little trick (more on that later). The tool handles several file types natively, and for the ones it doesn’t support it relies on Avisynth. The tool works great, it takes some time to complete so be patient, as it analyzes each frame on your encoded file and compares it against the source, and it can only run for one metric at the time (I suggest SSIM, is the most reliable).

The trick I did to be able to process HD files requires a little bit of a compromise, I used an avisynt script to crop the video (selecting only the center of the video) before passing it to the MSU VQMT, as most of the action occurs at the center of the frame I figure it’ll be fine, I did some tests cropping at the corners and at the sides, and to me the most significant info comes from the center of the frame. Here’s an example of such script for a 1080p (1920×1080) source:

loadplugin(“C:\Program Files\AviSynth 2.5\plugins\avss.dll”)
dss2(“C:\Users\xxx\AppData\Local\Temp\serie0675.2.mkv”,fps=23.976)
Crop(480,270,960,540)

It worked pretty well, I recommend this tool to everybody doing encodes.

ffms [error]: could not create index
lavf [info]: 1920x1080p 1:1 @ 24000/1001 fps (cfr)
x264 [info]: using SAR=1/1
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64
x264 [info]: profile High, level 4.1
x264 [info]: frame I:924   Avg QP:17.05  size:140499
x264 [info]: frame P:47360 Avg QP:19.86  size: 44167
x264 [info]: frame B:23332 Avg QP:21.35  size: 17213
x264 [info]: consecutive B-frames: 41.4% 41.9%  5.8% 10.9%
x264 [info]: mb I  I16..4: 22.8% 67.5%  9.7%
x264 [info]: mb P  I16..4:  4.5%  9.5%  0.6%  P16..4: 52.3% 11.6%  6.6%  0.0%  0
.0%    skip:14.9%
x264 [info]: mb B  I16..4:  0.4%  0.7%  0.0%  B16..8: 42.5%  0.4%  0.7%  direct:
7.2%  skip:48.1%  L0:40.5% L1:52.5% BI: 7.1%
x264 [info]: 8×8 transform intra:65.3% inter:78.5%
x264 [info]: coded y,uvDC,uvAC intra: 64.8% 71.9% 26.1% inter: 28.0% 37.2% 0.4%
x264 [info]: i16 v,h,dc,p: 30% 15% 27% 28%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 13% 43%  4%  5%  5%  5%  5%  5%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 16% 38%  5%  5%  5%  5%  4%  3%
x264 [info]: Weighted P-Frames: Y:0.3%
x264 [info]: ref P L0: 60.9% 19.7%  8.8% 10.5%  0.0%
x264 [info]: ref B L0: 81.2% 17.5%  1.4%
x264 [info]: ref B L1: 97.8%  2.2%
x264 [info]: kb/s:7025.66

encoded 71616 frames, 7.01 fps, 7025.66 kb/s

I had particular interest on the number of b-frames generated as this saves on the bitrate and helps provide more quality with smaller file sizes. I ran into the problem of some particular sessions not being recorded, and let me tell you that each one of this encodes took about 5-8 hours to complete, so I didn’t want to run them again just to verify the b-frame count. So after looking and looking in the Internet I found the Avinaptic  (http://forum.doom9.org/showthread.php?t=123076) tool, which does exactly what I wanted, analyzing an h.264 encoded file and generating all kind of useful stats in no time. here are some screenshots:

First Results:

To get the frame count you have to select the “Analyze DRF” option:

And

The tool runs really fast (a couple minutes for 1hr 30min of video) and it’s the only tool so far that I know can count the different types of frames (I,P,B) in an h.264 stream.

Compiling from source sounds complicated, and it is, at least for me. There are also two different ways (that I know of) of compiling from source.

Option 1. you can download the code straight from the SVN repository, apply the patches in SVN , install and you’re done. MythTV would work, but there’s no record in your linux distribution to keep track of this install and updates to your system may break the application and force you to do it all again.

Option 2. you can get the sources from a compiled repository, apply the patches in SVN, install using the package manager from your linux distribution and you’re done. In this case, your system keeps track of the packages and versions and updates won’t normaly affect your installation.

In both scenarios you have to have all the necessary building tools installed in your system before you start, this is not as difficult as it sounds, I’ll try to get to it in another post.

For option 1, the patches can be applied straight from the SVN tree, you may have some conflicts if you’re applying conflicting or similar functionality patches but normally is a straight process.

For option 2, the patches need to be in  dpatch format, the SVN patches are accepted and they will normally apply, but in my experience they bring a lot of issues, specially with recompiles, you really need to convert them.

I went through this process myself and added some patches that I think improved my overall experience, here is the list:

HDHR logging – this is not a trac patch, I did this myself to stop MythTV logging from the HDHR  from killing my hard drive. It changes the severity of some log messages related with the signal quality detected on an HDHR device, weird thing is that even when the HDHR was working fine, the log file would get hammered with quality messages. This patch reduces this load considerably.

7759.- Repeat picture. - This patch implements the correct video delays when frames with the repeat_pict flag are processed. This dramatically improves playback for HD 720p and 1080i broadcast material. Also included are some AV-sync improvements.

7766.- Skip Ahead subs. – A segfault occasionally occurs when skipping ahead (with the right arrow key) while EIA-708 (ATSC) closed captions are turned on, this patch corrects a null check that prevents those segfaults.

7775.- Various caption fixes. – This patch fixes various bugs in the EIA-708 (ATSC) closed captioning implementation. The bugs are as follows:
- Caption windows don’t appear immediatly you have to wait until a later command triggers a redraw.
- Caption windows which are not set as visible get drawn anyway, resulting in overlapping text on shows that alternate between two caption windows.
- On many shows, only the last line of captions is ever visible
- When a character is printed in the rightmost column of a caption window, the pen always moves to the beginning of the next line. This is only supposed to happen when the column lock is set and the row lock is not set. This bug causes extra blank lines to be inserted into scrolling captions, and causes some rows of non-scrolling captions to disappear

7776.- Black Background. – Analog closed captions can be displayed in front of a black background for maximum contrast, but not EIA-708 (ATSC) captions. This patch adds support for a black background behind ATSC closed captions. When the black background option is turned on, captions appearing over light backgrounds are much easier to read.

6719.- Channel Thread. – The signal monitor reports the status of tuning events to the mythfrontend. By adding the channel change process to this list of events, the user gets immediate feedback during a channel change. If the channel change fails or takes a long time, the user can see this, and choose to select a different channel or abort LiveTV. This patch helps the HDPVR implementation a lot.

6611.- HDPVR signal monitor. – Add a signal monitor for the HD-PVR recorder. Using a signal monitor, can make the HD-PVR slightly more responsive. Another great improvement for HD-PVR users.

6602.- Ringbuffer reset. – While watching LiveTV from an HDPVR streaming at 1080i, the frontend can start reading in the middle of a stream instead of waiting for an I or a keyframe after a RingBuffer? switch. This causes much mayhem on the frontend. Applying this patch also improves the HD-PVR experience.

6748.- Bluray subtiles. – This patch enables the display of embedded VobSub (DVD) and PGS (Bluray) subtitles in containers like MKV, M2TS among others, great patch, subtitles have always been hit or miss, forcing users to OCR file after file, with this patch you can just use subtitles as present in the container and watch them in their intented typeface and position.

I have adapted all of this patches along with JYA’s as dpatch’s and made them available in google code  if you are interested in downloading them and compiling from a distro package source, the link is:

http://code.google.com/p/myth-extra-patches/

Ticket 7307:  Add ITU BT709 colorspace support and studio levels (RGB 16-235) for VDPAU playback.

I always wondered why the same content look different if received from my regular TV antenna than if received from MythTV, colors were off, it didn’t look that bad, just very different, turns out that the VDPAU libraries use by default a color space more suited to PC monitors, this patches allows us to specify to MythTV that we are using a TV monitor instead, so it can use the proper color space, it worked pretty well in my case, I can’t notice the differences any longer. This patch has evolved and now adds more options for better performance and quality of HD content on VDPAU enabled video cards.

Ticket 7410: DTS-HD not properly decoded in 0.22-fixes.

This should actually read “TRUE-HD” instead of DTS-HD, linux (alsa library) doesn’t support HD formats, so whenever you play content in this format, you get a lot of grieve from the player, with this patch, mythtv downconverts Dolby Digital (AC3) TRUE-HD to plain Dolby Digital (AC3) allowing for a smooth playback. DTS-HD still is hit or miss, don’t try that content in Mythtv.

Altough this patches in general made me happy, I still had issues around the functionality of my particular devices.

1.- MythTV suddenly decided that the HDHR should log messages like hell (constantly filling my hard drives).

2.- The HD-PVR though supported, seemed clunky, crashing now and then, freezing, muting from time to time, etc.

3.- Embedded subtitles not working on MKV files, this one forced me to OCR subtitles for all my foreign content, I became a master of that, but it was no fun.

So I decided to dive into the mythtv contributed patches, the TRAC website, to search for code that could improve my experience. I found some pretty decent patches that I was then able to integrate into my installation, I’ll let you know more about this process in my next post.

If you are into MythTV you are probably familiar with the JYA repository (http://www.avenard.org/media/Home.html), Jean Yves has worked on mythtv for a while focusing on the quality of the content delivered by MythTV, and his repository contains some patches that really enhance the MythTV experience, his patches and repository are widely used and accepted by the MythTV community.

There are also several developers that submit patches to the MythTV development teams, some of those patches are also great and widely used, enhancing functionality of some not so common features, like subtitle handling or specific devices like the HDHR or HD-PVR, this patches though take a while to make it into an official version of MythTV so to be able to take advantge of them you have to build and install MythTV from source, which for many of us sounds extremely complex and time consuming.

Let me talk a little bit on how I understand the way MythTV development works. They have a development branch for the current version, which normally has the version number plus the “-fixes” (known as the “fixes” version) suffix, in this branch, MythTV developers focus in just correcting issues and problems with the current version, no new functionality or features are added there, releases on this branch are not often, and changes are normally minimal.

There’s also at least another branch for the next version, which is named using the next version number plus the “-trunk” suffix (know as the “trunk ” version), in this branch developers are very active creating new features and functionality, trying new algorithms for improved performance and adding support for new devices. Trunk is not very stable, releases are very frequent, and is not recommended for the general public.

Sometimes in trunk, there’s a patch that is identified as easy to backport, this means that the same patch could be applied to the fixes version without many changes and greatly improve a certain issue or functionality, and sometimes developers take the time to backport and test those fixes on an official release.

We also have other type of patches, the ones submitted by independent developers that are pending review, which means there are no active plans to include them either on fixes or trunk, sometimes there are real gems hiding in those patches, JYA’s patches used to fall in this category. Problem with those patches is that you have to be very careful selecting and applying them, as some are designed for trunk, some for fixes, and some of them are from previous versions, anyone can break your code, so be careful.

My current mythtv system is pretty typical, one computer serving as a backend frontend and a diskless frontend for the kids room, I receive OTA signals using an HD Homerun device (HDHR) and also through an HD-PVR. I watch a lot of foreign films so I have a need for subtitles in my content. The current version of MythTV pretty much serves my purposes, but let me work you through my experience.

I started on MythTV 0.21, looking at HD (1080p) content, my machine suffered in terms of performance, processing all those video frames was not an easy chore, that’s when I discovered JYA’s repository, it offered a compiled version of MythTV specific to my platform (Ubuntu), that contained some patches that allowed MythTV to offload the video processing to the Nvidia card using a protocol known as VDPAU. This was a big hit, as my machine was suddenly able to process high quality video on both frontends withtout a glitch, I’ve been using JYA’s repository ever since.

With mythtv 0.22, most of JYA’s patches were included in the main version, still there were some of them not included (color space and HD audio handling) that made the repository still worth of being my choice for updates.

I recommend this repository for everybody new to MythTV

BDMux has some interesting features, like being able to extract chapters of multi-episode BD’s (TV Series) into multiple MKV’s connecting to TheTVDB.com to retrieve the actual episode names for the resulting files. Also you can specify temporary and target directories, having those on separate drives speeds up the remuxing process a lot. There’s an FTP option to copy the file at the end of the remuxing process into a user specified server and directory, for users with media servers like XBMC, Boxee or Mythtv this is a good productivity improvement.

Why just a remuxer? to preserve maximum quality on the original file. Why not just use the .m2ts file that comes in the Bluray? maybe in the future, right now the format is not widely supported, the multiple file playlists are a killer for both users and players. Also included in the remuxing process is a conversion for DTS-HD tracks not currently supported in linux players, the conversion adds two tracks, a DTS only track with 5.1 channels and a FLAC track preserving the original quality and channels. Also the remuxing process converts all the PGS (Bluray subtitles) tracks to SUP/IDX (DVD Subtitles) which are more supported across players, this may not be need in the near future as PGS is better and getting wider support.

I tested all my files in my Mythtv install and they work just fine, showing embedded subtiles, processing high quality video and audio tracks without problems.

The application uses some external tools that have to be somewhere on your system:

1.- eac3to  an excellent BD extraction and audio conversion tool http://forum.doom9.org/showthread.php?t=125966

2.- BDSup2Sub a subtitle conversion tool http://forum.doom9.org/showthread.php?t=145277

3.- mkvtoolnix the defacto tools for processing matroska (MKV) files  http://www.bunkus.org/videotools/mkvtoolnix/

I sat on the shoulders of the excellent open source BDInfo tool (http://www.cinemasquid.com/blu-ray/tools/bdinfo) reusing a lot of their code for reading BD Track Information.

You can get BDMux from Google Code at http://code.google.com/p/bdmux/downloads/list

While hooking up the components to the Yamaha, I got into a serious limitation in both design and functionality as compared with the Denon. you’ll see, in the Denon the inputs can be reused, what I mean is that logical input number 1 will have HDMI, Coax, Toslink, Composite, RCA stereo connectors all associated to it, so you can connect any combination (depending on your component) to it, for example, your device may have composite video and Toslink audio outputs, and it will be ok, your DENON will handle it and play it as input 1 no problem. More to it, in certain models, individual connectors can be assigned to specific virtual inputs, you can mix audio from coax 1 with video from component 2 into a virtual input, this is great flexibility.

DENON AVR 1910

On Yamaha on the other side, the inputs are fixed, you cannot mix and match, and they also make a lot of assumptions on what is the equipment you’ll connect to it, for example they assume that if you have and HDMI component hooked in, you won’t need any additional audio connector for that input, which may or may not be true, you may have an old video component that doesn’t output audio via HDMI, well, you’ll be out of luck. Or my particular problem, I have a Wii with a Component cable for 480p resolution, which looks great, but the inputs that contain component video connectors on the Yamaha, only have toslink or coax audio connectors associated with it, and the Wii only outputs RCA stereo, so I had to switch to the old Wii cable with composite output that by the way looks like crap with the Yamaha video upconversion. For an extra $50-$60 USD, I could have gone with the Denon, it has better quality all around (just look at the speaker posts on the Yamaha). To solve my problem I had to buy and Startech converter which set me back $48, so bad deal. Are you interested in buying my V565, I’ll make you a good deal.

Yamaha RX-V565

As I mention, to fix my Wii issue I’m getting an RCA to SPDIF converter, the only one that I’m aware of, the Startech. This little converter receives one pair of cables for an RCA stereo connection and outputs both Toslink and Digital Coax signals, it is very small, and requires an external power source, which is inconvenient but understandable due to the ADC it has to run inside for the conversion. Sound is output as stereo (2 channel) with good (not great) quality,  now I can connect my Wii component cable back and enjoy the picture.

Startech RCA to SPDIF conversion.

More screen

Transcode screen

Playback will start just a little bit after that, fast, good quality, plays MKV files without a problem. DRM files are still no good, but there’s another cheap (not free) solution add-on for that, so it’s kind of ok. Bad thing about Transcode 360 is that its original creator at runtime360 has stopped development, in fact it didn’t really continued development after XP, according to the author, he liked TVersity better and didn’t find a reason to continue, what a shame. Another developer in UK (Transcode360.co.uk) picked up the code and upgraded it for Windows Vista, it works really well, but know seems like the code has been abandoned again, transcode360.co.uk has been down for a while now, you can still find the latest release (1.6.3) here.

While doing my transcode360 testing I meet one of vista’s nasty bugs. As I was happy with me current implementation, It worked even for ripped DVD folders (video_ts), I checked on the internet and found the registry tweak needed to indicate Vista Media Center that you have a DVD library (Gallery) on your hard drive. To do this, run “regedit” and navigate to:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Media Center\Settings\DvdSettings

and change the “ShowGallery” value to “Gallery”, I did a test and after everything was ok. I went on and moved a lot of video_ts folders into a USB Hard Drive connected to my VMC PC, I added the Hard Drive folder to the DVD library search path and was ready to start watching content in my extender. Suddenly I could play no content, errors in Vista Media Center all over the place, “Content cannot be played back” or something like that. I tried playing the content in windows media player to try to figure out what the problem was and surprise! WMP says that the DVD’s are in a region not supported by the player, which is not possible of course. Turns out that some buggy DRM code inside vista requires all the DVD folders to reside in directly connected drives like SATA or PATA and it marks folders in USB drives as invalid, a way around this is to connect the drive as a folder structure inside one of your main hard drives, this works ok, but what a pain it is, it has to be NTFS (most USB drives are originally formated as FAT32) and it breaks the purpose of the portability of a USB drive. VMC requires a lot of hacks so I’m thinking on getting out of it for my final solution.