#x264

A working file is a lossless file!

I’ve been using utvideo as my lossless codec of choice. (Remember, codec means encoder/decoder.) It compresses much like FLAC or ZIP files do: losslessly. And not just perceptual losslessness, but a mathematical one: what comes in will be exactly what comes out, bit for bit.

Download it here: https://github.com/umezawatakeshi/utvideo/releases

It’s an AVI VFW codec. In this instance, VFW means Video for Windows, and it’s just the... sort of universal API that any Windows program can call for. And AVI is the container, just like how MP4 and MKV are containers. MP4 as a file is not a video format, it’s a container. MPEG-4 AVC (aka H.264) is the video format specification you’re thinking of when you say “MP4″.

Here’s a typical AVI VFW window, you might have seen one in the wild already.

In apps that expose this setting, you can hit “configure” and set the prediction mode of utvideo to “median” to get some more efficient compression at the cost of slower decoding, but in practice this isn’t a problem.

Things to watch out for:

Any and all apps involved must support OpenDML AVIs. The original AVI spec is 2GB max only. This fixes that limitation. That’s normal, but make sure your apps support that. The OpenDML spec is from the mid-90s, so usually it’s not a problem. But for example, the SFM doesn’t support it.

The files WILL be very large. But they won’t be as large as they’d be if you had a truly uncompressed AVI.

SSDs are recommended within the bounds of reasonability, especially NVMe ones. 1080p30 should be within reach of traditional HDDs though.

utvideo will naturally perform better on CGI content rather than real-life footage and I would not recommend it at all for real-life footage, especially since you’re gonna get that in already-compressed form anyway. Do not convert your camera’s AVC/HEVC files to utvideo, it’s pointless. (Unless you were to do it as a proxy but still, kinda weird)

If you’re feeling adventurous, try out the YUV modes! They work great for matte passes, since those are often just luma-masks, so you don’t care about chroma subsampling.

If you don’t care about utvideo or don’t want to do AVIs for whatever reason, you could go the way of image sequences, but you’ll then be getting the OS-level overhead that comes with having dozens of thousands of files being accessed, etc.

They’re a valid option though. (Just not an efficient one in most cases.)

Some of my working files aren’t lossless...

Unfortunately we don’t all have 10 TB of storage in our computers. If you’re using compressed files as a source, make sure they get decoded properly by your video editing software. Make sure the colors, contrast, etc. match what you see in your “ground truth” player of choice. Make sure your “ground truth” player of choice really does represent the ground truth. Check with other devices if you can. You want to cross-reference to make sure.

One common thing that a lot of software screws up is BT.601 & BT.709 mixups. (It’s reds becoming a bit more orange.)

Ultimately you want your compressed footage to appear cohesive with your RGB footage. It should not have different ranges, different colors, etc. 

For reasons that I don’t fully understand myself, 99% of AVC/H.264 video is “limited range”. That means that internally it’s actually squeezed into 16-235 as opposed to the original starting 0-255 (which is full range). And a limited range video gets decoded back to 0-255 anyway.

Sony/Magix Vegas Pro will decode limited range video properly but it will NOT expand it back to full 0-255 range, so it will appear with grayish blacks and dimmer whites. You can go into the “Levels” Effects tab to apply a preset that fixes this.

Exporting your video.

A lot of video editors out there are going to “render” your video (that is to say, calculate and render what the frames of your video look like) and encode it at the same time with whatever’s bundled in the software.

Do not ever do this with Vegas Pro. Do not ever rely on the integrated AVC encoders of Vegas Pro. They expect full range input, and encode AVC video as if it were full range (yeah), so if you want normal looking video, you have to apply a Levels preset to squeeze it into 16-235 levels, but it’s... god, honestly, just save yourself the headache and don’t use them.

Instead, export a LOSSLESS AVI out of Vegas. (using utvideo!)

But you may be able to skip this step altogether if you use Adobe Media Encoder, or software that can interface directly with it.

Okay, what do I do with this lossless AVI?

Option 1: Adobe Media Encoder.

Premiere and AE integrate directly with Adobe Media Encoder. It’s good; it doesn’t mix up BT.601/709, for example. In this case, you won’t have to export an AVI, you should be able to export “straight from the software”.

However, the integrated AVC/HEVC encoders that Adobe has licensed (from MainConcept, I believe) aren’t at the top of their game. Even cranking up the bitrate super high won’t reach the level of pristine that you’d expect (it keeps on not really allocating bits to flatter parts of the image to make them fully clean), and they don’t expose a CRF mode (more on that later), so, technically, you could still go with something better.

But what I’m getting at is, it’s not wrong to go with AME. Just crank up the bitrate though. (Try to reach 0.3 bits per pixel.) Here’s my quick rough quick guideline of Adobe Media Encoder settings:

H.264/AVC (faster encode but far from the most efficient compression one can have)

Switch from Hardware to Software encoding (unless you’re really in a hurry... but if you’re gonna be using Hardware encoding you might as well switch to H.265/HEVC, see below.)

Set the profile to High (you may not be able to do this without the above)

Bitrate to... VBR 1-pass, 30mbps for 1080p, 90mbps for 4K. Set the maximum to x2. +50% to both target and max if fps = 60.

“Maximum Render Quality” doesn’t need to be ticked, this only affects scaling. Only tick it if you are changing the final resolution of the video during this encoder step (e.g. 1080p source to be encoded as 720p)

If using H.265/HEVC (smaller file size, better for using same file as archive)

Probably stick with hardware encoding due to how slow software encoding is.

Stick to Main profile & Main tier.

If hardware: quality: Highest (slowest)

If software: quality: Higher.

4K: set Level to 5.2, 60mbps

1440p: set Level to 5.1, 40mbps

1080p: keep Level to 5.0, 25mbps

If 60fps instead of 24/30: +50% to bitrate. In which case you might have to go up to Level 6.2, but this might cause local playback issues; more on "Levels” way further down the post.

Keep in mind however that hardware encoders are far less efficient in terms of compression, but boy howdy are they super fast. This is why they become kind of worth it when it comes to H.265/HEVC. Still won’t produce the kind of super pristine result I’d want, but acceptable for the vast majority of YouTube cases.

Option 2: other encoding GUIs...

Find software of your choice that integrates the x264 encoder, which is state-of-the-art. (Again, x264 is one encoder for the H.264/AVC codec specification. Just making sure there’s no confusion here.)

Handbrake is one common choice, but honestly, I haven’t used it enough to vouch for it. I don’t know if the settings it exposes are giving you proper control over the whole BT601/709 mess. It has some UI/UX choices which I find really questionable too.

If you’re feeling like a command-line masochist, you could try using ffmpeg, but be ready to pour over the documentation. (I haven’t managed to find out how to do the BT.709 conversion well in there yet.)

Personally, I use MeGUI, because it runs through Avisynth (a frameserver), which allows me to do some cool preprocessing and override some of the default behaviour that other encoder interfaces would do. It empowers you to get into the nitty gritty of things, with lots of plugins and scripts you can install, like this one:

http://avisynth.nl/index.php/Dither_tools (grab it)

Once you’re in MeGUI, and it has finished updating its modules, you gotta hit CTRL+R to open the automated script creator. Select your input, hit “File Indexer” (not “One Click Encoder”), then just hit “Queue” so that Avisynth’s internal thingamajigs start indexing your AVI file. Once that’s done, you’ll be greeted with a video player and a template script.

In the script, all you need to add is this at the bottom:

dither_convert_rgb_to_yuv(matrix="709",output="YV12",mode=7)

This will perform the proper colorspace conversion, AND it does so with dithering! It’s the only software I know of which can do it with dithering!! I kid you not! Mode 7 means it’s doing it using a noise distribution that scales better and doesn’t create weird patterns when resizing the video (I would know, I’ve tried them all).

Your script should look like this, just 3 lines

LoadPlugin("D:\(path to megui, etc)\LSMASHSource.dll")

LWLibavVideoSource("F:\yourvideo.avi")

dither_convert_rgb_to_yuv(matrix="709",output="YV12",mode=7)

The colors WILL look messed up in the preview window but that’s normal. It’s one more example of how you should always be wary when you see an issue. Sometimes you don’t know what is misbehaving, and at which stage. Always try to troubleshoot at every step along the way, otherwise you will be chasing red herrings. Anyway...

Now, back in the main MeGUI window, we’ve got our first line complete (AviSynth script), the “Video Output” path should be autofilled, now we’re gonna touch the third line: “Encoder settings”. Make sure x264 is selected and hit “config” on the right.

Tick “show advanced settings.”

Set the encoding mode to “Const. Quality” (that’s CRF, constant rate factor). Instead of being encoded with a fixed bitrate, and then achieving variable quality with that amount of bits available, CRF instead encodes for a fixed quality, with a variable bitrate (whatever needs to be done to achieve that quality).

CRF 20 is the default, and it’s alright, but you probably want to go up to 15 if you really want to be pristine. I’m going up to 10 because I am unreasonable. (Lower is better, higher numbers means quality is worse.)

Because we’re operating under a Constant Quality metric, CRF 15 at encoder presets “fast” vs. “slow” will produce the same perceptual quality, but at different file sizes. Slow being smaller, of course. 

You probably want to be at “slow” at least, there isn’t that much point in going to “slower” or “veryslow”, but you can always do it if you have the CPU horsepower to spare.

Make sure AVC Profile is set to High. The default would be Main, but High unlocks a few more features of the spec that increase compressability, especially at higher resolutions. (8x8 transforms & intra prediction, quantization scaling matrices, cb/cr controls, etc.)

Make sure to also select a Level. This doesn’t mean ANYTHING by itself, but thankfully the x264 config window here is smart enough to actually apply settings which are meaningful with regards to the level.

A short explanation is that different devices have different decoding capabilities. A decade ago, a mobile phone might have only supported level 3 in hardware, meaning that it could only do main profile at 30mbps max, and if you went over that, it would either not decode the video or do it using the CPU instead of its hardware acceleration, resulting in massive battery usage. The GPU in your computer also supports a maximum level. 5.0 is a safe bet though.

If you don’t restrict the level accordingly to what your video card supports, you might see funny things happen during playback:

It’s nothing that would actually affect YouTube (AFAIK), but still, it’s best to constrain.

Finally, head over to the “misc” tab of the x264 config panel and tick these.

If the command line preview looks like mine does (see the screenshot from a few paragraphs ago) then everything should be fine.

x264 is configured, now let’s take care of the audio.

Likewise, “Audio Input” and “Audio Output” should be prefilled if MeGUI detected an audio track in your AVI file. Just switch the audio encoder over to FLAC, hit config, crank the slider to “smallest file, slow encode” and you’re good to go. FLAC = mathematically lossless audio. Again, we want to not compress anything, or as little as possible until YouTube does its own compression job, so you might as well go with FLAC, which will equal roughly 700 to 1000kbps of audio, instead of going with 320kbps of MP3/AAC, which might be perceptually lossless, but is still compressed (bad). The added size is nothing next to the high-quality video track you’re about to pump out. 

FLAC is not an audio format supported by the MP4 container, so MeGUI should have automagically changed the output to be using the MKV (Matroska) container. If it hasn’t, do it yourself.

Now, hit the “Autoencode” button in the lower right of the main window. And STOP, do not be hasty: in the new window, make sure “no target size” is selected before you do anything else. If you were to keep “file size” selected, then you would be effectively switched over to 2-pass encoding, which is another form of (bit)rate control. We don’t want that. We want CRF. 

Hit queue and once it’s done processing, you should have a brand new pristine MKV file that constains lossless audio and extra clean video! Make sure to double-check that everything matches—take screenshots of the same frames in the AVI and MKV files and compare them.

Now all you’ve got to do is send it to YouTube!

For archival... well, you could just go and crank up the preset to Placebo and reduce CRF a little bit—OR you could use the 2-pass “File Size” mode which will ensure that your video stream will be the exact size (give or take a couple %) you want it to be. You could also use x265 for your archival file buuuut I haven’t used it enough (on account of how slow it is) to make sure that it has no problems anywhere with the whole BT.601/708 thing. It doesn’t expose those metadata settings so who knows how other software’s going to treat those files in the future... (god forbid they get read as BT.2020)

You can use Mediainfo (or any player that integrates it, like my favorite, MPC-HC) to check the metadata of the file.

Good luck out there!

And remember to always double-check the behaviour of decoders at every step of the way with your setup. 99% of the time I see people talk about YouTube messing with the contrast of their video, it’s because they weren’t aware of how quirky Vegas can be with H.264/AVC input & its integrated encoder.