
## About this site

This website hosts all the information related to the course _Video Encoding Systems_ taught at the
[University Pompeu Fabra](https://www.upf.edu/) in Barcelona, as part of its Engineering Audiovisual
Systems degree.

It aims to make it more comfortable for both students and teachers to manage, modify and share the
course information. If you see any mistake, typo, or you found information that could complement
what is showcased on this site, feel free to make a _pull request_ with your proposal using the
button at the bottom of the pages.

Additionally, it opens the knowledge to anyone interested in it. If you are not enrolled in the
course, be free to roam around the site and learn as much as you want! But of course, you will not
get any certificate or evaluation.

## Expected learnings

After completing this course and assimilating everything to offer, you should be able to:

- Apply knowledge of video coding systems to solve problems that may arise in engineering.

- Identify and model complex systems. Make qualitative analyzes and approximations, establishing the
  uncertainty of the results. Raise hypotheses and experimental methods to validate them. Identify
  key components and set commitments and priorities.

- Apply the skills acquired to carry out tasks autonomously. Identify the need for lifelong learning
  and develop your strategy for doing so.

## Contents

- T1: JPEG i JPEG2000
- T2: MPEG i MPEG2
- T3: MPEG4, h264 & Broadcasting standards
- T4: Future in codecs

These contents are available on this website and in the Aula Global in the University website.
Additionally, the teacher of this course, has recorded some video lessons to accompany the written
theory, which are also available on this site.

## Prerequisites

This course is part of the 4th year of an engineering degree, so it would be ideal to learn
beforehand all the knowledge that it implies. Nevertheless, the most important previous information
needed is:

- Basics in programming (ideally Python)
- Basics in signal processing

## Technical requirements

There are some technical requirements to follow the course:

1. Having a Debian-link Linux distribution (Ubuntu recommended). You can either install it natively
   or use a virtual machine. The first is advised since some exercises require considerable
   processing power, and would be too slow in a virtual machine.

It is also possible to use the Mackintosh terminal.

2. Install an IDE for Python. Some examples are PyCharm or Atom. Search for _linters_ for your
   preferred IDE and use the PEP-8 conventions.

3. If you are a student at UPF: Have a Github account. Be sure to include your full name in the
   profile. Also, it is preferable to use your laptops instead of the university ones, to avoid
   issues with permits.

## Teaching Methods

The subject is composed of theory, seminars, and practices.

Theory sessions will introduce video encoding and decoding concepts to be used in seminars and
practices, in a hands-on way, and with software programming.

Students will understand better the encoding and they will be able to create their encoding systems,
as well as to evaluate the results. Students will work in groups and individually.

## Evaluation

This section only addresses students enrolled in the university course.

The evaluation in this part of the subject is determined by:

- 50% Theory (Final exam)
- 50% Labs (That we will work on both on seminars and practice sessions)

This course is joined together with the _Audio Encoding Systems_ course. To pass, a minimum of 6 out
of 10 is needed in each of the blocks of both subjects.

For the theory blocks, there will be a joint exam in December. It will be scored out of 10 points, 5
points for the audio block, and 5 points for the video one.

If a student fails the ordinary call, there will be an extraordinary one in July to recover the
subject. In the subject recovery, the student will have an exam where all 4 blocks of the subject
will be evaluated.

Lecturers reserve the right to offer the student a research project as a substitution for the exam.
In that case, the student won’t be able to achieve a higher mark of 6 out of 10, with now a 5 being
the mínimum required to pass. If the final mark is higher than 6 out of 10, it will be truncated
to 6.

With that being said, let's start!

### Bibliography

The main reference of the course is:

- Image and Video Compression for Multimedia Engineering. Authors: Yun Q. Shin, Huifang Sun.


All the information about the course and this website.

Introduction and teaching plan


Before learning about video encoding, it is important to acquire some terminology and concepts. Most of them you might already know, but a refresh never hurts! 

Since video is by definition a succession of images, so we will start with understanding image formats.

## Basic concepts

The **pixel** (picture element) represents the intensity of a given color, and it's usually a
numeric value. For example:

- Red Pixel = 0 of green + 0 of blue + maximum of red
- Pink Pixel: 192 of green + 203 of blue + 255 of red.

We make use of a three-dimensional space where the intensity of each of the RGB colors is added to display the desired one.

![píxels](https://user-images.githubusercontent.com/40371955/146955382-13b583e6-7798-4544-acbf-7c54fa8c4c39.png)

There are other ways to encode a color image. For example, we can use an indexed palette where we
will only need a single byte to represent each pixel, instead of the 3 needed when using the RGB
model. In this kind of model, we can use a 2D matrix instead of a 3D one to represent the color, saving
memory but limiting our color options. 

For the first image format, the JPEG, we will only consider the first method.

![image](https://user-images.githubusercontent.com/40371955/146110480-640cb5df-ea37-4c6b-af00-32a832fd8281.png)

Another basic concept is **resolution**, which is the number of pixels in a single dimension.

## JPEG

As all standards, the JPEG codec was created by a board of experts who in this case, tried to agree on a normalized way of encoding images. 

This group was the **Joint Photographic Experts Group** or JPEG, which would also be the name of the format. 

It is a lossy compression method and supports true color (24 bits, or 16777216 different colors). Its ISO norm was created in 1983.

The following is the encoding / decoding diagrams of the JPEG compression:

![codingdiagram](https://user-images.githubusercontent.com/40371955/146962247-d7c31b42-e757-4890-9df1-f1a404dd5f64.png)

To understand better the JPEG, let's think about how the human eye works: we recognize objects equally well regardless of the image size. Based on this concept, the JPEG achieves considerable compression of data, making it extremely used in
computing and the Internet.

However, since it's a lossy method, the reconstructed image will never be equal to the original.

![comparison](https://user-images.githubusercontent.com/40371955/146963386-7a978c96-2e35-4029-823f-5558758f58eb.png)

In the previous comparison, we can see the difference between an uncompressed image and one
compressed with JPEG. The image goes from 83 to 10 kilobytes, which is about 1/8 of compression,
maintaining enough information so that the human eye understands the image.

## DCT

One of the most important parts of the encoding process is the **DCT** (Discrete Cosine Transform).
To understand how it works, let's use an example:

We assume that we want to compress a black and white image. The first step will be to fit the image into a grid. Then, for each cell, we will assign a value of intensity, or in other words, we will _quantize_ the image.

![dct1](https://user-images.githubusercontent.com/40371955/146971185-4e891ec3-eed6-433c-a815-60e92ffc1254.png)

Afterward, we can plot the numerical quantized values in a graph. The DCT will try to find the best approximation to the line created by the values. We draw a line along with the average value, and we try to match a sinusoidal wave to the obtained shape.

![dct2](https://user-images.githubusercontent.com/40371955/146970833-a8a14ee7-ca2a-470e-81a7-c47d1ad4d8b1.png)

Therefore, the DCT can compress a lot of numerical values in just the numbers that define the sinusoidal function.

![dct3](https://user-images.githubusercontent.com/40371955/146970866-96b8e363-7e5d-4764-9d8b-93eef28f1475.png)

The JPEG standards define a serpentine pattern to convert the DCT into a linear sequence of values.
Additionally, run-length encoding is applied, which swaps every line of zeroes for two digits, a
zero, and the number of consecutive zeroes, reducing, even more, the output.

![dct4](https://user-images.githubusercontent.com/40371955/146976636-f0e734b7-fc98-4c11-ba6c-daf89c1bb724.png)

### YCbCr

Until now, we used a black and white image as an example. But what about colored images? In that case, we just would need to repeat the operation three times, one for each channel RGB. Instead, JPEG introduced a new color model: the **YCbCr**. This stands for _Luminance_, _Chroma blue
difference_ and _ Chroma red difference_ respectively.

These values are obtained by applying a mathematical process to the RGB ones. Where R', G' and B' are
the gamma-corrected values and Kr, Kg, and Kb three defined constants:

![YUV](https://user-images.githubusercontent.com/40371955/146978104-c8731484-9b3b-475d-b147-e982e02cad2f.png)

```
    Y  = R *  0.29900 + G *  0.58700 + B *  0.11400
    Cb = R * -0.16874 + G * -0.33126 + B *  0.50000 + 128
    Cr = R *  0.50000 + G * -0.41869 + B * -0.08131 + 128

    R  = Y +                       + (Cr - 128) *  1.40200
    G  = Y + (Cb - 128) * -0.34414 + (Cr - 128) * -0.71414
    B  = Y + (Cb - 128) *  1.77200
```

This method of representation of color will make it easier to compress. 
Note: It is possible that you
see it also named as YPbPr, and although it is based on the same concept, that is applied only for
analog signals.

![yuv2](https://user-images.githubusercontent.com/40371955/146980975-4090d018-4c88-4178-96bf-2e9495275c55.png)

## JPEG 2000

To solve the quality loss issue with JPEG, the same board of experts presented in the year 2000 a new standard: JPEG2000. It was technologically the best possible engineering solution to the problem they had, but it failed to standardize, as it required the hardware to adapt to the new codec (in other words, it was not backward compatible with JPEG).

Its file extension was `.jp2`. More information about this codec can be found in slides pdf at the bottom of this page.

## Other Standards

Other methods of image compression succeeded in standardizing and are commonly used.

### GIF (Graphical Image Interface)

It stores 8 bits per pixel for each image, up to 256 different colors chosen from the 24-bit RGB color space, therefore, using the second method for representing color that we explained in the first section of this document. It uses the file extension `.gif`, and one of the most interesting
property is its animation support.

It is less suitable for color photographs but good enough for graphics or logos. Gif images are
compressed using the Lempel-Ziv-Welch (LZW) lossless data methods to reduce the file size without
degrading the visual quality.

### PNG (Portable Network Graphics)

Another image standard that succeeded more than JPEG2000, was the PNG. It was a non-patented, open-source format that aimed to substitute GIF.

It supports 24RGB or 32-bit RGBA (allowing opacity). The file extension is `.png` and uses a lossless codec, based on DEFLATE encoding (a mix of LZSS and Huffman).

---

#### Resources

<FileDownloadLink name='Slides T1' filepath='/files/T1 - Image, JPEG and others.pdf' />
<FileDownloadLink name='Seminar 1' filepath='/files/S1 - FFMPEG.pdf' />
<FileDownloadLink name='Practical 1' filepath='/files/P1 - JPEG.pdf' />


T1. JPEG & JPEG2000


## Basic concepts: aspect ratio and video

Let's start this second theory lesson by explaining another couple of basic concepts about images (and videos):

The **aspect ratio** describes the proportional relationship between the width and height of an image, video, or pixel. If we apply it to screens, we call it Display Aspect Ratio (DAR). It is
defined as two numbers divided by a semicolon, for example:

![aspectratio](https://user-images.githubusercontent.com/40371955/147374215-1e7ee221-62ea-48bc-86c9-32de4034ade4.png)

The concept can also be used for píxels, so the PAR will describe the relationship between height
and width or the shape of the píxel.

![pixelratio](https://user-images.githubusercontent.com/40371955/147374222-07a57c88-81e5-4faa-8eec-510b3a72aaac.png)

Before we explore the video formats, let's agree on what a **video** is. We can define a video as a succession of _n_ frames in time (which we can see as another dimension). This _n_ is the framerate of frames per second (fps).

![videodef](https://user-images.githubusercontent.com/40371955/147266739-aae68dde-13fd-473c-9475-840c8e84a089.png)

Furthermore, we define **bitrate** as the number of bits per second needed to show a video. `bitRate = width * height * bitDepth * FPS`. For example, if we have a video with 30fps, 24
bits/pixel and a 480x240 resolution, we will need 82944000 bits per second, or 82.944 Mbps (30 _ 480
_ 240 \* 24) if we don't employ any kind of compression.

When the bitrate is nearly constant, we call it CBR (constant bit rate). On the other hand, if it's variable, VBR (variable bit rate).

![variablebitrate](https://user-images.githubusercontent.com/40371955/147374235-da21a10d-de27-4738-baf5-b6d5c6d266a6.png)

There are also techniques to double the perceived framerate without consuming extra bandwidth. The
first one used was _interlaced video_, which sent half of the screen in one frame, and the other half in the other frame skipping every other line. Nowadays _progressive scan video_ is used, which displays, stores, or transmits moving images drawing all the lines of each frame in a sequence.

![intervsprog1](https://user-images.githubusercontent.com/40371955/147267979-520b1fae-d89c-466f-aca5-ce113fd56ed4.png)
![intervsprog2](https://user-images.githubusercontent.com/40371955/147267997-7b6a8e71-bc47-4fbf-a6e8-00d538f451a3.png)

Once these concepts are settled, we can start talking about video compression methods.

## MPEG

As we explained in the previous lesson, standardization is usually developed by groups of experts in
the subject. In this case, the Moving Picture Experts Group (MPEG) was established by ISO in 1988
and created the standard (named of course **MPEG-1**) in 1993.

MPEG-1 is a standard format for compressing and storing digital audio & video. This is a lossy compression method, but unlike images, all video compression formats are lossy, at least for now.

It was designed to compress VHS-quality raw digital video and CD audio down to 1.5 Mbit/s (26:1 and 6:1 compression ratios respectively). VHS had a resolution of only 360x240 pixels, which now would be low quality. It also allowed creating the Video CD format, which didn't succeed.

The conceptual idea behind the MPEG-1 was to create a smart encoder that runs on specialized
hardware, and then a simple decoder that could be used by cheap hardware at homes.

![mpegdiagram](https://user-images.githubusercontent.com/40371955/147271053-71f3811f-0876-494d-a2b3-789a761e4339.png)

In the previous lesson, we saw how JPEG could compress an image (or frame). The MPEG doesn't work
with each frame, instead uses an interframe or temporal compression. The following diagram describes
the method's workflow, which will be detailed in the next sections.

![mpegworkflow](https://user-images.githubusercontent.com/40371955/147272188-bba8c0e2-5a18-4cb4-bc14-e7509c8e0e9b.png)

### Reduction of resolution

This process can require a video downsize, and it translates from RGB to YUV (as in JPEG).

Additionally, it performs chroma subsampling, which consists in implementing less resolution from
chroma (U and V, or Cb and Cr) information than for luma (Y), taking advantage of the human visual
system's lower acuity for color differences than for luminance. The human eye contains about 120 million rod cells (which detect luminance) and only 6 million cones (which detect colors), hence the reduction of chroma.

![chroma sub](https://user-images.githubusercontent.com/40371955/147272899-b62f4622-5bcf-4c67-be7f-d72044532584.png)

The subsampling amount is described as _a:x:y_, where _a_ is the horizontal sampling reference (usually 4), _x_ the chroma samples in the first row, and _y_ the number of changes between the first and second row of pixels. Common subsampling schemes used in modern codecs are: 4:4:4 (no
subsampling), 4:2:2, 4:1:1, 4:2:0, 4:1:0 and 3:1:1.

![subsampl](https://user-images.githubusercontent.com/40371955/147273479-759fd20c-0ef0-4d32-906d-89ea7cb2a4b0.png)

As we can see, the human eye doesn't really tell the difference between the 4 examples.

### Motion estimation

To see how motion estimation works, we need to understand the three types of frames that we can
find, and how they work in MPEG. Let's assume we have a 30fps movie, and the following are the first 4 frames.

![frames](https://user-images.githubusercontent.com/40371955/147276035-b030141a-f931-4cd6-b4bc-67693379be26.png)

We can see there is a lot of repetition within the frames: the blue background or the different
still objects. We can divide the frames into three categories:

#### I Frame (intraframe, keyframe)

Which is a self-contained frame. It doesn't rely on anything to be rendered, looking similar to a
static photo. The first frame is usually an I-frame but we can see them inserted regularly in between other types of frames.

#### P frame (predicted)

It takes advantage of the fact that almost always the current picture can be rendered using the previous frame. For instance, in the second frame of our example, the only change is the ball moving. We can rebuild frame 1, using the difference and referencing to the previous frame.

![pframe](https://user-images.githubusercontent.com/40371955/147276085-7277e6fe-d917-4402-a2b2-764daeaa1e78.png)

#### B frame (Bidirectional predicted frame)

Using the same concept, we can not only move to the previous frame but also move to the following one. We'll be predicting the past and the future frame.

The frame types are used to provide better compression. When using this technique, MPEG sequences
and reorders the frames with the pattern IBBBPBBBP. An Iframe is the most 'expensive' frame, followed by a P frame and a B frame. Therefore, we can obtain a very convincing and compressed result by only using part of the video's information.

On the other hand, if some information is lost, we will not be able to recover the frames correctly and artifacts will appear:

![image](https://user-images.githubusercontent.com/40371955/147276892-9a542afc-9aae-4d09-9d64-9fae2279ffd5.png)

#### Motion vector

Motion estimation examines the movement of objects in an image sequence and tries to obtain vectors
representing the estimated motion. In this part of the encoding process, we will generate the motion vector data to then be consumed by the decoder.

> Note: motion estimation and motion compensation are considered as a single block in some resources, but are two different processes. Motion compensation is the use of motion estimation to achieve compression. If you can describe the motion, then you have to describe the changes that occur after compensating for that motion.

The following sections will be explained without a lot of depth into the formulas or mathematical
operations behind them, if you want to learn more check the full slides in PDF at the end of the lesson.

![motionvectors](https://user-images.githubusercontent.com/40371955/147281314-885b8e91-59c3-44b3-a443-14ff34c55790.png)

The motion vector represents the movement between two frames of the blocks of the image.

### Motion compensation

This process is split into three parts:

- Motion Analysis: reviews that the motion estimation part is correct.

- Prediction and differentiation: the present frame is predicted by using the estimated vectors and the previous frame. The prediction error is calculated:

  ![prediction](https://user-images.githubusercontent.com/40371955/147281850-d01aa8a7-881b-40df-b7e5-2293a30b76a4.png)

Once we have the motion vector for each block, we perform the **block matching** where we apply the
vector to the original block and make it match with the next frame, for every block. As before, further information can be found in the slides PDF.

![blockmatching](https://user-images.githubusercontent.com/40371955/147374245-bd63855f-cefb-46a1-b6e2-4ec313cd9174.png)

- Encoding: as performed in the JPEG format: DCT->Quantize->Run-length encoding->Entropy encoding
(Huffman)

Here's a real-life example of a MPEG encoder:

![mpegdiagramreal](https://user-images.githubusercontent.com/40371955/147282363-cb2b937f-a118-41ef-b462-562bd118e1f8.png)

#### What about audio?

MPEG-1 has its own codec based on psychoacoustics. MPEG-1 Layer I (.mp1) is defined in ISO/IEC
11172-3, which first version was published in 1993. It supports the following sampling rates and
bitrates:

- Sampling rates: 32, 44.1 and 48 kHz
- Bitrates: 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416 and 448 kbit/s.

## MPEG-2

The same group of experts developed an enhanced version of the format in 1996. it offered higher resolutions such as 720x480 and 1280x720 at 60fps, with full CD-quality audio. This was sufficient for all the major TV standards including NTSC and HDTV. MPEG-2 was used by DVD-ROMS and could compress a 2-hour video into a few gigabytes. Some of the improvements were the following.

- Coding performance related to supporting interlaced video input:
  - Field/frame prediction modes.
  - Field/frame DTC coding syntax
  - Downloadable quantization matrix and alternative scan order
  - Scalability extension
- Non-compression enhancements:
  - Syntax to facilitate 3:2 pull-down in the decoder
  - Pan and scan codes with 1=16 pixel resolution
  - Display flags indicating chromaticity, subcarrier amplitude, and phase (for NTSC=PAL=SECAM
    source material)
- Improved version of the audio codec:
  - Extensions, known as MPEG-BC, are backward compatible with MPEG-1 Audio. Defined in ISO/IEC 13818-3
  - Introduction of MPEG multichannel (backward compatible 5.1 channel surround sound)
  - Improved sampling and bit rates: 16000, 22050, 24000Hz and 8, 16, 24, 32, 40, 48, 56, 64, 80, 96, 112, 128, 144 and 160 kbit/s, respectively. Sampling rates are half of those in MPEG-1 and were added to maintain higher quality sound when encoding audio at lower bitrates.

MPEG-2 is used in DVD video, HDV, and cameras like MOD and TOD, XDCAM, in the Digital TV standards
and even in the first blue-ray video.

---

#### Resources

<FileDownloadLink name='Slides T2' filepath='/files/T2 - MPEG y MPEG2.pdf' />
<FileDownloadLink name='Practical 2' filepath='/files/P2 - Python  FFMPEG.pdf' />


Basic concepts about video and explanation of the first two video codecs, MPEG & MPEG2

T2. MPEG & MPEG2


## MPEG-4

In this lesson, we will talk about more recent video technologies. For starters, we will go over the
MPEG-4, which as the name suggests, was created by the same group, the MPEG, and it's an enhancement
of previous versions.

It was introduced in late 1998 and designated a standard for a group of audio and video coding
formats and related technology, agreed upon by the ISO/IEC MPEG (ISO/IEC JTC1/SC29/WG11) under the
formal standard ISO/IEC 14496 - Coding of audio-visual objects.

---

MPEG4 is a graphics and video compression algorithm standard based on MPEG-1, MPEG-2, and Apple
QuickTime technology. It is still almost everywhere, used for compression of AV data for web
(streaming media) and CD distribution, voice (telephone, videophone), and broadcast television
applications. Some of its improvements will be explained now:

- **Possibility to handle bigger resolutions**, and in this context, HDTV appears: 720p and 1080p.
  The p stands for progressive scan (not to confuse with 720i and 1080i: same resolution but with
  interlaced video):

  - 720p, _or standard HD_, is 1280x720 pixels: It’s a progressive HDTV signal format with 720
    horizontal lines and an aspect ratio (AR) of 16:9, normally known as widescreen HDTV (1.78:1).
    The frame rate is standards-dependent, and for conventional broadcasting appears in 50
    progressive frames per second in former PAL/SECAM countries (Europe, Australia, others), and
    59.94 frames per second in former NTSC countries (North America, Japan, Brazil, others). It can
    change or vary depending on the standard (i.e. broadcasting standards, the final chapter of this
    lesson).

  - 1080p, or _full HD_, is 1920x1080 pixels: The term usually assumes a widescreen aspect ratio of
    16:9, implying a resolution of 2.1 megapixels.
        Applications of the 1080p standard include television broadcasts, Blu-ray Discs, smartphones,
    Internet content such as YouTube videos and Netflix TV shows and movies, consumer-grade
    televisions and projectors, computer monitors, and videogame consoles. Small camcorders,
    smartphones, and digital cameras can capture still and moving images in 1080p resolution.

MPEG-4 files are smaller than JPEG or QuickTime files, so they are designated to transmit video and
images over a narrower bandwidth. Additionally, they can mix video with text, graphics, and 2D or 3D
animation layers. This was achieved with:

- **Video Objects and Video Object Planes:** A technique that allowed to add separate forms or
  _video objects_ to define the image.

![VOP](https://user-images.githubusercontent.com/40371955/147288327-420004a4-31f5-4c9f-bbad-fa9ec2720f8f.png)

- **Shape coding**: MPEG-4 video coding is the first to make effort at providing a standardized
  approach to compress the shape information of objects and contain the compressed results within a
  video bitstream. Video data can be coded on an object basis.

The information in the video signal is decomposed to shape, texture, and motion. This information is
then coded and transmitted within the bitstream. The shape information is provided in binary format
(with a pixel map) or gray-scale format.

- **Sprite coding:** A sprite is an especially composed VO that is visible throughout an entire
  video sequence. For example, the sprite generated from a panning sequence contains all the visible
  pixels of the background throughout the video sequence. A portion of the background may not be
  seen in certain frames due to the occlusion.

- **Interlaced video coding**
- **Wavelet-based texture coding**
- **Generalized Spatial and Temporal Scalability**

- **Error resilience**: MPEG4 provides error robustness and resilience to allow access to image and
  video data over a wide range of storage and transmission media.

- **The container concept** (Introduced in MPEG part 11, or `.mp4`: It enables that one file can
  have multiple tracks (video, audio, subtitles) that will be inside the video container.
 
 ![container](https://user-images.githubusercontent.com/40371955/147374591-fb238bee-d187-4f87-969d-831478ae2889.png)

When we used to see `.mpg` or `.mp2` files, those were unique videos with 1 track encoded with MPEG
and MPEG2 codec respectively. Now the container lets us mix codecs. I.e.: 1 MPEG video track, 2 MP3
audio tracks, 1 AAC audio track, 1 close caption/subtitles track...

- **Possibility to add DRM**: Digital rights management (DRM) tools or technological protection
  measures (TPM) are a set of access control technologies for restricting the use of proprietary
  hardware and copyrighted works. DRM technologies try to control the use, modification, and
  distribution of copyrighted works (such as software and multimedia content), as well as systems
  within devices that enforce these policies

- Lastly, the last versions of MPEG4 introduced a new codec for video: h264.

### h264

Also known as AVC (Advanced Video Coding) or MPEG-4 Part 10, Advanced Video Coding (MPEG4 AVC), is a
video compression standard based on block-oriented, motion-compensated integerDCT coding. It is the
most commonly used format for the recording, compression, and distribution of video content, used by
91% of video industry developers as of September 2019. It supports resolutions up to and including
8K UHD.

To provide a little historical context, until the launch of h264 (which will be explained in the
next sections), if you wanted to use MPEG-1 or MPEG-2 it was necessary to pay royalties to the owner
of the codec. And until then, it was a full monopoly of the MPEG. However, from 2001, h264 started
to compete with royalty-free codecs.

![royalty](https://user-images.githubusercontent.com/40371955/147291151-7db93354-7796-48bb-86e8-bbe952f0a7a5.png)

As MPEG2 was born due to achieve a broadcasting tech upgrade: compress interlaced broadcasting and
get digital video into satellite, cable, etc.; MPEG4 was born just to get better compression
bitrate, to send video through the internet. It had two main improvements:

- **Motion compensation:** Instead of 16x16 macroblocks, now they have a variable block size. This
  way, bigger blocks can be used for areas where the color is very similar.

![variableblocksize](https://user-images.githubusercontent.com/40371955/147374603-a312ac04-a828-4627-a0b1-b60e329775f1.png)

- **New entropy coding** at the end of the workflow: instead of using a variable-length coding based
  on Huffman, MPEG4 introduced two new algorithms -> CAVLC (Context Adaptative Variable Length
  Coding) and CABAC (Context Adaptive Binary Arithmetic Coding). They will substitute the RLE while
  being also lossless.

In H.264/MPEG-4 AVC, CAVLC is used to encode residual, zig-zag order, blocks of transform
coefficients. CABAC is notable for providing much better compression than most other entropy
encoding algorithms used in video encoding. CAVLC requires considerably less processing to decode
than CABAC, although it does not compress the data quite as effectively.

---

It's estimated the bitrate savings can be as much as 50% or more compared to MPEG-2.
[This video](https://www.youtube.com/watch?v=PmoEsPWEdOA) explains graphically the h264 compression
technology.

## Broadcasting standards

Although this section is more related to telecommunications, it is interesting to understand how TV
is broadcasted and see real-life applications of video encoding. Also, explanations about MPEG-TS
will be avoided, since it's not strictly necessary to understand the bigger picture. You just need
to know that Transport Stream exists. Think about that as an IPTV MPEG container.

The following map shows the four digital TV standards and their applications depending on each
country. As you can see their applications are very dependant on geopolitics and economic trades.

![tvmap](https://user-images.githubusercontent.com/40371955/147291729-a9c844f2-3692-42aa-aa8a-e5c4d04bb544.png)

Signals can either be: _terrestial_ (also named aerial) through antennas, _satellital_ or _cable_.
The last letter of each acronym corresponds to the medium in which they are transmitted. This can be
applied to all broadcasting standards: ISDB-T, ATSC-C, DVB-T, etc

### DVB (Digital Video Broadcasting)

DVB is a set of international, open standards for digital television. DVB standards are maintained
by the DVB Project, an international industry consortium, and are published by a Joint Technical
Committee (JTC) of the European Telecommunications Standards Institute (ETSI), European Committee
for Electrotechnical Standardization (CENELEC), and European Broadcasting Union (EBU).

For video, it uses MPEG2 and h.264. Usually is MPEG2 for SD channels and .h264 for HD channels. For
audio, it uses AAC, Dolby Digital (AC-3), and MP3.

The following screenshots are real examples from _Castilla la Mancha TV_ (Spain) and a German news
channel. In the first one, we can see the tracks inside the MPEG Transport Stream container, several
audio tracks in different codecs, and the main video track. On the other one, we see the same specs
in the video, several audio tracks, and two info tracks, subtitle and Teletext, all in the same
container.

![realexample](https://user-images.githubusercontent.com/40371955/147292769-832def9b-2e0a-4159-a96a-112e0ed4c7b3.png)

### ISDB

The Integrated Services Digital Broadcasting (ISDB; Japanese: 統合デジタル放送サービス, Tōgō
dejitaru hōsō sābisu) is a Japanese standard for digital television (DTV) and digital radio used by
the country's radio and television networks. ISDB replaced NTSC-J analog television system and the
previously used MUSE Hi-vision analog HDTV system in Japan and replaced NTSC, PAL-M, and PAL-N in
South America and the Philippines.

For video, as DVB: MPEG2, h.264. The first for SD channels and the latter for HD ones. Except on
ISDB-Tb (Brazilian & Latam), which uses only .h264. For audio it only uses AAC.

The following is a real-life example from public Ecuatorian TV. As you can see, it uses always h.264
for both SD and HD channels.

![quitoTV](https://user-images.githubusercontent.com/40371955/147293173-1adf2ee5-860d-4846-a6cb-90c123a29785.png)

### ATSC

ATSC, Advanced Television Systems Committee (ATSC) is largely a replacement for the analog NTSC
standard, and like that standard, used mostly in the United States, Mexico, and Canada. Other former
users of NTSC, like Japan, have not used ATSC during their digital television transition because
they adopted their system, ISDB.

ATSC includes two primary high-definition video formats, 1080i and 720p. It also includes
standard-definition formats, although initially only HDTV services were launched in the digital
format. ATSC can carry multiple channels of information on a single stream.

For video channels, it works as DVB: MPEG2 for SD and .h264 for HD. However, for audio it uses AC-3,
not a very performant codec ([here](<https://en.wikipedia.org/wiki/Grand_Alliance_(HDTV)>) you can
learn more about why).

As before, in the following image, there's a real-life example from USA and Mexican TV. As you can
see they use the AC-3 for audio, and the USA TV has an additional data track that sends timecodes to
inform where the advertisments are placed.

![usamexTV](https://user-images.githubusercontent.com/40371955/147293550-4a4b3caa-0163-4a0a-b010-1fbabfc8ee45.png)

### DTMB

Previously known as DMB-T/H (Digital Multimedia Broadcast-Terrestrial/Handheld), the DTMB is a
merger of the standards ADTB-T (developed by the Shanghai Jiao Tong University), DMB-T (developed by
Tsinghua University) and TiMi (Terrestrial Interactive Multiservice Infrastructure); this last one
is the standard proposed by the Academy of Broadcasting Science in 2002. The DTMB was created in
2004 and finally became an official DTT standard in 2006.

For video, it uses its own codecs: AVS, AVS+, and MPEG2, h.264. AVS is a Chinese codec very similar
to .h264. For audio, it uses DRA, AAC, AC-3, MP2, MP3. Being DRA (Dynamic Resolution Adaptation) the
Chinese own audio codec.

---

#### Resources

<FileDownloadLink
  name='Slides T3'
  filepath='/files/T3 - MPEG4, h.264 and Broadcasting standards.pdf'
/>
<FileDownloadLink name='Seminar 2' filepath='/files/S2 - More FFMPEG  More Python.pdf' />


Recent video codecs such as MPEG4 and h264, and broadcasting standards

T3. MPEG4, h.264 & broadcasting standards


## Codec Wars

Codec wars is a term used in the industry to refer to movements of technological companies towards
video codecs. The term started back around 2004 when internet access was democratized and higher
speeds were achieved. It was cheaper and easier to access fiber connections, deprecating the old
modems and letting users connect to high speeds through an ethernet cable.

In this context, video over IP or IPTV (Internet Protocol TeleVision) started to roll out. IPTV is a
wide concept to express all video delivered through the internet. As Moore's law dictates, encoding
processing is exponentially faster than before, allowing to do in real-time, encoding that was
previously impossible. In this environment, some companies related to video started to grow, such
as:

- **Netflix**: starting as an ordinary mail movie rental service, a competitor of _Blockbuster_.
  They started to do data mining and big data to make individual film documentation through their
  website and allowed their users to order the DVDs from there. The competition was hard, and just
  before bankruptcy in 2005, they started sharing online its content as a last resort strategy,
  which as you may have realized, worked pretty well.

- **Youtube**: created by 3 former Paypal employees in February 2005, aimed to be an enhancement for
  blogs and social networks, a service to upload content and insert it into user’s website. Myspace
  users started to adopt this service in their comments and profiles, increasing its popularity.
  Before _Youtube_, when a company wanted to have videos on their website, they needed to take care
  of encoding, storage, delivery, web player, etc. When _Nike_ launched a Ronaldinho ad in late 2005
  using _Youtube_, other companies started to focus on it, and finally, in October 2006 Google
  bought it for 1650M$.

### Current situation

The two companies listed before had been for years the biggest promoters of video through the
internet for commercial and non-commercial use. However, over the past few years, especially after
the covid pandemic, a lot of video-on-demand companies have appeared to share the pie:

![useVoDuse](https://user-images.githubusercontent.com/40371955/147357779-ba812548-089e-4d28-a826-e52cb8aa4dd9.png)

Video traffic through the internet has increased exponentially, representing now an estimated
**70% - 80% of internet traffic**.

This situation changed the approach towards future video codecs, which as we've seen, are always
created to solve technological challenges. These are the technological challenges of today, focused
on IPTV:

- **Reduce data** to decrease storage costs
- **Reduce bitrate** to make delivery faster
- **Reduce complexity** to reduce latency

And all of those challenges, of course, **without losing quality**.

Taking this into account, new codecs appeared, starting to break the monopoly of the MPEG, and
started the _codec wars_. Companies wanted to explore royalty-free codecs to reduce costs (the same
way _Apple_ started to use their audio codec to avoid paying royalties to MP3). This generated
competition between codecs to become the standard. This is used by companies as a weapon against
rivals, making better or worse support of codecs in their hardware or browsers.

![codeccomparison](https://user-images.githubusercontent.com/40371955/147374642-ba7a2fe8-6e9c-41be-89d7-986bf42af312.png)

A summary of these technological-economic wars is: If you get a patent, you get paid. If you develop
better codecs, you can offer better services and user experiences for your customers. You can push
or block other companies’ codecs in your hardware to avoid them to succeed. This is not only done
through codecs (i.e. Twitch vs Youtube). A clear historical example of this technological format
competition can be _VHS vs BetaMax_. It’s was the first time that big companies were fighting with
codecs to get a bigger share of the market. VHS was developed by _Panasonic_ and _Sony_ developed
Betamax. The latter was technologically superior, but the relations of Panasonic at the time with
the cinema industry forced Betamax to become obsolete.

## VP8

Despite being open-source, VP8 is owned by Google (_codec wars!_). In May 2010, after the purchase
of On2 Technologies, Google provided an irrevocable patent promise on its patents for implementing
the VP8 format and released a specification under the Creative Commons Attribution 3.0 license. That
same year, Google also released libvpx, the reference implementation of VP8, under the revised BSD
license.

Opera, Firefox, Chrome, and Chromium support playing VP8 video in HTML5 video tag. Internet Explorer
officially supports VP8 with a separate codec. According to Google VP8 is mainly used in connection
with WebRTC and as a format for short looped animations, as a replacement for the Graphics
Interchange Format (GIF).

![vp8diagram](https://user-images.githubusercontent.com/40371955/147359177-32bc18df-05d0-465b-87f6-f6533038e91d.png)

As you can see, all video codecs are based on the same concept, but with some differences. VP8 is
very similar to h264, it changes the prediction modes (motion compensation) and uses DCT with 4x4
macroblocks and Hadamard transform.

VP8 only supports progressive scan video signals with 4:2:0 chroma subsampling and 8 bits per
sample. Macroblocks can comprise 4×4, 8×8, or 16×16 samples, and motion vectors have quarter-pixel
precision.

An interesting change in this codec is the addiction of _altframes_ (alternative frames), which can
serve as reference-only frames and can be deactivated. The encoder can fill them with arbitrary
useful image data, even from future frames, and thereby serve the same purpose as the b-frames of
the MPEG formats.

### Matroska files

The VP8 also added its kind of file, the **Matroska** `.mkv`. The Matroska Multimedia Container is a
free, open standard container format, a file format that can hold an unlimited number of video,
audio, picture, or subtitle tracks in one file.

Matroska is similar in concept to other containers like AVI, MP4, or Advanced Systems Format (ASF),
but is entirely open in specification, with implementations consisting mostly of open-source
software.

In 2010, it was announced that the WebM audio/video format would be based on a profile of the
Matroska container format together with VP8 video and Vorbis audio. It’s also used for illegal
BlueRay Disc rips and it added support to the next codec we’ll see: h265.

## h265

Also known as HEVC (High-Efficiency Video Coding) is the .h264 successor, and offers from 25% to 50%
better data compression at the same level of video quality, or substantially improved video quality
at the same bit rate. It supports resolutions up to 8192×4320, including 8K UHD.

While .h264 uses the integer discrete cosine transform (DCT) with 4x4 and 8x8 block sizes, HEVC uses
integer DCT and DST transforms with varied block sizes between 4x4 and 32x32. The High-Efficiency
Image Format (HEIF) is based on HEVC. As of 2019, HEVC is used by 43% of video developers and is the
second most widely used video coding format after .h264. It has a basic free version, but it’s
patented and theoretically, you should pay if you use .mp4 files.

The HEVC video coding layer uses the same "hybrid" approach used in all modern video standards: it
uses inter-/intra-picture prediction and 2D transform coding. The HEVC encoder first proceeds by
splitting a picture into block-shaped regions for the first picture, or the first picture of a
random access point, which uses intra-picture prediction. After the prediction methods are finished
and the picture goes through the loop filters, the final picture representation is stored in the
decoded picture buffer, which then can be used for the prediction of other pictures. Another
characteristic is its lack of native support of interlaced video.

HEVC replaces 16×16 pixel macroblocks, which were used with previous standards, with coding tree
units (CTUs) which can use larger block structures of up to 64x64 samples and can better
sub-partition the picture into variable-sized structures. HEVC initially divides the picture into
CTUs which can be 64×64, 32×32, or 16×16 with a larger pixel block size usually increasing the
coding efficiency.

It also uses a new version of CABAC and introduces new modes for motion estimation (33 vector
possibilities). It has a longer filter for chroma subsampling, and lots of other filters, and small
internal upgrades.

![comparisonh264h265](https://user-images.githubusercontent.com/40371955/147360234-bf25086e-cefa-4414-b6c1-fa93ebc585ff.png)

The most important aspect of the codec is its huge reduction of the size of the files compared to
h264, and the containment of video quality. The estimated ratio is between -25% and -50% bitrate
reduction. On the other hand, it's not fully supported on Chrome & Firefox (_codec wars!_).

## VP9

VP9 is an open and royalty-free video coding format developed by Google. Its the successor to VP8
and competes mainly with MPEG's HEVC (H.265, as recently seen). At first, VP9 was mainly used on
Google's video platform, YouTube, however, the emergence of the Alliance for Open Media, and its
support for the ongoing development of the successor AV1, in which Google took part, led to growing
interest in the format. In contrast to HEVC, VP9 support is common among web browsers. The
combination of VP9 video and Opus audio in the WebM container, as served by YouTube, was supported
by roughly 80% of the browser market (mobile included) as of June 2018.

The single holdout among major modern-day browsers is Apple's Safari (_codec wars!_). Android has
supported VP9 since version 4.4 KitKat. Parts of the format are covered by patents held by Google.
The company grants free usage of its related patents based on reciprocity, i.e. as long as the user
does not engage in patent litigations.

VP9 has many design improvements compared to VP8:

- Support for the use of coding units of 64×64 pixels. This is especially useful with
  high-resolution video.
- Improved prediction of motion vectors. In addition to VP8's four modes (average/"DC", "true
  motion", horizontal, vertical), VP9 supports six oblique directions for linear extrapolation of
  pixels in intraframe prediction.
- Other small upgrades such as:
  - eighth-pixel precision for motion vectors
  - three different switchable 8-tap sub-pixel interpolation filters
  - improved selection of reference motion vectors
  - improved coding of offsets of motion vectors to their reference
  - improved entropy coding
  - improved and adapted (to new block sizes) loop filtering
  - the asymmetric discrete sine transform (ADST)
  - larger discrete cosine transforms (DCT, 16×16 and 32×32)
  - improved segmentation of frames into areas with specific similarities (e.g. fore-/background)

Another peculiarity is that transform coefficients are scanned in a round pattern (increasing
distance from the corners):
![vp9pattern](https://user-images.githubusercontent.com/40371955/147360735-1e788602-ab42-4cf1-8e71-fb0574b7230b.png)

### WebM

As the previous codec, it has its video file `.WEBM`. WebM is an audiovisual media file format,
which is primarily intended to offer a royalty-free alternative to use in the HTML5 video and the
HTML5 audio elements. It has a sister project WebP for images. The development of the format is
sponsored by Google, and the corresponding software is distributed under a BSD license.

The WebM container is based on a profile of Matroska. WebM initially supported VP8 video and Vorbis
audio streams. In 2013, it was updated to accommodate VP9 video and Opus audio.

Native WebM support by Mozilla Firefox, Opera, and Google Chrome was announced at the 2010 Google
I/O conference. Didn’t have support on Microsoft Edge/Explorer until 2017 and it is still not
supported on Safari (_you already know what to say here_).

## AV1 - the real future?

AOMedia Video 1 (AV1) is an open, royalty-free video coding format designed for video transmissions
over the Internet. It was developed as a successor to VP9 by the Alliance for Open Media (AOMedia),
a consortium founded in 2015 that includes semiconductor firms, video on demand providers, video
content producers, software development companies, and web browser vendors.

The AV1 bitstream specification includes a reference video codec. In Facebook testing that
approximates real-world conditions AV1 achieved **34%, 46.2%, and 50.3%** higher data compression
than libvpx-vp9, x264 high profile, and x264 main profile respectively. This codec is very
promising, with the only drawback of needing very powerful machines to encode.

![av1diagram](https://user-images.githubusercontent.com/40371955/147361919-cf37e949-e895-43d9-916b-e752588e5bc7.png)

Several of the Alliance's founding members, including Google (VP10), Mozilla (Daala), and Cisco
(Thor), had ongoing research projects into royalty-free video and incorporated aspects of each into
AV1, but the patent owner is Google.

As for the technology, if you want to learn about how it works, go to the slides in PDF.

Some companies such as _Youtube, Netflix, Facebook_ or _Twitch_ are starting to use this codec.

---

#### Resources

<FileDownloadLink name='Slides T4' filepath='/files/T4 - Future in codecs.pdf' />
<FileDownloadLink name='Final Exercices' filepath='/files/SP3 - Final exercises.pdf' />


An overview of the future of video codecs, including some of the most recent ones such as VP9 or h265

>>>
Learn everything about
video encoding

Video Encoding
Course

Learning material
from the Pompeu Fabra University
'Audio and Video Encoding Systems' course

Last updates

Introduction and teaching plan

T1. JPEG & JPEG2000

T2. MPEG & MPEG2

>>>Learn everything about video encoding

Video Encoding Course

Learning material from the Pompeu Fabra University 'Audio and Video Encoding Systems' course

Last updates

Introduction and teaching plan

T1. JPEG & JPEG2000

T2. MPEG & MPEG2

>>>
Learn everything about
video encoding

Video Encoding
Course

Learning material
from the Pompeu Fabra University
'Audio and Video Encoding Systems' course