Entering the world of audio and video - RGB and YUV formats

在图像的世界里,一般使用RGB作为存储格式.And in the world of video,一般使用YUVas a compressed storage format.Sometimes the interviewer will ask：Why video is usedYUVto compress storage,而不用RGB？YUV与RGB有什么区别,How are the two converted？常见的RGB格式有哪些,常见的YUVWhat are the formats？What is the preview format of the phone camera,如何转换为YUV420P的？我们带着这些问题,来揭开RGB与YUVFormat veil.

一、RGB格式

RGBis an image storage format,Also three primary colors,取值范围[0, 255].R代表Red红色,G代表Green绿色,B代表Blue蓝色.在openCV中,一般使用BGR格式.在图像中,一般使用32位色的ARGB(或RGBA)代表一个像素,其中A代表Alpha透明度.

1、RGBA8888

关于RGBA8888格式,each channel station8位,即一个字节.Four channels make up a pixel,总共占32位.排列顺序如下图所示：

 2、RGB565

关于RGB565格式,其中R占5位,G占6位,B占5位.Three channels make up a pixel,总共占16位.排列顺序如下图所示：

 3、The pixel array of the image

An image consists of widthxtall pixel array,为了内存对齐,会使用stride来填充.如下图所示,由4x3composed of pixel arrays,其中P代表pixel：

二、YUV格式

YUVIt is a video compression storage format.其中Y代表Luma亮度,U代表Chroma色度,V代表Contrast对比度.常见的YUVThe sampling ratio is as follows：

• 4:4:4 表示完全采样
• 4:2:2 表示水平2:1采样,垂直完全采样
• 4:2:0 表示水平2:1采样,垂直2:1采样
• 4:1:1 表示水平4:1采样,垂直完全采样

常见的YUV格式有：YUV420p、YUV420sp、NV21等.

1、YUV420p

YUV420pIt belongs to flat storage,YUVThe weight ratio is 4:1:1,即每4个Y共享一组UV.排列如下图所示：

 2、YUV420sp

YUV420spIt belongs to interleaved storage,YUVThe weight ratio is 4:1:1,即每4个Y共享一组UV.排列如下图所示：

 3、NV21

NV21It belongs to interleaved storage,YUVThe weight ratio is 4:1:1,即每4个Y共享一组UV.和YUV420sp的区别是,NV21是VUVU这样排列,如下图所示：

三、RGB与YUV转换

关于YUV与RGB的转换公式,可参考ITU标准：https://www.itu.int/rec/R-REC-BT.601.See also Wikipedia：https://zh.wikipedia.org/wiki/YUV.Let's look at the conversion formula：

以rgb转yuv为例,示例代码如下：

void rgb_to_yuv(int8_t *yuv, int *rgb, int width, int height) {
    int rgbIndex = 0;
    int yIndex = 0;
    int uIndex = width * height;
    int vIndex = width * height * 5 / 4;
    int R, G, B;
    float Y, U, V;

    // 遍历图像,获取所有像素点 
    for (int i = 0; i < height; i++) {
        for (int j = 0; j < width; j++) {
            // Obtained from pixelsR、G、B分量
            R = (rgb[rgbIndex] & 0xFF0000) >> 16;
            G = (rgb[rgbIndex] & 0xFF00) >> 8;
            B = (rgb[rgbIndex] & 0xFF);
            // Use the formula to putRGB转成YUV
            Y = 0.299 * R + 0.587 * G + 0.114 * B;
            U = -0.147 * R - 0.289 * G + 0.436 * B;
            V = 0.615 * R - 0.515 * G - 0.100 * B;
            // YUVThe component is assigned toyuv数组
            yuv[yIndex++] = (int8_t)Y;
            if (i % 2 == 0 && j % 2 == 0) {
                yuv[uIndex++] = (int8_t) U;
                yuv[vIndex++] = (int8_t) V;
            }
            rgbIndex++;
        }
        
    } 
}

四、NV21转换为YUV420p

由于NV21是交错存储,4个Y共享一组UV,而且是VUVU这样排列.所以,We need to put even numbersV分量、奇数的UQuantity is read out,然后赋值给YUV420p.代码如下：

static void nv21_to_yuv420p(int8_t *dst, int8_t *src, int len) {
    memcpy(dst, src, len); // y
    for (int i = 0; i < len / 4; ++i) {
        *(dst + len + i) = *(src + len + i * 2 + 1);  // u
        *(dst + len * 5 / 4 + i) = *(src + len + i * 2); // v
    }
}

五、YUV旋转

YUVThe storage is the presence of a rotation angle.when shooting with a mobile phone,Look counterclockwise,Landscape is to the left0度,vertical screen down90度,Landscape is to the right180度,Vertical screen is up270度.Since there is a rotation angle,我们就需要对YUV进行旋转处理,代码如下：

static void yuv420p_rotate90(int8_t *dst, const int8_t *src, int width, int height) {
    int n = 0;
    int wh = width * height;
    int half_width = width / 2;
    int half_height = height / 2;
    // y
    for (int j = 0; j < width; j++) {
        for (int i = height - 1; i >= 0; i--) {
            dst[n++] = src[width * i + j];
        }
    }
    // u
    for (int i = 0; i < half_width; i++) {
        for (int j = 1; j <= half_height; j++) {
            dst[n++] = src[wh + ((half_height - j) * half_width + i)];
        }
    }
    // v
    for (int i = 0; i < half_width; i++) {
        for (int j = 1; j <= half_height; j++) {
            dst[n++] = src[wh + wh / 4 + ((half_height - j) * half_width + i)];
        }
    }
}

static void yuv420p_rotate180(int8_t *dst, const int8_t *src, int width, int height) {
    int n = 0;
    int half_width = width / 2;
    int half_height = height / 2;
    // y
    for (int j = height - 1; j >= 0; j--) {
        for (int i = width; i > 0; i--) {
            dst[n++] = src[width * j + i - 1];
        }
    }
    // u
    int offset = width * height;
    for (int j = half_height - 1; j >= 0; j--) {
        for (int i = half_width; i > 0; i--) {
            dst[n++] = src[offset + half_width * j + i - 1];
        }
    }
    // v
    offset += half_width * half_height;
    for (int j = half_height - 1; j >= 0; j--) {
        for (int i = half_width; i > 0; i--) {
            dst[n++] = src[offset + half_width * j + i - 1];
        }
    }
}

static void yuv420p_rotate270(int8_t *dst, const int8_t *src, int width, int height) {

    for (int j = 0; j < width; j++) {
        for (int i = 1; i <= height; i++) {
            *dst++ = *(src + i * width - j);
        }
    }

    auto *src_u = const_cast<int8_t *>(src + width * height);
    for (int j = 0; j < width / 2; j++) {
        for (int i = 1; i <= height / 2; i++) {
            *dst++ = *(src_u + i * width / 2 - j);
        }
    }

    auto *src_v = const_cast<int8_t *>(src + width * height * 5 / 4);
    for (int j = 0; j < width / 2; j++) {
        for (int i = 1; i <= height / 2; i++) {
            *dst++ = *(src_v + i * width / 2 - j);
        }
    }
}

static void yuv420p_rotate(int8_t *dst, int8_t *src, int width, int height, int degree) {
    switch(degree) {
        case 0:
            memcpy(dst, src, width * height * 3 / 2);
            break;
        case 90:
            yuv420p_rotate90(dst, src, width, height);
            break;
        case 180:
            yuv420p_rotate180(dst, src, width, height);
            break;
        case 270:
            yuv420p_rotate270(dst, src, width, height);
            break;
        default:
            break;
    }
}