wangcong/story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: checkpoint 第一版
ci / verify (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
Codex Local
2026-09-06 17:04:11 +08:00
parent a8cdec98d1
commit 0c2151ace9
659 changed files with 185855 additions and 554 deletions
Download Patch File Download Diff File Expand all files Collapse all files
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/CHANGELOG.md
+16
View File
@@ -0,0 +1,16 @@
# Changelog
## v0.7.1
### Features
- Add QP and GOP setting support for video encoder element
## v0.7.0
### Features
This is the initial release of the `gmf_video` component, introducing the following features:
* Core video processing elements: decoder, encoder, frame rate converter, overlay mixer, and hardware-accelerated Pixel Processing Accelerator (PPA)
* Software-based video effects: cropper, scaler, color converter, and rotator
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/CMakeLists.txt
+8
View File
@@ -0,0 +1,8 @@
if (${IDF_TARGET} STREQUAL "esp32p4")
list(APPEND COMPONENT_REQUIRES esp_driver_ppa esp_hw_support esp_mm)
endif()
idf_component_register(SRC_DIRS "./"
INCLUDE_DIRS "./include"
REQUIRES ${COMPONENT_REQUIRES}
PRIV_INCLUDE_DIRS "./private_include")
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/LICENSE
+22
View File
@@ -0,0 +1,22 @@
Espressif Modified MIT License
Copyright (c) 2025 Espressif Systems (Shanghai) CO., LTD
Permission is hereby granted for use EXCLUSIVELY with Espressif Systems products.
This includes the right to use, copy, modify, merge, publish, distribute, and sublicense
the Software, subject to the following conditions:
1. This Software MUST BE USED IN CONJUNCTION WITH ESPRESSIF SYSTEMS PRODUCTS.
2. The above copyright notice and this permission notice shall be included in all copies
or substantial portions of the Software.
3. Redistribution of the Software in source or binary form FOR USE WITH NON-ESPRESSIF PRODUCTS
is strictly prohibited.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/README.md
+93
View File
@@ -0,0 +1,93 @@
# ESP-GMF-Video
- [![Component Registry](https://components.espressif.com/components/espressif/gmf_video/badge.svg)](https://components.espressif.com/components/espressif/gmf_video)
- [中文版](./README_CN.md)
ESP GMF Video is a suite of video processing elements designed for a variety of tasks such as video encoding, and applying video effects. These elements can be combined to form a complete video processing pipeline.
## Supported Video Modules
The following table lists the currently supported video processing modules and their detailed information:
|Name|TAG|Function|Method|Input Port|Output Port|Hardware Acceleration|Dependent on Video Information|
|:----:|:----:|:-----:|:----:|:----:|:----:|:----:|:----|
| VIDEO_ENC | vid_enc | Video encoder: H264, MJPEG | `set_bitrate`<br>`set_dst_codec`<br>`get_src_fmts`<br>`preset`<br>`get_frame_size` | Single | Single | Yes | Yes |
| VIDEO_DEC | vid_dec | Video decoder: H264, MJPEG | `set_dst_fmt`<br>`set_src_codec`<br>`get_dst_fmts` | Single | Single | No | No |
| VIDEO_PPA | vid_ppa | Pixel processing accelerator: Color conversion,<br>Scaling, Cropping, Rotation | `set_dst_format`<br>`set_dst_resolution`<br>`set_rotation`<br>`set_cropped_rgn` | Single | Single | Yes | Yes |
| FPS_CVT | vid_fps_cvt | Frame rate conversion | `set_fps` | Single | Single | No | Yes |
| OVERLAY_MIXER | vid_overlay | Video overlay mixer | `overlay_enable`<br>`set_rgn`<br>`set_port`<br>`set_alpha` | Multiple | Single | No | Yes |
| COLOR_CVT | vid_color_cvt | Software color conversion | `set_dst_fmt` | Single | Single | No | Yes |
| CROP | vid_crop | Software video cropping | `set_crop_rgn` | Single | Single | No | Yes |
| SCALE | vid_scale | Software video scaling | `set_dst_res` | Single | Single | No | Yes |
| ROTATE | vid_rotate | Software video rotation | `set_angle` | Single | Single | No | Yes |
## Elements
### Video Encoder
The Video Encoder element uses the `esp_video_codec` to convert raw video frames into compressed video streams. It currently supports two codecs:
- **H264**
- **MJPEG**
### Video Decoder
The video decoder element use the `esp_video_codec` to convert compressed video streams into raw video frames. It currently supports two codecs：
- **H264**
- **MJPEG**
### Video PPA (Pixel Processing Accelerator)
The Video PPA element is a compact element, support multiple functions. It is currently available on the ESP32P4 board and includes following functionalities:
- **Color Conversion:**
The ESP32P4 supports color conversion through two hardware modules:
- **2D-DMA:** Automatically selected for better efficiency if supported.
- **PPA:** Used as a fallback when 2D-DMA not supported
- **Resizing:**
Resizing functionality is provided by the PPA module.
- **Cropping:**
Cropping functionality is provided by the PPA module.
- **Rotation:**
Supports rotations at 0°, 90°, 180°, and 270°.
### Video FPS Converter
This module adjusts the frame rate of the video. It decreases the input frame rate to a specified output rate, using the Presentation Time Stamp (PTS) embedded in the input data to accurately schedule frames.
### Video Overlay Mixer
The Video Overlay Mixer module allows users to overlay additional graphics onto a video frame. By receiving overlay data via a user-defined port, it can blend elements such as timestamps, watermarks, or other images into a designated region of the original video frame.
### Video Pixel Processor Elements
Following elements are wrapped for [Video Pixel Processor](https://github.com/espressif/esp-adf-libs/tree/master/esp_image_effects) which implemented software video processing.
#### Video Color Converter
Element to do software color conversion for video image
#### Video Cropper
Element to do software video cropper for video image
#### Video Scaler
Element to do software video scaler for video image
#### Video Rotator
Element to do software video rotator for video image
## ESP-GMF-Video Release and SoC Compatibility
The following table summarizes the support for ESP-GMF-Video elements across Espressif SoCs in the current release.
A check mark (&#10004;) indicates that the element is supported, while a cross mark (&#10006;) indicates it is not supported.
| Element | ESP32 | ESP32-S2 | ESP32-S3 | ESP32-P4 |
|-----------------|:-----------:|:-----------:|:-----------:|:-----------:|
| Video PPA | &#10006; | &#10006; | &#10006; | &#10004; |
| FPS Converter | &#10004; | &#10004; | &#10004; | &#10004; |
| Overlay Mixer | &#10004; | &#10004; | &#10004; | &#10004; |
| Video Decoder | MJPEG only | MJPEG only | &#10004; | &#10004; |
| Video Encoder | MJPEG only | MJPEG only | &#10004; | &#10004; |
| Video Pixel Processor | &#10004; | &#10004; | &#10004; | &#10004; |
## Notes
- **Video PPA** is currently supported **only** on the ESP32-P4.
- **FPS Converter** and **Overlay Mixer** are supported on **all SoCs**.
- **Video Decoder** supports MJPEG decoding on ESP32 and ESP32-S2
- **Video Encoder** supports MJPEG encoding on ESP32 and ESP32-S2
## Usage
ESP GMF Video modules are often used together to build a complete video processing pipeline. For example, you might first convert its colors or size, adjust the frame rate, and overlay and finally output through video encoder. For a practical implementation, please refer to the example in [test_app](../test_apps/main/elements/gmf_video_el_test.c).
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/README_CN.md
+93
View File
@@ -0,0 +1,93 @@
# ESP-GMF-Video
- [![Component Registry](https://components.espressif.com/components/espressif/gmf_video/badge.svg)](https://components.espressif.com/components/espressif/gmf_video)
- [English Version](./README.md)
ESP GMF Video 是一套专为视频编解码和视频转换等设计的视频处理模块。这些模块可以组合在一起，形成完整的视频处理流水线。
## 支持的视频模块
下表列出了当前支持的视频处理模块及其详细信息：
| 名称 | 标签 | 功能 | 函数方法 | 输入端口 | 输出端口 | 硬件加速 |依赖视频信息 |
|:----:|:-----:|:----:|:----:|:----:|:----:|:----:|:----|
| VIDEO_ENC | vid_enc | 视频编码器：H264, MJPEG | `set_bitrate`<br>`set_dst_codec`<br>`get_src_fmts`<br>`preset`<br>`get_frame_size` | 单个 | 单个 | 是 | 是 |
| VIDEO_DEC | vid_dec | 视频解码器：H264, MJPEG | `set_dst_fmt`<br>`set_src_codec`<br>`get_dst_fmts` | 单个 | 单个 | 否 | 否 |
| VIDEO_PPA | vid_ppa | 像素处理加速器：颜色转换，缩放，<br>裁剪，旋转 | `set_dst_format`<br>`set_dst_resolution`<br>`set_rotation`<br>`set_cropped_rgn` | 单个 | 单个 | 是 | 是 |
| FPS_CVT | vid_fps_cvt | 帧率转换 | `set_fps` | 单个 | 单个 | 否 | 是 |
| OVERLAY_MIXER | vid_overlay | 视频叠加混合器 | `overlay_enable`<br>`set_rgn`<br>`set_port`<br>`set_alpha` | 多个 | 单个 | 否 | 是 |
| COLOR_CVT | vid_color_cvt | 软件颜色转换 | `set_dst_fmt` | 单个 | 单个 | 否 | 是 |
| CROP | vid_crop | 软件视频裁剪 | `set_crop_rgn` | 单个 | 单个 | 否 | 是 |
| SCALE | vid_scale | 软件视频缩放 | `set_dst_res` | 单个 | 单个 | 否 | 是 |
| ROTATE | vid_rotate | 软件视频旋转 | `set_angle` | 单个 | 单个 | 否 | 是 |
## 元素
### 视频编码器
视频编码器模块使用 `esp_video_codec` 将原始视频帧转换为压缩视频流。目前支持以下两种编码格式：
- **H264**
- **MJPEG**
### 视频解码器
视频编码器模块使用 `esp_video_codec` 将压缩视频流解码为原始视频帧。目前支持以下两种解码格式：
- **H264**
- **MJPEG**
### 视频像素加速器 (PPA)
视频像素加速器模块是一个复合模块，目前仅适用于 ESP32P4 ，包含以下功能：
- **颜色转换：**
ESP32P4 通过两个硬件模块支持颜色转换：
- **2D-DMA：** 如果支持，将自动选择以提高效率。
- **PPA：** 当 2D-DMA 不支持时作为备用方案。
- **缩放：**
由 PPA 模块提供缩放功能。
- **裁剪:**
由 PPA 模块提供裁剪功能。
- **旋转：**
支持 0°、90°、180° 和 270° 旋转。
### 视频帧率转换
此模块用于调整视频的帧率。它会根据输入数据中嵌入的演示时间戳（PTS）来准确地调整视频帧，从而将输入帧率降低到指定的输出帧率。
### 视频叠加混合器
视频叠加混合器模块允许用户在视频帧上叠加额外的图像。它通过用户定义的端口接收叠加数据，并将诸如时间戳、水印或其他图像等元素混合到原始视频帧的指定区域。
### 视频像素处理器元素
以下元素是为[视频像素处理器]（https://github.com/espressif/esp-adf-libs/tree/master/esp_image_effects）包装的，它实现了软件视频处理。
#### 视频彩色转换器
软件实现视频图像不同颜色转换
#### 视频剪辑器
软件实现视频图像任意位置及大小的裁剪
#### 视频缩放器
软件实现视频图像放大和缩小
#### 视频旋转器
软件实现视频图像任意角度的旋转
# ESP-GMF-Video SoC 兼容性
下表总结了当前版本中 ESP-GMF-Video 元素在乐鑫各 SoC 上的支持情况。
&#10004; 表示支持，&#10006; 表示不支持。
| 元素 | ESP32 | ESP32-S2 | ESP32-S3 | ESP32-P4 |
|----------------|:-----------:|:-----------:|:-----------:|:-----------:|
| 视频像素加速器 | &#10006; | &#10006; | &#10006; | &#10004; |
| 帧率转换器 | &#10004; | &#10004; | &#10004; | &#10004; |
| 叠加混合器 | &#10004; | &#10004; | &#10004; | &#10004; |
| 视频解码器 | 仅支持 MJPEG | 仅支持 MJPEG | &#10004; | &#10004; |
| 视频编码器 | 仅支持 MJPEG | 仅支持 MJPEG | &#10004; | &#10004; |
| 视频像素处理器 | &#10004; | &#10004; | &#10004; | &#10004; |
## 备注
- **视频像素加速器** 目前仅在 **ESP32-P4** 上支持。
- **帧率转换器** 和 **叠加混合器** 在**所有 SoC**上均支持。
- **视频解码器** 在 ESP32 和 ESP32-S2 上仅支持 MJPEG 解码。
- **视频编码器** 在 ESP32 和 ESP32-S2 上仅支持 MJPEG 编码。
## 使用方法
ESP GMF Video 模块通常组合使用，以构建完整的视频处理流水线。例如，您可以先调整帧率，然后转换颜色或调整大小，叠加特效，最终通过视频编码器输出。有关具体用法，请参考 [test_app](../test_apps/main/elements/gmf_video_el_test.c) 示例。
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_color_convert.c
+314
View File
@@ -0,0 +1,314 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include <string.h>
#include <inttypes.h>
#include "esp_log.h"
#include "esp_gmf_oal_mutex.h"
#include "esp_gmf_oal_mem.h"
#include "esp_gmf_err.h"
#include "esp_gmf_node.h"
#include "esp_gmf_video_element.h"
#include "esp_gmf_video_color_convert.h"
#include "esp_gmf_video_methods_def.h"
#include "gmf_video_common.h"
#include "esp_gmf_caps_def.h"
static const char *TAG = "IMGFX_CLRCVT_EL";
typedef struct _gmf_imgfx_color_convert_t {
esp_gmf_video_element_t parent;
esp_imgfx_color_convert_handle_t hd;
bool need_recfg;
} esp_gmf_color_convert_hd_t;
static esp_gmf_job_err_t video_cc_el_open(esp_gmf_element_handle_t self, void *para)
{
esp_gmf_color_convert_hd_t *video_el = (esp_gmf_color_convert_hd_t *)self;
// Get and check configuration
esp_imgfx_color_convert_cfg_t *cfg = (esp_imgfx_color_convert_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_CHECK(TAG, cfg, return ESP_GMF_JOB_ERR_FAIL, "Failed to get color convert config");
// Open color convert module
esp_imgfx_color_convert_open(cfg, &video_el->hd);
ESP_GMF_CHECK(TAG, video_el->hd, return ESP_GMF_JOB_ERR_FAIL, "Failed to create color convert handle");
// Get video size
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->in_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size));
esp_imgfx_get_image_size(cfg->out_pixel_fmt, &cfg->in_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size));
// Report information to the next element, the next element will use this information to configure
gmf_video_update_info(self, cfg->in_res.width, cfg->in_res.height, cfg->out_pixel_fmt);
// The video_el->hd has been opened using newest configuration, so it can set the need_recfg to false
video_el->need_recfg = false;
ESP_LOGD(TAG, "Open, %p", self);
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_job_err_t video_cc_el_process(esp_gmf_element_handle_t self, void *para)
{
esp_gmf_color_convert_hd_t *video_el = (esp_gmf_color_convert_hd_t *)self;
esp_imgfx_data_t in_image;
esp_imgfx_data_t out_image;
esp_imgfx_color_convert_cfg_t *cfg = (esp_imgfx_color_convert_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_NULL_CHECK(TAG, cfg, return ESP_GMF_JOB_ERR_FAIL);
bool bypass = cfg->in_pixel_fmt == cfg->out_pixel_fmt;
if (video_el->need_recfg) {
// reset color convert config
esp_imgfx_err_t imgfx_ret = esp_imgfx_color_convert_set_cfg(video_el->hd, cfg);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
return ESP_GMF_JOB_ERR_FAIL;
}
// Get video size
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->in_res, (uint32_t *)&ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size);
esp_imgfx_get_image_size(cfg->out_pixel_fmt, &cfg->in_res, (uint32_t *)&ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size);
video_el->need_recfg = false;
}
esp_gmf_port_handle_t in_port = ESP_GMF_ELEMENT_GET(self)->in;
esp_gmf_port_handle_t out_port = ESP_GMF_ELEMENT_GET(self)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
esp_gmf_job_err_t ret = ESP_GMF_JOB_ERR_OK;
esp_gmf_err_io_t load_ret = ESP_GMF_IO_OK;
ret = video_el_acquire_payload(in_port, out_port, &in_load, &out_load, ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size,
ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size, bypass);
if (ret != ESP_GMF_JOB_ERR_OK) {
goto __release;
}
/* It is for bypass */
if (in_load == out_load) {
goto __release;
}
/* finished */
if (in_load->is_done) {
out_load->is_done = in_load->is_done;
out_load->pts = in_load->pts;
ret = ESP_GMF_JOB_ERR_DONE;
ESP_LOGD(TAG, "It's done, out: %d", in_load->valid_size);
goto __release;
}
in_image.data = in_load->buf;
in_image.data_len = in_load->valid_size;
out_image.data = out_load->buf;
out_image.data_len = out_load->buf_length;
esp_imgfx_err_t imgfx_ret = esp_imgfx_color_convert_process(video_el->hd, &in_image, &out_image);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Image effects color convert process failed, ret: %d-%p", imgfx_ret, video_el);
ret = ESP_GMF_JOB_ERR_FAIL;
goto __release;
}
// Copy the information to the output payload. The next element will use this information to process
out_load->is_done = in_load->is_done;
out_load->valid_size = ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size;
out_load->pts = in_load->pts;
__release:
// Release in and out port
if (out_load != NULL) {
load_ret = esp_gmf_port_release_out(out_port, out_load, ESP_GMF_MAX_DELAY);
if ((load_ret < ESP_GMF_IO_OK) && (load_ret != ESP_GMF_IO_ABORT)) {
ESP_LOGE(TAG, "OUT port release error, ret:%d", load_ret);
ret = ESP_GMF_JOB_ERR_FAIL;
}
}
if (in_load != NULL) {
load_ret = esp_gmf_port_release_in(in_port, in_load, ESP_GMF_MAX_DELAY);
if ((load_ret < ESP_GMF_IO_OK) && (load_ret != ESP_GMF_IO_ABORT)) {
ESP_LOGE(TAG, "IN port release error, ret:%d", load_ret);
ret = ESP_GMF_JOB_ERR_FAIL;
}
}
return ret;
}
static esp_gmf_job_err_t video_cc_el_close(esp_gmf_element_handle_t self, void *para)
{
ESP_LOGD(TAG, "Closed, %p", self);
esp_gmf_color_convert_hd_t *video_el = (esp_gmf_color_convert_hd_t *)self;
if (video_el) {
if (video_el->hd) {
esp_imgfx_color_convert_close(video_el->hd);
video_el->hd = NULL;
}
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_cc_el_delete(esp_gmf_obj_handle_t handle)
{
ESP_LOGD(TAG, "Deleted, %p", handle);
void *cfg = OBJ_GET_CFG(handle);
if (cfg) {
esp_gmf_oal_free(cfg);
}
esp_gmf_video_el_deinit(handle);
esp_gmf_oal_free(handle);
return ESP_GMF_ERR_OK;
}
static inline esp_gmf_err_t esp_gmf_video_color_convert_set_cfg(esp_gmf_element_handle_t self, esp_imgfx_color_convert_cfg_t *config)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_JOB_ERR_FAIL);
ESP_GMF_NULL_CHECK(TAG, config, return ESP_GMF_JOB_ERR_FAIL);
esp_gmf_color_convert_hd_t *video_el = (esp_gmf_color_convert_hd_t *)self;
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// reset obj config
if (obj->cfg) {
memcpy(obj->cfg, config, sizeof(esp_imgfx_color_convert_cfg_t));
video_el->need_recfg = true;
return ESP_GMF_JOB_ERR_OK;
}
ESP_LOGE(TAG, " %s-%p is no configured yet", __func__, self);
return ESP_GMF_JOB_ERR_FAIL;
}
static inline esp_gmf_err_t esp_gmf_video_color_convert_get_cfg(esp_gmf_element_handle_t self, esp_imgfx_color_convert_cfg_t *config)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_JOB_ERR_FAIL);
esp_gmf_color_convert_hd_t *video_el = (esp_gmf_color_convert_hd_t *)self;
// If IMGFX color converter opened, get it from it or-else get from object
if (video_el->hd) {
esp_imgfx_err_t imgfx_ret = esp_imgfx_color_convert_get_cfg(video_el->hd, config);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Get video effects color convert cfg failed, hd:%p, ret: %d", self, imgfx_ret);
return ESP_GMF_JOB_ERR_FAIL;
}
return ESP_GMF_JOB_ERR_OK;
} else {
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// Get obj config
if (obj->cfg) {
memcpy(config, obj->cfg, sizeof(esp_imgfx_color_convert_cfg_t));
return ESP_GMF_JOB_ERR_OK;
}
}
ESP_LOGE(TAG, " %s-%p is no configured yet", __func__, self);
return ESP_GMF_JOB_ERR_FAIL;
}
static esp_gmf_err_t video_set_dst_format(void *handle, esp_gmf_args_desc_t *arg_desc, uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, buf, return ESP_GMF_ERR_INVALID_ARG);
return esp_gmf_video_color_convert_dst_format(handle, *(esp_imgfx_pixel_fmt_t *)buf);
}
static esp_gmf_err_t video_cc_el_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t dec_caps = {0};
dec_caps.cap_eightcc = ESP_GMF_CAPS_VIDEO_COLOR_CONVERT;
dec_caps.attr_fun = NULL;
int ret = esp_gmf_cap_append(&caps, &dec_caps);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to create capability");
esp_gmf_element_t *el = (esp_gmf_element_t *)handle;
el->caps = caps;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_cc_el_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_method_t *method = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_args_desc_t *set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CLR_CVT, SET_DST_FMT, FMT), ESP_GMF_ARGS_TYPE_INT8, sizeof(esp_imgfx_pixel_fmt_t), 0);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append color convert destination format");
ret = esp_gmf_method_append(&method, VMETHOD(CLR_CVT, SET_DST_FMT), video_set_dst_format, set_args);
ESP_GMF_RET_ON_ERROR(TAG, ret, return ret, "Failed to register %s method", VMETHOD(CLR_CVT, SET_DST_FMT));
esp_gmf_element_t *el = (esp_gmf_element_t *)handle;
el->method = method;
return ret;
}
static esp_gmf_err_t video_cc_el_received_event_handler(esp_gmf_event_pkt_t *evt, void *ctx)
{
ESP_GMF_NULL_CHECK(TAG, ctx, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, evt, return ESP_GMF_ERR_INVALID_ARG);
if ((evt->type != ESP_GMF_EVT_TYPE_REPORT_INFO)
|| (evt->sub != ESP_GMF_INFO_VIDEO)
|| (evt->payload == NULL)) {
return ESP_GMF_ERR_OK;
}
esp_gmf_element_handle_t self = (esp_gmf_element_handle_t)ctx;
esp_gmf_element_handle_t el = evt->from;
esp_gmf_event_state_t state = ESP_GMF_EVENT_STATE_NONE;
esp_gmf_element_get_state(self, &state);
esp_gmf_info_video_t *info = (esp_gmf_info_video_t *)evt->payload;
esp_gmf_video_el_set_src_info(self, info);
esp_imgfx_color_convert_cfg_t *config = (esp_imgfx_color_convert_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_NULL_CHECK(TAG, config, return ESP_GMF_ERR_FAIL);
esp_gmf_color_convert_hd_t *video_el = (esp_gmf_color_convert_hd_t *)self;
GMF_VIDEO_UPDATE_CONFIG(config, info, video_el->need_recfg);
ESP_LOGD(TAG, "RECV element info, from: %s-%p, next: %p, self: %s-%p, type: %x, state: %s, width: %d, height: %d, pixel format: %lx",
OBJ_GET_TAG(el), el, esp_gmf_node_for_next((esp_gmf_node_t *)el), OBJ_GET_TAG(self), self, evt->type,
esp_gmf_event_get_state_str(state), info->width, info->height, info->format_id);
// It is for the first time to receive event from previous element
if (state == ESP_GMF_EVENT_STATE_NONE) {
esp_gmf_element_set_state(self, ESP_GMF_EVENT_STATE_INITIALIZED);
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_cc_el_new(void *config, esp_gmf_obj_handle_t *handle)
{
return esp_gmf_video_color_convert_init((esp_imgfx_color_convert_cfg_t *)config, handle);
}
esp_gmf_err_t esp_gmf_video_color_convert_init(esp_imgfx_color_convert_cfg_t *config, esp_gmf_obj_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
*handle = NULL;
esp_gmf_color_convert_hd_t *video_el = esp_gmf_oal_calloc(1, sizeof(esp_gmf_color_convert_hd_t));
ESP_GMF_MEM_CHECK(TAG, video_el, return ESP_GMF_ERR_MEMORY_LACK);
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// Set element tag
ret = esp_gmf_obj_set_tag(video_el, "vid_color_cvt");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto __init_exit, "Failed set OBJ tag");
// Configure object callbacks
obj->new_obj = video_cc_el_new;
obj->del_obj = video_cc_el_delete;
// Set element config
esp_imgfx_color_convert_cfg_t *cfg = esp_gmf_oal_calloc(1, sizeof(esp_imgfx_color_convert_cfg_t));
ESP_GMF_MEM_VERIFY(TAG, cfg, {ret = ESP_GMF_ERR_MEMORY_LACK; goto __init_exit;}, "color convert configuration", sizeof(esp_imgfx_color_convert_cfg_t));
esp_gmf_obj_set_config(obj, cfg, sizeof(esp_imgfx_color_convert_cfg_t));
if (config) {
memcpy(cfg, config, sizeof(esp_imgfx_color_convert_cfg_t));
} else {
esp_imgfx_color_convert_cfg_t dcfg = DEFAULT_ESP_GMF_COLOR_CONVERT_CONFIG();
memcpy(cfg, &dcfg, sizeof(esp_imgfx_color_convert_cfg_t));
}
// Configure element
esp_gmf_element_cfg_t el_cfg = {
.dependency = true,
};
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
// Initialize element
ret = esp_gmf_video_el_init(video_el, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto __init_exit, "Failed initialize video element");
// Register element's callbacks
esp_gmf_video_element_t *parent = (esp_gmf_video_element_t *)video_el;
parent->base.ops.open = video_cc_el_open;
parent->base.ops.process = video_cc_el_process;
parent->base.ops.close = video_cc_el_close;
parent->base.ops.event_receiver = video_cc_el_received_event_handler;
parent->base.ops.load_caps = video_cc_el_load_caps;
parent->base.ops.load_methods = video_cc_el_load_methods;
*handle = (esp_gmf_obj_handle_t)video_el;
ESP_LOGD(TAG, "Initialization, %s-%p", OBJ_GET_TAG(obj), obj);
return ret;
__init_exit:
video_cc_el_delete(video_el);
return ret;
}
esp_gmf_err_t esp_gmf_video_color_convert_dst_format(esp_gmf_element_handle_t handle, uint32_t format)
{
esp_imgfx_color_convert_cfg_t config;
esp_gmf_err_t ret = esp_gmf_video_color_convert_get_cfg(handle, &config);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to set destination pixel format");
config.out_pixel_fmt = format;
return esp_gmf_video_color_convert_set_cfg(handle, &config);
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_crop.c
+324
View File
@@ -0,0 +1,324 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include <string.h>
#include <inttypes.h>
#include "esp_log.h"
#include "esp_gmf_oal_mutex.h"
#include "esp_gmf_oal_mem.h"
#include "esp_gmf_err.h"
#include "esp_gmf_node.h"
#include "esp_gmf_video_crop.h"
#include "esp_gmf_video_methods_def.h"
#include "esp_gmf_video_element.h"
#include "gmf_video_common.h"
#include "esp_gmf_video_types.h"
#include "esp_gmf_caps_def.h"
static const char *TAG = "IMGFX_CROP_EL";
typedef struct _gmf_imgfx_crop_t {
esp_gmf_video_element_t parent;
esp_imgfx_crop_handle_t hd;
bool need_recfg;
} esp_gmf_crop_hd_t;
static esp_gmf_job_err_t video_crop_el_open(esp_gmf_element_handle_t self, void *para)
{
esp_gmf_crop_hd_t *video_el = (esp_gmf_crop_hd_t *)self;
// Get and check config
esp_imgfx_crop_cfg_t *cfg = (esp_imgfx_crop_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_CHECK(TAG, cfg, return ESP_GMF_JOB_ERR_FAIL, "Failed to get crop config");
// Open crop module
esp_imgfx_crop_open(cfg, &video_el->hd);
ESP_GMF_CHECK(TAG, video_el->hd, return ESP_GMF_JOB_ERR_FAIL, "Failed to create crop handle");
// Get video size
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->in_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size));
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->cropped_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size));
// Report information to the next element, the next element will use this information to configure
gmf_video_update_info(self, cfg->cropped_res.width, cfg->cropped_res.height, cfg->in_pixel_fmt);
// The video_el->hd has been opened using newest configuration, so it can set the need_recfg to false
video_el->need_recfg = false;
ESP_LOGD(TAG, "Open, %p", self);
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_job_err_t video_crop_el_process(esp_gmf_element_handle_t self, void *para)
{
esp_gmf_crop_hd_t *video_el = (esp_gmf_crop_hd_t *)self;
esp_imgfx_data_t in_image;
esp_imgfx_data_t out_image;
if (video_el->need_recfg) {
esp_imgfx_crop_cfg_t *cfg = (esp_imgfx_crop_cfg_t *)OBJ_GET_CFG(self);
// Reset crop configuration
esp_imgfx_err_t imgfx_ret = esp_imgfx_crop_set_cfg(video_el->hd, cfg);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
return ESP_GMF_JOB_ERR_FAIL;
}
// Get video size
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->in_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size));
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->cropped_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size));
video_el->need_recfg = false;
}
esp_gmf_port_handle_t in_port = ESP_GMF_ELEMENT_GET(self)->in;
esp_gmf_port_handle_t out_port = ESP_GMF_ELEMENT_GET(self)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
esp_gmf_job_err_t ret = ESP_GMF_JOB_ERR_OK;
esp_gmf_err_io_t load_ret = ESP_GMF_IO_OK;
ret = video_el_acquire_payload(in_port, out_port, &in_load, &out_load, (ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size), (ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size),
((ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size) == (ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size)));
if (ret != ESP_GMF_JOB_ERR_OK) {
goto __release;
}
/* It is for bypass */
if (in_load == out_load) {
goto __release;
}
if (in_load->is_done) {
out_load->is_done = in_load->is_done;
out_load->pts = in_load->pts;
ret = ESP_GMF_JOB_ERR_DONE;
ESP_LOGD(TAG, "It's done, out: %d", in_load->valid_size);
goto __release;
}
in_image.data = in_load->buf;
in_image.data_len = in_load->valid_size;
out_image.data = out_load->buf;
out_image.data_len = out_load->buf_length;
esp_imgfx_err_t imgfx_ret = esp_imgfx_crop_process(video_el->hd, &in_image, &out_image);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Image effects crop process failed, ret: %d-%p", imgfx_ret, video_el);
ret = ESP_GMF_JOB_ERR_FAIL;
goto __release;
}
// Copy the information to the output payload. The next element will use this information to process
out_load->is_done = in_load->is_done;
out_load->valid_size = (ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size);
out_load->pts = in_load->pts;
__release:
// Release in and out port
if (out_load != NULL) {
load_ret = esp_gmf_port_release_out(out_port, out_load, ESP_GMF_MAX_DELAY);
if ((load_ret < ESP_GMF_IO_OK) && (load_ret != ESP_GMF_IO_ABORT)) {
ESP_LOGE(TAG, "OUT port release error, ret:%d", load_ret);
ret = ESP_GMF_JOB_ERR_FAIL;
}
}
if (in_load != NULL) {
load_ret = esp_gmf_port_release_in(in_port, in_load, ESP_GMF_MAX_DELAY);
if ((load_ret < ESP_GMF_IO_OK) && (load_ret != ESP_GMF_IO_ABORT)) {
ESP_LOGE(TAG, "IN port release error, ret:%d", load_ret);
ret = ESP_GMF_JOB_ERR_FAIL;
}
}
return ret;
}
static esp_gmf_job_err_t video_crop_el_close(esp_gmf_element_handle_t self, void *para)
{
ESP_LOGD(TAG, "Closed, %p", self);
esp_gmf_crop_hd_t *video_el = (esp_gmf_crop_hd_t *)self;
if (video_el) {
if (video_el->hd) {
esp_imgfx_crop_close(video_el->hd);
video_el->hd = NULL;
}
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_crop_el_delete(esp_gmf_obj_handle_t handle)
{
ESP_LOGD(TAG, "Deleted, %p", handle);
void *cfg = OBJ_GET_CFG(handle);
if (cfg) {
esp_gmf_oal_free(cfg);
}
esp_gmf_video_el_deinit(handle);
esp_gmf_oal_free(handle);
return ESP_GMF_ERR_OK;
}
static inline esp_gmf_err_t esp_gmf_video_crop_set_cfg(esp_gmf_element_handle_t self, esp_imgfx_crop_cfg_t *config)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_JOB_ERR_FAIL);
ESP_GMF_NULL_CHECK(TAG, config, return ESP_GMF_JOB_ERR_FAIL);
esp_gmf_crop_hd_t *video_el = (esp_gmf_crop_hd_t *)self;
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// reset obj config
if (obj->cfg) {
memcpy(obj->cfg, config, sizeof(esp_imgfx_crop_cfg_t));
video_el->need_recfg = true;
return ESP_GMF_JOB_ERR_OK;
}
ESP_LOGE(TAG, " %s-%p is no configured yet", __func__, self);
return ESP_GMF_JOB_ERR_FAIL;
}
static inline esp_gmf_err_t esp_gmf_video_crop_get_cfg(esp_gmf_element_handle_t self, esp_imgfx_crop_cfg_t *config)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_JOB_ERR_FAIL);
esp_gmf_crop_hd_t *video_el = (esp_gmf_crop_hd_t *)self;
if (video_el->hd) {
esp_imgfx_err_t imgfx_ret = esp_imgfx_crop_get_cfg(video_el->hd, config);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Failed to get video crop cfg, hd:%p, ret: %d", self, imgfx_ret);
return ESP_GMF_JOB_ERR_FAIL;
}
return ESP_GMF_JOB_ERR_OK;
} else {
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// Get obj config
if (obj->cfg) {
memcpy(config, obj->cfg, sizeof(esp_imgfx_crop_cfg_t));
return ESP_GMF_JOB_ERR_OK;
}
}
ESP_LOGE(TAG, " %s-%p is no configured yet", __func__, self);
return ESP_GMF_JOB_ERR_FAIL;
}
static esp_gmf_err_t video_set_dst_rgn(void *handle, esp_gmf_args_desc_t *arg_desc, uint8_t *buf, int buf_len)
{
ESP_GMF_MEM_CHECK(TAG, buf, return ESP_GMF_ERR_INVALID_ARG);
return esp_gmf_video_crop_rgn(handle, (esp_gmf_video_rgn_t *)buf);
}
static esp_gmf_err_t video_crop_el_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t dec_caps = {0};
dec_caps.cap_eightcc = ESP_GMF_CAPS_VIDEO_CROP;
dec_caps.attr_fun = NULL;
int ret = esp_gmf_cap_append(&caps, &dec_caps);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to create capability");
esp_gmf_element_t *el = (esp_gmf_element_t *)handle;
el->caps = caps;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_crop_el_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_method_t *method = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_args_desc_t *set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CROP, SET_CROP_RGN, X), ESP_GMF_ARGS_TYPE_UINT16,
sizeof(uint16_t), offsetof(esp_gmf_video_rgn_t, x));
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append x of cropper region");
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CROP, SET_CROP_RGN, Y), ESP_GMF_ARGS_TYPE_UINT16,
sizeof(uint16_t), offsetof(esp_gmf_video_rgn_t, y));
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append y of cropper region");
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CROP, SET_CROP_RGN, WIDTH), ESP_GMF_ARGS_TYPE_UINT16,
sizeof(uint16_t), offsetof(esp_gmf_video_rgn_t, width));
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append width of cropper region");
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CROP, SET_CROP_RGN, HEIGHT), ESP_GMF_ARGS_TYPE_UINT16,
sizeof(uint16_t), offsetof(esp_gmf_video_rgn_t, height));
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append height of cropper region");
ret = esp_gmf_method_append(&method, VMETHOD(CROP, SET_CROP_RGN), video_set_dst_rgn, set_args);
ESP_GMF_RET_ON_ERROR(TAG, ret, return ret, "Failed to register %s method", VMETHOD(CROP, SET_CROP_RGN));
esp_gmf_element_t *el = (esp_gmf_element_t *)handle;
el->method = method;
return ret;
}
static esp_gmf_err_t video_crop_el_received_event_handler(esp_gmf_event_pkt_t *evt, void *ctx)
{
ESP_GMF_NULL_CHECK(TAG, ctx, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, evt, return ESP_GMF_ERR_INVALID_ARG);
if ((evt->type != ESP_GMF_EVT_TYPE_REPORT_INFO)
|| (evt->sub != ESP_GMF_INFO_VIDEO)
|| (evt->payload == NULL)) {
return ESP_GMF_ERR_OK;
}
esp_gmf_element_handle_t self = (esp_gmf_element_handle_t)ctx;
esp_gmf_element_handle_t el = evt->from;
esp_gmf_event_state_t state = ESP_GMF_EVENT_STATE_NONE;
esp_gmf_element_get_state(self, &state);
esp_gmf_info_video_t *info = (esp_gmf_info_video_t *)evt->payload;
esp_gmf_video_el_set_src_info(self, info);
esp_imgfx_crop_cfg_t *config = (esp_imgfx_crop_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_NULL_CHECK(TAG, config, return ESP_GMF_ERR_FAIL);
esp_gmf_crop_hd_t *video_el = (esp_gmf_crop_hd_t *)self;
GMF_VIDEO_UPDATE_CONFIG(config, info, video_el->need_recfg);
ESP_LOGD(TAG, "RECV element info, from: %s-%p, next: %p, self: %s-%p, type: %x, state: %s, width: %d, height: %d, pixel format: %lx",
OBJ_GET_TAG(el), el, esp_gmf_node_for_next((esp_gmf_node_t *)el), OBJ_GET_TAG(self), self, evt->type,
esp_gmf_event_get_state_str(state), info->width, info->height, info->format_id);
// It is for the first time to receive event from previous element
if (state == ESP_GMF_EVENT_STATE_NONE) {
esp_gmf_element_set_state(self, ESP_GMF_EVENT_STATE_INITIALIZED);
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_crop_el_new(void *config, esp_gmf_obj_handle_t *handle)
{
return esp_gmf_video_crop_init((esp_imgfx_crop_cfg_t *)config, handle);
}
esp_gmf_err_t esp_gmf_video_crop_init(esp_imgfx_crop_cfg_t *config, esp_gmf_obj_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
*handle = NULL;
esp_gmf_crop_hd_t *video_el = esp_gmf_oal_calloc(1, sizeof(esp_gmf_crop_hd_t));
ESP_GMF_MEM_CHECK(TAG, video_el, return ESP_GMF_ERR_MEMORY_LACK);
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// Set element tag
ret = esp_gmf_obj_set_tag(video_el, "vid_crop");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto __init_exit, "Failed set OBJ tag");
// Configure object callbacks
obj->new_obj = video_crop_el_new;
obj->del_obj = video_crop_el_delete;
// Set element config
esp_imgfx_crop_cfg_t *cfg = esp_gmf_oal_calloc(1, sizeof(esp_imgfx_crop_cfg_t));
ESP_GMF_MEM_VERIFY(TAG, cfg, {ret = ESP_GMF_ERR_MEMORY_LACK; goto __init_exit;}, "crop configuration", sizeof(esp_imgfx_crop_cfg_t));
esp_gmf_obj_set_config(obj, cfg, sizeof(esp_imgfx_crop_cfg_t));
if (config) {
memcpy(cfg, config, sizeof(esp_imgfx_crop_cfg_t));
} else {
esp_imgfx_crop_cfg_t dcfg = DEFAULT_ESP_GMF_CROP_CONFIG();
memcpy(cfg, &dcfg, sizeof(esp_imgfx_crop_cfg_t));
}
// Configure element
esp_gmf_element_cfg_t el_cfg = {
.dependency = true,
};
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
// Initialize element
ret = esp_gmf_video_el_init(video_el, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto __init_exit, "Failed initialize video element");
// Register element's callbacks
esp_gmf_video_element_t *parent = (esp_gmf_video_element_t *)video_el;
parent->base.ops.open = video_crop_el_open;
parent->base.ops.process = video_crop_el_process;
parent->base.ops.close = video_crop_el_close;
parent->base.ops.event_receiver = video_crop_el_received_event_handler;
parent->base.ops.load_caps = video_crop_el_load_caps;
parent->base.ops.load_methods = video_crop_el_load_methods;
*handle = (esp_gmf_obj_handle_t)video_el;
ESP_LOGD(TAG, "Initialization, %s-%p", OBJ_GET_TAG(obj), obj);
return ret;
__init_exit:
video_crop_el_delete(video_el);
return ret;
}
esp_gmf_err_t esp_gmf_video_crop_rgn(esp_gmf_element_handle_t handle, esp_gmf_video_rgn_t *rgn)
{
esp_imgfx_crop_cfg_t config;
esp_gmf_err_t ret = esp_gmf_video_crop_get_cfg(handle, &config);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to get crop configuration");
config.x_pos = rgn->x;
config.y_pos = rgn->y;
config.cropped_res.width = rgn->width;
config.cropped_res.height = rgn->height;
return esp_gmf_video_crop_set_cfg(handle, &config);
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_dec.c
+434
View File
@@ -0,0 +1,434 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include "esp_gmf_video_dec.h"
#include "esp_gmf_caps_def.h"
#include "esp_gmf_video_element.h"
#include "esp_video_dec.h"
#include "esp_video_codec_utils.h"
#include "esp_gmf_video_element.h"
#include "gmf_video_common.h"
#include "esp_gmf_video_methods_def.h"
#include "esp_log.h"
#define TAG "VDEC_EL"
#define COPY_ARG(dst, src, offset, size, total_len) \
if (offset + size <= total_len) { \
memcpy(&(dst), src + offset, size); \
offset += size; \
}
#define ADD_ARG_DESC(desc, type, size, offset) \
ret = esp_gmf_args_desc_append(&set_args, desc, type, size, offset); \
GMF_VIDEO_BREAK_ON_FAIL(ret); \
offset += size;
/**
* @brief Video decoder definition
*/
typedef struct {
esp_gmf_video_element_t parent; /*!< Video element parent */
uint32_t out_format; /*!< Video decoder output codec */
uint32_t codec_cc; /*!< Codec FourCC used to find decoder if user set it */
bool vdec_bypass; /*!< Whether decoder is bypassed or not */
bool header_parsed; /*!< Whether video header parsed or not */
esp_video_dec_handle_t dec_handle; /*!< Video decoder handle */
} vdec_t;
static inline uint32_t get_prefer_codec(vdec_t *vdec)
{
esp_gmf_video_dec_cfg_t *cfg = (esp_gmf_video_dec_cfg_t *)OBJ_GET_CFG(vdec);
return cfg ? cfg->codec_cc : 0;
}
static int vdec_get_out_fmts(vdec_t *vdec, uint32_t src_codec, const uint32_t **fmts, uint8_t *num)
{
esp_video_dec_caps_t caps = {};
esp_video_codec_query_t query = {
.codec_type = (esp_video_codec_type_t)src_codec,
.codec_cc = get_prefer_codec(vdec),
};
esp_vc_err_t ret = esp_video_dec_query_caps(&query, &caps);
if (ret != ESP_VC_ERR_OK) {
return ESP_GMF_ERR_NOT_SUPPORT;
}
*fmts = (const uint32_t *)caps.out_fmts;
*num = caps.out_fmt_num;
return ESP_GMF_ERR_OK;
}
static bool vdec_is_codec_supported(vdec_t *vdec, uint32_t in_codec, uint32_t out_format)
{
esp_video_dec_caps_t caps = {};
esp_video_codec_query_t query = {
.codec_type = (esp_video_codec_type_t)in_codec,
.codec_cc = get_prefer_codec(vdec),
};
esp_vc_err_t ret = esp_video_dec_query_caps(&query, &caps);
if (ret != ESP_VC_ERR_OK) {
return false;
}
for (uint8_t i = 0; i < caps.out_fmt_num; i++) {
if (out_format == (uint32_t)caps.out_fmts[i]) {
return true;
}
}
return false;
}
static esp_gmf_job_err_t vdec_el_open(esp_gmf_video_element_handle_t self, void *para)
{
vdec_t *vdec = (vdec_t *)self;
esp_gmf_info_video_t *src_info = &vdec->parent.src_info;
vdec->vdec_bypass = (src_info->format_id == vdec->out_format);
if (vdec->vdec_bypass) {
// Report video info to next element directly
esp_gmf_element_notify_vid_info(self, src_info);
return ESP_GMF_JOB_ERR_OK;
}
// Open video decoder now
esp_video_dec_cfg_t dec_cfg = {
.codec_type = (esp_video_codec_type_t)src_info->format_id,
.codec_cc = get_prefer_codec(vdec),
.out_fmt = (esp_video_codec_pixel_fmt_t)vdec->out_format,
};
bool supported = vdec_is_codec_supported(vdec, src_info->format_id, vdec->out_format);
if (supported == false) {
ESP_LOGE(TAG, "Format not supported in:%s out:%s",
esp_gmf_video_get_format_string(src_info->format_id),
esp_gmf_video_get_format_string(vdec->out_format));
return ESP_GMF_JOB_ERR_FAIL;
}
int ret = esp_video_dec_open(&dec_cfg, &vdec->dec_handle);
if (ret != ESP_VC_ERR_OK) {
return ESP_GMF_JOB_ERR_FAIL;
}
// Get alignment request
uint8_t in_frame_align = 0;
uint8_t out_frame_align = 0;
esp_video_dec_get_frame_align(vdec->dec_handle, &in_frame_align, &out_frame_align);
ESP_GMF_ELEMENT_GET(vdec)->in_attr.port.buf_addr_aligned = in_frame_align;
ESP_GMF_ELEMENT_GET(vdec)->out_attr.port.buf_addr_aligned = out_frame_align;
esp_gmf_info_video_t out_info = vdec->parent.src_info;
out_info.format_id = vdec->out_format;
esp_gmf_element_notify_vid_info(self, &out_info);
return ESP_GMF_JOB_ERR_OK;
}
static int vdec_bypass(vdec_t *vdec, esp_gmf_port_t *in, esp_gmf_port_t *out)
{
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
esp_gmf_err_io_t ret = esp_gmf_port_acquire_in(in, &in_load, ESP_GMF_ELEMENT_GET(vdec)->in_attr.data_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Acquire on in port, ret:%d", ret);
return ret == ESP_GMF_IO_ABORT ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
bool is_done = in_load->is_done;
out_load = in_load;
ret = esp_gmf_port_acquire_out(out, &out_load, out_load->valid_size, -1);
// Not release out if error
if (ret < 0) {
esp_gmf_port_release_in(in, in_load, 0);
return ret;
}
esp_gmf_port_release_out(out, out_load, 0);
esp_gmf_port_release_in(in, in_load, 0);
if (is_done) {
ret = ESP_GMF_JOB_ERR_DONE;
}
return ret;
}
static esp_gmf_job_err_t vdec_el_process(esp_gmf_video_element_handle_t self, void *para)
{
esp_gmf_element_handle_t hd = (esp_gmf_element_handle_t)self;
vdec_t *vdec = (vdec_t *)self;
esp_gmf_port_t *in = ESP_GMF_ELEMENT_GET(hd)->in;
esp_gmf_port_t *out = ESP_GMF_ELEMENT_GET(hd)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
if (vdec->vdec_bypass) {
// This code is added to bypass raw input data only
return vdec_bypass(vdec, in, out);
}
// TODO make sure input frame alignment meet request
int ret = esp_gmf_port_acquire_in(in, &in_load, ESP_GMF_ELEMENT_GET(vdec)->in_attr.data_size, -1);
if (ret < 0) {
ESP_LOGE(TAG, "Acquire in port error ret:%d", ret);
return ret == ESP_GMF_IO_ABORT ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
// Suppose input data must be frame boundary
esp_video_dec_in_frame_t in_frame = {
.pts = in_load->pts,
.data = in_load->buf,
.size = in_load->valid_size,
};
esp_video_dec_out_frame_t decoded_frame = {};
ret = ESP_GMF_JOB_ERR_FAIL;
do {
// Check whether done
if (in_load->valid_size == 0 && in_load->is_done) {
ret = ESP_GMF_JOB_ERR_DONE;
break;
}
// Check alignment
if ((intptr_t)in_load->buf & (ESP_GMF_ELEMENT_GET(vdec)->in_attr.port.buf_addr_aligned - 1)) {
ESP_LOGE(TAG, "Input alignment not meet %d", ESP_GMF_ELEMENT_GET(vdec)->in_attr.port.buf_addr_aligned);
ret = ESP_GMF_JOB_ERR_FAIL;
break;
}
if (vdec->header_parsed == false) {
// Workaround use small buffer, so that can get frame info
uint32_t out_size = 32;
uint8_t out_frame_align = ESP_GMF_ELEMENT_GET(vdec)->out_attr.port.buf_addr_aligned;
uint8_t *out_data = esp_video_codec_align_alloc(out_frame_align, out_size, &out_size);
if (out_data == NULL) {
ESP_LOGE(TAG, "No enough memory for parse header");
ret = ESP_GMF_JOB_ERR_FAIL;
break;
}
decoded_frame.data = out_data;
decoded_frame.size = out_size;
ret = esp_video_dec_process(vdec->dec_handle, &in_frame, &decoded_frame);
esp_video_codec_free(out_data);
esp_video_codec_frame_info_t frame_info = {};
ret = esp_video_dec_get_frame_info(vdec->dec_handle, &frame_info);
if ((ret != ESP_VC_ERR_OK) || (frame_info.res.width == 0) || (frame_info.res.height) == 0) {
ESP_LOGE(TAG, "Fail to get frame info%d", ret);
ret = ESP_GMF_JOB_ERR_CONTINUE;
break;
}
ESP_LOGI(TAG, "Dec frame size %dx%d", (int)frame_info.res.width, (int)frame_info.res.height);
// Update output frame size
uint32_t out_frame_size = esp_video_codec_get_image_size(vdec->out_format, &frame_info.res);
out_frame_size = GMF_VIDEO_ALIGN_UP(out_frame_size, out_frame_align);
ESP_GMF_ELEMENT_GET(vdec)->out_attr.data_size = out_frame_size;
// Report info to next element when parse header finished
esp_gmf_info_video_t out_info = {
.format_id = vdec->out_format,
.width = frame_info.res.width,
.height = frame_info.res.height,
.fps = frame_info.fps,
};
esp_gmf_element_notify_vid_info(self, &out_info);
vdec->header_parsed = true;
}
// We do not allow decode change resolution in middle currently
ret = esp_gmf_port_acquire_out(out, &out_load, ESP_GMF_ELEMENT_GET(vdec)->out_attr.data_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Write data error, ret:%d, line:%d", ret, __LINE__);
ret = (ret == ESP_GMF_IO_ABORT) ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
break;
}
decoded_frame.data = out_load->buf;
decoded_frame.size = ESP_GMF_ELEMENT_GET(vdec)->out_attr.data_size;
ret = esp_video_dec_process(vdec->dec_handle, &in_frame, &decoded_frame);
// Skip data if decode failed
if (ret != ESP_VC_ERR_OK) {
ESP_LOGE(TAG, "Fail to decode ret %d", ret);
ret = ESP_GMF_JOB_ERR_CONTINUE;
break;
}
out_load->valid_size = decoded_frame.decoded_size;
out_load->pts = in_load->pts;
ret = ESP_GMF_JOB_ERR_OK;
} while (0);
if (out_load) {
esp_gmf_port_release_out(out, out_load, 0);
}
esp_gmf_port_release_in(in, in_load, 0);
return ret;
}
static esp_gmf_job_err_t vdec_el_close(esp_gmf_video_element_handle_t self, void *para)
{
vdec_t *vdec = (vdec_t *)self;
if (vdec->dec_handle) {
esp_video_dec_close(vdec->dec_handle);
vdec->dec_handle = NULL;
}
vdec->header_parsed = false;
ESP_LOGI(TAG, "Closed, %p", self);
return ESP_OK;
}
static esp_gmf_err_t vdec_el_new(void *cfg, esp_gmf_obj_handle_t *handle)
{
return esp_gmf_video_dec_init((esp_gmf_video_dec_cfg_t *)cfg, (esp_gmf_element_handle_t*)handle);
}
static esp_gmf_err_t vdec_el_destroy(esp_gmf_obj_handle_t self)
{
esp_gmf_video_el_deinit(self);
void *cfg = OBJ_GET_CFG(self);
if (cfg) {
esp_gmf_oal_free(cfg);
}
esp_gmf_oal_free(self);
return ESP_OK;
}
static esp_gmf_err_t set_out_format(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
vdec_t *vdec = (vdec_t *)handle;
vdec->out_format = *((uint32_t*) buf);
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t set_src_codec(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
vdec_t *vdec = (vdec_t *)handle;
vdec->parent.src_info.format_id = *((uint32_t*) buf);
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t get_out_formats(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
vdec_t *vdec = (vdec_t *)handle;
int offset = 0;
uint32_t in_codec = 0;
const uint32_t **out_fmts = NULL;
uint8_t *out_fmt_num = NULL;
COPY_ARG(in_codec, buf, offset, sizeof(uint32_t), buf_len);
COPY_ARG(out_fmts, buf, offset, sizeof(void*), buf_len);
COPY_ARG(out_fmt_num, buf, offset, sizeof(void*), buf_len);
return vdec_get_out_fmts(vdec, in_codec, out_fmts, out_fmt_num);
}
static esp_gmf_err_t vdec_el_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t cap = {0};
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
do {
cap.cap_eightcc = ESP_GMF_CAPS_VIDEO_DECODER;
cap.attr_fun = NULL;
ret = esp_gmf_cap_append(&caps, &cap);
GMF_VIDEO_BREAK_ON_FAIL(ret);
((esp_gmf_element_t *) handle)->caps = caps;
return ret;
} while (0);
if (caps) {
esp_gmf_cap_destroy(caps);
}
return ret;
}
static esp_gmf_err_t vdec_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_args_desc_t *set_args = NULL;
esp_gmf_method_t *methods = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
int offset = 0;
do {
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CLR_CVT, SET_DST_FMT, FMT), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(CLR_CVT, SET_DST_FMT), set_out_format, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
offset = 0;
ADD_ARG_DESC(VMETHOD_ARG(DECODER, SET_SRC_CODEC, CODEC), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), offset);
ret = esp_gmf_method_append(&methods, VMETHOD(DECODER, SET_SRC_CODEC), set_src_codec, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
offset = 0;
ADD_ARG_DESC(VMETHOD_ARG(DECODER, GET_DST_FMTS, SRC_CODEC), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), offset);
ADD_ARG_DESC(VMETHOD_ARG(DECODER, GET_DST_FMTS, DST_FMTS_PTR), ESP_GMF_ARGS_TYPE_UINT32, sizeof(void *), offset);
ADD_ARG_DESC(VMETHOD_ARG(DECODER, GET_DST_FMTS, DST_FMTS_NUM_PTR), ESP_GMF_ARGS_TYPE_UINT32, sizeof(void *), offset);
ret = esp_gmf_method_append(&methods, VMETHOD(DECODER, GET_DST_FMTS), get_out_formats, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
((esp_gmf_element_t *) handle)->method = methods;
return ESP_GMF_ERR_OK;
} while (0);
ESP_LOGE(TAG, "Fail to load methods");
if (set_args) {
esp_gmf_args_desc_destroy(set_args);
}
if (methods) {
esp_gmf_method_destroy(methods);
}
return ESP_GMF_ERR_MEMORY_LACK;
}
esp_gmf_err_t esp_gmf_video_dec_init(esp_gmf_video_dec_cfg_t *cfg, esp_gmf_element_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
vdec_t *vdec = esp_gmf_oal_calloc(1, sizeof(vdec_t));
ESP_GMF_MEM_CHECK(TAG, vdec, return ESP_GMF_ERR_MEMORY_LACK);
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)vdec;
obj->new_obj = vdec_el_new;
obj->del_obj = vdec_el_destroy;
if (cfg) {
esp_gmf_video_dec_cfg_t *dec_cfg = (esp_gmf_video_dec_cfg_t *) calloc(1, sizeof(esp_gmf_video_dec_cfg_t));
ESP_GMF_MEM_CHECK(TAG, vdec, { goto VDEC_FAIL; });
*dec_cfg = *cfg;
esp_gmf_obj_set_config(obj, dec_cfg, sizeof(esp_gmf_video_dec_cfg_t));
}
esp_gmf_err_t ret = esp_gmf_obj_set_tag(obj, "vid_dec");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto VDEC_FAIL, "Failed set OBJ tag");
uint8_t align = esp_gmf_oal_get_spiram_cache_align();
esp_gmf_element_cfg_t el_cfg = {
.dependency = true,
};
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_SINGLE, align, align,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, align, align,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ret = esp_gmf_video_el_init(vdec, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto VDEC_FAIL, "Failed to init video decoder element");
ESP_GMF_ELEMENT_GET(vdec)->ops.open = vdec_el_open;
ESP_GMF_ELEMENT_GET(vdec)->ops.process = vdec_el_process;
ESP_GMF_ELEMENT_GET(vdec)->ops.close = vdec_el_close;
ESP_GMF_ELEMENT_GET(vdec)->ops.event_receiver = esp_gmf_video_handle_events;
ESP_GMF_ELEMENT_GET(vdec)->ops.load_caps = vdec_el_load_caps;
ESP_GMF_ELEMENT_GET(vdec)->ops.load_methods = vdec_load_methods;
*handle = (esp_gmf_element_handle_t) obj;
ESP_LOGD(TAG, "Create %s-%p", OBJ_GET_TAG(obj), obj);
return ESP_GMF_ERR_OK;
VDEC_FAIL:
esp_gmf_obj_delete(obj);
return ret;
}
esp_gmf_err_t esp_gmf_video_dec_set_dst_format(esp_gmf_element_handle_t handle, uint32_t dst_fmt)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
vdec_t *vdec = (vdec_t *)handle;
vdec->out_format = dst_fmt;
return ESP_GMF_ERR_OK;
}
esp_gmf_err_t esp_gmf_video_dec_get_dst_formats(esp_gmf_element_handle_t handle,
uint32_t in_codec,
const uint32_t **dst_fmts,
uint8_t *dst_fmts_num)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
vdec_t *vdec = (vdec_t *)handle;
return vdec_get_out_fmts(vdec, in_codec, dst_fmts, dst_fmts_num);
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_enc.c
+578
View File
@@ -0,0 +1,578 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include <string.h>
#include "esp_log.h"
#include "esp_gmf_info.h"
#include "esp_gmf_oal_mem.h"
#include "esp_gmf_err.h"
#include "esp_gmf_video_element.h"
#include "esp_video_enc.h"
#include "esp_video_enc_reg.h"
#include "esp_video_codec_utils.h"
#include "esp_gmf_video_enc.h"
#include "gmf_video_common.h"
#include "esp_fourcc.h"
#include "esp_gmf_video_methods_def.h"
#include "esp_gmf_caps_def.h"
#define MAX_ENC_TYPE_SUPPORT 4
// TODO this is estimated value so that each frame not over the size
#define MJPEG_ENC_MIN_COMPRESS_RATIO 10
#define H264_ENC_MIN_COMPRESS_RATIO 2
#define COPY_ARG(dst, src, offset, size, total_len) \
if (offset + size <= total_len) { \
memcpy(dst, src + offset, size); \
offset += size; \
}
#define ADD_ARG_DESC(desc, type, size, offset) \
ret = esp_gmf_args_desc_append(&set_args, desc, type, size, offset); \
GMF_VIDEO_BREAK_ON_FAIL(ret); \
offset += size;
static const char *TAG = "VENC_EL";
/**
* @brief Video encoder extra setting mask
*/
typedef enum {
VENC_EXTRA_SET_MASK_NONE = 0,
VENC_EXTRA_SET_MASK_BITRATE = (1 << 0),
VENC_EXTRA_SET_MASK_QP = (1 << 1),
VENC_EXTRA_SET_MASK_GOP = (1 << 2),
VENC_EXTRA_SET_MASK_ALL = (0xFF),
} venc_extra_set_mask_t;
/**
* @brief Video encoder extra setting
*/
typedef struct {
uint32_t bitrate;
uint32_t min_qp;
uint32_t max_qp;
uint32_t gop;
venc_extra_set_mask_t mask;
} venc_extra_set_t;
/**
* @brief Video encoder definition
*/
typedef struct {
esp_gmf_video_element_t parent; /*!< Video element parent */
esp_video_codec_type_t dst_codec; /*!< Video encoder destination codec */
bool venc_bypass; /*!< Whether video encoder is bypassed or not */
uint32_t codec_cc; /*!< FourCC used to find encoder if set */
esp_video_enc_handle_t enc_handle; /*!< Video encoder handle */
venc_extra_set_t extra_set; /*!< Video encoder extra setting */
} venc_t;
static int venc_get_input_codecs(venc_t *venc, uint32_t dst_codec, const uint32_t **codecs, uint8_t *num)
{
esp_video_enc_caps_t caps = {};
esp_video_codec_query_t query = {
.codec_type = (esp_video_codec_type_t)dst_codec,
.codec_cc = venc->codec_cc,
};
int ret = esp_video_enc_query_caps(&query, &caps);
if (ret != ESP_VC_ERR_OK) {
return ESP_GMF_ERR_NOT_SUPPORT;
}
*codecs = (const uint32_t *)caps.in_fmts;
*num = caps.in_fmt_num;
return ESP_GMF_ERR_OK;
}
static bool venc_is_codec_supported(venc_t *venc, uint32_t src_codec, uint32_t dst_codec)
{
esp_video_enc_caps_t caps = {};
esp_video_codec_query_t query = {
.codec_type = (esp_video_codec_type_t)dst_codec,
.codec_cc = venc->codec_cc,
};
int ret = esp_video_enc_query_caps(&query, &caps);
if (ret != ESP_VC_ERR_OK) {
return false;
}
for (uint8_t i = 0; i < caps.in_fmt_num; i++) {
if (src_codec == (uint32_t)caps.in_fmts[i]) {
return true;
}
}
return false;
}
static int venc_get_frame_size(venc_t *venc, int *in_frame_size, int *out_frame_size)
{
esp_gmf_info_video_t *src_info = &venc->parent.src_info;
esp_video_codec_resolution_t res = {
.width = src_info->width,
.height = src_info->height,
};
*in_frame_size = (int)esp_video_codec_get_image_size(
(esp_video_codec_pixel_fmt_t)src_info->format_id, &res);
if (venc->dst_codec == ESP_FOURCC_MJPG) {
*out_frame_size = *in_frame_size / MJPEG_ENC_MIN_COMPRESS_RATIO;
} else if (venc->dst_codec == ESP_FOURCC_H264) {
uint8_t align = esp_gmf_oal_get_spiram_cache_align();
*out_frame_size = *in_frame_size / H264_ENC_MIN_COMPRESS_RATIO;
*out_frame_size = GMF_VIDEO_ALIGN_UP(*out_frame_size, align);
} else {
return ESP_GMF_ERR_NOT_SUPPORT;
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t venc_el_apply_settings(venc_t *venc, venc_extra_set_mask_t mask)
{
venc_extra_set_t *extra_set = &venc->extra_set;
if (venc->enc_handle == NULL ||
extra_set->mask == VENC_EXTRA_SET_MASK_NONE) {
return ESP_GMF_ERR_OK;
}
esp_vc_err_t ret = ESP_VC_ERR_OK;
if (extra_set->mask & (mask & VENC_EXTRA_SET_MASK_BITRATE)) {
ret |= esp_video_enc_set_bitrate(venc->enc_handle, extra_set->bitrate);
}
if (extra_set->mask & (mask & VENC_EXTRA_SET_MASK_QP)) {
ret |= esp_video_enc_set_qp(venc->enc_handle, extra_set->min_qp, extra_set->max_qp);
}
if (extra_set->mask & (mask & VENC_EXTRA_SET_MASK_GOP)) {
ret |= esp_video_enc_set_gop(venc->enc_handle, extra_set->gop);
}
return ret == ESP_VC_ERR_OK ? ESP_GMF_ERR_OK : ESP_GMF_ERR_NOT_SUPPORT;
}
static esp_gmf_job_err_t venc_el_open(esp_gmf_video_element_handle_t self, void *para)
{
venc_t *venc = (venc_t *)self;
esp_gmf_video_enc_cfg_t *cfg = (esp_gmf_video_enc_cfg_t *)OBJ_GET_CFG(self);
esp_gmf_info_video_t *src_info = &venc->parent.src_info;
venc->venc_bypass = (src_info->format_id == venc->dst_codec);
if (venc->venc_bypass == false) {
esp_video_enc_cfg_t enc_cfg = {
.codec_type = venc->dst_codec,
.resolution = {
.width = src_info->width,
.height = src_info->height,
},
.in_fmt = (esp_video_codec_pixel_fmt_t)src_info->format_id,
.fps = src_info->fps,
};
if (cfg) {
enc_cfg.codec_cc = cfg->codec_cc;
venc->codec_cc = cfg->codec_cc;
}
esp_vc_err_t ret = esp_video_enc_open(&enc_cfg, &venc->enc_handle);
if (ret != ESP_VC_ERR_OK) {
ESP_LOGE(TAG, "Fail to open encoder ret %d", ret);
return ESP_GMF_ERR_NOT_SUPPORT;
}
int in_frame_size = 0;
int out_frame_size = 0;
venc_get_frame_size(venc, &in_frame_size, &out_frame_size);
ESP_GMF_ELEMENT_GET(venc)->in_attr.data_size = in_frame_size;
ESP_GMF_ELEMENT_GET(venc)->out_attr.data_size = out_frame_size;
venc_el_apply_settings(venc, VENC_EXTRA_SET_MASK_ALL);
}
// Report info to next element
esp_gmf_info_video_t out_info = venc->parent.src_info;
out_info.format_id = venc->dst_codec;
esp_gmf_element_notify_vid_info(self, &out_info);
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_job_err_t venc_el_process(esp_gmf_video_element_handle_t self, void *para)
{
esp_gmf_element_handle_t hd = (esp_gmf_element_handle_t)self;
venc_t *venc = (venc_t *)self;
esp_gmf_port_t *in = ESP_GMF_ELEMENT_GET(hd)->in;
esp_gmf_port_t *out = ESP_GMF_ELEMENT_GET(hd)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
int ret = esp_gmf_port_acquire_in(in, &in_load, ESP_GMF_ELEMENT_GET(venc)->in_attr.data_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Acquire size:%d on in port, ret:%d", ESP_GMF_ELEMENT_GET(venc)->in_attr.data_size, ret);
return (ret == ESP_GMF_IO_ABORT) ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
if (in_load->is_done && in_load->valid_size == 0) {
esp_gmf_port_release_in(in, in_load, 0);
return ESP_GMF_JOB_ERR_DONE;
}
if (venc->venc_bypass) {
out_load = in_load;
}
int out_frame_size = ESP_GMF_ELEMENT_GET(venc)->out_attr.data_size;
ret = esp_gmf_port_acquire_out(out, &out_load, out_frame_size, -1);
if (ret < 0) {
esp_gmf_port_release_in(in, in_load, 0);
ESP_LOGE(TAG, "Acquire size:%d on out port, ret:%d", out_frame_size, ret);
return ret;
}
do {
if (venc->venc_bypass) {
break;
}
esp_video_enc_in_frame_t in_frame = {
.pts = in_load->pts,
.data = in_load->buf,
.size = in_load->valid_size,
};
esp_video_enc_out_frame_t out_frame = {
.data = out_load->buf,
.size = out_frame_size,
};
ret = esp_video_enc_process(venc->enc_handle, &in_frame, &out_frame);
ESP_LOGD(TAG, "I-b:%p,I-sz:%d, O-b:%p,O-sz:%d, in_sz:%d", in_load->buf, in_load->valid_size,
out_load->buf, out_load->valid_size, out_frame_size);
// Handle buffer not enough case
if (ret == ESP_VC_ERR_BUF_NOT_ENOUGH) {
ret = 0;
break;
}
if (ret != ESP_VC_ERR_OK) {
ESP_LOGE(TAG, "Video encoder encode frame error, ret:%d", ret);
ret = ESP_GMF_JOB_ERR_FAIL;
break;
}
out_load->pts = in_load->pts;
out_load->valid_size = out_frame.encoded_size;
ret = out_load->valid_size;
} while (0);
esp_gmf_port_release_in(in, in_load, 0);
esp_gmf_port_release_out(out, out_load, 0);
if (in_load->is_done) {
ret = ESP_GMF_JOB_ERR_DONE;
}
return ret;
}
static esp_gmf_job_err_t venc_el_close(esp_gmf_video_element_handle_t self, void *para)
{
venc_t *venc = (venc_t *)self;
if (venc->enc_handle) {
esp_video_enc_close(venc->enc_handle);
venc->enc_handle = NULL;
}
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_err_t venc_el_destroy(esp_gmf_video_element_handle_t self)
{
venc_t *venc = (venc_t *)self;
if (venc->enc_handle) {
esp_video_enc_close(venc->enc_handle);
venc->enc_handle = NULL;
}
esp_gmf_video_el_deinit(self);
void *cfg = OBJ_GET_CFG(self);
if (cfg) {
esp_gmf_oal_free(cfg);
}
esp_gmf_oal_free(self);
return ESP_OK;
}
static esp_gmf_err_t venc_el_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t cap = {0};
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
do {
cap.cap_eightcc = ESP_GMF_CAPS_VIDEO_ENCODER;
cap.attr_fun = NULL;
ret = esp_gmf_cap_append(&caps, &cap);
GMF_VIDEO_BREAK_ON_FAIL(ret);
((esp_gmf_element_t *) handle)->caps = caps;
return ret;
} while (0);
if (caps) {
esp_gmf_cap_destroy(caps);
}
return ret;
}
static esp_gmf_err_t venc_el_new(void *cfg, esp_gmf_obj_handle_t *obj)
{
return esp_gmf_video_enc_init((esp_gmf_video_enc_cfg_t *)cfg, (esp_gmf_element_handle_t*)obj);
}
static esp_gmf_err_t set_bitrate(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)handle;
int offset = 0;
COPY_ARG(&venc->extra_set.bitrate, buf, offset, sizeof(uint32_t), buf_len);
venc->extra_set.mask |= VENC_EXTRA_SET_MASK_BITRATE;
return venc_el_apply_settings(venc, VENC_EXTRA_SET_MASK_BITRATE);
}
static esp_gmf_err_t get_in_formats(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)handle;
int offset = 0;
uint32_t dst_codec = 0;
const uint32_t **in_fmts = NULL;
uint8_t *in_fmts_num = NULL;
COPY_ARG(&dst_codec, buf, offset, sizeof(uint32_t), buf_len);
COPY_ARG(&in_fmts, buf, offset, sizeof(void *), buf_len);
COPY_ARG(&in_fmts_num, buf, offset, sizeof(void *), buf_len);
return venc_get_input_codecs(venc, dst_codec, in_fmts, in_fmts_num);
}
static esp_gmf_err_t set_format(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)handle;
esp_gmf_info_video_t video_info = {};
uint32_t dst_codec = 0;
int offset = 0;
COPY_ARG(&video_info.format_id, buf, offset, sizeof(uint32_t), buf_len);
COPY_ARG(&video_info.width, buf, offset, sizeof(uint16_t), buf_len);
COPY_ARG(&video_info.height, buf, offset, sizeof(uint16_t), buf_len);
COPY_ARG(&video_info.fps, buf, offset, sizeof(uint8_t), buf_len);
COPY_ARG(&video_info.bitrate, buf, offset, sizeof(uint32_t), buf_len);
COPY_ARG(&dst_codec, buf, offset, sizeof(uint32_t), buf_len);
uint32_t in_fmt = video_info.format_id;
if (venc_is_codec_supported(venc, in_fmt, dst_codec) == false) {
ESP_LOGE(TAG, "Not support encode from %s to %s",
esp_gmf_video_get_format_string(video_info.format_id),
esp_gmf_video_get_format_string(dst_codec));
return ESP_GMF_ERR_NOT_SUPPORT;
}
venc->parent.src_info = video_info;
venc->dst_codec = (esp_video_codec_type_t)(dst_codec);
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t set_dst_codec(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)handle;
venc->dst_codec = *(uint32_t *) buf;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t get_frame_size(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)handle;
uint32_t *frame_size = NULL;
int offset = 0;
COPY_ARG(&frame_size, buf, offset, sizeof(void *), buf_len);
if (frame_size == NULL) {
return ESP_GMF_ERR_INVALID_ARG;
}
*frame_size = 0;
if (venc->parent.src_info.format_id == 0 || venc->dst_codec == ESP_VIDEO_CODEC_TYPE_NONE) {
return ESP_GMF_ERR_NOT_SUPPORT;
}
if (venc->venc_bypass) {
return ESP_GMF_ERR_OK;
}
int in_frame_size = 0, out_frame_size = 0;
venc_get_frame_size(venc, &in_frame_size, &out_frame_size);
*frame_size = out_frame_size;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t venc_el_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_args_desc_t *set_args = NULL;
esp_gmf_method_t *methods = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
int offset = 0;
do {
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(ENCODER, SET_BITRATE, BITRATE), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(ENCODER, SET_BITRATE), set_bitrate, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
offset = 0;
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, SET_DST_CODEC, CODEC), ESP_GMF_ARGS_TYPE_UINT32, sizeof(void *), offset);
ret = esp_gmf_method_append(&methods, VMETHOD(ENCODER, SET_DST_CODEC), set_dst_codec, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
offset = 0;
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, GET_SRC_FMTS, DST_CODEC), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), offset);
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, GET_SRC_FMTS, SRC_FMTS_PTR), ESP_GMF_ARGS_TYPE_UINT32, sizeof(void *), offset);
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, GET_SRC_FMTS, SRC_FMTS_NUM_PTR), ESP_GMF_ARGS_TYPE_UINT32, sizeof(void *), offset);
ret = esp_gmf_method_append(&methods, VMETHOD(ENCODER, GET_SRC_FMTS), get_in_formats, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
offset = 0;
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, PRESET, SRC_FMT), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), offset);
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, PRESET, SRC_WIDTH), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), offset);
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, PRESET, SRC_HEIGHT), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), offset);
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, PRESET, SRC_FPS), ESP_GMF_ARGS_TYPE_UINT8, sizeof(uint8_t), offset);
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, PRESET, SRC_BITRATE), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), offset);
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, PRESET, DST_CODEC), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), offset);
ret = esp_gmf_method_append(&methods, VMETHOD(ENCODER, PRESET), set_format, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
offset = 0;
ADD_ARG_DESC(VMETHOD_ARG(ENCODER, GET_FRAME_SIZE, DST_FRM_SIZE_PTR), ESP_GMF_ARGS_TYPE_UINT32, sizeof(void *), offset);
ret = esp_gmf_method_append(&methods, VMETHOD(ENCODER, GET_FRAME_SIZE), get_frame_size, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
((esp_gmf_element_t *) handle)->method = methods;
return ESP_GMF_ERR_OK;
} while (0);
ESP_LOGE(TAG, "Fail to load methods");
if (set_args) {
esp_gmf_args_desc_destroy(set_args);
}
if (methods) {
esp_gmf_method_destroy(methods);
}
return ESP_GMF_ERR_MEMORY_LACK;
}
esp_gmf_err_t esp_gmf_video_enc_get_out_size(esp_gmf_element_handle_t self, uint32_t *frame_size)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, frame_size, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)self;
*frame_size = 0;
if (venc->parent.src_info.format_id == 0 || venc->dst_codec == ESP_VIDEO_CODEC_TYPE_NONE) {
return ESP_GMF_ERR_NOT_SUPPORT;
}
if (venc->venc_bypass) {
return ESP_GMF_ERR_OK;
}
int in_frame_size = 0, out_frame_size = 0;
venc_get_frame_size(venc, &in_frame_size, &out_frame_size);
*frame_size = out_frame_size;
return ESP_GMF_ERR_OK;
}
esp_gmf_err_t esp_gmf_video_enc_set_bitrate(esp_gmf_element_handle_t self, uint32_t bitrate)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)self;
venc->extra_set.bitrate = bitrate;
venc->extra_set.mask |= VENC_EXTRA_SET_MASK_BITRATE;
return venc_el_apply_settings(venc, VENC_EXTRA_SET_MASK_BITRATE);
}
esp_gmf_err_t esp_gmf_video_enc_set_gop(esp_gmf_element_handle_t self, uint32_t gop)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)self;
venc->extra_set.gop = gop;
venc->extra_set.mask |= VENC_EXTRA_SET_MASK_GOP;
return venc_el_apply_settings(venc, VENC_EXTRA_SET_MASK_GOP);
}
esp_gmf_err_t esp_gmf_video_enc_set_qp(esp_gmf_element_handle_t self, uint32_t min_qp, uint32_t max_qp)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)self;
venc->extra_set.min_qp = min_qp;
venc->extra_set.max_qp = max_qp;
venc->extra_set.mask |= VENC_EXTRA_SET_MASK_QP;
venc_el_apply_settings(venc, VENC_EXTRA_SET_MASK_QP);
return ESP_GMF_ERR_OK;
}
esp_gmf_err_t esp_gmf_video_enc_set_dst_codec(esp_gmf_element_handle_t handle, uint32_t dst_codec)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)handle;
if (venc->enc_handle) {
ESP_LOGE(TAG, "Not support set encoder handle during run");
return ESP_GMF_ERR_NOT_SUPPORT;
}
venc->dst_codec = (esp_video_codec_type_t)dst_codec;
return ESP_GMF_ERR_OK;
}
esp_gmf_err_t esp_gmf_video_enc_preset(esp_gmf_element_handle_t self, esp_gmf_info_video_t *video_info, uint32_t dst_codec)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)self;
if (venc_is_codec_supported(venc, video_info->format_id, dst_codec) == false) {
ESP_LOGE(TAG, "Not support encode from %s to %s",
esp_gmf_video_get_format_string(video_info->format_id), esp_gmf_video_get_format_string(dst_codec));
return ESP_GMF_ERR_NOT_SUPPORT;
}
venc->parent.src_info = *video_info;
venc->dst_codec = (esp_video_codec_type_t)(dst_codec);
return 0;
}
esp_gmf_err_t esp_gmf_video_enc_get_src_formats(esp_gmf_element_handle_t self,
uint32_t dst_codec,
const uint32_t **input_codecs,
uint8_t *input_codec_num)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
venc_t *venc = (venc_t *)self;
return venc_get_input_codecs(venc, dst_codec, input_codecs, input_codec_num);
}
esp_gmf_err_t esp_gmf_video_enc_init(esp_gmf_video_enc_cfg_t *cfg, esp_gmf_element_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_ERR_INVALID_ARG);
venc_t *venc = esp_gmf_oal_calloc(1, sizeof(venc_t));
ESP_GMF_MEM_CHECK(TAG, venc, return ESP_ERR_NO_MEM);
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)venc;
obj->new_obj = venc_el_new;
obj->del_obj = venc_el_destroy;
int ret = esp_gmf_obj_set_tag(obj, "vid_enc");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto VENC_FAIL, "Failed set OBJ tag");
uint8_t align = esp_gmf_oal_get_spiram_cache_align();
esp_gmf_element_cfg_t el_cfg = {
.dependency = true,
};
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_SINGLE, align, align,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, align, align,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ret = esp_gmf_video_el_init(venc, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto VENC_FAIL, "Failed init video encoder el");
if (cfg) {
esp_gmf_video_enc_cfg_t *enc_cfg = (esp_gmf_video_enc_cfg_t *) calloc(1, sizeof(esp_gmf_video_enc_cfg_t));
ESP_GMF_MEM_CHECK(TAG, enc_cfg, { goto VENC_FAIL; });
memcpy(enc_cfg, cfg, sizeof(esp_gmf_video_enc_cfg_t));
esp_gmf_obj_set_config(obj, enc_cfg, sizeof(esp_gmf_video_enc_cfg_t));
}
ESP_GMF_ELEMENT_GET(venc)->ops.open = venc_el_open;
ESP_GMF_ELEMENT_GET(venc)->ops.process = venc_el_process;
ESP_GMF_ELEMENT_GET(venc)->ops.close = venc_el_close;
ESP_GMF_ELEMENT_GET(venc)->ops.event_receiver = esp_gmf_video_handle_events;
ESP_GMF_ELEMENT_GET(venc)->ops.load_caps = venc_el_load_caps;
ESP_GMF_ELEMENT_GET(venc)->ops.load_methods = venc_el_load_methods;
ESP_LOGI(TAG, "Create %s-%p", OBJ_GET_TAG(obj), obj);
*handle = (esp_gmf_element_handle_t) obj;
return ESP_OK;
VENC_FAIL:
esp_gmf_obj_delete(obj);
return ret;
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_fps_cvt.c
+213
View File
@@ -0,0 +1,213 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include <string.h>
#include "esp_log.h"
#include "esp_gmf_err.h"
#include "esp_gmf_node.h"
#include "esp_gmf_oal_mem.h"
#include "esp_gmf_video_fps_cvt.h"
#include "esp_gmf_video_methods_def.h"
#include "esp_gmf_caps_def.h"
#include "esp_gmf_element.h"
#include "esp_gmf_video_element.h"
#include "esp_gmf_info.h"
#include "gmf_video_common.h"
static const char *TAG = "VID_FPS_CVT";
/**
* @brief Video frame rate convert definition
*/
typedef struct {
esp_gmf_video_element_t parent; /*!< Video element parent */
uint16_t dst_fps; /*!< Destination frame rate */
uint32_t frame_num; /*!< Accumulated frame number that received */
uint64_t start_pts; /*!< Started PTS for first received frame */
} gmf_vid_rate_cvt_t;
static esp_gmf_job_err_t gmf_vid_rate_cvt_open(esp_gmf_element_handle_t self, void *para)
{
gmf_vid_rate_cvt_t *rate_cvt = (gmf_vid_rate_cvt_t *)self;
esp_gmf_info_video_t *src_info = &rate_cvt->parent.src_info;
if (rate_cvt->dst_fps == 0 || rate_cvt->dst_fps > src_info->fps) {
ESP_LOGE(TAG, "dst fps %d > src fps %d", rate_cvt->dst_fps, src_info->fps);
return ESP_GMF_JOB_ERR_FAIL;
}
esp_gmf_info_video_t vid_info = *src_info;
vid_info.fps = rate_cvt->dst_fps;
esp_gmf_element_notify_vid_info(self, &vid_info);
rate_cvt->frame_num = 0;
rate_cvt->start_pts = 0;
return ESP_GMF_JOB_ERR_OK;
}
static bool rate_control_need_drop(gmf_vid_rate_cvt_t *rate_cvt, esp_gmf_payload_t *in_load)
{
esp_gmf_info_video_t *src_info = &rate_cvt->parent.src_info;
if (rate_cvt->dst_fps == src_info->fps) {
return false;
}
if (rate_cvt->frame_num == 0) {
rate_cvt->start_pts = in_load->pts;
rate_cvt->frame_num++;
return false;
}
uint64_t expected_pts = rate_cvt->start_pts + (rate_cvt->frame_num) * 1000 / rate_cvt->dst_fps;
if (in_load->pts >= expected_pts) {
rate_cvt->frame_num++;
return false;
}
return true;
}
static esp_gmf_job_err_t gmf_vid_rate_cvt_process(esp_gmf_element_handle_t self, void *para)
{
gmf_vid_rate_cvt_t *rate_cvt = (gmf_vid_rate_cvt_t *)self;
int ret = 0;
esp_gmf_port_handle_t in_port = ESP_GMF_ELEMENT_GET(self)->in;
esp_gmf_port_handle_t out_port = ESP_GMF_ELEMENT_GET(self)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
ret = esp_gmf_port_acquire_in(in_port, &in_load, ESP_GMF_ELEMENT_GET(self)->in_attr.data_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Read data error, ret:%d, line:%d", ret, __LINE__);
return ret == ESP_GMF_IO_ABORT ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
if (in_load->is_done && in_load->valid_size == 0) {
esp_gmf_port_release_in(in_port, in_load, 0);
return ESP_GMF_JOB_ERR_DONE;
}
// Handle drop logic
if (rate_control_need_drop(rate_cvt, in_load)) {
esp_gmf_port_release_in(in_port, in_load, ESP_GMF_MAX_DELAY);
return ESP_GMF_JOB_ERR_CONTINUE;
}
out_load = in_load;
ret = esp_gmf_port_acquire_out(out_port, &out_load, in_load->valid_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Write data error, ret:%d, line:%d", ret, __LINE__);
return ret == ESP_GMF_IO_ABORT ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
esp_gmf_port_release_out(out_port, out_load, ESP_GMF_MAX_DELAY);
esp_gmf_port_release_in(in_port, in_load, ESP_GMF_MAX_DELAY);
return ret;
}
static esp_gmf_job_err_t gmf_vid_rate_cvt_close(esp_gmf_element_handle_t self, void *para)
{
gmf_vid_rate_cvt_t *rate_cvt = (gmf_vid_rate_cvt_t *)self;
rate_cvt->frame_num = 0;
rate_cvt->start_pts = 0;
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_err_t set_dst_fps(esp_gmf_element_handle_t self, esp_gmf_args_desc_t *arg_desc, uint8_t *buf, int buf_len)
{
gmf_vid_rate_cvt_t *rate_cvt = (gmf_vid_rate_cvt_t *)self;
rate_cvt->dst_fps = *(uint16_t *)buf;
// After change fps reset frame number
rate_cvt->frame_num = 0;
return ESP_OK;
}
static esp_gmf_err_t gmf_vid_rate_cvt_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_args_desc_t *set_args = NULL;
esp_gmf_method_t *methods = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
do {
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(FPS_CVT, SET_FPS, FPS), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(FPS_CVT, SET_FPS), set_dst_fps, set_args);
((esp_gmf_element_t *) handle)->method = methods;
return ESP_GMF_ERR_OK;
} while (0);
ESP_LOGE(TAG, "Fail to load methods");
if (set_args) {
esp_gmf_args_desc_destroy(set_args);
}
if (methods) {
esp_gmf_method_destroy(methods);
}
return ESP_GMF_ERR_MEMORY_LACK;
}
static esp_gmf_err_t gmf_vid_rate_cvt_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t cap = {
.cap_eightcc = ESP_GMF_CAPS_VIDEO_FPS_CVT
};
esp_gmf_err_t ret = esp_gmf_cap_append(&caps, &cap);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append caps");
((esp_gmf_element_t *) handle)->caps = caps;
return ret;
}
static esp_gmf_err_t gmf_vid_rate_cvt_new(void *cfg, esp_gmf_obj_handle_t *handle)
{
return esp_gmf_video_fps_cvt_init(cfg, (esp_gmf_element_handle_t*)handle);
}
static esp_gmf_err_t gmf_vid_rate_cvt_destroy(esp_gmf_element_handle_t self)
{
gmf_vid_rate_cvt_t *rate_cvt = (gmf_vid_rate_cvt_t *)self;
esp_gmf_video_el_deinit(self);
if (rate_cvt) {
esp_gmf_oal_free(rate_cvt);
}
return ESP_GMF_ERR_OK;
}
esp_gmf_err_t esp_gmf_video_fps_cvt_init(void *config, esp_gmf_element_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
gmf_vid_rate_cvt_t *vid_rate_cvt = esp_gmf_oal_calloc(1, sizeof(gmf_vid_rate_cvt_t));
ESP_GMF_MEM_CHECK(TAG, vid_rate_cvt, return ESP_GMF_ERR_MEMORY_LACK);
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)vid_rate_cvt;
obj->new_obj = gmf_vid_rate_cvt_new;
obj->del_obj = gmf_vid_rate_cvt_destroy;
esp_gmf_err_t ret = esp_gmf_obj_set_tag(obj, "vid_fps_cvt");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto _cvt_init_fail, "Failed set OBJ tag");
esp_gmf_element_cfg_t el_cfg = { 0 };
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
el_cfg.dependency = true;
ret = esp_gmf_video_el_init(obj, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto _cvt_init_fail, "Failed init video rate convert");
vid_rate_cvt->parent.base.ops.open = gmf_vid_rate_cvt_open;
vid_rate_cvt->parent.base.ops.process = gmf_vid_rate_cvt_process;
vid_rate_cvt->parent.base.ops.close = gmf_vid_rate_cvt_close;
vid_rate_cvt->parent.base.ops.event_receiver = esp_gmf_video_handle_events;
vid_rate_cvt->parent.base.ops.load_methods = gmf_vid_rate_cvt_load_methods;
vid_rate_cvt->parent.base.ops.load_caps = gmf_vid_rate_cvt_load_caps;
*handle = obj;
return ESP_GMF_ERR_OK;
_cvt_init_fail:
esp_gmf_obj_delete(obj);
return ret;
}
esp_gmf_err_t esp_gmf_video_fps_cvt_set_fps(esp_gmf_element_handle_t handle, uint16_t fps)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method((esp_gmf_element_handle_t)handle, &method_head);
esp_gmf_method_found(method_head, VMETHOD(FPS_CVT, SET_FPS), &method);
uint8_t buf[2] = { 0 };
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(FPS_CVT, SET_FPS, FPS), buf, (uint8_t *)&fps, sizeof(fps));
return esp_gmf_element_exe_method((esp_gmf_element_handle_t)handle, VMETHOD(FPS_CVT, SET_FPS), buf, sizeof(buf));
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_overlay.c
+442
View File
@@ -0,0 +1,442 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include <string.h>
#include "esp_log.h"
#include "esp_gmf_err.h"
#include "esp_fourcc.h"
#include "esp_gmf_node.h"
#include "esp_gmf_oal_mem.h"
#include "esp_gmf_video_element.h"
#include "esp_gmf_info.h"
#include "esp_gmf_video_overlay.h"
#include "esp_gmf_video_methods_def.h"
#include "esp_gmf_caps_def.h"
#include "gmf_video_common.h"
static const char *TAG = "OVERLAY_MIXER";
/**
* @brief Video pixel data definition
*/
typedef struct {
uint8_t *data; /*!< Video pixel data */
uint32_t size; /*!< Video pixel data size */
} esp_gmf_video_pixel_data_t;
/**
* @brief Video overlay definition
*/
typedef struct {
esp_gmf_video_element_t parent; /*!< Video element parent */
esp_gmf_port_handle_t overlay_port; /*!< GMF port for overlay */
bool enable; /*!< Overlay enable setting from user */
bool overlay_enabled; /*!< Whether overlay enabled or not */
esp_gmf_overlay_rgn_info_t overlay_rgn; /*!< Overlay region info */
uint8_t window_alpha; /*!< Overlay window alpha */
bool is_open; /*!< Whether element is open or not */
} gmf_vid_overlay_t;
static esp_gmf_err_t sw_mixer_open(gmf_vid_overlay_t *mixer)
{
// Currently only support RGB565
if (mixer->parent.src_info.format_id != ESP_FOURCC_RGB16) {
ESP_LOGE(TAG, "Only support RGB565");
return ESP_GMF_ERR_NOT_SUPPORT;
}
return ESP_GMF_ERR_OK;
}
static inline uint16_t sw_mix_rgb565(uint16_t pixel_a, uint16_t pixel_b, uint8_t alpha)
{
uint8_t inv_alpha = 255 - alpha;
uint8_t r_a = (pixel_a >> 11) & 0x1F;
uint8_t g_a = (pixel_a >> 5) & 0x3F;
uint8_t b_a = pixel_a & 0x1F;
uint8_t r_b = (pixel_b >> 11) & 0x1F;
uint8_t g_b = (pixel_b >> 5) & 0x3F;
uint8_t b_b = pixel_b & 0x1F;
uint8_t r = (r_a * inv_alpha + r_b * alpha) >> 8;
uint8_t g = (g_a * inv_alpha + g_b * alpha) >> 8;
uint8_t b = (b_a * inv_alpha + b_b * alpha) >> 8;
return (r << 11) | (g << 5) | b;
}
static esp_gmf_err_t sw_mixer_process(gmf_vid_overlay_t *mixer, esp_gmf_video_pixel_data_t *dst,
esp_gmf_video_pixel_data_t *window_data)
{
if (mixer->window_alpha == 0) {
return ESP_GMF_ERR_OK;
}
esp_gmf_video_rgn_t *window = &mixer->overlay_rgn.dst_rgn;
esp_gmf_info_video_t *src_info = &mixer->parent.src_info;
if (mixer->overlay_rgn.format_id == ESP_FOURCC_RGB16) {
// size check
int dst_size = src_info->width * src_info->height * 2;
int window_size = window->width * window->height * 2;
if (dst->size < dst_size || window_data->size < window_size) {
return ESP_GMF_ERR_INVALID_ARG;
}
uint16_t *dst_pixel = ((uint16_t *)dst->data) + window->y * src_info->width;
dst_pixel += window->x;
uint16_t *window_pixel = (uint16_t *)window_data->data;
if (mixer->window_alpha == 255) {
for (int i = 0; i < window->height; i++) {
memcpy(dst_pixel, window_pixel, window->width * 2);
dst_pixel += src_info->width;
window_pixel += window->width;
}
return ESP_GMF_ERR_OK;
} else {
for (int i = 0; i < window->height; i++) {
uint16_t *cur_dst = dst_pixel;
uint16_t *cur_window = window_pixel;
uint16_t *line_end = cur_window + window->width;
while (cur_window < line_end) {
*cur_dst = sw_mix_rgb565(*cur_dst, *cur_window, mixer->window_alpha);
cur_dst++;
cur_window++;
}
dst_pixel += src_info->width;
window_pixel = line_end;
}
return ESP_GMF_ERR_OK;
}
}
return ESP_GMF_ERR_NOT_SUPPORT;
}
static esp_gmf_err_t overlay_enable(gmf_vid_overlay_t *overlay_mixer)
{
esp_gmf_info_video_t *src_info = &overlay_mixer->parent.src_info;
// Element not open or no overlay set
if (overlay_mixer->overlay_port == NULL || overlay_mixer->enable == false || overlay_mixer->is_open == false) {
return ESP_GMF_ERR_OK;
}
if (overlay_mixer->overlay_enabled) {
return ESP_GMF_ERR_OK;
}
esp_gmf_video_rgn_t *rgn = &overlay_mixer->overlay_rgn.dst_rgn;
if (overlay_mixer->overlay_rgn.format_id != src_info->format_id ||
rgn->x + rgn->width > src_info->width ||
rgn->y + rgn->height > src_info->height) {
ESP_LOGE(TAG, "Wrong overlay region or codec settings");
return ESP_GMF_ERR_NOT_SUPPORT;
}
esp_gmf_err_t ret = sw_mixer_open(overlay_mixer);
if (ret != ESP_GMF_ERR_OK) {
return ret;
}
overlay_mixer->overlay_enabled = true;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t overlay_disable(gmf_vid_overlay_t *overlay_mixer)
{
if (overlay_mixer->overlay_enabled == false) {
return ESP_GMF_ERR_INVALID_STATE;
}
overlay_mixer->overlay_enabled = false;
return ESP_GMF_ERR_OK;
}
static esp_gmf_job_err_t gmf_vid_overlay_open(esp_gmf_element_handle_t self, void *para)
{
gmf_vid_overlay_t *overlay_mixer = (gmf_vid_overlay_t *)self;
overlay_mixer->is_open = true;
esp_gmf_err_t ret = overlay_enable(overlay_mixer);
if (ret != ESP_GMF_ERR_OK) {
return ESP_GMF_JOB_ERR_FAIL;
}
esp_gmf_element_notify_vid_info(self, &overlay_mixer->parent.src_info);
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_job_err_t gmf_vid_overlay_process(esp_gmf_element_handle_t self, void *para)
{
gmf_vid_overlay_t *overlay_mixer = (gmf_vid_overlay_t *)self;
int ret = 0;
esp_gmf_port_handle_t in_port = ESP_GMF_ELEMENT_GET(self)->in;
esp_gmf_port_handle_t out_port = ESP_GMF_ELEMENT_GET(self)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
ret = esp_gmf_port_acquire_in(in_port, &in_load, ESP_GMF_ELEMENT_GET(self)->in_attr.data_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Read data error, ret:%d, line:%d", ret, __LINE__);
return ret == ESP_GMF_IO_ABORT ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
if (in_load->is_done && in_load->valid_size == 0) {
esp_gmf_port_release_in(in_port, in_load, 0);
return ESP_GMF_JOB_ERR_DONE;
}
out_load = in_load;
if (overlay_mixer->overlay_enabled && overlay_mixer->overlay_port) {
// Fetch overlay data
esp_gmf_payload_t *overlay_load = NULL;
ret = esp_gmf_port_acquire_in(overlay_mixer->overlay_port, &overlay_load, ESP_GMF_ELEMENT_GET(self)->in_attr.data_size,
ESP_GMF_MAX_DELAY);
if (ret >= 0) {
// TODO workaround use PTS to store alpha
uint8_t alpha = (uint8_t)overlay_load->pts;
overlay_mixer->window_alpha = alpha;
// Do mixer
esp_gmf_video_pixel_data_t dst_frame = {
.data = in_load->buf,
.size = in_load->valid_size,
};
esp_gmf_video_pixel_data_t overlay_frame = {
.data = overlay_load->buf,
.size = overlay_load->valid_size,
};
ret = sw_mixer_process(overlay_mixer, &dst_frame, &overlay_frame);
esp_gmf_port_release_in(overlay_mixer->overlay_port, overlay_load, ESP_GMF_MAX_DELAY);
} else {
ESP_LOGE(TAG, "Fail to fetch overlay data ret %d", ret);
}
}
ret = esp_gmf_port_acquire_out(out_port, &out_load, in_load->valid_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Write data error, ret:%d, line:%d", ret, __LINE__);
return ret == ESP_GMF_IO_ABORT ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
esp_gmf_port_release_out(out_port, out_load, ESP_GMF_MAX_DELAY);
esp_gmf_port_release_in(in_port, in_load, ESP_GMF_MAX_DELAY);
return ret;
}
static esp_gmf_job_err_t gmf_vid_overlay_close(esp_gmf_element_handle_t self, void *para)
{
gmf_vid_overlay_t *overlay_mixer = (gmf_vid_overlay_t *)self;
overlay_disable(overlay_mixer);
overlay_mixer->is_open = false;
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_err_t gmf_vid_overlay_destroy(esp_gmf_element_handle_t self)
{
gmf_vid_overlay_t *overlay_mixer = (gmf_vid_overlay_t *)self;
esp_gmf_video_el_deinit(self);
if (overlay_mixer != NULL) {
esp_gmf_oal_free(overlay_mixer);
}
return ESP_GMF_ERR_OK;
}
esp_gmf_err_t esp_gmf_video_overlay_set_overlay_port(esp_gmf_element_handle_t self, esp_gmf_port_handle_t port)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, port, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method(self, &method_head);
esp_gmf_method_found(method_head, VMETHOD(OVERLAY, SET_PORT), &method);
uint8_t buf[sizeof(esp_gmf_port_handle_t)];
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(OVERLAY, SET_PORT, PORT), buf, (uint8_t *)&port, sizeof(esp_gmf_port_handle_t));
return esp_gmf_element_exe_method(self, VMETHOD(OVERLAY, SET_PORT), buf, sizeof(buf));
}
esp_gmf_err_t esp_gmf_video_overlay_set_alpha(esp_gmf_element_handle_t self, uint8_t alpha)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method(self, &method_head);
esp_gmf_method_found(method_head, VMETHOD(OVERLAY, SET_ALPHA), &method);
uint8_t buf[sizeof(uint8_t)];
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(OVERLAY, SET_ALPHA, ALPHA), buf, (uint8_t *)&alpha, sizeof(esp_gmf_port_handle_t));
return esp_gmf_element_exe_method(self, VMETHOD(OVERLAY, SET_ALPHA), buf, sizeof(buf));
}
esp_gmf_err_t esp_gmf_video_overlay_set_rgn(esp_gmf_element_handle_t self, esp_gmf_overlay_rgn_info_t *rgn_info)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, rgn_info, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method(self, &method_head);
esp_gmf_method_found(method_head, VMETHOD(OVERLAY, SET_RGN), &method);
uint8_t buf[sizeof(esp_gmf_overlay_rgn_info_t)];
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(OVERLAY, SET_RGN, FMT), buf, (uint8_t *)&rgn_info->format_id, sizeof(uint32_t));
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(OVERLAY, SET_RGN, X), buf, (uint8_t *)&rgn_info->dst_rgn.x, sizeof(uint16_t));
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(OVERLAY, SET_RGN, Y), buf, (uint8_t *)&rgn_info->dst_rgn.y, sizeof(uint16_t));
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(OVERLAY, SET_RGN, WIDTH), buf, (uint8_t *)&rgn_info->dst_rgn.width, sizeof(uint16_t));
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(OVERLAY, SET_RGN, HEIGHT), buf, (uint8_t *)&rgn_info->dst_rgn.height, sizeof(uint16_t));
return esp_gmf_element_exe_method(self, VMETHOD(OVERLAY, SET_RGN), buf, sizeof(buf));
}
esp_gmf_err_t esp_gmf_video_overlay_enable(esp_gmf_element_handle_t self, bool enable)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method(self, &method_head);
esp_gmf_method_found(method_head, VMETHOD(OVERLAY, OVERLAY_ENABLE), &method);
uint8_t buf[sizeof(bool)];
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(OVERLAY, OVERLAY_ENABLE, ENABLE), buf, (uint8_t *)&enable, sizeof(bool));
return esp_gmf_element_exe_method(self, VMETHOD(OVERLAY, OVERLAY_ENABLE), buf, sizeof(buf));
}
static esp_gmf_err_t set_mixer_enable(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
gmf_vid_overlay_t *overlay_mixer = (gmf_vid_overlay_t *)handle;
overlay_mixer->enable = *(bool *)buf;
return overlay_mixer->enable ? overlay_enable(overlay_mixer) : overlay_disable(overlay_mixer);
}
static esp_gmf_err_t set_mixer_rgn(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
gmf_vid_overlay_t *overlay_mixer = (gmf_vid_overlay_t *)handle;
overlay_mixer->overlay_rgn.format_id = *(uint32_t *)buf;
buf += sizeof(uint32_t);
overlay_mixer->overlay_rgn.dst_rgn.x = *(uint16_t *)(buf);
overlay_mixer->overlay_rgn.dst_rgn.y = *(uint16_t *)(buf + 2);
overlay_mixer->overlay_rgn.dst_rgn.width = *(uint16_t *)(buf + 4);
overlay_mixer->overlay_rgn.dst_rgn.height = *(uint16_t *)(buf + 6);
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t set_mixer_port(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
gmf_vid_overlay_t *overlay_mixer = (gmf_vid_overlay_t *)handle;
overlay_mixer->overlay_port = *(esp_gmf_port_handle_t *)buf;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t set_mixer_alpha(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
gmf_vid_overlay_t *overlay_mixer = (gmf_vid_overlay_t *)handle;
overlay_mixer->window_alpha = *(uint8_t *)buf;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t gmf_vid_overlay_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_args_desc_t *set_args = NULL;
esp_gmf_method_t *methods = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
do {
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(OVERLAY, OVERLAY_ENABLE, ENABLE), ESP_GMF_ARGS_TYPE_UINT8, sizeof(uint8_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(OVERLAY, OVERLAY_ENABLE), set_mixer_enable, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(OVERLAY, SET_RGN, FMT), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(OVERLAY, SET_RGN, X), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 4);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(OVERLAY, SET_RGN, Y), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 6);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(OVERLAY, SET_RGN, WIDTH), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 8);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(OVERLAY, SET_RGN, HEIGHT), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 10);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(OVERLAY, SET_RGN), set_mixer_rgn, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(OVERLAY, SET_PORT, PORT), ESP_GMF_ARGS_TYPE_UINT32, sizeof(intptr_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(OVERLAY, SET_PORT), set_mixer_port, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(OVERLAY, SET_ALPHA, ALPHA), ESP_GMF_ARGS_TYPE_UINT8, sizeof(uint8_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(OVERLAY, SET_ALPHA), set_mixer_alpha, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
((esp_gmf_element_t *) handle)->method = methods;
return ESP_GMF_ERR_OK;
} while (0);
ESP_LOGE(TAG, "Fail to load methods");
if (set_args) {
esp_gmf_args_desc_destroy(set_args);
}
if (methods) {
esp_gmf_method_destroy(methods);
}
return ESP_GMF_ERR_MEMORY_LACK;
}
static esp_gmf_err_t gmf_vid_overlay_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t cap = {0};
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
do {
cap.cap_eightcc = ESP_GMF_CAPS_VIDEO_OVERLAY;
cap.attr_fun = NULL;
ret = esp_gmf_cap_append(&caps, &cap);
GMF_VIDEO_BREAK_ON_FAIL(ret);
((esp_gmf_element_t *) handle)->caps = caps;
return ret;
} while (0);
if (caps) {
esp_gmf_cap_destroy(caps);
}
return ret;
}
static esp_gmf_err_t gmf_vid_overlay_new(void *cfg, esp_gmf_obj_handle_t *handle)
{
return esp_gmf_video_overlay_init(cfg, (esp_gmf_element_handle_t *)handle);
}
esp_gmf_err_t esp_gmf_video_overlay_init(void *config, esp_gmf_element_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
gmf_vid_overlay_t *overlay_mixer = esp_gmf_oal_calloc(1, sizeof(gmf_vid_overlay_t));
ESP_GMF_MEM_CHECK(TAG, overlay_mixer, return ESP_GMF_ERR_MEMORY_LACK);
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)overlay_mixer;
obj->new_obj = gmf_vid_overlay_new;
obj->del_obj = gmf_vid_overlay_destroy;
int ret = esp_gmf_obj_set_tag(obj, "vid_overlay");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto _overlay_init_fail, "Failed set OBJ tag");
esp_gmf_element_cfg_t el_cfg = { 0 };
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_MULTI, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
el_cfg.dependency = true;
ret = esp_gmf_video_el_init(obj, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto _overlay_init_fail, "Failed init video rate convert");
overlay_mixer->parent.base.ops.open = gmf_vid_overlay_open;
overlay_mixer->parent.base.ops.process = gmf_vid_overlay_process;
overlay_mixer->parent.base.ops.close = gmf_vid_overlay_close;
overlay_mixer->parent.base.ops.event_receiver = esp_gmf_video_handle_events;
overlay_mixer->parent.base.ops.load_methods = gmf_vid_overlay_load_methods;
overlay_mixer->parent.base.ops.load_caps = gmf_vid_overlay_load_caps;
*handle = obj;
ESP_LOGI(TAG, "Create video overlay, %s-%p", OBJ_GET_TAG(obj), obj);
return ESP_GMF_ERR_OK;
_overlay_init_fail:
esp_gmf_obj_delete(obj);
return ret;
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_param.c
+162
View File
@@ -0,0 +1,162 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include "esp_gmf_video_param.h"
#define PREPARE_VMETHOD_SETTING() \
esp_gmf_method_exec_ctx_t exec_ctx = {}; \
const esp_gmf_method_t* method_head = NULL; \
esp_gmf_element_get_method(self, &method_head); \
esp_gmf_err_t ret = esp_gmf_method_prepare_exec_ctx(method_head, method_name, &exec_ctx); \
if (ret != ESP_GMF_ERR_OK) { \
return ret; \
}
#define SET_METHOD_ARG(arg_name, value) \
esp_gmf_args_set_value(exec_ctx.method->args_desc, arg_name, exec_ctx.exec_buf, (uint8_t*) &value, sizeof(value));
#define RELEASE_VMETHOD_SETTING() \
ret = exec_ctx.method->func(self, exec_ctx.method->args_desc, exec_ctx.exec_buf, exec_ctx.buf_size); \
esp_gmf_method_release_exec_ctx(&exec_ctx); \
return ret;
esp_gmf_err_t esp_gmf_video_param_set_dst_format(esp_gmf_element_handle_t self, uint32_t dst_fmt)
{
const char *method_name = VMETHOD(CLR_CVT, SET_DST_FMT);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(CLR_CVT, SET_DST_FMT, FMT), dst_fmt);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_set_fps(esp_gmf_element_handle_t self, uint16_t fps)
{
const char *method_name = VMETHOD(FPS_CVT, SET_FPS);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(FPS_CVT, SET_FPS, FPS), fps);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_set_dst_resolution(esp_gmf_element_handle_t self, esp_gmf_video_resolution_t *res)
{
const char *method_name = VMETHOD(SCALER, SET_DST_RES);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(SCALER, SET_DST_RES, WIDTH), res->width);
SET_METHOD_ARG(VMETHOD_ARG(SCALER, SET_DST_RES, HEIGHT), res->height);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_set_dst_codec(esp_gmf_element_handle_t self, uint32_t dst_codec)
{
const char *method_name = VMETHOD(ENCODER, SET_DST_CODEC);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, SET_DST_CODEC, CODEC), dst_codec);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_venc_preset(esp_gmf_element_handle_t self, esp_gmf_info_video_t *vid_info,
uint32_t dst_codec)
{
const char *method_name = VMETHOD(ENCODER, PRESET);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, PRESET, SRC_FMT), vid_info->format_id);
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, PRESET, SRC_WIDTH), vid_info->width);
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, PRESET, SRC_HEIGHT), vid_info->height);
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, PRESET, SRC_FPS), vid_info->fps);
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, PRESET, SRC_BITRATE), vid_info->bitrate);
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, PRESET, DST_CODEC), dst_codec);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_get_src_fmts_by_codec(esp_gmf_element_handle_t self, uint32_t dst_codec,
const uint32_t **src_fmts, uint8_t *src_fmts_num)
{
const char *method_name = VMETHOD(ENCODER, GET_SRC_FMTS);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, GET_SRC_FMTS, DST_CODEC), dst_codec);
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, GET_SRC_FMTS, SRC_FMTS_PTR), src_fmts);
SET_METHOD_ARG(VMETHOD_ARG(ENCODER, GET_SRC_FMTS, SRC_FMTS_NUM_PTR), src_fmts_num);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_set_src_codec(esp_gmf_element_handle_t self, uint32_t src_codec)
{
const char *method_name = VMETHOD(DECODER, SET_SRC_CODEC);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(DECODER, SET_SRC_CODEC, CODEC), src_codec);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_set_rotate_angle(esp_gmf_element_handle_t self, uint16_t degree)
{
const char *method_name = VMETHOD(ROTATOR, SET_ANGLE);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(ROTATOR, SET_ANGLE, DEGREE), degree);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_set_cropped_region(esp_gmf_element_handle_t self, esp_gmf_video_rgn_t *rgn)
{
const char *method_name = VMETHOD(CROP, SET_CROP_RGN);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(CROP, SET_CROP_RGN, X), rgn->x);
SET_METHOD_ARG(VMETHOD_ARG(CROP, SET_CROP_RGN, Y), rgn->y);
SET_METHOD_ARG(VMETHOD_ARG(CROP, SET_CROP_RGN, WIDTH), rgn->width);
SET_METHOD_ARG(VMETHOD_ARG(CROP, SET_CROP_RGN, HEIGHT), rgn->height);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_overlay_enable(esp_gmf_element_handle_t self, bool enable)
{
const char *method_name = VMETHOD(OVERLAY, OVERLAY_ENABLE);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(OVERLAY, OVERLAY_ENABLE, ENABLE), enable);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_set_overlay_port(esp_gmf_element_handle_t self, void *port)
{
const char *method_name = VMETHOD(OVERLAY, SET_PORT);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(OVERLAY, SET_PORT, PORT), port);
RELEASE_VMETHOD_SETTING();
}
esp_gmf_err_t esp_gmf_video_param_set_overlay_rgn(esp_gmf_element_handle_t self, esp_gmf_overlay_rgn_info_t *rgn)
{
const char *method_name = VMETHOD(OVERLAY, SET_RGN);
PREPARE_VMETHOD_SETTING();
SET_METHOD_ARG(VMETHOD_ARG(OVERLAY, SET_RGN, FMT), rgn->format_id);
SET_METHOD_ARG(VMETHOD_ARG(OVERLAY, SET_RGN, X), rgn->dst_rgn.x);
SET_METHOD_ARG(VMETHOD_ARG(OVERLAY, SET_RGN, Y), rgn->dst_rgn.y);
SET_METHOD_ARG(VMETHOD_ARG(OVERLAY, SET_RGN, WIDTH), rgn->dst_rgn.width);
SET_METHOD_ARG(VMETHOD_ARG(OVERLAY, SET_RGN, HEIGHT), rgn->dst_rgn.height);
RELEASE_VMETHOD_SETTING();
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_ppa.c
+926
View File
@@ -0,0 +1,926 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include <string.h>
#include "esp_log.h"
#include "esp_gmf_err.h"
#include "esp_gmf_node.h"
#include "esp_gmf_oal_mem.h"
#include "esp_gmf_video_ppa.h"
#include "esp_gmf_video_methods_def.h"
#include "esp_gmf_caps_def.h"
#include "esp_gmf_element.h"
#include "esp_gmf_video_element.h"
#include "esp_gmf_info.h"
#include "gmf_video_common.h"
#include "esp_heap_caps.h"
#include "esp_fourcc.h"
#include "esp_cache.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/semphr.h"
#include "esp_idf_version.h"
#if CONFIG_IDF_TARGET_ESP32P4
#include "driver/ppa.h"
#include "esp_private/dma2d.h"
#include "hal/dma2d_types.h"
#include "hal/color_types.h"
#include "soc/dma2d_channel.h"
#endif
static const char *TAG = "VCVT_EL";
#if CONFIG_IDF_TARGET_ESP32P4
/**
* @brief 2D-DMA EOF callback
*/
typedef bool (*dma2d_m2m_trans_eof_callback_t)(void *user_data);
/**
* @brief 2D-DMA M2M transport configuration
*/
typedef struct {
intptr_t tx_desc_base_addr; /*!< 2D-DMA TX descriptor address */
intptr_t rx_desc_base_addr; /*!< 2D-DMA RX descriptor address */
dma2d_m2m_trans_eof_callback_t trans_eof_cb; /*!< Transfer EOF callback */
void *user_data; /*!< User registered data to be passed into `trans_eof_cb` callback */
dma2d_transfer_ability_t transfer_ability; /*!< Transfer ability */
dma2d_strategy_config_t *tx_strategy_config; /*!< Pointer to a collection of 2D-DMA TX strategy configuration */
dma2d_strategy_config_t *rx_strategy_config; /*!< Pointer to a collection of 2D-DMA RX strategy configuration */
dma2d_csc_config_t *tx_csc_config; /*!< Pointer to a collection of 2D-DMA TX color space conversion configuration */
dma2d_csc_config_t *rx_csc_config; /*!< Pointer to a collection of 2D-DMA RX color space conversion configuration */
} dma2d_m2m_trans_config_t;
/**
* @brief 2D-DMA M2M transaction definition
*/
typedef struct {
dma2d_m2m_trans_config_t m2m_trans_desc; /*!< M2M transport description configuration */
dma2d_trans_config_t dma_chan_desc; /*!< 2D-DMA transport configuration */
uint8_t dma_trans_placeholder_head[64]; /*!< DMA transport description place holder */
} dma2d_m2m_transaction_t;
/**
* @brief 2D-DMA information definition
*/
typedef struct {
dma2d_descriptor_t *rx_desc; /*!< Pointer to 2D-DMA RX description */
dma2d_descriptor_t *tx_desc; /*!< Pointer to 2D-DMA TX description */
dma2d_pool_handle_t handle; /*!< 2D-DMA pool handle */
SemaphoreHandle_t sema; /*!< Semaphore for wait transfer done */
dma2d_m2m_transaction_t trans; /*!< 2D-DMA M2M transaction */
dma2d_csc_config_t tx_cvt; /*!< 2D-DMA TX color space conversion configuration */
} dma2d_info_t;
#endif
/**
* @brief Video PPA (Pixel Processing Accelerator) definition
*/
typedef struct {
esp_gmf_video_element_t parent; /*!< Video element parent */
uint32_t dst_format; /*!< Color converter destination format */
uint16_t dst_width; /*!< Scale destination width */
uint16_t dst_height; /*!< Scale destination height */
uint16_t rotate_degree; /*!< Rotation angle setting */
esp_gmf_video_rgn_t crop_rgn; /*!< Cropped region setting */
uint32_t out_frame_size; /*!< Output frame size of PPA */
bool bypass; /*!< Whether PPA is bypassed or not */
#if CONFIG_IDF_TARGET_ESP32P4
ppa_client_handle_t ppa_handle; /*!< PPA client handle */
ppa_srm_oper_config_t ppa_config; /*!< PPA SRM operation configuration */
bool supported; /*!< Whether setting supported or not */
bool use_ppa; /*!< Whether use PPA or 2D-DMA */
dma2d_info_t dma2d_info; /*!< 2D-DMA information */
#endif
} gmf_video_ppa_t;
#if CONFIG_IDF_TARGET_ESP32P4
static uint32_t get_frame_size(gmf_video_ppa_t *vid_cvt, uint32_t codec)
{
switch (codec) {
case ESP_FOURCC_RGB24:
case ESP_FOURCC_BGR24:
return vid_cvt->dst_width * vid_cvt->dst_height * 3;
case ESP_FOURCC_RGB16:
case ESP_FOURCC_RGB16_BE:
return vid_cvt->dst_width * vid_cvt->dst_height * 2;
case ESP_FOURCC_YUV420P:
case ESP_FOURCC_OUYY_EVYY:
return vid_cvt->dst_width * vid_cvt->dst_height * 3 / 2;
case ESP_FOURCC_YUYV:
return vid_cvt->dst_width * vid_cvt->dst_height * 2;
default:
return 0;
}
}
static ppa_srm_color_mode_t get_ppa_clr_mode(uint32_t codec)
{
switch (codec) {
case ESP_FOURCC_RGB24:
case ESP_FOURCC_BGR24:
return PPA_SRM_COLOR_MODE_RGB888;
case ESP_FOURCC_RGB16_BE:
case ESP_FOURCC_RGB16:
return PPA_SRM_COLOR_MODE_RGB565;
case ESP_FOURCC_YUV420P:
case ESP_FOURCC_OUYY_EVYY:
return PPA_SRM_COLOR_MODE_YUV420;
default:
return (ppa_srm_color_mode_t)0;
}
}
static bool check_2ddma_supported(gmf_video_ppa_t *vid_cvt)
{
dma2d_info_t *dma2d = &vid_cvt->dma2d_info;
esp_gmf_info_video_t *src_info = &vid_cvt->parent.src_info;
if (src_info->format_id == ESP_FOURCC_RGB16_BE && vid_cvt->dst_format == ESP_FOURCC_BGR24) {
dma2d->tx_cvt.pre_scramble = 1;
} else if ((src_info->format_id == ESP_FOURCC_RGB16 && vid_cvt->dst_format == ESP_FOURCC_BGR24) || (src_info->format_id == ESP_FOURCC_BGR24 && vid_cvt->dst_format == ESP_FOURCC_RGB16)) {
dma2d->tx_cvt.pre_scramble = 0;
} else if ((src_info->format_id == ESP_FOURCC_RGB24 && vid_cvt->dst_format == ESP_FOURCC_RGB16)) {
dma2d->tx_cvt.pre_scramble = 5;
} else {
return false;
}
return true;
}
static bool check_ppa_supported(gmf_video_ppa_t *vid_cvt)
{
esp_gmf_info_video_t *src_info = &vid_cvt->parent.src_info;
if (src_info->format_id == vid_cvt->dst_format) {
return true;
}
if ((src_info->format_id == ESP_FOURCC_RGB16_BE && vid_cvt->dst_format == ESP_FOURCC_RGB16) ||
(src_info->format_id == ESP_FOURCC_RGB16 && vid_cvt->dst_format == ESP_FOURCC_RGB16_BE)) {
vid_cvt->ppa_config.byte_swap = 1;
return true;
}
if ((src_info->format_id == ESP_FOURCC_RGB16 && vid_cvt->dst_format == ESP_FOURCC_BGR24) ||
(src_info->format_id == ESP_FOURCC_OUYY_EVYY && vid_cvt->dst_format == ESP_FOURCC_RGB16) ||
(src_info->format_id == ESP_FOURCC_OUYY_EVYY && vid_cvt->dst_format == ESP_FOURCC_BGR24) ||
(src_info->format_id == ESP_FOURCC_RGB16 && vid_cvt->dst_format == ESP_FOURCC_OUYY_EVYY) ||
(src_info->format_id == ESP_FOURCC_RGB16_BE && vid_cvt->dst_format == ESP_FOURCC_OUYY_EVYY)) {
return true;
}
if ((src_info->format_id == ESP_FOURCC_RGB24 && vid_cvt->dst_format == ESP_FOURCC_RGB16) ||
(src_info->format_id == ESP_FOURCC_RGB16 && vid_cvt->dst_format == ESP_FOURCC_RGB24) ||
(src_info->format_id == ESP_FOURCC_RGB16 && vid_cvt->dst_format == ESP_FOURCC_OUYY_EVYY) ||
(src_info->format_id == ESP_FOURCC_RGB24 && vid_cvt->dst_format == ESP_FOURCC_BGR24) ||
(src_info->format_id == ESP_FOURCC_BGR24 && vid_cvt->dst_format == ESP_FOURCC_RGB24)) {
vid_cvt->ppa_config.rgb_swap = 1;
return true;
}
return false;
}
static bool check_rotation_angle(uint16_t degree, ppa_srm_rotation_angle_t *angle)
{
switch (degree) {
case 0:
*angle = PPA_SRM_ROTATION_ANGLE_0;
break;
case 90:
*angle = PPA_SRM_ROTATION_ANGLE_90;
break;
case 180:
*angle = PPA_SRM_ROTATION_ANGLE_180;
break;
case 270:
*angle = PPA_SRM_ROTATION_ANGLE_270;
break;
default:
return false;
}
return true;
}
static int open_ppa(gmf_video_ppa_t *vid_cvt)
{
esp_gmf_info_video_t *src_info = &vid_cvt->parent.src_info;
ppa_client_config_t ppa_client_config = {
.oper_type = PPA_OPERATION_SRM,
.max_pending_trans_num = 1,
};
ppa_register_client(&ppa_client_config, &vid_cvt->ppa_handle);
ESP_GMF_MEM_CHECK(TAG, vid_cvt->ppa_handle, return ESP_GMF_ERR_NOT_ENOUGH);
memset(&vid_cvt->ppa_config, 0, sizeof(ppa_srm_oper_config_t));
uint32_t in_block_w = src_info->width;
uint32_t in_block_h = src_info->height;
float scale_x, scale_y;
if (vid_cvt->crop_rgn.width) {
in_block_w = vid_cvt->crop_rgn.width;
in_block_h = vid_cvt->crop_rgn.height;
vid_cvt->ppa_config.in.block_offset_x = vid_cvt->crop_rgn.x;
vid_cvt->ppa_config.in.block_offset_y = vid_cvt->crop_rgn.y;
}
if (vid_cvt->rotate_degree == 0 || vid_cvt->rotate_degree == 180) {
scale_x = 1.0 * vid_cvt->dst_width / in_block_w;
scale_y = 1.0 * vid_cvt->dst_height / in_block_h;
} else {
scale_x = 1.0 * vid_cvt->dst_height / in_block_w;
scale_y = 1.0 * vid_cvt->dst_width / in_block_h;
}
vid_cvt->ppa_config.in.pic_w = src_info->width;
vid_cvt->ppa_config.in.pic_h = src_info->height;
vid_cvt->ppa_config.in.block_w = in_block_w;
vid_cvt->ppa_config.in.block_h = in_block_h;
vid_cvt->ppa_config.in.srm_cm = get_ppa_clr_mode(src_info->format_id);
vid_cvt->ppa_config.out.pic_w = vid_cvt->dst_width;
vid_cvt->ppa_config.out.pic_h = vid_cvt->dst_height;
vid_cvt->ppa_config.out.srm_cm = get_ppa_clr_mode(vid_cvt->dst_format);
vid_cvt->ppa_config.rgb_swap = 0;
vid_cvt->ppa_config.byte_swap = 0;
check_ppa_supported(vid_cvt);
vid_cvt->ppa_config.mode = PPA_TRANS_MODE_BLOCKING;
vid_cvt->ppa_config.scale_x = scale_x;
vid_cvt->ppa_config.scale_y = scale_y;
if (check_rotation_angle(vid_cvt->rotate_degree, &vid_cvt->ppa_config.rotation_angle) == false) {
return ESP_GMF_ERR_NOT_SUPPORT;
}
return ESP_GMF_ERR_OK;
}
static int close_ppa(gmf_video_ppa_t *vid_cvt)
{
if (vid_cvt->ppa_handle) {
ppa_unregister_client(vid_cvt->ppa_handle);
vid_cvt->ppa_handle = NULL;
}
return 0;
}
static int ppa_convert(gmf_video_ppa_t *vid_cvt, esp_gmf_payload_t *in_load, esp_gmf_payload_t *out_load)
{
if (vid_cvt->supported == false) {
return ESP_GMF_ERR_NOT_SUPPORT;
}
ESP_GMF_NULL_CHECK(TAG, vid_cvt->ppa_handle, return ESP_GMF_ERR_NOT_SUPPORT);
vid_cvt->ppa_config.in.buffer = in_load->buf;
vid_cvt->ppa_config.out.buffer = out_load->buf;
vid_cvt->ppa_config.out.buffer_size = out_load->buf_length;
int err = ppa_do_scale_rotate_mirror(vid_cvt->ppa_handle, &vid_cvt->ppa_config);
return err;
}
static bool need_ppa(gmf_video_ppa_t *vid_cvt)
{
esp_gmf_info_video_t *src_info = &vid_cvt->parent.src_info;
if (src_info->width != vid_cvt->dst_width || src_info->height != vid_cvt->dst_height) {
return true;
}
if (src_info->format_id == ESP_FOURCC_YUV420P ||
vid_cvt->dst_format == ESP_FOURCC_YUV420P ||
src_info->format_id == ESP_FOURCC_OUYY_EVYY ||
vid_cvt->dst_format == ESP_FOURCC_OUYY_EVYY) {
return true;
}
if (check_2ddma_supported(vid_cvt)) {
return false;
}
return true;
}
static bool dma2d_m2m_transaction_done_cb(dma2d_channel_handle_t dma2d_chan, dma2d_event_data_t *event_data, void *user_data)
{
bool need_yield = false;
dma2d_m2m_transaction_t *trans_config = (dma2d_m2m_transaction_t*) user_data;
dma2d_m2m_trans_config_t *m2m_trans_desc = &trans_config->m2m_trans_desc;
if (m2m_trans_desc->trans_eof_cb) {
need_yield |= m2m_trans_desc->trans_eof_cb(m2m_trans_desc->user_data);
}
return need_yield;
}
static bool dma2d_m2m_transaction_on_picked(uint32_t channel_num, const dma2d_trans_channel_info_t *dma2d_chans, void *user_config)
{
dma2d_m2m_transaction_t *trans_config = (dma2d_m2m_transaction_t*) user_config;
dma2d_m2m_trans_config_t *m2m_trans_desc = &trans_config->m2m_trans_desc;
// Get the required 2D-DMA channel handles
uint32_t dma_tx_chan_idx = 0;
uint32_t dma_rx_chan_idx = 1;
if (dma2d_chans[0].dir == DMA2D_CHANNEL_DIRECTION_RX) {
dma_tx_chan_idx = 1;
dma_rx_chan_idx = 0;
}
dma2d_channel_handle_t dma_tx_chan = dma2d_chans[dma_tx_chan_idx].chan;
dma2d_channel_handle_t dma_rx_chan = dma2d_chans[dma_rx_chan_idx].chan;
dma2d_trigger_t trig_periph = {
.periph = DMA2D_TRIG_PERIPH_M2M,
.periph_sel_id = SOC_DMA2D_TRIG_PERIPH_M2M_TX,
};
dma2d_connect(dma_tx_chan, &trig_periph);
trig_periph.periph_sel_id = SOC_DMA2D_TRIG_PERIPH_M2M_RX;
dma2d_connect(dma_rx_chan, &trig_periph);
dma2d_set_transfer_ability(dma_tx_chan, &m2m_trans_desc->transfer_ability);
dma2d_set_transfer_ability(dma_rx_chan, &m2m_trans_desc->transfer_ability);
if (m2m_trans_desc->tx_strategy_config) {
dma2d_apply_strategy(dma_tx_chan, m2m_trans_desc->tx_strategy_config);
}
if (m2m_trans_desc->rx_strategy_config) {
dma2d_apply_strategy(dma_rx_chan, m2m_trans_desc->rx_strategy_config);
}
if (m2m_trans_desc->tx_csc_config) {
dma2d_configure_color_space_conversion(dma_tx_chan, m2m_trans_desc->tx_csc_config);
}
if (m2m_trans_desc->rx_csc_config) {
dma2d_configure_color_space_conversion(dma_rx_chan, m2m_trans_desc->rx_csc_config);
}
dma2d_rx_event_callbacks_t dma_cbs = {
.on_recv_eof = dma2d_m2m_transaction_done_cb,
};
dma2d_register_rx_event_callbacks(dma_rx_chan, &dma_cbs, (void *)trans_config);
dma2d_set_desc_addr(dma_tx_chan, m2m_trans_desc->tx_desc_base_addr);
dma2d_set_desc_addr(dma_rx_chan, m2m_trans_desc->rx_desc_base_addr);
dma2d_start(dma_tx_chan);
dma2d_start(dma_rx_chan);
return false;
}
static bool IRAM_ATTR dma2d_m2m_suc_eof_event_cb(void *user_data)
{
BaseType_t xHigherPriorityTaskWoken = pdFALSE;
SemaphoreHandle_t sem = (SemaphoreHandle_t) user_data;
xSemaphoreGiveFromISR(sem, &xHigherPriorityTaskWoken);
return (xHigherPriorityTaskWoken == pdTRUE);
}
static void dma2d_link_dscr_init(dma2d_descriptor_t *dma2d, uint32_t *next, void *buf_ptr, uint32_t ha, uint32_t va,
uint32_t hb, uint32_t vb, uint32_t eof, uint32_t en_2d, uint32_t pbyte, uint32_t mod,
uint32_t bias_x, uint32_t bias_y)
{
dma2d->owner = DMA2D_DESCRIPTOR_BUFFER_OWNER_DMA;
dma2d->suc_eof = eof;
dma2d->dma2d_en = en_2d;
dma2d->err_eof = 0;
dma2d->hb_length = hb;
dma2d->vb_size = vb;
dma2d->pbyte = pbyte;
dma2d->ha_length = ha;
dma2d->va_size = va;
dma2d->mode = mod;
dma2d->y = bias_y;
dma2d->x = bias_x;
dma2d->buffer = buf_ptr;
dma2d->next = (dma2d_descriptor_t*) next;
}
static int open_dma2d(gmf_video_ppa_t *vid_cvt)
{
esp_gmf_info_video_t *src_info = &vid_cvt->parent.src_info;
dma2d_pool_config_t pool_config = {
.pool_id = 0,
};
dma2d_info_t *dma2d = &vid_cvt->dma2d_info;
int ret = dma2d_acquire_pool(&pool_config, &dma2d->handle);
if (ret != ESP_OK) {
ESP_LOGE(TAG, "Fail to allocate for DMA2D");
return ret;
}
uint8_t align = esp_gmf_oal_get_spiram_cache_align();
dma2d->tx_desc = (dma2d_descriptor_t *)heap_caps_aligned_calloc(align, 1, 64, MALLOC_CAP_SPIRAM);
dma2d->rx_desc = (dma2d_descriptor_t *)heap_caps_aligned_calloc(align, 1, 64, MALLOC_CAP_SPIRAM);
dma2d->sema = xSemaphoreCreateCounting(1, 0);
if (dma2d->sema == NULL || dma2d->rx_desc == NULL || dma2d->tx_desc == NULL) {
ESP_GMF_MEM_CHECK(TAG, NULL, return ESP_GMF_ERR_MEMORY_LACK);
}
dma2d->trans.dma_chan_desc.tx_channel_num = 1;
dma2d->trans.dma_chan_desc.rx_channel_num = 1;
dma2d->trans.dma_chan_desc.channel_flags = DMA2D_CHANNEL_FUNCTION_FLAG_SIBLING;
dma2d->trans.dma_chan_desc.specified_tx_channel_mask = 0;
dma2d->trans.dma_chan_desc.specified_rx_channel_mask = 0;
dma2d->trans.dma_chan_desc.user_config = (void*) &dma2d->trans;
dma2d->trans.dma_chan_desc.on_job_picked = dma2d_m2m_transaction_on_picked;
if ((src_info->format_id == ESP_FOURCC_RGB24 || src_info->format_id == ESP_FOURCC_BGR24) &&
(vid_cvt->dst_format == ESP_FOURCC_RGB16 || vid_cvt->dst_format == ESP_FOURCC_RGB16_BE)) {
dma2d->tx_cvt.tx_csc_option = DMA2D_CSC_TX_RGB888_TO_RGB565;
dma2d->tx_cvt.pre_scramble = DMA2D_SCRAMBLE_ORDER_BYTE0_1_2;
dma2d->trans.m2m_trans_desc.tx_csc_config = &dma2d->tx_cvt;
} else if ((src_info->format_id == ESP_FOURCC_RGB16_BE || src_info->format_id == ESP_FOURCC_RGB16) &&
(vid_cvt->dst_format == ESP_FOURCC_RGB24 || vid_cvt->dst_format == ESP_FOURCC_BGR24)) {
dma2d->tx_cvt.tx_csc_option = DMA2D_CSC_TX_RGB565_TO_RGB888;
dma2d->tx_cvt.pre_scramble = DMA2D_SCRAMBLE_ORDER_BYTE2_1_0;
dma2d->trans.m2m_trans_desc.tx_csc_config = &dma2d->tx_cvt;
}
check_2ddma_supported(vid_cvt);
if (dma2d->trans.m2m_trans_desc.tx_csc_config) {
dma2d->trans.dma_chan_desc.channel_flags |= DMA2D_CHANNEL_FUNCTION_FLAG_TX_CSC;
}
if (dma2d->trans.m2m_trans_desc.rx_csc_config) {
dma2d->trans.dma_chan_desc.channel_flags |= DMA2D_CHANNEL_FUNCTION_FLAG_RX_CSC;
}
// TODO support other color conversion
dma2d->trans.m2m_trans_desc.tx_desc_base_addr = (intptr_t) dma2d->tx_desc;
dma2d->trans.m2m_trans_desc.rx_desc_base_addr = (intptr_t) dma2d->rx_desc;
dma2d->trans.m2m_trans_desc.trans_eof_cb = dma2d_m2m_suc_eof_event_cb;
dma2d->trans.m2m_trans_desc.user_data = (void*) dma2d->sema;
dma2d_transfer_ability_t* trans_ability = &dma2d->trans.m2m_trans_desc.transfer_ability;
trans_ability->data_burst_length = DMA2D_DATA_BURST_LENGTH_128, trans_ability->desc_burst_en = true,
trans_ability->mb_size = DMA2D_MACRO_BLOCK_SIZE_NONE;
int src_size = get_frame_size(vid_cvt, src_info->format_id);
int dst_size = get_frame_size(vid_cvt, vid_cvt->dst_format);
dma2d_link_dscr_init(dma2d->tx_desc, NULL, NULL,
src_size >> 14, src_size >> 14,
src_size & 0x3FFF, src_size & 0x3FFF,
1, 0, DMA2D_DESCRIPTOR_PBYTE_1B0_PER_PIXEL,
DMA2D_DESCRIPTOR_BLOCK_RW_MODE_SINGLE, 0, 0);
dma2d_link_dscr_init(dma2d->rx_desc, NULL, NULL,
0, dst_size >> 14,
0, dst_size & 0x3FFF,
0, 0, DMA2D_DESCRIPTOR_PBYTE_1B0_PER_PIXEL,
DMA2D_DESCRIPTOR_BLOCK_RW_MODE_SINGLE, 0, 0);
return ESP_GMF_ERR_OK;
}
static void flush_src_dst_data(gmf_video_ppa_t *vid_cvt, esp_gmf_payload_t *in_load, esp_gmf_payload_t *out_load)
{
esp_cache_msync((void *)in_load->buf, in_load->buf_length, ESP_CACHE_MSYNC_FLAG_DIR_C2M);
esp_cache_msync((void *)out_load->buf, out_load->buf_length, ESP_CACHE_MSYNC_FLAG_DIR_C2M);
}
static void invalid_dst_data(gmf_video_ppa_t *vid_cvt, esp_gmf_payload_t *out_load)
{
esp_cache_msync(out_load->buf, out_load->buf_length, ESP_CACHE_MSYNC_FLAG_DIR_M2C);
}
static int dm2d_convert(gmf_video_ppa_t *vid_cvt, esp_gmf_payload_t *in_load, esp_gmf_payload_t *out_load)
{
if (vid_cvt->supported == false) {
return ESP_GMF_ERR_NOT_SUPPORT;
}
dma2d_info_t *dma2d = &vid_cvt->dma2d_info;
flush_src_dst_data(vid_cvt, in_load, out_load);
// Set buffer for TX, RX desc
dma2d->tx_desc->buffer = (void *)in_load->buf;
dma2d->rx_desc->buffer = (void *)out_load->buf;
esp_cache_msync(dma2d->tx_desc, 64, ESP_CACHE_MSYNC_FLAG_DIR_C2M | ESP_CACHE_MSYNC_FLAG_UNALIGNED);
esp_cache_msync(dma2d->rx_desc, 64, ESP_CACHE_MSYNC_FLAG_DIR_C2M | ESP_CACHE_MSYNC_FLAG_UNALIGNED);
int ret = dma2d_enqueue(dma2d->handle, &dma2d->trans.dma_chan_desc,
(dma2d_trans_t *)dma2d->trans.dma_trans_placeholder_head);
if (ret != ESP_OK) {
return ret;
}
// Wait for DMA transfer to complete
xSemaphoreTake(dma2d->sema, portMAX_DELAY);
invalid_dst_data(vid_cvt, out_load);
return 0;
}
static void close_dma2d(gmf_video_ppa_t *vid_cvt)
{
dma2d_info_t *dma2d = &vid_cvt->dma2d_info;
if (dma2d->handle) {
dma2d_release_pool(dma2d->handle);
dma2d->handle = NULL;
}
if (dma2d->sema) {
vSemaphoreDelete(dma2d->sema);
dma2d->sema = NULL;
}
if (dma2d->tx_desc) {
heap_caps_free(dma2d->tx_desc);
dma2d->tx_desc = NULL;
}
if (dma2d->rx_desc) {
heap_caps_free(dma2d->rx_desc);
dma2d->rx_desc = NULL;
}
}
#endif
static esp_gmf_job_err_t gmf_video_ppa_open(esp_gmf_element_handle_t self, void *para)
{
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)self;
esp_gmf_info_video_t *src_info = &vid_cvt->parent.src_info;
esp_gmf_info_video_t vid_info = vid_cvt->parent.src_info;
// Set Default info
if (vid_cvt->dst_width == 0) {
vid_cvt->dst_width = src_info->width;
if (vid_cvt->rotate_degree != 0 && vid_cvt->rotate_degree != 180) {
vid_cvt->dst_width = src_info->height;
}
}
if (vid_cvt->dst_height == 0) {
vid_cvt->dst_height = src_info->height;
if (vid_cvt->rotate_degree != 0 && vid_cvt->rotate_degree != 180) {
vid_cvt->dst_height = src_info->width;
}
}
if (vid_cvt->dst_format == 0) {
vid_cvt->dst_format = src_info->format_id;
}
if (vid_cvt->crop_rgn.width) {
if ((vid_cvt->crop_rgn.x + vid_cvt->crop_rgn.width > src_info->width) || (vid_cvt->crop_rgn.y + vid_cvt->crop_rgn.height > src_info->height)) {
ESP_LOGE(TAG, "Crop over limited w: %d + %d > %d h: %d + %d > %d",
vid_cvt->crop_rgn.x, vid_cvt->crop_rgn.width, src_info->width,
vid_cvt->crop_rgn.y, vid_cvt->crop_rgn.height, src_info->height);
return ESP_GMF_JOB_ERR_FAIL;
}
}
vid_cvt->bypass = false;
if (vid_cvt->dst_width == src_info->width && vid_cvt->dst_height == src_info->height && vid_cvt->dst_format == src_info->format_id) {
vid_cvt->bypass = true;
}
int ret = 0;
if (vid_cvt->bypass == false) {
#if CONFIG_IDF_TARGET_ESP32P4
vid_cvt->supported = check_ppa_supported(vid_cvt) || check_2ddma_supported(vid_cvt);
if (vid_cvt->supported == false) {
ESP_LOGE(TAG, "Not support convert from %s to %s",
esp_gmf_video_get_format_string(src_info->format_id), esp_gmf_video_get_format_string(vid_cvt->dst_format));
return ESP_GMF_ERR_NOT_SUPPORT;
}
// TODO decide to use DMA2D or PPA
vid_cvt->out_frame_size = get_frame_size(vid_cvt, vid_cvt->dst_format);
// Allocate memory for output frame
int cache_line_size = CONFIG_CACHE_L2_CACHE_LINE_SIZE;
ESP_GMF_ELEMENT_GET(vid_cvt)->out_attr.data_size = GMF_VIDEO_ALIGN_UP(vid_cvt->out_frame_size, cache_line_size);
vid_cvt->use_ppa = need_ppa(vid_cvt);
if (vid_cvt->use_ppa) {
ret = open_ppa(vid_cvt);
} else {
ret = open_dma2d(vid_cvt);
}
vid_info.format_id = vid_cvt->dst_format;
vid_info.width = vid_cvt->dst_width;
vid_info.height = vid_cvt->dst_height;
ESP_LOGI(TAG, "Convert in %s %dx%d to %s %dx%d ppa:%d",
esp_gmf_video_get_format_string(src_info->format_id), (int)src_info->width, (int)src_info->height,
esp_gmf_video_get_format_string(vid_info.format_id), (int)vid_info.width, (int)vid_info.height,
vid_cvt->use_ppa);
#else
ESP_LOGE(TAG, "Not support video convert except esp32p4");
return ESP_GMF_JOB_ERR_FAIL;
#endif
}
esp_gmf_element_notify_vid_info(self, &vid_info);
return ret;
}
static esp_gmf_job_err_t gmf_video_ppa_process(esp_gmf_element_handle_t self, void *para)
{
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)self;
int ret = 0;
esp_gmf_port_handle_t in_port = ESP_GMF_ELEMENT_GET(self)->in;
esp_gmf_port_handle_t out_port = ESP_GMF_ELEMENT_GET(self)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
ret = esp_gmf_port_acquire_in(in_port, &in_load, ESP_GMF_ELEMENT_GET(self)->in_attr.data_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Read data error, ret:%d, line:%d", ret, __LINE__);
return ret == ESP_GMF_IO_ABORT ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
if (in_load->is_done && in_load->valid_size == 0) {
esp_gmf_port_release_in(in_port, in_load, 0);
return ESP_GMF_JOB_ERR_DONE;
}
uint32_t wanted_size = 0;
if (vid_cvt->bypass) {
out_load = in_load;
wanted_size = in_load->valid_size;
} else {
out_load = NULL;
wanted_size = ESP_GMF_ELEMENT_GET(vid_cvt)->out_attr.data_size;
}
ret = esp_gmf_port_acquire_out(out_port, &out_load, wanted_size, ESP_GMF_MAX_DELAY);
if (ret < 0) {
ESP_LOGE(TAG, "Write data error, ret:%d, line:%d", ret, __LINE__);
return ret == ESP_GMF_IO_ABORT ? ESP_GMF_JOB_ERR_OK : ESP_GMF_JOB_ERR_FAIL;
}
if (vid_cvt->bypass == false) {
#if CONFIG_IDF_TARGET_ESP32P4
if (vid_cvt->use_ppa) {
ret = ppa_convert(vid_cvt, in_load, out_load);
} else {
ret = dm2d_convert(vid_cvt, in_load, out_load);
}
if (ret == 0) {
out_load->valid_size = vid_cvt->out_frame_size;
out_load->pts = in_load->pts;
}
#else
ret = ESP_GMF_JOB_ERR_FAIL;
#endif
}
esp_gmf_port_release_out(out_port, out_load, ESP_GMF_MAX_DELAY);
esp_gmf_port_release_in(in_port, in_load, ESP_GMF_MAX_DELAY);
return ret;
}
static esp_gmf_job_err_t gmf_video_ppa_close(esp_gmf_element_handle_t self, void *para)
{
#if CONFIG_IDF_TARGET_ESP32P4
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)self;
if (vid_cvt->use_ppa) {
close_ppa(vid_cvt);
} else {
close_dma2d(vid_cvt);
}
#endif
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_err_t gmf_video_ppa_destroy(esp_gmf_element_handle_t self)
{
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)self;
esp_gmf_video_el_deinit(self);
if (vid_cvt != NULL) {
esp_gmf_oal_free(vid_cvt);
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t set_dst_format(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)handle;
vid_cvt->dst_format = *(uint32_t *)buf;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t set_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)handle;
uint16_t *v = (uint16_t *)buf;
vid_cvt->dst_width = *(v++);
vid_cvt->dst_height = *v;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t set_rotation(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)handle;
uint16_t *v = (uint16_t *)buf;
vid_cvt->rotate_degree = *v;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t set_crop(esp_gmf_element_handle_t handle, esp_gmf_args_desc_t *arg_desc,
uint8_t *buf, int buf_len)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, arg_desc, return ESP_GMF_ERR_INVALID_ARG);
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)handle;
uint16_t *v = (uint16_t *)buf;
vid_cvt->crop_rgn.x = *(v++);
vid_cvt->crop_rgn.y = *(v++);
vid_cvt->crop_rgn.width = *(v++);
vid_cvt->crop_rgn.height = *(v++);
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t gmf_video_ppa_new(void *cfg, esp_gmf_obj_handle_t *handle)
{
return esp_gmf_video_ppa_init(cfg, (esp_gmf_element_handle_t*)handle);
}
static esp_gmf_err_t gmf_video_ppa_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_args_desc_t *set_args = NULL;
esp_gmf_method_t *methods = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
do {
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CLR_CVT, SET_DST_FMT, FMT), ESP_GMF_ARGS_TYPE_UINT32, sizeof(uint32_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(CLR_CVT, SET_DST_FMT), set_dst_format, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(SCALER, SET_DST_RES, WIDTH), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(SCALER, SET_DST_RES, HEIGHT), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), sizeof(uint16_t));
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(SCALER, SET_DST_RES), set_dst_resolution, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(ROTATOR, SET_ANGLE, DEGREE), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(ROTATOR, SET_ANGLE), set_rotation, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CROP, SET_CROP_RGN, X), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 0);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CROP, SET_CROP_RGN, Y), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 2);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CROP, SET_CROP_RGN, WIDTH), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 4);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(CROP, SET_CROP_RGN, HEIGHT), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 6);
GMF_VIDEO_BREAK_ON_FAIL(ret);
ret = esp_gmf_method_append(&methods, VMETHOD(CROP, SET_CROP_RGN), set_crop, set_args);
GMF_VIDEO_BREAK_ON_FAIL(ret);
((esp_gmf_element_t *) handle)->method = methods;
return ESP_GMF_ERR_OK;
} while (0);
ESP_LOGE(TAG, "Fail to load methods");
if (set_args) {
esp_gmf_args_desc_destroy(set_args);
}
if (methods) {
esp_gmf_method_destroy(methods);
}
return ESP_GMF_ERR_MEMORY_LACK;
}
static esp_gmf_err_t gmf_video_ppa_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t cap = {0};
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
do {
cap.cap_eightcc = ESP_GMF_CAPS_VIDEO_COLOR_CONVERT;
cap.attr_fun = NULL;
ret = esp_gmf_cap_append(&caps, &cap);
GMF_VIDEO_BREAK_ON_FAIL(ret);
cap.cap_eightcc = ESP_GMF_CAPS_VIDEO_SCALE;
ret = esp_gmf_cap_append(&caps, &cap);
GMF_VIDEO_BREAK_ON_FAIL(ret);
cap.cap_eightcc = ESP_GMF_CAPS_VIDEO_CROP;
ret = esp_gmf_cap_append(&caps, &cap);
GMF_VIDEO_BREAK_ON_FAIL(ret);
cap.cap_eightcc = ESP_GMF_CAPS_VIDEO_ROTATE;
ret = esp_gmf_cap_append(&caps, &cap);
GMF_VIDEO_BREAK_ON_FAIL(ret);
((esp_gmf_element_t *) handle)->caps = caps;
return ret;
} while (0);
if (caps) {
esp_gmf_cap_destroy(caps);
}
return ret;
}
esp_gmf_err_t esp_gmf_video_ppa_init(void *config, esp_gmf_element_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
gmf_video_ppa_t *video_cvt = esp_gmf_oal_calloc(1, sizeof(gmf_video_ppa_t));
ESP_GMF_MEM_CHECK(TAG, video_cvt, return ESP_GMF_ERR_MEMORY_LACK);
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_cvt;
obj->new_obj = gmf_video_ppa_new;
obj->del_obj = gmf_video_ppa_destroy;
esp_gmf_err_t ret = esp_gmf_obj_set_tag(obj, "vid_ppa");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto _cvt_init_fail, "Failed set OBJ tag");
uint8_t align = esp_gmf_oal_get_spiram_cache_align();
esp_gmf_element_cfg_t el_cfg = {
.dependency = true,
};
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_SINGLE, align, align,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, align, align,
ESP_GMF_PORT_TYPE_BLOCK | ESP_GMF_PORT_TYPE_BYTE, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ret = esp_gmf_video_el_init(obj, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto _cvt_init_fail, "Failed to init video convert");
// Bind API
gmf_video_ppa_t *rate_cvt = (gmf_video_ppa_t *)obj;
rate_cvt->parent.base.ops.open = gmf_video_ppa_open;
rate_cvt->parent.base.ops.process = gmf_video_ppa_process;
rate_cvt->parent.base.ops.close = gmf_video_ppa_close;
rate_cvt->parent.base.ops.event_receiver = esp_gmf_video_handle_events;
rate_cvt->parent.base.ops.load_methods = gmf_video_ppa_load_methods;
rate_cvt->parent.base.ops.load_caps = gmf_video_ppa_load_caps;
*handle = (esp_gmf_element_handle_t) rate_cvt;
return ESP_GMF_ERR_OK;
_cvt_init_fail:
esp_gmf_obj_delete(obj);
return ret;
}
esp_gmf_err_t esp_gmf_video_ppa_set_dst_format(esp_gmf_element_handle_t handle, uint32_t codec)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method((esp_gmf_element_handle_t)handle, &method_head);
esp_gmf_method_found(method_head, VMETHOD(CLR_CVT, SET_DST_FMT), &method);
ESP_GMF_NULL_CHECK(TAG, method_head, return ESP_GMF_ERR_NOT_SUPPORT);
uint8_t buf[4] = { 0 };
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(CLR_CVT, SET_DST_FMT, FMT), buf, (uint8_t *)&codec, sizeof(uint32_t));
return esp_gmf_element_exe_method((esp_gmf_element_handle_t)handle, VMETHOD(CLR_CVT, SET_DST_FMT), buf, sizeof(buf));
}
esp_gmf_err_t esp_gmf_video_ppa_set_cropped_rgn(esp_gmf_element_handle_t handle, esp_gmf_video_rgn_t *rgn)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method((esp_gmf_element_handle_t)handle, &method_head);
esp_gmf_method_found(method_head, VMETHOD(CROP, SET_CROP_RGN), &method);
ESP_GMF_NULL_CHECK(TAG, method_head, return ESP_GMF_ERR_NOT_SUPPORT);
uint8_t buf[8];
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(CROP, SET_CROP_RGN, X), buf, (uint8_t *)&rgn->x, sizeof(uint16_t));
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(CROP, SET_CROP_RGN, Y), buf, (uint8_t *)&rgn->y, sizeof(uint16_t));
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(CROP, SET_CROP_RGN, WIDTH), buf, (uint8_t *)&rgn->width, sizeof(uint16_t));
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(CROP, SET_CROP_RGN, HEIGHT), buf, (uint8_t *)&rgn->height, sizeof(uint16_t));
return esp_gmf_element_exe_method((esp_gmf_element_handle_t)handle, VMETHOD(CROP, SET_CROP_RGN), buf, sizeof(buf));
}
esp_gmf_err_t esp_gmf_video_ppa_set_rotation(esp_gmf_element_handle_t handle, uint16_t degree)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method((esp_gmf_element_handle_t)handle, &method_head);
esp_gmf_method_found(method_head, VMETHOD(ROTATOR, SET_ANGLE), &method);
ESP_GMF_NULL_CHECK(TAG, method_head, return ESP_GMF_ERR_NOT_SUPPORT);
uint8_t buf[2] = { 0 };
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(ROTATOR, SET_ANGLE, DEGREE), buf, (uint8_t *)&degree, sizeof(uint16_t));
return esp_gmf_element_exe_method((esp_gmf_element_handle_t)handle, VMETHOD(ROTATOR, SET_ANGLE), buf, sizeof(buf));
}
esp_gmf_err_t esp_gmf_video_ppa_set_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_video_resolution_t *res)
{
ESP_GMF_NULL_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, res, return ESP_GMF_ERR_INVALID_ARG);
const esp_gmf_method_t *method_head = NULL;
const esp_gmf_method_t *method = NULL;
esp_gmf_element_get_method((esp_gmf_element_handle_t)handle, &method_head);
esp_gmf_method_found(method_head, VMETHOD(SCALER, SET_DST_RES), &method);
ESP_GMF_NULL_CHECK(TAG, method_head, return ESP_GMF_ERR_NOT_SUPPORT);
uint8_t buf[4] = { 0 };
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(SCALER, SET_DST_RES, WIDTH), buf,
(uint8_t *)&res->width, sizeof(uint16_t));
esp_gmf_args_set_value(method->args_desc, VMETHOD_ARG(SCALER, SET_DST_RES, HEIGHT),
buf, (uint8_t *)&res->height, sizeof(uint16_t));
return esp_gmf_element_exe_method((esp_gmf_element_handle_t)handle, VMETHOD(SCALER, SET_DST_RES), buf, sizeof(buf));
}
/**
* @brief This API is for debug only
*/
int gmf_video_ppa_test(uint32_t from_codec, int32_t to_codec, uint32_t width, uint32_t height, uint8_t *src, uint8_t *dst, int v)
{
esp_gmf_obj_handle_t cvt = NULL;
gmf_video_ppa_new(NULL, &cvt);
gmf_video_ppa_t *vid_cvt = (gmf_video_ppa_t *)cvt;
esp_gmf_info_video_t vid_info = {
.format_id = from_codec,
.width = width,
.height = height,
};
vid_cvt->parent.src_info = vid_info;
vid_cvt->dst_format = to_codec;
vid_cvt->dst_width = width;
vid_cvt->dst_height = height;
gmf_video_ppa_open(cvt, NULL);
int ret = 0;
#if CONFIG_IDF_TARGET_ESP32P4
esp_gmf_payload_t in_load = {
.buf = src,
.buf_length = get_frame_size(vid_cvt, from_codec),
};
esp_gmf_payload_t out_load = {
.buf = dst,
.buf_length = get_frame_size(vid_cvt, to_codec),
};
if (v != -1) {
vid_cvt->ppa_config.rgb_swap = v & 0x1;
vid_cvt->ppa_config.byte_swap = v & 0x2;
dma2d_info_t *dma2d = &vid_cvt->dma2d_info;
dma2d->tx_cvt.pre_scramble = v;
}
ESP_LOGI(TAG, "RGB swap:%d byteswap:%d scramble:%d", vid_cvt->ppa_config.rgb_swap, vid_cvt->ppa_config.byte_swap, v);
if (vid_cvt->use_ppa) {
ret = ppa_convert(vid_cvt, &in_load, &out_load);
} else {
ret = dm2d_convert(vid_cvt, &in_load, &out_load);
}
#endif
gmf_video_ppa_close(cvt, NULL);
gmf_video_ppa_destroy(cvt);
return ret;
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_rotate.c
+316
View File
@@ -0,0 +1,316 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include <string.h>
#include <inttypes.h>
#include "esp_log.h"
#include "esp_gmf_oal_mutex.h"
#include "esp_gmf_oal_mem.h"
#include "esp_gmf_err.h"
#include "esp_gmf_node.h"
#include "esp_gmf_video_rotate.h"
#include "esp_gmf_video_methods_def.h"
#include "esp_gmf_video_element.h"
#include "gmf_video_common.h"
#include "esp_gmf_caps_def.h"
static const char *TAG = "IMGFX_ROTATE_EL";
typedef struct _gmf_imgfx_rotate_t {
esp_gmf_video_element_t parent;
esp_imgfx_rotate_handle_t hd;
bool need_recfg;
} esp_gmf_rotate_hd_t;
static esp_gmf_job_err_t video_rotate_el_open(esp_gmf_element_handle_t self, void *para)
{
esp_gmf_rotate_hd_t *video_el = (esp_gmf_rotate_hd_t *)self;
// Get and check config
esp_imgfx_rotate_cfg_t *cfg = (esp_imgfx_rotate_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_CHECK(TAG, cfg, return ESP_GMF_JOB_ERR_FAIL, "Failed to get rotate config");
// Open rotate module
esp_imgfx_rotate_open(cfg, &video_el->hd);
ESP_GMF_CHECK(TAG, video_el->hd, return ESP_GMF_JOB_ERR_FAIL, "Failed to create rotate handle");
// Get video size
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->in_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size));
esp_imgfx_resolution_t res;
esp_imgfx_rotate_get_rotated_resolution(video_el->hd, &res);
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size));
// Report information to the next element, the next element will use this information to configure
gmf_video_update_info(self, res.width, res.height, cfg->in_pixel_fmt);
// The video_el->hd has been opened using newest configuration, so it can set the need_recfg to false
video_el->need_recfg = false;
ESP_LOGD(TAG, "Open, %p", self);
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_job_err_t video_rotate_el_process(esp_gmf_element_handle_t self, void *para)
{
esp_gmf_rotate_hd_t *video_el = (esp_gmf_rotate_hd_t *)self;
esp_imgfx_data_t in_image;
esp_imgfx_data_t out_image;
esp_imgfx_rotate_cfg_t *cfg = (esp_imgfx_rotate_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_MEM_CHECK(TAG, cfg, return ESP_GMF_JOB_ERR_FAIL);
bool bypass = cfg->degree % 360 == 0;
if (video_el->need_recfg) {
// Reset rotate configuration
esp_imgfx_err_t imgfx_ret = esp_imgfx_rotate_set_cfg(video_el->hd, cfg);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
return ESP_GMF_JOB_ERR_FAIL;
}
// Get video size
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->in_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size));
esp_imgfx_resolution_t res;
esp_imgfx_rotate_get_rotated_resolution(video_el->hd, &res);
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size));
video_el->need_recfg = false;
}
esp_gmf_port_handle_t in_port = ESP_GMF_ELEMENT_GET(self)->in;
esp_gmf_port_handle_t out_port = ESP_GMF_ELEMENT_GET(self)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
esp_gmf_job_err_t ret = ESP_GMF_JOB_ERR_OK;
esp_gmf_err_io_t load_ret = ESP_GMF_IO_OK;
ret = video_el_acquire_payload(in_port, out_port, &in_load, &out_load, (ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size),
(ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size), bypass);
if (ret != ESP_GMF_JOB_ERR_OK) {
goto __release;
}
/* It is for bypass */
if (in_load == out_load) {
goto __release;
}
if (in_load->is_done) {
out_load->is_done = in_load->is_done;
out_load->pts = in_load->pts;
ret = ESP_GMF_JOB_ERR_DONE;
ESP_LOGD(TAG, "It's done, out: %d", in_load->valid_size);
goto __release;
}
in_image.data = in_load->buf;
in_image.data_len = in_load->valid_size;
out_image.data = out_load->buf;
out_image.data_len = out_load->buf_length;
esp_imgfx_err_t imgfx_ret = esp_imgfx_rotate_process(video_el->hd, &in_image, &out_image);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Image effects rotate process failed, ret: %d-%p", imgfx_ret, video_el);
ret = ESP_GMF_JOB_ERR_FAIL;
goto __release;
}
// Copy the information to the output payload. The next element will use this information to process
out_load->is_done = in_load->is_done;
out_load->valid_size = (ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size);
out_load->pts = in_load->pts;
__release:
// Release in and out port
if (out_load != NULL) {
load_ret = esp_gmf_port_release_out(out_port, out_load, ESP_GMF_MAX_DELAY);
if ((load_ret < ESP_GMF_IO_OK) && (load_ret != ESP_GMF_IO_ABORT)) {
ESP_LOGE(TAG, "OUT port release error, ret:%d", load_ret);
ret = ESP_GMF_JOB_ERR_FAIL;
}
}
if (in_load != NULL) {
load_ret = esp_gmf_port_release_in(in_port, in_load, ESP_GMF_MAX_DELAY);
if ((load_ret < ESP_GMF_IO_OK) && (load_ret != ESP_GMF_IO_ABORT)) {
ESP_LOGE(TAG, "IN port release error, ret:%d", load_ret);
ret = ESP_GMF_JOB_ERR_FAIL;
}
}
return ret;
}
static esp_gmf_job_err_t video_rotate_el_close(esp_gmf_element_handle_t self, void *para)
{
ESP_LOGD(TAG, "Closed, %p", self);
esp_gmf_rotate_hd_t *video_el = (esp_gmf_rotate_hd_t *)self;
if (video_el) {
if (video_el->hd) {
esp_imgfx_rotate_close(video_el->hd);
video_el->hd = NULL;
}
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_rotate_el_delete(esp_gmf_obj_handle_t handle)
{
ESP_LOGD(TAG, "Deleted, %p", handle);
void *cfg = OBJ_GET_CFG(handle);
if (cfg) {
esp_gmf_oal_free(cfg);
}
esp_gmf_video_el_deinit(handle);
esp_gmf_oal_free(handle);
return ESP_GMF_ERR_OK;
}
static inline esp_gmf_err_t esp_gmf_video_rotate_set_cfg(esp_gmf_element_handle_t self, esp_imgfx_rotate_cfg_t *config)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_JOB_ERR_FAIL);
ESP_GMF_NULL_CHECK(TAG, config, return ESP_GMF_JOB_ERR_FAIL);
esp_gmf_rotate_hd_t *video_el = (esp_gmf_rotate_hd_t *)self;
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// reset obj config
if (obj->cfg) {
memcpy(obj->cfg, config, sizeof(esp_imgfx_rotate_cfg_t));
video_el->need_recfg = true;
return ESP_GMF_JOB_ERR_OK;
}
ESP_LOGE(TAG, " %s-%p is no configured yet", __func__, self);
return ESP_GMF_JOB_ERR_FAIL;
}
static inline esp_gmf_err_t esp_gmf_video_rotate_get_cfg(esp_gmf_element_handle_t self, esp_imgfx_rotate_cfg_t *config)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_JOB_ERR_FAIL);
esp_gmf_rotate_hd_t *video_el = (esp_gmf_rotate_hd_t *)self;
// Get cfg from rotate handle if handle valid or-else get from object
if (video_el->hd) {
esp_imgfx_err_t imgfx_ret = esp_imgfx_rotate_get_cfg(video_el->hd, config);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Get video effects rotate cfg failed, hd:%p, ret: %d", self, imgfx_ret);
return ESP_GMF_JOB_ERR_FAIL;
}
return ESP_GMF_JOB_ERR_OK;
} else {
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// Get obj config
if (obj->cfg) {
memcpy(config, obj->cfg, sizeof(esp_imgfx_rotate_cfg_t));
return ESP_GMF_JOB_ERR_OK;
}
}
ESP_LOGE(TAG, " %s-%p is no configured yet", __func__, self);
return ESP_GMF_JOB_ERR_FAIL;
}
static esp_gmf_err_t video_set_dst_degree(void *handle, esp_gmf_args_desc_t *arg_desc, uint8_t *buf, int buf_len)
{
ESP_GMF_MEM_CHECK(TAG, buf, return ESP_GMF_ERR_INVALID_ARG);
return esp_gmf_video_rotate_set_rotation(handle, *(uint16_t *)buf);
}
static esp_gmf_err_t video_rotate_el_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t dec_caps = {0};
dec_caps.cap_eightcc = ESP_GMF_CAPS_VIDEO_ROTATE;
dec_caps.attr_fun = NULL;
int ret = esp_gmf_cap_append(&caps, &dec_caps);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to create capability");
esp_gmf_element_t *el = (esp_gmf_element_t *)handle;
el->caps = caps;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_rotate_el_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_method_t *method = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_args_desc_t *set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(ROTATOR, SET_ANGLE, DEGREE), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), 0);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append rotation degree");
ret = esp_gmf_method_append(&method, VMETHOD(ROTATOR, SET_ANGLE), video_set_dst_degree, set_args);
ESP_GMF_RET_ON_ERROR(TAG, ret, return ret, "Failed to register %s method", VMETHOD(ROTATOR, SET_ANGLE));
esp_gmf_element_t *el = (esp_gmf_element_t *)handle;
el->method = method;
return ret;
}
static esp_gmf_err_t video_rotate_el_received_event_handler(esp_gmf_event_pkt_t *evt, void *ctx)
{
ESP_GMF_NULL_CHECK(TAG, ctx, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, evt, return ESP_GMF_ERR_INVALID_ARG);
if ((evt->type != ESP_GMF_EVT_TYPE_REPORT_INFO)
|| (evt->sub != ESP_GMF_INFO_VIDEO)
|| (evt->payload == NULL)) {
return ESP_GMF_ERR_OK;
}
esp_gmf_element_handle_t self = (esp_gmf_element_handle_t)ctx;
esp_gmf_element_handle_t el = evt->from;
esp_gmf_event_state_t state = ESP_GMF_EVENT_STATE_NONE;
esp_gmf_element_get_state(self, &state);
esp_gmf_info_video_t *info = (esp_gmf_info_video_t *)evt->payload;
esp_gmf_video_el_set_src_info(self, info);
esp_imgfx_rotate_cfg_t *config = (esp_imgfx_rotate_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_NULL_CHECK(TAG, config, return ESP_GMF_ERR_FAIL);
esp_gmf_rotate_hd_t *video_el = (esp_gmf_rotate_hd_t *)self;
GMF_VIDEO_UPDATE_CONFIG(config, info, video_el->need_recfg);
ESP_LOGD(TAG, "RECV element info, from: %s-%p, next: %p, self: %s-%p, type: %x, state: %s, width: %d, height: %d, pixel format: %lx",
OBJ_GET_TAG(el), el, esp_gmf_node_for_next((esp_gmf_node_t *)el), OBJ_GET_TAG(self), self, evt->type,
esp_gmf_event_get_state_str(state), info->width, info->height, info->format_id);
// It is for the first time to receive event from previous element
if (state == ESP_GMF_EVENT_STATE_NONE) {
esp_gmf_element_set_state(self, ESP_GMF_EVENT_STATE_INITIALIZED);
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_rotate_el_new(void *config, esp_gmf_obj_handle_t *handle)
{
return esp_gmf_video_rotate_init((esp_imgfx_rotate_cfg_t *)config, handle);
}
esp_gmf_err_t esp_gmf_video_rotate_init(esp_imgfx_rotate_cfg_t *config, esp_gmf_obj_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
*handle = NULL;
esp_gmf_rotate_hd_t *video_el = esp_gmf_oal_calloc(1, sizeof(esp_gmf_rotate_hd_t));
ESP_GMF_MEM_CHECK(TAG, video_el, return ESP_GMF_ERR_MEMORY_LACK);
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// Set element tag
ret = esp_gmf_obj_set_tag(video_el, "vid_rotate");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto __init_exit, "Failed set OBJ tag");
// Configure object callbacks
obj->new_obj = video_rotate_el_new;
obj->del_obj = video_rotate_el_delete;
esp_imgfx_rotate_cfg_t *cfg = esp_gmf_oal_calloc(1, sizeof(esp_imgfx_rotate_cfg_t));
ESP_GMF_MEM_VERIFY(TAG, cfg, {ret = ESP_GMF_ERR_MEMORY_LACK; goto __init_exit;}, "rotate configuration", sizeof(esp_imgfx_rotate_cfg_t));
esp_gmf_obj_set_config(obj, cfg, sizeof(esp_imgfx_rotate_cfg_t));
if (config) {
memcpy(cfg, config, sizeof(esp_imgfx_rotate_cfg_t));
} else {
esp_imgfx_rotate_cfg_t dcfg = DEFAULT_ESP_GMF_ROTATE_CONFIG();
memcpy(cfg, &dcfg, sizeof(esp_imgfx_rotate_cfg_t));
}
// Configure element
esp_gmf_element_cfg_t el_cfg = {
.dependency = true,
};
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
// Initialize element
ret = esp_gmf_video_el_init(video_el, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto __init_exit, "Failed initialize video element");
// Register element's callbacks
esp_gmf_video_element_t *parent = (esp_gmf_video_element_t *)video_el;
parent->base.ops.open = video_rotate_el_open;
parent->base.ops.process = video_rotate_el_process;
parent->base.ops.close = video_rotate_el_close;
parent->base.ops.event_receiver = video_rotate_el_received_event_handler;
parent->base.ops.load_caps = video_rotate_el_load_caps;
parent->base.ops.load_methods = video_rotate_el_load_methods;
*handle = (esp_gmf_obj_handle_t)video_el;
ESP_LOGD(TAG, "Initialization, %s-%p", OBJ_GET_TAG(obj), obj);
return ret;
__init_exit:
video_rotate_el_delete(video_el);
return ret;
}
esp_gmf_err_t esp_gmf_video_rotate_set_rotation(esp_gmf_element_handle_t handle, uint16_t degree)
{
esp_imgfx_rotate_cfg_t config;
esp_gmf_err_t ret = esp_gmf_video_rotate_get_cfg(handle, &config);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to set destination degree");
config.degree = degree;
return esp_gmf_video_rotate_set_cfg(handle, &config);
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/esp_gmf_video_scale.c
+317
View File
@@ -0,0 +1,317 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include <string.h>
#include <inttypes.h>
#include "esp_log.h"
#include "esp_gmf_oal_mutex.h"
#include "esp_gmf_oal_mem.h"
#include "esp_gmf_err.h"
#include "esp_gmf_node.h"
#include "esp_gmf_video_scale.h"
#include "esp_gmf_video_methods_def.h"
#include "gmf_video_common.h"
#include "esp_gmf_video_element.h"
#include "esp_gmf_video_types.h"
#include "esp_gmf_caps_def.h"
static const char *TAG = "IMGFX_SCALE_EL";
typedef struct _gmf_imgfx_scale_t {
esp_gmf_video_element_t parent;
esp_imgfx_scale_handle_t hd;
bool need_recfg;
} esp_gmf_scale_hd_t;
static esp_gmf_job_err_t video_scale_el_open(esp_gmf_element_handle_t self, void *para)
{
esp_gmf_scale_hd_t *video_el = (esp_gmf_scale_hd_t *)self;
// Get and check config
esp_imgfx_scale_cfg_t *cfg = (esp_imgfx_scale_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_CHECK(TAG, cfg, return ESP_GMF_JOB_ERR_FAIL, "Failed to get scale config");
// Open scale module
esp_imgfx_scale_open(cfg, &video_el->hd);
ESP_GMF_CHECK(TAG, video_el->hd, return ESP_GMF_JOB_ERR_FAIL, "Failed to create scale handle");
// Get video size
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->in_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size));
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->scale_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size));
// Report information to the next element, the next element will use this information to configure
gmf_video_update_info(self, cfg->scale_res.width, cfg->scale_res.height, cfg->in_pixel_fmt);
// The video_el->hd has been opened using newest configuration, so it can set the need_recfg to false
video_el->need_recfg = false;
ESP_LOGD(TAG, "Open, %p", self);
return ESP_GMF_JOB_ERR_OK;
}
static esp_gmf_job_err_t video_scale_el_process(esp_gmf_element_handle_t self, void *para)
{
esp_gmf_scale_hd_t *video_el = (esp_gmf_scale_hd_t *)self;
esp_imgfx_data_t in_image;
esp_imgfx_data_t out_image;
if (video_el->need_recfg) {
esp_imgfx_scale_cfg_t *cfg = (esp_imgfx_scale_cfg_t *)OBJ_GET_CFG(self);
// Reset scale configuration
esp_imgfx_err_t imgfx_ret = esp_imgfx_scale_set_cfg(video_el->hd, cfg);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Image effects color convert set cfg failed, ret: %d", imgfx_ret);
return ESP_GMF_JOB_ERR_FAIL;
}
// Get video size
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->in_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size));
esp_imgfx_get_image_size(cfg->in_pixel_fmt, &cfg->scale_res, (uint32_t *)&(ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size));
video_el->need_recfg = false;
}
esp_gmf_port_handle_t in_port = ESP_GMF_ELEMENT_GET(self)->in;
esp_gmf_port_handle_t out_port = ESP_GMF_ELEMENT_GET(self)->out;
esp_gmf_payload_t *in_load = NULL;
esp_gmf_payload_t *out_load = NULL;
esp_gmf_job_err_t ret = ESP_GMF_JOB_ERR_OK;
esp_gmf_err_io_t load_ret = ESP_GMF_IO_OK;
ret = video_el_acquire_payload(in_port, out_port, &in_load, &out_load, (ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size), (ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size),
((ESP_GMF_ELEMENT_GET(video_el)->in_attr.data_size) == (ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size)));
if (ret != ESP_GMF_JOB_ERR_OK) {
goto __release;
}
/* It is for bypass */
if (in_load == out_load) {
goto __release;
}
/* finished */
if (in_load->is_done) {
out_load->is_done = in_load->is_done;
out_load->pts = in_load->pts;
ret = ESP_GMF_JOB_ERR_DONE;
ESP_LOGD(TAG, "It's done, out: %d", in_load->valid_size);
goto __release;
}
in_image.data = in_load->buf;
in_image.data_len = in_load->valid_size;
out_image.data = out_load->buf;
out_image.data_len = out_load->buf_length;
esp_imgfx_err_t imgfx_ret = esp_imgfx_scale_process(video_el->hd, &in_image, &out_image);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Image effects scale process failed, ret: %d-%p", imgfx_ret, video_el);
ret = ESP_GMF_JOB_ERR_FAIL;
goto __release;
}
// Copy the information to the output payload. The next element will use this information to process
out_load->is_done = in_load->is_done;
out_load->valid_size = (ESP_GMF_ELEMENT_GET(video_el)->out_attr.data_size);
out_load->pts = in_load->pts;
__release:
// Release in and out port
if (out_load != NULL) {
load_ret = esp_gmf_port_release_out(out_port, out_load, ESP_GMF_MAX_DELAY);
if ((load_ret < ESP_GMF_IO_OK) && (load_ret != ESP_GMF_IO_ABORT)) {
ESP_LOGE(TAG, "OUT port release error, ret:%d", load_ret);
ret = ESP_GMF_JOB_ERR_FAIL;
}
}
if (in_load != NULL) {
load_ret = esp_gmf_port_release_in(in_port, in_load, ESP_GMF_MAX_DELAY);
if ((load_ret < ESP_GMF_IO_OK) && (load_ret != ESP_GMF_IO_ABORT)) {
ESP_LOGE(TAG, "IN port release error, ret:%d", load_ret);
ret = ESP_GMF_JOB_ERR_FAIL;
}
}
return ret;
}
static esp_gmf_job_err_t video_scale_el_close(esp_gmf_element_handle_t self, void *para)
{
ESP_LOGD(TAG, "Closed, %p", self);
esp_gmf_scale_hd_t *video_el = (esp_gmf_scale_hd_t *)self;
if (video_el) {
if (video_el->hd) {
esp_imgfx_scale_close(video_el->hd);
video_el->hd = NULL;
}
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_scale_el_delete(esp_gmf_obj_handle_t handle)
{
ESP_LOGD(TAG, "Deleted, %p", handle);
void *cfg = OBJ_GET_CFG(handle);
if (cfg) {
esp_gmf_oal_free(cfg);
}
esp_gmf_video_el_deinit(handle);
esp_gmf_oal_free(handle);
return ESP_GMF_ERR_OK;
}
static inline esp_gmf_err_t esp_gmf_video_scale_set_cfg(esp_gmf_element_handle_t self, esp_imgfx_scale_cfg_t *config)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_JOB_ERR_FAIL);
ESP_GMF_NULL_CHECK(TAG, config, return ESP_GMF_JOB_ERR_FAIL);
esp_gmf_scale_hd_t *video_el = (esp_gmf_scale_hd_t *)self;
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// reset obj config
if (obj->cfg) {
memcpy(obj->cfg, config, sizeof(esp_imgfx_scale_cfg_t));
video_el->need_recfg = true;
return ESP_GMF_JOB_ERR_OK;
}
ESP_LOGE(TAG, " %s-%p is no configured yet", __func__, self);
return ESP_GMF_JOB_ERR_FAIL;
}
static inline esp_gmf_err_t esp_gmf_video_scale_get_cfg(esp_gmf_element_handle_t self, esp_imgfx_scale_cfg_t *config)
{
ESP_GMF_NULL_CHECK(TAG, self, return ESP_GMF_JOB_ERR_FAIL);
esp_gmf_scale_hd_t *video_el = (esp_gmf_scale_hd_t *)self;
// first get cfg from scale handle. If it is NULL, get cfg from obj
if (video_el->hd) {
esp_imgfx_err_t imgfx_ret = esp_imgfx_scale_get_cfg(video_el->hd, config);
if (imgfx_ret != ESP_IMGFX_ERR_OK) {
ESP_LOGE(TAG, "Get video effects scale cfg failed, hd:%p, ret: %d", self, imgfx_ret);
return ESP_GMF_JOB_ERR_FAIL;
}
return ESP_GMF_JOB_ERR_OK;
} else {
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// Get obj config
if (obj->cfg) {
memcpy(config, obj->cfg, sizeof(esp_imgfx_scale_cfg_t));
return ESP_GMF_JOB_ERR_OK;
}
}
ESP_LOGE(TAG, " %s-%p is no configured yet", __func__, self);
return ESP_GMF_JOB_ERR_FAIL;
}
static esp_gmf_err_t video_set_dst_res(void *handle, esp_gmf_args_desc_t *arg_desc, uint8_t *buf, int buf_len)
{
ESP_GMF_MEM_CHECK(TAG, buf, return ESP_GMF_ERR_INVALID_ARG);
return esp_gmf_video_scale_dst_resolution(handle, (esp_gmf_video_resolution_t *)buf);
}
static esp_gmf_err_t video_scale_el_load_caps(esp_gmf_element_handle_t handle)
{
esp_gmf_cap_t *caps = NULL;
esp_gmf_cap_t dec_caps = {0};
dec_caps.cap_eightcc = ESP_GMF_CAPS_VIDEO_SCALE;
dec_caps.attr_fun = NULL;
int ret = esp_gmf_cap_append(&caps, &dec_caps);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to create capability");
esp_gmf_element_t *el = (esp_gmf_element_t *)handle;
el->caps = caps;
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_scale_el_load_methods(esp_gmf_element_handle_t handle)
{
esp_gmf_method_t *method = NULL;
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_args_desc_t *set_args = NULL;
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(SCALER, SET_DST_RES, WIDTH), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), offsetof(esp_gmf_video_resolution_t, width));
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append scale destination width of resolution");
ret = esp_gmf_args_desc_append(&set_args, VMETHOD_ARG(SCALER, SET_DST_RES, HEIGHT), ESP_GMF_ARGS_TYPE_UINT16, sizeof(uint16_t), offsetof(esp_gmf_video_resolution_t, height));
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to append scale destination height of resolution");
ret = esp_gmf_method_append(&method, VMETHOD(SCALER, SET_DST_RES), video_set_dst_res, set_args);
ESP_GMF_RET_ON_ERROR(TAG, ret, return ret, "Failed to register %s method", VMETHOD(SCALER, SET_DST_RES));
esp_gmf_element_t *el = (esp_gmf_element_t *)handle;
el->method = method;
return ret;
}
static esp_gmf_err_t video_scale_el_received_event_handler(esp_gmf_event_pkt_t *evt, void *ctx)
{
ESP_GMF_NULL_CHECK(TAG, ctx, return ESP_GMF_ERR_INVALID_ARG);
ESP_GMF_NULL_CHECK(TAG, evt, return ESP_GMF_ERR_INVALID_ARG);
if ((evt->type != ESP_GMF_EVT_TYPE_REPORT_INFO)
|| (evt->sub != ESP_GMF_INFO_VIDEO)
|| (evt->payload == NULL)) {
return ESP_GMF_ERR_OK;
}
esp_gmf_element_handle_t self = (esp_gmf_element_handle_t)ctx;
esp_gmf_element_handle_t el = evt->from;
esp_gmf_event_state_t state = ESP_GMF_EVENT_STATE_NONE;
esp_gmf_element_get_state(self, &state);
esp_gmf_info_video_t *info = (esp_gmf_info_video_t *)evt->payload;
esp_gmf_video_el_set_src_info(self, info);
esp_imgfx_scale_cfg_t *config = (esp_imgfx_scale_cfg_t *)OBJ_GET_CFG(self);
ESP_GMF_NULL_CHECK(TAG, config, return ESP_GMF_ERR_FAIL);
esp_gmf_scale_hd_t *video_el = (esp_gmf_scale_hd_t *)self;
GMF_VIDEO_UPDATE_CONFIG(config, info, video_el->need_recfg);
ESP_LOGD(TAG, "RECV element info, from: %s-%p, next: %p, self: %s-%p, type: %x, state: %s, width: %d, height: %d, pixel format: %lx",
OBJ_GET_TAG(el), el, esp_gmf_node_for_next((esp_gmf_node_t *)el), OBJ_GET_TAG(self), self, evt->type,
esp_gmf_event_get_state_str(state), info->width, info->height, info->format_id);
// It is for the first time to receive event from previous element
if (state == ESP_GMF_EVENT_STATE_NONE) {
esp_gmf_element_set_state(self, ESP_GMF_EVENT_STATE_INITIALIZED);
}
return ESP_GMF_ERR_OK;
}
static esp_gmf_err_t video_scale_el_new(void *config, esp_gmf_obj_handle_t *handle)
{
return esp_gmf_video_scale_init((esp_imgfx_scale_cfg_t *)config, handle);
}
esp_gmf_err_t esp_gmf_video_scale_init(esp_imgfx_scale_cfg_t *config, esp_gmf_obj_handle_t *handle)
{
ESP_GMF_MEM_CHECK(TAG, handle, return ESP_GMF_ERR_INVALID_ARG);
*handle = NULL;
esp_gmf_scale_hd_t *video_el = esp_gmf_oal_calloc(1, sizeof(esp_gmf_scale_hd_t));
ESP_GMF_MEM_CHECK(TAG, video_el, return ESP_GMF_ERR_MEMORY_LACK);
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_obj_t *obj = (esp_gmf_obj_t *)video_el;
// Set element tag
ret = esp_gmf_obj_set_tag(video_el, "vid_scale");
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto __init_exit, "Failed set OBJ tag");
// Configure object callbacks
obj->new_obj = video_scale_el_new;
obj->del_obj = video_scale_el_delete;
// Set element config
esp_imgfx_scale_cfg_t *cfg = esp_gmf_oal_calloc(1, sizeof(esp_imgfx_scale_cfg_t));
ESP_GMF_MEM_VERIFY(TAG, cfg, {ret = ESP_GMF_ERR_MEMORY_LACK; goto __init_exit;}, "scale configuration", sizeof(esp_imgfx_scale_cfg_t));
esp_gmf_obj_set_config(obj, cfg, sizeof(esp_imgfx_scale_cfg_t));
if (config) {
memcpy(cfg, config, sizeof(esp_imgfx_scale_cfg_t));
} else {
esp_imgfx_scale_cfg_t dcfg = DEFAULT_ESP_GMF_SCALE_CONFIG();
memcpy(cfg, &dcfg, sizeof(esp_imgfx_scale_cfg_t));
}
// Configure element
esp_gmf_element_cfg_t el_cfg = {
.dependency = true,
};
ESP_GMF_ELEMENT_IN_PORT_ATTR_SET(el_cfg.in_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
ESP_GMF_ELEMENT_OUT_PORT_ATTR_SET(el_cfg.out_attr, ESP_GMF_EL_PORT_CAP_SINGLE, 0, 0,
ESP_GMF_PORT_TYPE_BLOCK, ESP_GMF_ELEMENT_PORT_DATA_SIZE_DEFAULT);
// Initialize element
ret = esp_gmf_video_el_init(video_el, &el_cfg);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, goto __init_exit, "Failed initialize video element");
// Register element's callbacks
esp_gmf_video_element_t *parent = (esp_gmf_video_element_t *)video_el;
parent->base.ops.open = video_scale_el_open;
parent->base.ops.process = video_scale_el_process;
parent->base.ops.close = video_scale_el_close;
parent->base.ops.event_receiver = video_scale_el_received_event_handler;
parent->base.ops.load_caps = video_scale_el_load_caps;
parent->base.ops.load_methods = video_scale_el_load_methods;
*handle = (esp_gmf_obj_handle_t)video_el;
ESP_LOGD(TAG, "Initialization, %s-%p", OBJ_GET_TAG(obj), obj);
return ret;
__init_exit:
video_scale_el_delete(video_el);
return ret;
}
esp_gmf_err_t esp_gmf_video_scale_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_video_resolution_t *res)
{
esp_imgfx_scale_cfg_t config;
esp_gmf_err_t ret = esp_gmf_video_scale_get_cfg(handle, &config);
ESP_GMF_RET_ON_NOT_OK(TAG, ret, return ret, "Failed to set destination pixel format");
config.scale_res.height = res->height;
config.scale_res.width = res->width;
return esp_gmf_video_scale_set_cfg(handle, &config);
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/gmf_video_common.c
+67
View File
@@ -0,0 +1,67 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include "esp_fourcc.h"
#include "gmf_video_common.h"
#include "esp_gmf_video_element.h"
#include "esp_log.h"
#define TAG "VIDEO_COMM"
const char *esp_gmf_video_get_format_string(uint32_t codec)
{
switch (codec) {
case ESP_FOURCC_H264:
return "h264";
case ESP_FOURCC_MJPG:
return "mjpeg";
case ESP_FOURCC_RGB16:
return "rgb565";
case ESP_FOURCC_RGB16_BE:
return "rgb565_be";
case ESP_FOURCC_RGB24:
return "rgb888";
case ESP_FOURCC_BGR24:
return "bgr888";
case ESP_FOURCC_YUV420P:
return "yuv420p";
case ESP_FOURCC_YUV422P:
return "yuv422p";
case ESP_FOURCC_YUYV:
return "yuv422";
case ESP_FOURCC_OUYY_EVYY:
return "o_uyy_e_vyy";
default:
return "none";
}
}
esp_gmf_err_t esp_gmf_video_handle_events(esp_gmf_event_pkt_t *evt, void *ctx)
{
esp_gmf_err_t ret = ESP_GMF_ERR_OK;
esp_gmf_element_handle_t self = (esp_gmf_element_handle_t) ctx;
// Only process report info currently
if ((evt->type != ESP_GMF_EVT_TYPE_REPORT_INFO)
|| (evt->sub != ESP_GMF_INFO_VIDEO)
|| (evt->payload == NULL)) {
return ESP_GMF_ERR_OK;
}
esp_gmf_event_state_t state = -1;
esp_gmf_element_get_state(self, &state);
// State must matched
esp_gmf_info_video_t *vid_info = evt->payload;
esp_gmf_video_element_t *video_el = (esp_gmf_video_element_t *)self;
video_el->src_info = *vid_info;
ESP_LOGI(TAG, "Video info for %s-%p format:%s %dx%d %dfps",
OBJ_GET_TAG(self), self,
esp_gmf_video_get_format_string(vid_info->format_id),
(int)vid_info->width, (int)vid_info->height, (int)vid_info->fps);
if (state == ESP_GMF_EVENT_STATE_NONE) {
esp_gmf_element_set_state(self, ESP_GMF_EVENT_STATE_INITIALIZED);
}
return ret;
}
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/idf_component.yml
+17
View File
@@ -0,0 +1,17 @@
version: "0.7.1"
description: Espressif GMF Video Elements Module
url: https://github.com/espressif/esp-gmf/tree/main/elements/gmf_video
documentation: "https://github.com/espressif/esp-gmf/blob/main/elements/gmf_video/README.md"
issues: "https://github.com/espressif/esp-gmf/issues"
tags:
- "gmf"
- "video"
dependencies:
espressif/esp_video_codec: "~0.5.2"
espressif/esp_image_effects:
require: public
version: "~1.0.0"
espressif/gmf_core:
version: "~0.7.0"
require: public
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_color_convert.h
+57
View File
@@ -0,0 +1,57 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#include "esp_imgfx_color_convert.h"
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
#define DEFAULT_ESP_GMF_COLOR_CONVERT_CONFIG() { \
.in_res = { \
.width = 320, \
.height = 240 , \
}, \
.in_pixel_fmt = ESP_IMGFX_PIXEL_FMT_YUYV, \
.out_pixel_fmt = ESP_IMGFX_PIXEL_FMT_RGB565_LE, \
.color_space_std = ESP_IMGFX_COLOR_SPACE_STD_BT601, \
}
/**
* @brief Initialize the video color convert element with the specified configuration
*
* @param[in] config Pointer to a color convert configuration
* @param[out] handle Pointer to a `esp_gmf_element_handle_t` where the initialized video color convert
* handle will be stored
*
* @return
* - ESP_GMF_ERR_OK Initialization successful
* - ESP_GMF_ERR_INVALID_ARG Invalid arguments
* - ESP_GMF_ERR_MEMORY_LACK No memory for video color convert element
*/
esp_gmf_err_t esp_gmf_video_color_convert_init(esp_imgfx_color_convert_cfg_t *config, esp_gmf_element_handle_t *handle);
/**
* @brief Set video color convert destination format
*
* @note This API have no special timing request, user can call it freely
*
* @param[in] handle Video color convert handle
* @param[in] format Video color convert destination format (GMF FourCC representation of video format)
*
* @return
* - ESP_GMF_ERR_OK Successful
* - ESP_GMF_ERR_INVALID_ARG Invalid arguments
*/
esp_gmf_err_t esp_gmf_video_color_convert_dst_format(esp_gmf_element_handle_t handle, uint32_t format);
#ifdef __cplusplus
}
#endif /* __cplusplus */
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_crop.h
+63
View File
@@ -0,0 +1,63 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#include "esp_gmf_video_types.h"
#include "esp_imgfx_crop.h"
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
#define DEFAULT_ESP_GMF_CROP_CONFIG() { \
.in_res = { \
.width = 320, \
.height = 240, \
}, \
.in_pixel_fmt = ESP_IMGFX_PIXEL_FMT_RGB565_LE, \
.cropped_res = { \
.width = 160, \
.height = 120, \
}, \
.x_pos = 0, \
.y_pos = 0, \
}
/**
* @brief Initialize the video crop element with the specified configuration
*
* @param[in] config Pointer to a `esp_imgfx_crop_cfg_t` structure that contains the
* configuration settings for the video crop
* @param[out] handle Pointer to a `esp_gmf_element_handle_t` where the initialized video crop
* handle will be stored
*
* @return
* - ESP_GMF_ERR_OK Initialization successful
* - ESP_GMF_ERR_INVALID_ARG Invalid arguments
* - ESP_GMF_ERR_MEMORY_LACK No memory for video crop element
*/
esp_gmf_err_t esp_gmf_video_crop_init(esp_imgfx_crop_cfg_t *config, esp_gmf_element_handle_t *handle);
/**
* @brief Set video cropped region
*
* @note This API have no special timing request, user can call it freely
*
* @param[in] handle Video video crop handle
* @param[in] rgn Region to be kept in original video frame
*
* @return
* - ESP_GMF_ERR_OK Successful
* - ESP_GMF_ERR_INVALID_ARG Invalid arguments
*/
esp_gmf_err_t esp_gmf_video_crop_rgn(esp_gmf_element_handle_t handle, esp_gmf_video_rgn_t *rgn);
#ifdef __cplusplus
}
#endif /* __cplusplus */
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_dec.h
+87
View File
@@ -0,0 +1,87 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#ifdef __cplusplus
extern "C" {
#endif
/**
* @brief Video decoder configuration (optional)
*/
typedef struct {
uint32_t codec_cc; /*!< FourCC code to force a specific decoder implementation
See `esp_video_codec_desc_t` in `esp_video_codec_types.h` for details
If zero, the system will select a suitable decoder (HW preferred) */
} esp_gmf_video_dec_cfg_t;
/**
* @brief Initializes the GMF Video decoder with the provided configuration
*
* @param[in] cfg Pointer to the Video decoder configuration (optional)
* @param[out] handle Pointer to the Video decoder handle to be initialized
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_dec_init(esp_gmf_video_dec_cfg_t *cfg, esp_gmf_element_handle_t *handle);
/**
* @brief Get Video decoder output frame size
*
* @note This API should only called after element running or `esp_gmf_video_dec_set_dst_format` is called
*
* @param[in] handle Video decoder element handle
* @param[out] frame_size Output frame size to store
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_dec_get_out_size(esp_gmf_element_handle_t handle, uint32_t *frame_size);
/**
* @brief Setting for video decoder destination format (e.g., YUV420P, RGB565LE)
*
* @note This API should only called before element running
*
* @param[in] handle Video decoder element handle
* @param[in] dst_fmt Destination format (GMF FourCC representation of video codec)
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_dec_set_dst_format(esp_gmf_element_handle_t handle, uint32_t dst_fmt);
/**
* @brief Get supported output formats for a given video decoder type
* This function retrieves the list of output formats (e.g., YUV420P, RGB565LE) that the decoder can produce
* when decoding from the specified source codec
*
* @param[in] handle Video decoder element handle
* @param[in] src_codec Decode type (GMF FourCC representation of video codec)
* @param[out] dst_fmts Output video formats to store (GMF FourCC representation of video format)
* @param[out] dst_fmts_num Output video format number to store
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_dec_get_dst_formats(esp_gmf_element_handle_t handle,
uint32_t src_codec,
const uint32_t **dst_fmts,
uint8_t *dst_fmts_num);
#ifdef __cplusplus
}
#endif /* __cplusplus */
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_enc.h
+153
View File
@@ -0,0 +1,153 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#ifdef __cplusplus
extern "C" {
#endif
/**
* @brief Video encoder configuration
*/
typedef struct {
uint32_t codec_cc; /*!< FourCC code to force a specific encoder implementation
See `esp_video_codec_desc_t` in `esp_video_codec_types.h` for details
If zero, the system will select a suitable encoder (HW preferred) */
} esp_gmf_video_enc_cfg_t;
/**
* @brief Initializes the GMF video encoder with the provided configuration
*
* @param[in] cfg Pointer to the video encoder configuration
* @param[out] handle Pointer to the video encoder handle to be initialized
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_enc_init(esp_gmf_video_enc_cfg_t *cfg, esp_gmf_element_handle_t *handle);
/**
* @brief Setting for video encoder destination codec (e.g., H.264, MJPEG)
*
* @note This API should only called before element running
*
* @param[in] handle Video encoder element handle
* @param[in] dst_codec Video encode destination codec (use GMF FourCC presentation)
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_enc_set_dst_codec(esp_gmf_element_handle_t handle, uint32_t dst_codec);
/**
* @brief Do presetting for video encoder
*
* @note This function pre-configures the video encoder with both source and destination information
* This allows querying encoder capabilities before runtime
* Normally, source information is reported from dependent element in the running state
* Call this function only before element running
*
* @param[in] handle Video encoder element handle
* @param[in] src_info Video encode input video information
* @param[in] dst_codec Destination video codec (use GMF FourCC presentation)
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_enc_preset(esp_gmf_element_handle_t handle, esp_gmf_info_video_t *src_info,
uint32_t dst_codec);
/**
* @brief Get video encoder output frame size
*
* @note This API should only be called after `esp_gmf_video_enc_preset` set or after element running
*
* @param[in] handle Video encoder element handle
* @param[out] frame_size Output frame size to store
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_enc_get_out_size(esp_gmf_element_handle_t handle, uint32_t *frame_size);
/**
* @brief Get supported source formats (e.g., YUV420P, RGB565LE etc) for a given video encoder type
* This function retrieves the list of supported input formats that can be encoded
* into the specified destination codec
*
* @param[in] handle Video encoder element handle
* @param[in] dst_codec Video encode destination codec (GMF FourCC representation of video codec)
* @param[out] src_fmts Source formats to stored (GMF FourCC representation of video format)
* @param[out] src_fmts_num Source format number to store
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_enc_get_src_formats(esp_gmf_element_handle_t handle,
uint32_t dst_codec,
const uint32_t **src_fmts,
uint8_t *src_fmts_num);
/**
* @brief Setting for video encoder bitrate
*
* @note Support setting both before and after element running
*
* @param[in] handle Video encoder element handle
* @param[in] bitrate Output bitrate
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_enc_set_bitrate(esp_gmf_element_handle_t handle, uint32_t bitrate);
/**
* @brief Set the GOP (Group of Pictures) for video encoder.
*
* @note This setting applies specifically to the H.264 encoder
* The H.264 encoder generates an I-frame at the start of each GOP
* Can be configured both before and after the encoder element is running
*
* @param[in] handle Handle to the video encoder element
* @param[in] gop GOP size for H.264 (in number of frames)
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_enc_set_gop(esp_gmf_element_handle_t handle, uint32_t gop);
/**
* @brief Set the QP (Quantization Parameter) range for video encoder
*
* @note This setting applies specifically to the H.264 encoder
* Higher QP values generally result in lower image quality
* Can be configured both before and after the encoder element is running
*
* @param[in] handle Handle to the video encoder element
* @param[in] min_qp Minimum QP value
* @param[in] max_qp Maximum QP value
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_enc_set_qp(esp_gmf_element_handle_t handle, uint32_t min_qp, uint32_t max_qp);
#ifdef __cplusplus
}
#endif /* __cplusplus */
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_fps_cvt.h
+45
View File
@@ -0,0 +1,45 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#ifdef __cplusplus
extern "C" {
#endif
/**
* @brief Initializes the GMF video frame rate convert
*
* @param[in] config No need to set
* @param[out] handle Pointer to the video frame rate convert handle to be initialized
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_fps_cvt_init(void *config, esp_gmf_element_handle_t *handle);
/**
* @brief Set GMF video frame rate converter output frame rate
*
* @note This API should only called before element running
*
* @param[in] handle Video frame rate converter handle
* @param[out] fps Destination frame rate to set (unit frame per second)
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_fps_cvt_set_fps(esp_gmf_element_handle_t handle, uint16_t fps);
#ifdef __cplusplus
}
#endif
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_methods_def.h
+176
View File
@@ -0,0 +1,176 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_method_helper.h"
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
/**
* @brief Video general method argument definition
*/
#define ESP_GMF_VIDEO_FORMAT_ARG "format"
#define ESP_GMF_VIDEO_WIDTH_ARG "width"
#define ESP_GMF_VIDEO_HEIGHT_ARG "height"
#define ESP_GMF_VIDEO_FPS_ARG "fps"
#define ESP_GMF_VIDEO_BITRATE_ARG "bitrate"
#define ESP_GMF_VIDEO_CODEC_ARG "codec"
/**
* @brief Video general region argument definition
*/
#define ESP_GMF_VIDEO_RGN_ARG_X "x"
#define ESP_GMF_VIDEO_RGN_ARG_Y "y"
#define ESP_GMF_VIDEO_RGN_ARG_WIDTH ESP_GMF_VIDEO_WIDTH_ARG
#define ESP_GMF_VIDEO_RGN_ARG_HEIGHT ESP_GMF_VIDEO_HEIGHT_ARG
/**
* @brief GMF video method definition
*/
#define VMETHOD_DEF(module, method, str) ESP_GMF_METHOD_DEF(VIDEO, module, method, str)
/**
* @brief Get GMF video method string
*/
#define VMETHOD(module, method) ESP_GMF_METHOD_STR(VIDEO, module, method)
/**
* @brief GMF video method argument definition
*/
#define VMETHOD_ARG_DEF(module, method, arg, str) ESP_GMF_METHOD_ARG_DEF(VIDEO, module, method, arg, str)
/**
* @brief Get GMF video method argument string
*/
#define VMETHOD_ARG(module, method, arg) ESP_GMF_METHOD_ARG_STR(VIDEO, module, method, arg)
/**
* @brief Video FPS converter method definition for set frame per second
*/
VMETHOD_DEF(FPS_CVT, SET_FPS, "set_fps");
VMETHOD_ARG_DEF(FPS_CVT, SET_FPS, FPS, ESP_GMF_VIDEO_FPS_ARG);
/**
* @brief Video overlay mixer method definition for enable overlay
*/
VMETHOD_DEF(OVERLAY, OVERLAY_ENABLE, "overlay_enable");
VMETHOD_ARG_DEF(OVERLAY, OVERLAY_ENABLE, ENABLE, "enable");
/**
* @brief Video overlay mixer method definition for set overlay port
*/
VMETHOD_DEF(OVERLAY, SET_PORT, "overlay_port");
VMETHOD_ARG_DEF(OVERLAY, SET_PORT, PORT, "port");
/**
* @brief Video overlay mixer method definition for set overlay region information
*/
VMETHOD_DEF(OVERLAY, SET_RGN, "overlay_rgn");
VMETHOD_ARG_DEF(OVERLAY, SET_RGN, FMT, ESP_GMF_VIDEO_FORMAT_ARG);
VMETHOD_ARG_DEF(OVERLAY, SET_RGN, X, ESP_GMF_VIDEO_RGN_ARG_X);
VMETHOD_ARG_DEF(OVERLAY, SET_RGN, Y, ESP_GMF_VIDEO_RGN_ARG_Y);
VMETHOD_ARG_DEF(OVERLAY, SET_RGN, WIDTH, ESP_GMF_VIDEO_RGN_ARG_WIDTH);
VMETHOD_ARG_DEF(OVERLAY, SET_RGN, HEIGHT, ESP_GMF_VIDEO_RGN_ARG_HEIGHT);
/**
* @brief Video overlay mixer method definition for set overlay region alpha
*/
VMETHOD_DEF(OVERLAY, SET_ALPHA, "set_alpha");
VMETHOD_ARG_DEF(OVERLAY, SET_ALPHA, ALPHA, "alpha");
/**
* @brief Video color convert method definition for set destination format
*/
VMETHOD_DEF(CLR_CVT, SET_DST_FMT, "set_dst_fmt");
VMETHOD_ARG_DEF(CLR_CVT, SET_DST_FMT, FMT, ESP_GMF_VIDEO_FORMAT_ARG);
/**
* @brief Video scaler method definition for set destination resolution
*/
VMETHOD_DEF(SCALER, SET_DST_RES, "set_dst_res");
VMETHOD_ARG_DEF(SCALER, SET_DST_RES, WIDTH, ESP_GMF_VIDEO_RGN_ARG_WIDTH);
VMETHOD_ARG_DEF(SCALER, SET_DST_RES, HEIGHT, ESP_GMF_VIDEO_RGN_ARG_HEIGHT);
/**
* @brief Video cropper method definition for set cropped region
*/
VMETHOD_DEF(CROP, SET_CROP_RGN, "set_crop_rgn");
VMETHOD_ARG_DEF(CROP, SET_CROP_RGN, X, ESP_GMF_VIDEO_RGN_ARG_X);
VMETHOD_ARG_DEF(CROP, SET_CROP_RGN, Y, ESP_GMF_VIDEO_RGN_ARG_Y);
VMETHOD_ARG_DEF(CROP, SET_CROP_RGN, WIDTH, ESP_GMF_VIDEO_RGN_ARG_WIDTH);
VMETHOD_ARG_DEF(CROP, SET_CROP_RGN, HEIGHT, ESP_GMF_VIDEO_RGN_ARG_HEIGHT);
/**
* @brief Video rotator method definition for set rotate degree
*/
VMETHOD_DEF(ROTATOR, SET_ANGLE, "set_angle");
VMETHOD_ARG_DEF(ROTATOR, SET_ANGLE, DEGREE, "degree");
/**
* @brief Video encoder method definition for set output codec (e.g., H.264, MJPEG)
*/
VMETHOD_DEF(ENCODER, SET_DST_CODEC, "set_dst_codec");
VMETHOD_ARG_DEF(ENCODER, SET_DST_CODEC, CODEC, ESP_GMF_VIDEO_CODEC_ARG);
/**
* @brief Video encoder method definition for set output bitrate
*/
VMETHOD_DEF(ENCODER, SET_BITRATE, "set_bitrate");
VMETHOD_ARG_DEF(ENCODER, SET_BITRATE, BITRATE, ESP_GMF_VIDEO_BITRATE_ARG);
/**
* @brief Video encoder method definition for get input formats according output codec
*/
VMETHOD_DEF(ENCODER, GET_SRC_FMTS, "get_src_fmts");
VMETHOD_ARG_DEF(ENCODER, GET_SRC_FMTS, DST_CODEC, "dst_codec");
VMETHOD_ARG_DEF(ENCODER, GET_SRC_FMTS, SRC_FMTS_PTR, "src_fmts");
VMETHOD_ARG_DEF(ENCODER, GET_SRC_FMTS, SRC_FMTS_NUM_PTR, "src_fmts_num");
/**
* @brief Video encoder method definition for presetting including input video information and output codec
*
* @note This method is specially used when user want to query things from encoder and provide encode basic settings
*/
VMETHOD_DEF(ENCODER, PRESET, "venc_preset");
VMETHOD_ARG_DEF(ENCODER, PRESET, SRC_FMT, "src_fmt");
VMETHOD_ARG_DEF(ENCODER, PRESET, SRC_WIDTH, ESP_GMF_VIDEO_WIDTH_ARG);
VMETHOD_ARG_DEF(ENCODER, PRESET, SRC_HEIGHT, ESP_GMF_VIDEO_HEIGHT_ARG);
VMETHOD_ARG_DEF(ENCODER, PRESET, SRC_FPS, ESP_GMF_VIDEO_FPS_ARG);
VMETHOD_ARG_DEF(ENCODER, PRESET, SRC_BITRATE, ESP_GMF_VIDEO_BITRATE_ARG);
VMETHOD_ARG_DEF(ENCODER, PRESET, DST_CODEC, "dst_codec");
/**
* @brief Video encoder method definition for get encode frame size
*/
VMETHOD_DEF(ENCODER, GET_FRAME_SIZE, "get_frm_size");
VMETHOD_ARG_DEF(ENCODER, GET_FRAME_SIZE, DST_FRM_SIZE_PTR, "dst_frm_size");
/**
* @brief Video decoder method definition for set source codec (e.g., H.264, MJPEG)
*/
VMETHOD_DEF(DECODER, SET_SRC_CODEC, "set_src_codec");
VMETHOD_ARG_DEF(DECODER, SET_SRC_CODEC, CODEC, ESP_GMF_VIDEO_CODEC_ARG);
/**
* @brief Video decoder method definition for set destination format (e.g., YUV420P, RGB565LE)
* Reuse `VMETHOD_DEF(CLR_CVT, SET_DST_FMT, "set_dst_fmt")` for same function
*/
/**
* @brief Video decoder method definition for get output formats according input codec
*/
VMETHOD_DEF(DECODER, GET_DST_FMTS, "get_dst_fmts");
VMETHOD_ARG_DEF(DECODER, GET_DST_FMTS, SRC_CODEC, "src_codec");
VMETHOD_ARG_DEF(DECODER, GET_DST_FMTS, DST_FMTS_PTR, "dst_fmts");
VMETHOD_ARG_DEF(DECODER, GET_DST_FMTS, DST_FMTS_NUM_PTR, "dst_fmts_num");
#ifdef __cplusplus
}
#endif /* __cplusplus */
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_overlay.h
+111
View File
@@ -0,0 +1,111 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#include "esp_gmf_video_types.h"
#ifdef __cplusplus
extern "C" {
#endif
/**
* @brief Initializes the GMF overlay mixer with the provided configuration
*
* @note This module mixes the overlay plane (foreground) into the original plane (background) using alpha blending
* The formula is:
* Output_Plane_Pixel = Original_Plane_Pixel * (255 - alpha) + Overlay_Plane_Pixel * alpha
* where `alpha` is the transparency level (0 = fully transparent, 255 = fully opaque)
*
* Diagram:
* +----------------------------+
* | Original Plane |
* | (Background) |
* | +---------------------+ |
* | | Overlay Plane | |
* | | (Foreground) | |
* | +---------------------+ |
* +----------------------------+
*
* @param[in] config No need to set
* @param[out] handle Pointer to the overlay mixer handle to be initialized
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_overlay_init(void *config, esp_gmf_element_handle_t *handle);
/**
* @brief Set the output region for video overlay
*
* @note This API should only called before element running
*
* @param[in] handle Video overlay element handle
* @param[out] rgn_info Region information
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_overlay_set_rgn(esp_gmf_element_handle_t handle, esp_gmf_overlay_rgn_info_t *rgn_info);
/**
* @brief Set the GMF port for video overlay input
*
* @note The video overlay element will use this port to acquire overlay data,
* mix it into the original video plane, and then release the port
* The port is managed by the user, the overlay element only accesses it
* This API can be called before element running and allow reset
*
* @param[in] handle Video overlay element handle
* @param[in] port GMF port for video overlay data
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_overlay_set_overlay_port(esp_gmf_element_handle_t handle, esp_gmf_port_handle_t port);
/**
* @brief Set the alpha value of the overlay window
*
* @note Allow to set both before and after element running
*
* @param[in] handle Video overlay element handle
* @param[in] alpha Overlay window alpha setting:
* 0 means fully transparent, 255 means fully opaque
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_overlay_set_alpha(esp_gmf_element_handle_t handle, uint8_t alpha);
/**
* @brief Enable overlay mixer or not
*
* @note When overlay mixer is disabled will not acquire data from overlay port and do mixer anymore
* Original plane data leave unchanged
* Allow to set both before and after element running
*
* @param[in] handle Video overlay mixer handle
* @param[in] enable Enable mixer or not
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_overlay_enable(esp_gmf_element_handle_t handle, bool enable);
#ifdef __cplusplus
}
#endif /* __cplusplus */
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_param.h
+204
View File
@@ -0,0 +1,204 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#include "esp_gmf_video_types.h"
#include "esp_gmf_video_methods_def.h"
#ifdef __cplusplus
extern "C" {
#endif
/**
* @brief This file provides general parameter setting functions for common video processing using GMF method
*
* These functions simplify setting parameters for video processing elements,
* handling the underlying method calls. They do not enforce a specific order
* or argument count, but apply the basic settings where supported. Refer to
* the individual element APIs for details on supported settings and timing.
*/
/**
* @brief Set destination format for video element (e.g., color converter, video decoder)
*
* @param[in] handle Video element handle
* @param[in] dst_fmt Destination format
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_dst_format(esp_gmf_element_handle_t handle, uint32_t dst_fmt);
/**
* @brief Set frame rate for video element (mainly for FPS converter)
*
* @param[in] handle Video element handle
* @param[in] fps Destination frame rate
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_fps(esp_gmf_element_handle_t handle, uint16_t fps);
/**
* @brief Set destination resolution for video element (mainly for video scaler)
*
* @param[in] handle Video element handle
* @param[in] res Destination resolution
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_video_resolution_t *res);
/**
* @brief Set destination codec for video element (e.g., video encoder)
*
* @param[in] handle Video element handle
* @param[in] dst_codec Destination codec
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_dst_codec(esp_gmf_element_handle_t handle, uint32_t dst_codec);
/**
* @brief Preset source video information and destination codec for video encoder
*
* @note In special cases call this API then query some information from video encoder before element running
*
* @param[in] handle Video element handle
* @param[in] vid_info Source video information
* @param[in] dst_codec Destination codec
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_venc_preset(esp_gmf_element_handle_t handle, esp_gmf_info_video_t *vid_info,
uint32_t dst_codec);
/**
* @brief Get supported source formats by codec for video element (mainly for video encoder)
*
* @param[in] handle Video element handle
* @param[in] dst_codec Destination codec
* @param[in] src_fmts Pointer to store source formats array
* @param[in] src_fmts_num Pointer to store source formats number
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_get_src_fmts_by_codec(esp_gmf_element_handle_t handle, uint32_t dst_codec,
const uint32_t **src_fmts, uint8_t *src_fmts_num);
/**
* @brief Set source codec for video element (e.g., video decoder)
*
* @param[in] handle Video element handle
* @param[in] src_codec Source codec
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_src_codec(esp_gmf_element_handle_t handle, uint32_t src_codec);
/**
* @brief Set the rotation angle for the video element (mainly for video rotator)
*
* @param[in] handle Video element handle
* @param[in] degree Rotation angle in degrees (e.g., 0, 90, 180, 270)
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_rotate_angle(esp_gmf_element_handle_t handle, uint16_t degree);
/**
* @brief Set the cropped region for video element (mainly for video cropper)
*
* @param[in] handle Video element handle
* @param[in] rgn Kept region after crop
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_cropped_region(esp_gmf_element_handle_t handle, esp_gmf_video_rgn_t *rgn);
/**
* @brief Enable overlay for video element (mainly for video overlay)
*
* @param[in] handle Video element handle
* @param[in] enable Whether enable video overlay or not
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_overlay_enable(esp_gmf_element_handle_t self, bool enable);
/**
* @brief Set overlay port for video element (mainly for video overlay)
*
* @param[in] handle Video element handle
* @param[in] port Handle to the input GMF port providing overlay data
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_overlay_port(esp_gmf_element_handle_t self, void *port);
/**
* @brief Set overlay region information for video element (mainly for video overlay)
*
* @param[in] handle Video element handle
* @param[in] rgn Overlay region information
*
* @return
* - ESP_GMF_ERR_OK On success
* - ESP_GMF_ERR_NOT_FOUND Not found the method
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
* - Others Failed to apply method
*/
esp_gmf_err_t esp_gmf_video_param_set_overlay_rgn(esp_gmf_element_handle_t self, esp_gmf_overlay_rgn_info_t *rgn);
#ifdef __cplusplus
}
#endif
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_ppa.h
+99
View File
@@ -0,0 +1,99 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#include "esp_gmf_element.h"
#include "esp_gmf_video_types.h"
#ifdef __cplusplus
extern "C" {
#endif
/**
* @brief Video PPA (Pixel Processing Accelerator) is a compact module
* Currently it only support on ESP32P4 and including following video processing:
* 1) Color convert
* 2) Scale
* 3) Rotate
* 5) Crop
* These operations can be mixed so that can do them all together for better performance
*/
/**
* @brief Initializes the GMF Video PPA
*
* @param[in] config No need to set
* @param[out] handle Video PPA handle to store
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_ppa_init(void *config, esp_gmf_element_handle_t *handle);
/**
* @brief Set video PPA destination resolution
*
* @note This API should only called before element running
*
* @param[in] handle Video PPA handle
* @param[in] res Video PPA output resolution
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_ppa_set_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_video_resolution_t *res);
/**
* @brief Set video PPA destination format
*
* @note This API should only called before element running
*
* @param[in] handle Video PPA handle
* @param[in] format Video PPA destination format (GMF FourCC representation of video format)
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_ppa_set_dst_format(esp_gmf_element_handle_t handle, uint32_t format);
/**
* @brief Set video PPA cropped region
*
* @note This API should only called before element running
*
* @param[in] handle Video PPA handle
* @param[in] rgn Region to be kept in original video frame
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
* - ESP_GMF_ERR_MEMORY_LACK Failed to allocate memory
*/
esp_gmf_err_t esp_gmf_video_ppa_set_cropped_rgn(esp_gmf_element_handle_t handle, esp_gmf_video_rgn_t *rgn);
/**
* @brief Set video PPA rotation degree
*
* @note This API should only called before element running
*
* @param[in] handle Video PPA handle
* @param[in] degree Currently only support (0, 90, 180, 270) unit(one degree)
*
* @return
* - ESP_GMF_ERR_OK Success
* - ESP_GMF_ERR_INVALID_ARG Invalid configuration provided
*/
esp_gmf_err_t esp_gmf_video_ppa_set_rotation(esp_gmf_element_handle_t handle, uint16_t degree);
#ifdef __cplusplus
}
#endif
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_rotate.h
+57
View File
@@ -0,0 +1,57 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#include "esp_imgfx_rotate.h"
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
#define DEFAULT_ESP_GMF_ROTATE_CONFIG() { \
.in_res = { \
.width = 320, \
.height = 240, \
}, \
.in_pixel_fmt = ESP_IMGFX_PIXEL_FMT_RGB565_LE, \
.degree = 0, \
}
/**
* @brief Initialize the video rotate element with the specified configuration
*
* @param[in] config Pointer to a `esp_imgfx_rotate_cfg_t` structure that contains the
* configuration settings for the video rotate
* @param[out] handle Pointer to a `esp_gmf_element_handle_t` where the initialized video rotate
* handle will be stored
*
* @return
* - ESP_GMF_ERR_OK Initialization successful
* - ESP_GMF_ERR_INVALID_ARG Invalid arguments
* - ESP_GMF_ERR_MEMORY_LACK No memory for video rotate element
*/
esp_gmf_err_t esp_gmf_video_rotate_init(esp_imgfx_rotate_cfg_t *config, esp_gmf_element_handle_t *handle);
/**
* @brief Set video rotate rotation degree
*
* @note This API have no special timing request, user can call it freely
*
* @param[in] handle Video video rotate handle
* @param[in] degree Any angle, unit(one degree)
*
* @return
* - ESP_GMF_ERR_OK Successful
* - ESP_GMF_ERR_INVALID_ARG Invalid arguments
*/
esp_gmf_err_t esp_gmf_video_rotate_set_rotation(esp_gmf_element_handle_t handle, uint16_t degree);
#ifdef __cplusplus
}
#endif /* __cplusplus */
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_scale.h
+62
View File
@@ -0,0 +1,62 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#include "esp_gmf_video_types.h"
#include "esp_imgfx_scale.h"
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
#define DEFAULT_ESP_GMF_SCALE_CONFIG() { \
.in_res = { \
.width = 320, \
.height = 240, \
}, \
.in_pixel_fmt = ESP_IMGFX_PIXEL_FMT_RGB565_LE, \
.scale_res = { \
.width = 160, \
.height = 120, \
}, \
.filter_type = ESP_IMGFX_SCALE_FILTER_TYPE_DOWN_RESAMPLE \
}
/**
* @brief Initialize the video scale element with the specified configuration
*
* @param[in] config Pointer to a `esp_imgfx_scale_cfg_t` structure that contains the
* configuration settings for the video scale
* @param[out] handle Pointer to a `esp_gmf_element_handle_t` where the initialized video scale
* handle will be stored
*
* @return
* - ESP_GMF_ERR_OK Initialization successful
* - ESP_GMF_ERR_INVALID_ARG Invalid arguments
* - ESP_GMF_ERR_MEMORY_LACK No memory for video scale element
*/
esp_gmf_err_t esp_gmf_video_scale_init(esp_imgfx_scale_cfg_t *config, esp_gmf_element_handle_t *handle);
/**
* @brief Set video scale destination resolution
*
* @note This API have no special timing request, user can call it freely
*
* @param[in] handle Video video scale handle
* @param[in] res Video video scale output resolution
*
* @return
* - ESP_GMF_ERR_OK Successful
* - ESP_GMF_ERR_INVALID_ARG Invalid arguments
*/
esp_gmf_err_t esp_gmf_video_scale_dst_resolution(esp_gmf_element_handle_t handle, esp_gmf_video_resolution_t *res);
#ifdef __cplusplus
}
#endif /* __cplusplus */
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/include/esp_gmf_video_types.h
+47
View File
@@ -0,0 +1,47 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include "esp_gmf_element.h"
#ifdef __cplusplus
extern "C" {
#endif
/**
* @brief Video resolution definition
*/
typedef struct {
uint16_t width; /*!< Width of the video in pixels */
uint16_t height; /*!< Height of the video in pixels */
} esp_gmf_video_resolution_t;
/**
* @brief Video region definition
* Represents a rectangular region of the video frame
*/
typedef struct {
uint16_t x; /*!< X-coordinate of the top-left corner */
uint16_t y; /*!< Y-coordinate of the top-left corner */
uint16_t width; /*!< Width of the region in pixels */
uint16_t height; /*!< Height of the region in pixels */
} esp_gmf_video_rgn_t;
/**
* @brief Specifies the format and destination region for a video overlay
*
* @note This structure defines the parameters needed to position and format an overlay on a video frame
*/
typedef struct {
uint32_t format_id; /*!< FourCC representation of overlay format */
esp_gmf_video_rgn_t dst_rgn; /*!< Region position to put the overlay */
} esp_gmf_overlay_rgn_info_t;
#ifdef __cplusplus
}
#endif
firmware/echoear-platform-agent/components/esp-gmf/elements/gmf_video/private_include/gmf_video_common.h
+147
View File
@@ -0,0 +1,147 @@
/*
* SPDX-FileCopyrightText: 2025 Espressif Systems (Shanghai) CO., LTD
* SPDX-License-Identifier: LicenseRef-Espressif-Modified-MIT
*
* See LICENSE file for details.
*/
#pragma once
#include <stdint.h>
#include <stdatomic.h>
#include "esp_gmf_event.h"
#include "esp_gmf_video_element.h"
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
/**
* @brief Atomic declare and operations
*/
#define ATOM_VAR atomic_int
#define GET_ATOM(v) atomic_load(&v);
#define SET_ATOM(atom, v) atomic_store(&atom, v)
#define ATOM_SET_BITS(atom, bits) atomic_fetch_or(&atom, bits)
#define ATOM_CLEAR_BITS(atom, bits) atomic_fetch_and(&atom, ~bits)
#define GMF_VIDEO_BREAK_ON_FAIL(ret) if (ret != ESP_GMF_ERR_OK) { \
break; \
}
#define GMF_VIDEO_ALIGN_UP(size, align) (((size) + ((align) - 1)) & ~((align) - 1))
#define GMF_VIDEO_UPDATE_CONFIG(config, info, need_recfg) do { \
(need_recfg) = ((config)->in_pixel_fmt != (info)->format_id) \
|| ((config)->in_res.width != (info)->width) \
|| ((config)->in_res.height != (info)->height); \
(config)->in_res.width = (info)->width; \
(config)->in_res.height = (info)->height; \
(config)->in_pixel_fmt = (info)->format_id; \
} while (0);
/**
* @brief Get video format string representation
*
* @param[in] format Video format FourCC representation
*
* @return
* - "none" Not valid codec type
* - Others String representation of codec
*/
const char *esp_gmf_video_get_format_string(uint32_t format);
/**
* @brief Common video event handler
*
* @note This API implement basic handler of GMF report info event, element should derive from `esp_gmf_video_element_t`
*
* @param[in] evt GMT event packet
* @param[in] ctx Event context
*
* @return
* - ESP_GMF_ERR_OK On success
*/
esp_gmf_err_t esp_gmf_video_handle_events(esp_gmf_event_pkt_t *evt, void *ctx);
/**
* @brief Update basic information of video element
* This function updates the width, height, and pixel format of a video element.
* Typically called when video stream configuration changes (e.g., resolution adjustment
* or format switching) to synchronize the latest parameters
*
* @param[in] self Video element handle pointing to the target video element object.
* Must be initialized and non-NULL.
* @param[in] width New video width in pixels
* @param[in] height New video height in pixels
* @param[in] pixel_fmt Pixel format
*
*/
static inline void gmf_video_update_info(esp_gmf_video_element_handle_t self, uint16_t width, uint16_t height, uint32_t pixel_fmt)
{
esp_gmf_info_video_t vid_info = {0};
esp_gmf_video_el_get_src_info(self, &vid_info);
vid_info.height = height;
vid_info.width = width;
vid_info.format_id = pixel_fmt;
esp_gmf_element_notify_vid_info(self, &vid_info);
}
/**
* @brief Acquire input/output payload for processing
*
* @param[in] in_port Handle to the input port
* @param[in] out_port Handle to the output port
* @param[out] in_load Double pointer to receive the acquired input payload buffer
* The buffer will be allocated/reserved according to in_wanted_size
* @param[out] out_load Double pointer to receive the acquired output payload buffer
* The buffer will be allocated/reserved according to out_wanted_size
* @param[in] in_wanted_size Desired minimum size for input payload buffer (in bytes)
* @param[in] out_wanted_size Desired minimum size for output payload buffer (in bytes)
* @param[in] is_bypass Bypass flag indicating whether to use zero-copy optimization (true)
* or allow data modification (false)
*
* @return
* - ESP_GMF_JOB_ERR_OK On success
* - ESP_GMF_JOB_ERR_FAIL On failure
*
*/
static inline esp_gmf_job_err_t video_el_acquire_payload(esp_gmf_port_handle_t in_port, esp_gmf_port_handle_t out_port, esp_gmf_payload_t **in_load,
esp_gmf_payload_t **out_load, uint32_t in_wanted_size, uint32_t out_wanted_size, bool is_bypass)
{
esp_gmf_err_io_t load_ret = ESP_GMF_IO_OK;
load_ret = esp_gmf_port_acquire_in(in_port, in_load, in_wanted_size, ESP_GMF_MAX_DELAY);
if (load_ret < ESP_GMF_IO_OK) {
if (load_ret == ESP_GMF_IO_ABORT) {
return ESP_GMF_JOB_ERR_OK;
}
ESP_LOGE("Imgfx_el", "Acquire size: %ld on in port, ret: %d", in_wanted_size, load_ret);
return ESP_GMF_JOB_ERR_FAIL;
}
if (((*in_load)->valid_size) < (in_wanted_size)) {
ESP_LOGE("Imgfx_el", "Acquire size %ld-%d on in port is not enough", in_wanted_size, (*in_load)->valid_size);
return ESP_GMF_JOB_ERR_FAIL;
}
if ((in_port->is_shared == 1)
&& (is_bypass)) {
/* If it is share buffer and is_bypass is true, just copy it */
*out_load = (*in_load);
}
load_ret = esp_gmf_port_acquire_out(out_port, out_load, out_wanted_size, ESP_GMF_MAX_DELAY);
if (load_ret < ESP_GMF_IO_OK) {
if (load_ret == ESP_GMF_IO_ABORT) {
return ESP_GMF_JOB_ERR_OK;
}
ESP_LOGE("Imgfx_el", "Acquire size: %ld on out port, ret: %d", out_wanted_size, load_ret);
return ESP_GMF_JOB_ERR_FAIL;
}
if ((*out_load)->buf_length < out_wanted_size) {
ESP_LOGE("Imgfx_el", "Acquire size %ld-%d on out port is not enough", out_wanted_size, (*out_load)->buf_length);
return ESP_GMF_JOB_ERR_FAIL;
}
return ESP_GMF_JOB_ERR_OK;
}
#ifdef __cplusplus
}
#endif /* __cplusplus */