story/firmware/echoear-platform-agent/components/av_processor

AV Processor Component (Audio/Video Processing)

This component provides a unified encapsulation for audio recording/playback/data feeding and video capture/decoding/rendering on ESP platforms, implemented based on middleware such as GMF (General Media Framework), esp-audio-render, esp-video-codec, and esp-capture.

Feature Overview 🎛️

• Audio 🎧:
  • Recording (AEC/VAD/AFE optional)
  • Playback (supports URL and local files, decoding depends on configuration)
  • Data feeding (feeder, suitable for scenarios like RTC/streaming where receiving and playing happen simultaneously)
  • Optional mixer with volume fade and focus control between multiple streams
• Video 🎥:
  • Multi-channel video capture
  • Video decoding and rendering callbacks

---

Quick Start

1. Audio Usage

1. Initialize audio manager:

// Method 1: Use default configuration macro (recommended)
audio_manager_config_t cfg = DEFAULT_AUDIO_MANAGER_CONFIG();
cfg.play_dev = your_play_dev;
cfg.rec_dev = your_rec_dev;
strcpy(cfg.mic_layout, "RMNM"); // Microphone layout (optional, for AFE configuration)
cfg.board_sample_rate = 16000;
cfg.board_bits = 32;
cfg.board_channels = 2;
cfg.play_volume = 80;
cfg.rec_volume = 60;
cfg.rec_ref_volume = 60;
cfg.enable_mixer = true; // Enable when mixing feeder and playback is needed

audio_manager_init(&cfg);

2. Recording (polling read):

// Recording event callback (optional)
void recorder_event_cb(void *event, void *ctx) {
    // Handle recording events (e.g., AFE events)
}

// Configure recorder (using default configuration macro)
audio_recorder_config_t recorder_cfg = DEFAULT_AUDIO_RECORDER_CONFIG();
// Optional: Configure AFE (runtime configuration, takes priority over Kconfig)
recorder_cfg.afe_config.vad_enable = true;
recorder_cfg.afe_config.vad_mode = 4;
recorder_cfg.afe_config.agc_enable = true;
recorder_cfg.recorder_event_cb = recorder_event_cb;

// Open recorder
audio_recorder_open(&recorder_cfg);
uint8_t buf[2048];
audio_recorder_read_data(buf, sizeof(buf));

audio_recorder_close();

3. Feeder playback (suitable for external stream → local playback):

// Configure feeder (using default configuration macro)
audio_feeder_config_t feeder_cfg = DEFAULT_AUDIO_FEEDER_CONFIG();
// Optional: When using OPUS decoder, task_stack is recommended to be >= 4096 * 10
// feeder_cfg.feeder_task_config.task_stack = 4096 * 10;

audio_feeder_open(&feeder_cfg);
audio_feeder_run();
// Feed external data blocks multiple times as needed
audio_feeder_feed_data(pkt, pkt_len);

audio_feeder_stop();
audio_feeder_close();

4. Normal playback (URL or local):

// Configure player (using default configuration macro)
audio_playback_config_t playback_cfg = DEFAULT_AUDIO_PLAYBACK_CONFIG();

audio_playback_open(&playback_cfg);
audio_playback_play("http://<ip>:<port>/audio.mp3");

audio_playback_pause();
audio_playback_resume();
audio_playback_stop();
audio_playback_close();

5. Focus/fade control (when mixer is enabled):

// Note: Mixer must be opened after both playback and feeder are opened
audio_processor_mixer_open();
audio_processor_ramp_control(AUDIO_MIXER_FOCUS_FEEDER);     // Focus on feeder audio
audio_processor_ramp_control(AUDIO_MIXER_FOCUS_PLAYBACK);   // Focus on playback audio
audio_processor_mixer_close();

2. Video Usage

1. Render (decode) callback:

void decoded_cb(void *ctx, const uint8_t *data, size_t size) {
    // Handle decoded frame data
}

video_render_config_t rcfg = {
    .decode_cfg = your_vdec_cfg,
    .resolution = {.width = 640, .height = 480},
    .decode_cb = decoded_cb,
};
video_render_handle_t r = video_render_open(&rcfg);
video_render_start(r);

video_frame_t f = {.data = enc_frame, .size = enc_size};
video_render_frame_feed(r, &f);

video_render_stop(r);
video_render_close(r);

1. Capture (multi-channel):

void capture_cb(void *ctx, int index, video_frame_t *frame) {
    // Handle captured video frame
}

video_capture_config_t ccfg = {0};
ccfg.camera_config = &your_cam_cfg;
ccfg.sink_num = 2;
ccfg.sink_cfg[0] = your_sink0_cfg;
ccfg.sink_cfg[1] = your_sink1_cfg;
ccfg.capture_frame_cb = capture_cb;

video_capture_handle_t c = video_capture_open(&ccfg);
video_capture_start(c);
video_capture_stop(c);
video_capture_close(c);

---

Media Dump 💾

• Enable condition ✅: Takes effect after enabling MEDIA_DUMP_ENABLE in menuconfig.
• Save methods 📤: Supports SD card and UDP outputs (choose one).
  • SD card 💾: Enable CONFIG_MEDIA_DUMP_SINK_SDCARD, output to file CONFIG_MEDIA_DUMP_SDCARD_FILENAME, duration controlled by CONFIG_MEDIA_DUMP_DURATION_SEC.
  • UDP 📡: Enable CONFIG_MEDIA_DUMP_SINK_UDP, send raw media data via CONFIG_MEDIA_DUMP_UDP_IP and CONFIG_MEDIA_DUMP_UDP_PORT (can use script/udp_reciver.py script).
• Save point 🎚️ (before/after AEC processing): Select in "Audio Save Mode (AEC point)" in menuconfig
  • MEDIA_DUMP_AUDIO_BEFORE_AEC (Save Before AEC): Save raw microphone audio before AEC (for analyzing noise/feedback)
  • MEDIA_DUMP_AUDIO_AFTER_AEC (Save After AEC): Save audio after AEC processing (for evaluating AEC effectiveness)
• Typical usage 🔍: Export raw audio/video data when reproducing issues, check saturation, feedback, and format correctness offline using tools like Audacity.

Configuration ⚙️

Runtime Configuration (Recommended)

AFE configuration can be set at runtime via the afe_config field in the audio_recorder_config_t structure, which takes priority over Kconfig configuration:

audio_recorder_config_t recorder_cfg = DEFAULT_AUDIO_RECORDER_CONFIG();
// Configure AFE
recorder_cfg.afe_config.ai_mode_wakeup = false;        // AI mode: true=wakeup mode, false=direct mode
recorder_cfg.afe_config.vad_enable = true;              // Enable VAD
recorder_cfg.afe_config.vad_mode = 4;                   // VAD mode (1-4), larger values are more sensitive
recorder_cfg.afe_config.vad_min_speech_ms = 64;        // Minimum speech duration (ms)
recorder_cfg.afe_config.vad_min_noise_ms = 1000;       // Minimum noise duration (ms)
recorder_cfg.afe_config.agc_enable = true;              // Enable AGC
recorder_cfg.afe_config.enable_vcmd_detect = false;     // Enable VCMD
recorder_cfg.afe_config.vcmd_timeout_ms = 5000;         // VCMD timeout (ms)
recorder_cfg.afe_config.mn_language = "cn";             // Model language: "cn" or "en"
recorder_cfg.afe_config.wakeup_time_ms = 10000;         // Wakeup time (ms)
recorder_cfg.afe_config.wakeup_end_time_ms = 2000;      // Wakeup end time (ms)

// Pass configuration when opening recorder
audio_recorder_open(&recorder_cfg);

Kconfig Configuration (Alternative)

If fields in audio_recorder_config_t.afe_config use default values or are not set, Kconfig configuration will be used. Configure via idf.py menuconfig:

💾 Media Dump Configuration

Configure under "Component config" -> "Audio/Video Processor Configuration" -> "Media Dump" in menuconfig:

• MEDIA_DUMP_ENABLE: Enable media data dump functionality, disabled by default

  • When enabled, raw audio/video data can be saved for debugging and analysis

• MEDIA_DUMP_AUDIO_POINT: Audio save point selection (choice type)

  • MEDIA_DUMP_AUDIO_BEFORE_AEC: Save raw audio before AEC processing
    • Useful for analyzing noise and feedback issues
  • MEDIA_DUMP_AUDIO_AFTER_AEC: Save audio after AEC processing
    • Useful for evaluating AEC processing effectiveness
  • MEDIA_DUMP_AUDIO_NONE: Do not save audio (default)

• MEDIA_DUMP_DURATION_SEC: Save duration in seconds, default 20

  • Type: integer (int)
  • Range: 1-3600 seconds

• MEDIA_DUMP_SINK: Save method selection (choice type)

  • MEDIA_DUMP_SINK_SDCARD: Save to SD card file (default)
    • File path configuration: CONFIG_MEDIA_DUMP_SDCARD_FILENAME (default: /sdcard/media_dump.bin)
    • Ensure SD card is properly mounted
  • MEDIA_DUMP_SINK_UDP: Send via UDP
    • Target IP configuration: CONFIG_MEDIA_DUMP_UDP_IP (default: 192.168.1.100)
    • Target port configuration: CONFIG_MEDIA_DUMP_UDP_PORT (default: 5000)
    • Can use script/udp_reciver.py script to receive data

Task Configuration

The audio processing component uses multiple FreeRTOS tasks internally, which can be customized via task configuration fields in each module's configuration structure:

Task Configuration for Each Module

• Recorder (audio_recorder_config_t):

  • afe_feed_task_config: AFE feed task configuration (default stack size 3KB)
  • afe_fetch_task_config: AFE fetch task configuration (default stack size 3KB)
  • recorder_task_config: Recorder task configuration (default stack size 5KB, recommend >= 40KB when using OPUS encoder)

• Player (audio_playback_config_t):

  • playback_task_config: Playback task configuration (default stack size 4KB)

• Feeder (audio_feeder_config_t):

  • feeder_task_config: Feeder task configuration (default stack size 5KB, recommend >= 40KB when using OPUS decoder)

Important Notes:

• When using OPUS encoder or decoder, task_stack of recorder_task_config and feeder_task_config should be set to at least 4096 * 10 bytes (40KB).
• When task_stack in task configuration is set to 0, default values will be used.
• When task_stack_in_ext is set to true, task stack will be allocated in external memory, helping to save internal RAM.
• Task configuration should be set in each module's configuration structure (e.g., audio_recorder_config_t, audio_playback_config_t, audio_feeder_config_t), not in audio_manager_config_t.

---

API Reference 📚

Header files:

• include/audio_processor.h
• include/video_processor.h
• include/av_processor_type.h

Common functions (selection):

• Audio management:
  • audio_manager_init/audio_manager_deinit: Initialize/deinitialize audio manager
• Recording:
  • audio_recorder_open: Open recorder (supports encoder configuration and event callback)
  • audio_recorder_read_data: Read recording data
  • audio_recorder_pause/audio_recorder_resume: Pause/resume recording
  • audio_recorder_get_afe_manager_handle: Get AFE manager handle
  • audio_recorder_close: Close recorder
• Feeder:
  • audio_feeder_open: Open feeder (supports decoder configuration)
  • audio_feeder_run: Start feeder
  • audio_feeder_feed_data: Feed audio data
  • audio_feeder_stop: Stop feeder
  • audio_feeder_close: Close feeder
• Playback:
  • audio_playback_open/audio_playback_close: Open/close player
  • audio_playback_play: Play audio (supports URL or local file, e.g., http://... or file:///sdcard/...)
  • audio_playback_stop: Stop playback
  • audio_playback_pause/audio_playback_resume: Pause/resume playback
  • audio_playback_get_state: Get playback state
• Mixer:
  • audio_processor_mixer_open: Open mixer (must be called after both playback and feeder are opened)
  • audio_processor_mixer_close: Close mixer
  • audio_processor_ramp_control: Control audio focus and volume fade
• Video rendering:
  • video_render_open/video_render_close: Open/close video renderer
  • video_render_start/video_render_stop: Start/stop rendering
  • video_render_frame_feed: Feed video frame
• Video capture:
  • video_capture_open/video_capture_close: Open/close video capturer
  • video_capture_start/video_capture_stop: Start/stop capture

---

FAQ ❓

Audio Issues

• "Self-questioning" phenomenon occurs (device playback sound is repeatedly captured by microphone):

  • Enable MEDIA_DUMP_ENABLE in menuconfig, reproduce the issue and export saved audio data for further analysis
  • Open the exported audio file using Audacity, observe waveform/spectrum to determine if saturation or feedback exists
  • If saturation clipping occurs, reduce microphone gain (esp_codec_dev_set_in_gain); or appropriately reduce speaker volume (esp_codec_dev_set_out_vol) to avoid excessive feedback

• Stack overflow when using OPUS encoder/decoder:

  • Ensure task_stack of recorder_task_config and feeder_task_config is set to at least 4096 * 10 bytes (40KB)
  • Consider allocating task stack in external memory (set task_stack_in_ext = true)

• Mixer not working properly:

  • Ensure that both player (audio_playback_open) and feeder (audio_feeder_open) are opened before calling audio_processor_mixer_open()
  • Ensure enable_mixer = true is set when calling audio_manager_init

Development and Debugging

• Modifying component source code (espressif__av_processor):

  • If you need to modify code within espressif__av_processor, you can execute the script/move_av_processor.py script
  • The script will copy managed_components/espressif__av_processor to your project's PROJECT_DIR/components/espressif__av_processor directory
  • The script will automatically delete the original directory under managed_components and update idf_component.yml
  • Before using, set the environment variable: export PROJECT_DIR=/path/to/project

• Media data dump (Media Dump):

  • UDP method: Can use script/udp_reciver.py script to receive UDP data
  • SD card method: Ensure SD card is properly mounted and file path is accessible

---

Version Information

Current version: v0.5.0

License

This component follows the Apache-2.0 license. See the LICENSE file included in the repository for details.

Related Resources

• Project repository: https://github.com/espressif/esp-gmf/tree/main/components/av_processor
• Issue reporting: https://github.com/espressif/esp-gmf/issues
• Documentation: https://github.com/espressif/esp-gmf/blob/main/components/av_processor/README.md