wangcong/story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0c2151ace9614db18a5ef4d34917565612b5608a
story/firmware/echoear-platform-agent/components/av_processor/README.md
T
Codex Local 0c2151ace9
ci / verify (push) Canceled after 0s
Details
chore: checkpoint 第一版
2026-09-06 17:04:11 +08:00

14 KiB
Raw Blame History

AV Processor Component (Audio/Video Processing)

This component provides a unified encapsulation for audio recording/playback/data feeding and video capture/decoding/rendering on ESP platforms, implemented based on middleware such as GMF (General Media Framework), esp-audio-render, esp-video-codec, and esp-capture.

Feature Overview 🎛️

  • Audio 🎧:
    • Recording (AEC/VAD/AFE optional)
    • Playback (supports URL and local files, decoding depends on configuration)
    • Data feeding (feeder, suitable for scenarios like RTC/streaming where receiving and playing happen simultaneously)
    • Optional mixer with volume fade and focus control between multiple streams
  • Video 🎥:
    • Multi-channel video capture
    • Video decoding and rendering callbacks

Quick Start

1. Audio Usage

  1. Initialize audio manager:
// Method 1: Use default configuration macro (recommended)
audio_manager_config_t cfg = DEFAULT_AUDIO_MANAGER_CONFIG();
cfg.play_dev = your_play_dev;
cfg.rec_dev = your_rec_dev;
strcpy(cfg.mic_layout, "RMNM"); // Microphone layout (optional, for AFE configuration)
cfg.board_sample_rate = 16000;
cfg.board_bits = 32;
cfg.board_channels = 2;
cfg.play_volume = 80;
cfg.rec_volume = 60;
cfg.rec_ref_volume = 60;
cfg.enable_mixer = true; // Enable when mixing feeder and playback is needed

audio_manager_init(&cfg);
  1. Recording (polling read):
// Recording event callback (optional)
void recorder_event_cb(void *event, void *ctx) {
    // Handle recording events (e.g., AFE events)
}

// Configure recorder (using default configuration macro)
audio_recorder_config_t recorder_cfg = DEFAULT_AUDIO_RECORDER_CONFIG();
// Optional: Configure AFE (runtime configuration, takes priority over Kconfig)
recorder_cfg.afe_config.vad_enable = true;
recorder_cfg.afe_config.vad_mode = 4;
recorder_cfg.afe_config.agc_enable = true;
recorder_cfg.recorder_event_cb = recorder_event_cb;

// Open recorder
audio_recorder_open(&recorder_cfg);
uint8_t buf[2048];
audio_recorder_read_data(buf, sizeof(buf));

audio_recorder_close();
  1. Feeder playback (suitable for external stream → local playback):
// Configure feeder (using default configuration macro)
audio_feeder_config_t feeder_cfg = DEFAULT_AUDIO_FEEDER_CONFIG();
// Optional: When using OPUS decoder, task_stack is recommended to be >= 4096 * 10
// feeder_cfg.feeder_task_config.task_stack = 4096 * 10;

audio_feeder_open(&feeder_cfg);
audio_feeder_run();
// Feed external data blocks multiple times as needed
audio_feeder_feed_data(pkt, pkt_len);

audio_feeder_stop();
audio_feeder_close();
  1. Normal playback (URL or local):
// Configure player (using default configuration macro)
audio_playback_config_t playback_cfg = DEFAULT_AUDIO_PLAYBACK_CONFIG();

audio_playback_open(&playback_cfg);
audio_playback_play("http://<ip>:<port>/audio.mp3");

audio_playback_pause();
audio_playback_resume();
audio_playback_stop();
audio_playback_close();
  1. Focus/fade control (when mixer is enabled):
// Note: Mixer must be opened after both playback and feeder are opened
audio_processor_mixer_open();
audio_processor_ramp_control(AUDIO_MIXER_FOCUS_FEEDER);     // Focus on feeder audio
audio_processor_ramp_control(AUDIO_MIXER_FOCUS_PLAYBACK);   // Focus on playback audio
audio_processor_mixer_close();

2. Video Usage

  1. Render (decode) callback:
void decoded_cb(void *ctx, const uint8_t *data, size_t size) {
    // Handle decoded frame data
}

video_render_config_t rcfg = {
    .decode_cfg = your_vdec_cfg,
    .resolution = {.width = 640, .height = 480},
    .decode_cb = decoded_cb,
};
video_render_handle_t r = video_render_open(&rcfg);
video_render_start(r);

video_frame_t f = {.data = enc_frame, .size = enc_size};
video_render_frame_feed(r, &f);

video_render_stop(r);
video_render_close(r);
  1. Capture (multi-channel):
void capture_cb(void *ctx, int index, video_frame_t *frame) {
    // Handle captured video frame
}

video_capture_config_t ccfg = {0};
ccfg.camera_config = &your_cam_cfg;
ccfg.sink_num = 2;
ccfg.sink_cfg[0] = your_sink0_cfg;
ccfg.sink_cfg[1] = your_sink1_cfg;
ccfg.capture_frame_cb = capture_cb;

video_capture_handle_t c = video_capture_open(&ccfg);
video_capture_start(c);
video_capture_stop(c);
video_capture_close(c);

Media Dump 💾

  • Enable condition ✅: Takes effect after enabling MEDIA_DUMP_ENABLE in menuconfig.
  • Save methods 📤: Supports SD card and UDP outputs (choose one).
    • SD card 💾: Enable CONFIG_MEDIA_DUMP_SINK_SDCARD, output to file CONFIG_MEDIA_DUMP_SDCARD_FILENAME, duration controlled by CONFIG_MEDIA_DUMP_DURATION_SEC.
    • UDP 📡: Enable CONFIG_MEDIA_DUMP_SINK_UDP, send raw media data via CONFIG_MEDIA_DUMP_UDP_IP and CONFIG_MEDIA_DUMP_UDP_PORT (can use script/udp_reciver.py script).
  • Save point 🎚️ (before/after AEC processing): Select in "Audio Save Mode (AEC point)" in menuconfig
    • MEDIA_DUMP_AUDIO_BEFORE_AEC (Save Before AEC): Save raw microphone audio before AEC (for analyzing noise/feedback)
    • MEDIA_DUMP_AUDIO_AFTER_AEC (Save After AEC): Save audio after AEC processing (for evaluating AEC effectiveness)
  • Typical usage 🔍: Export raw audio/video data when reproducing issues, check saturation, feedback, and format correctness offline using tools like Audacity.

Configuration ⚙️

Runtime Configuration (Recommended)

AFE configuration can be set at runtime via the afe_config field in the audio_recorder_config_t structure, which takes priority over Kconfig configuration:

audio_recorder_config_t recorder_cfg = DEFAULT_AUDIO_RECORDER_CONFIG();
// Configure AFE
recorder_cfg.afe_config.ai_mode_wakeup = false;        // AI mode: true=wakeup mode, false=direct mode
recorder_cfg.afe_config.vad_enable = true;              // Enable VAD
recorder_cfg.afe_config.vad_mode = 4;                   // VAD mode (1-4), larger values are more sensitive
recorder_cfg.afe_config.vad_min_speech_ms = 64;        // Minimum speech duration (ms)
recorder_cfg.afe_config.vad_min_noise_ms = 1000;       // Minimum noise duration (ms)
recorder_cfg.afe_config.agc_enable = true;              // Enable AGC
recorder_cfg.afe_config.enable_vcmd_detect = false;     // Enable VCMD
recorder_cfg.afe_config.vcmd_timeout_ms = 5000;         // VCMD timeout (ms)
recorder_cfg.afe_config.mn_language = "cn";             // Model language: "cn" or "en"
recorder_cfg.afe_config.wakeup_time_ms = 10000;         // Wakeup time (ms)
recorder_cfg.afe_config.wakeup_end_time_ms = 2000;      // Wakeup end time (ms)

// Pass configuration when opening recorder
audio_recorder_open(&recorder_cfg);

Kconfig Configuration (Alternative)

If fields in audio_recorder_config_t.afe_config use default values or are not set, Kconfig configuration will be used. Configure via idf.py menuconfig:

💾 Media Dump Configuration

Configure under "Component config" -> "Audio/Video Processor Configuration" -> "Media Dump" in menuconfig:

  • MEDIA_DUMP_ENABLE: Enable media data dump functionality, disabled by default

    • When enabled, raw audio/video data can be saved for debugging and analysis

  • MEDIA_DUMP_AUDIO_POINT: Audio save point selection (choice type)

    • MEDIA_DUMP_AUDIO_BEFORE_AEC: Save raw audio before AEC processing
      • Useful for analyzing noise and feedback issues
    • MEDIA_DUMP_AUDIO_AFTER_AEC: Save audio after AEC processing
      • Useful for evaluating AEC processing effectiveness
    • MEDIA_DUMP_AUDIO_NONE: Do not save audio (default)

  • MEDIA_DUMP_DURATION_SEC: Save duration in seconds, default 20

    • Type: integer (int)
    • Range: 1-3600 seconds

  • MEDIA_DUMP_SINK: Save method selection (choice type)

    • MEDIA_DUMP_SINK_SDCARD: Save to SD card file (default)
      • File path configuration: CONFIG_MEDIA_DUMP_SDCARD_FILENAME (default: /sdcard/media_dump.bin)
      • Ensure SD card is properly mounted
    • MEDIA_DUMP_SINK_UDP: Send via UDP
      • Target IP configuration: CONFIG_MEDIA_DUMP_UDP_IP (default: 192.168.1.100)
      • Target port configuration: CONFIG_MEDIA_DUMP_UDP_PORT (default: 5000)
      • Can use script/udp_reciver.py script to receive data

Task Configuration

The audio processing component uses multiple FreeRTOS tasks internally, which can be customized via task configuration fields in each module's configuration structure:

Task Configuration for Each Module

  • Recorder (audio_recorder_config_t):

    • afe_feed_task_config: AFE feed task configuration (default stack size 3KB)
    • afe_fetch_task_config: AFE fetch task configuration (default stack size 3KB)
    • recorder_task_config: Recorder task configuration (default stack size 5KB, recommend >= 40KB when using OPUS encoder)

  • Player (audio_playback_config_t):

    • playback_task_config: Playback task configuration (default stack size 4KB)

  • Feeder (audio_feeder_config_t):

    • feeder_task_config: Feeder task configuration (default stack size 5KB, recommend >= 40KB when using OPUS decoder)

Important Notes:

  • When using OPUS encoder or decoder, task_stack of recorder_task_config and feeder_task_config should be set to at least 4096 * 10 bytes (40KB).
  • When task_stack in task configuration is set to 0, default values will be used.
  • When task_stack_in_ext is set to true, task stack will be allocated in external memory, helping to save internal RAM.
  • Task configuration should be set in each module's configuration structure (e.g., audio_recorder_config_t, audio_playback_config_t, audio_feeder_config_t), not in audio_manager_config_t.

API Reference 📚

Header files:

  • include/audio_processor.h
  • include/video_processor.h
  • include/av_processor_type.h

Common functions (selection):

  • Audio management:
    • audio_manager_init/audio_manager_deinit: Initialize/deinitialize audio manager
  • Recording:
    • audio_recorder_open: Open recorder (supports encoder configuration and event callback)
    • audio_recorder_read_data: Read recording data
    • audio_recorder_pause/audio_recorder_resume: Pause/resume recording
    • audio_recorder_get_afe_manager_handle: Get AFE manager handle
    • audio_recorder_close: Close recorder
  • Feeder:
    • audio_feeder_open: Open feeder (supports decoder configuration)
    • audio_feeder_run: Start feeder
    • audio_feeder_feed_data: Feed audio data
    • audio_feeder_stop: Stop feeder
    • audio_feeder_close: Close feeder
  • Playback:
    • audio_playback_open/audio_playback_close: Open/close player
    • audio_playback_play: Play audio (supports URL or local file, e.g., http://... or file:///sdcard/...)
    • audio_playback_stop: Stop playback
    • audio_playback_pause/audio_playback_resume: Pause/resume playback
    • audio_playback_get_state: Get playback state
  • Mixer:
    • audio_processor_mixer_open: Open mixer (must be called after both playback and feeder are opened)
    • audio_processor_mixer_close: Close mixer
    • audio_processor_ramp_control: Control audio focus and volume fade
  • Video rendering:
    • video_render_open/video_render_close: Open/close video renderer
    • video_render_start/video_render_stop: Start/stop rendering
    • video_render_frame_feed: Feed video frame
  • Video capture:
    • video_capture_open/video_capture_close: Open/close video capturer
    • video_capture_start/video_capture_stop: Start/stop capture

FAQ ❓

Audio Issues

  • "Self-questioning" phenomenon occurs (device playback sound is repeatedly captured by microphone):

    • Enable MEDIA_DUMP_ENABLE in menuconfig, reproduce the issue and export saved audio data for further analysis
    • Open the exported audio file using Audacity, observe waveform/spectrum to determine if saturation or feedback exists
    • If saturation clipping occurs, reduce microphone gain (esp_codec_dev_set_in_gain); or appropriately reduce speaker volume (esp_codec_dev_set_out_vol) to avoid excessive feedback

  • Stack overflow when using OPUS encoder/decoder:

    • Ensure task_stack of recorder_task_config and feeder_task_config is set to at least 4096 * 10 bytes (40KB)
    • Consider allocating task stack in external memory (set task_stack_in_ext = true)

  • Mixer not working properly:

    • Ensure that both player (audio_playback_open) and feeder (audio_feeder_open) are opened before calling audio_processor_mixer_open()
    • Ensure enable_mixer = true is set when calling audio_manager_init

Development and Debugging

  • Modifying component source code (espressif__av_processor):

    • If you need to modify code within espressif__av_processor, you can execute the script/move_av_processor.py script
    • The script will copy managed_components/espressif__av_processor to your project's PROJECT_DIR/components/espressif__av_processor directory
    • The script will automatically delete the original directory under managed_components and update idf_component.yml
    • Before using, set the environment variable: export PROJECT_DIR=/path/to/project

  • Media data dump (Media Dump):

    • UDP method: Can use script/udp_reciver.py script to receive UDP data
    • SD card method: Ensure SD card is properly mounted and file path is accessible

Version Information

Current version: v0.5.0

License

This component follows the Apache-2.0 license. See the LICENSE file included in the repository for details.

Related Resources

Reference in New Issue View Git Blame Copy Permalink