Codex Local
@@ -1,8 +1,45 @@
|
||||
# Windows 语音终端模拟器
|
||||
# Windows 硬件对话智能体测试程序
|
||||
|
||||
该程序使用Electron和火山引擎Web RTC SDK模拟一台真实语音终端。它采集Windows麦克风、加入平台分配的RTC房间、发布本地音频、订阅并播放豆包智能体音频，并通过设备v2接口完成激活、状态等待、心跳、Token续签和结束通话。进房后还会启动RTC实时字幕，收集长者与智能体的最终话轮并通过现有`turns:batch`设备接口可靠上传；字幕未成功启动时拒绝继续激活通话，避免生成没有聊天记录的完成通话。
|
||||
这是方案 1 的 Windows 端到端验收工具。默认页面不再使用平台代管的 RTC/StartVoiceChat，而是按火山“硬件对话智能体低负载标准协议”直接连接 `wss://ai-gateway.vei.volces.com/v1/realtime`。传统 RTC 模拟器仍保留在页面右上角的“传统 RTC 测试页”。
|
||||
|
||||
创建通话返回的隐私策略允许原录音时，模拟器同时通过RTC原始音频帧回调采集长者`near`和智能体`far`轨道。结束通话后自动封装为单声道WAV PCM16、按平台限制分片、申请短期对象地址直传，并等待后台完整性与容器校验完成。家庭工作台的通话记录会标记“可听聊天语音”，获得独立PLAY授权的账号可在详情中选择“听长者语音”或“听智能体语音”。
|
||||
## 实际覆盖的流程
|
||||
|
||||
1. 使用平台设备 ID、时间戳、随机数、正文摘要和 HMAC-SHA256 调用设备 v2/v3 接口。
|
||||
2. 调用平台 `/device/v3/hardware-agent-credentials:provision`；平台自动应用产品级Instance ID、Product Key、默认Bot、传输和最低SDK配置，并使用KMS/Secret挂载的 Product Secret 代设备调用 DynamicRegister。
|
||||
3. 火山在动态注册时从产品额度中分配 License；平台只把本设备 Device Secret 下发给设备，Windows 使用 Safe Storage 加密保存后重新拉取配置版本并进入 `ACTIVE`。
|
||||
4. 创建 `/device/v3/agent-sessions`，从平台取得产品默认 Bot ID、`low_load` 传输方案、本次冻结的人设、背景、最近摘要、确认记忆和授权范围。
|
||||
5. 按火山规定的请求头签名建立 WebSocket，等待 `session.created`，提交本次 `session.update`，收到 `session.updated` 后才向平台回报 `session_started`并开放麦克风。
|
||||
6. Windows 麦克风被转换为 PCM16、16kHz、单声道、20ms 分片，通过 `input_audio_buffer.append` 实时上行；`response.audio.delta` 按火山 low-load WebSocket SDK 的 PCM16、16kHz、单声道格式实时播放。
|
||||
7. 消费服务端 VAD、用户最终识别、智能体最终字幕、Function Calling 和错误事件；双方最终话轮通过 v3 `turns:batch` 幂等上传。
|
||||
8. 智能体触发 `memory_search`、`search_memory` 或 `retrieve_memory` 时，程序使用平台签发的当前会话短期令牌调用同源记忆工具，再把结果作为 `function_call_output` 返回火山。
|
||||
9. 周期提交 `session_updated`。平台返回 `stop_session` 时立即关麦、补传字幕、关闭火山连接并回报 `session_ended`。
|
||||
10. 页面显示 13 项验收门禁，可导出不包含任何密钥的 JSON 报告。
|
||||
|
||||
## 使用前配置
|
||||
|
||||
平台维护中心只需把测试设备配置为：
|
||||
|
||||
- `voice_runtime = volc_hardware_agent`
|
||||
|
||||
平台部署需配置并关联同一火山产品和 Bot：
|
||||
|
||||
- Instance ID
|
||||
- Product Key
|
||||
- 默认 Bot ID
|
||||
- 传输方式 `low_load`
|
||||
- 最低 SDK 版本不高于 `1.1.0-windows-standard-protocol`
|
||||
- Product Secret（只进入平台密钥挂载，不进入Windows程序）
|
||||
- Device Name 由平台按设备ID稳定生成，或使用设备绑定中预先登记的名称
|
||||
- 可用的硬件对话智能体 License
|
||||
|
||||
Windows程序不提供 Product Secret、Device Secret 或 Bot ID 输入框。Bot ID、Instance ID、Product Key 和 Device Name 均来自平台；设备不能覆盖产品默认路由。
|
||||
|
||||
开始测试前还需按控制台中的 Bot 模型选择“智能体模型类型”：
|
||||
|
||||
- 通用 LLM：使用官方 `config.LLMConfig.SystemMessages` 注入平台冻结的人设、背景与授权上下文；
|
||||
- 豆包端到端语音：使用官方 `config.S2SConfig.ProviderParams.dialog` 注入 `system_role`、`speaking_style` 和 `dialog_id`。
|
||||
|
||||
两种模式都使用官方 `session.object/config/agent_config/input_audio_format/output_audio_format` 报文结构。字段依据火山[初始化会话配置](https://www.volcengine.com/docs/6348/2139862?lang=zh)与[标准协议自定义接入](https://www.volcengine.com/docs/6348/1904602?lang=zh)文档。
|
||||
|
||||
## 构建
|
||||
|
||||
@@ -12,36 +49,70 @@ npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
输出：`release/voice-device-simulator-0.1.0-windows-x64.exe`。
|
||||
输出文件：
|
||||
|
||||
连接公网测试服务器时，可以直接双击EXE，也可以使用`launch-cloud-device.ps1`启动。Electron主进程优先读取启动脚本传入的临时环境变量，否则从当前Windows用户的`%LOCALAPPDATA%\DataTerminalVoiceDevice\server-device-secret`读取设备签名密钥。密钥不写入源码、安装包或浏览器存储，也不会回传到渲染进程；输入框留空并显示“已从Windows安全凭据文件加载”即表示可直接检测。
|
||||
|
||||
## 自动连续性验证
|
||||
|
||||
本地回归时可让模拟器自动启动通话、播放一段测试音频，并把双方最终字幕写入JSON结果文件。该入口只接受本机文件和回环地址，不影响正常交互使用：
|
||||
|
||||
```powershell
|
||||
$env:SIMULATOR_AUTO_TEST = '1'
|
||||
$env:SIMULATOR_AUTOMATED_AUDIO_PATH = 'C:\path\to\question.wav'
|
||||
$env:SIMULATOR_AUTO_RESULT_PATH = 'C:\path\to\result.json'
|
||||
& .\release\voice-device-simulator-0.1.0-windows-x64.exe
|
||||
```text
|
||||
release/hardware-agent-tester-0.3.5-windows-x64.exe
|
||||
```
|
||||
|
||||
测试会等待长者与智能体的最终字幕，结束通话后退出；结果文件包含通话ID、最终话轮和通过/失败状态。测试音频支持WAV、MP3和OGG，大小不得超过10 MB。
|
||||
当前测试包未配置商业代码签名证书，Windows 可能显示“未知发布者”。交付前可用同目录 SHA-256 文件校验完整性；正式面向终端用户分发时应由甲方或发布主体使用其代码签名证书签名。
|
||||
|
||||
`npm run build` 会依次执行协议单测、TypeScript 检查、双页面构建和 Windows x64 便携包封装。
|
||||
|
||||
## 本地完整自动回归
|
||||
|
||||
先启动仓库根目录的 Compose 开发环境，然后运行：
|
||||
|
||||
```powershell
|
||||
Set-Location .\device-simulator-windows
|
||||
pwsh -NoProfile -File .\test\run-hardware-e2e.ps1
|
||||
pwsh -NoProfile -File .\test\run-hardware-e2e.ps1 -ModelMode s2s
|
||||
pwsh -NoProfile -File .\test\run-hardware-e2e.ps1 -ExecutablePath .\release\hardware-agent-tester-0.3.5-windows-x64.exe
|
||||
```
|
||||
|
||||
第一条验证通用 LLM 报文，第二条验证端到端语音报文。回归会临时创建 `low_load` 设备绑定，启动一个校验真实火山签名和事件格式的本机 WebSocket 网关，再启动实际 Electron 窗口。测试只有在以下证据全部成立时才返回 PASS：
|
||||
|
||||
- 动态上下文确认后才开放麦克风；
|
||||
- PCM 上行和音频下行均发生；
|
||||
- 用户与智能体各产生一条最终字幕并写入 PostgreSQL；
|
||||
- 平台下发停止状态，Windows 程序通过下一次 `session_updated` 收到 `stop_session`；
|
||||
- 会话、硬件会话均为 `COMPLETED`；
|
||||
- SaaS `conversation.started` 和 `conversation.completed` 事件均已产生；
|
||||
- 页面 13 项验收门禁全部通过。
|
||||
|
||||
测试结束会删除临时会话、事件和设备绑定，不污染日常开发数据。
|
||||
|
||||
## 安全边界
|
||||
|
||||
- RTC AppKey、火山AK/SK和豆包Access Token只存在本地后端的受保护文件中；
|
||||
- 模拟器安装包只接收平台签发的短期RTC Token；
|
||||
- 设备签名密钥只驻留Electron主进程内存，不写入浏览器存储；
|
||||
- 当前测试构建只允许设备API地址指向`localhost`、`127.0.0.1`或固定测试服务器`https://121.41.46.238`，避免误连其他环境；
|
||||
- 关闭软件或结束通话时停止音频采集并销毁RTC引擎。
|
||||
- 未取得`record_near`或`record_far`授权的轨道不会注册PCM回调；`upload_allowed=false`时不会采集或上传任何原录音。
|
||||
- Product Secret 只存在于平台KMS/Secret挂载和注册调用内存，不到达设备，不写入数据库明文、日志或报告。
|
||||
- 平台返回的本设备Device Secret使用 Electron `safeStorage` 加密后保存在当前 Windows 用户目录，不进入源码和安装包；License由火山产品侧分配，不下发Windows程序。
|
||||
- 平台设备密钥只驻留 Electron 主进程，渲染页面不能读取已加载的明文。
|
||||
- 记忆工具地址必须与配置的平台 API 同源，防止短期令牌被发送给第三方地址。
|
||||
- 默认只允许连接火山官方硬件网关；本机 `ws://127.0.0.1` 仅供自动回归。
|
||||
- 平台确认 `session.updated` 前不会启动麦克风；收到 `stop_session` 后不会继续发送音频。
|
||||
- 导出的报告包含设备标识、Bot、配置摘要、事件和字幕，但不包含 Product Secret、Device Secret、平台签名密钥或记忆工具令牌。
|
||||
- 自动回归通过仅在`SIMULATOR_AUTO_TEST=1`时生效的进程环境注入mock一机一密，不写入正式Windows凭据缓存。
|
||||
|
||||
## 实时字幕前提
|
||||
## 真实验收与本地模拟的区别
|
||||
|
||||
测试所用RTC AppID必须在火山RTC控制台开启“实时字幕”和“流式语音识别”。流式语音识别凭据来自豆包语音控制台创建的“流式语音识别”应用，包括APP ID、Access Token和Cluster ID；“音视频字幕生成”类型不能替代该应用。未开通时程序会显示`ServiceAccessDenied`并清理本次未激活通话。
|
||||
### 0.3.5 播放速度修复
|
||||
|
||||
最终字幕会在程序的“实时字幕与上传”区域显示。每条最终字幕使用通话内稳定键和严格递增序号上传；网络失败时留在内存队列，并在心跳及结束通话前重试。存在未上传字幕时，程序不会结束并丢弃当前通话。
|
||||
- 根据火山 low-load WebSocket SDK 示例，将智能体下行 PCM 的源采样率从错误的 24kHz 修正为 16kHz，避免 1.5 倍速和音调升高。
|
||||
- Windows 输出设备不是 16kHz 时，由 AudioWorklet 按实际系统输出采样率连续重采样，不改变语速和音高。
|
||||
|
||||
完整本地启动顺序见[本地真实RTC联调说明](../deploy/local-rtc/README.md)。
|
||||
### 0.3.4 连续音频播放修复
|
||||
|
||||
- 火山返回的 `pcm_s16le/16kHz/mono` 音频进入常驻 AudioWorklet 环形缓冲，不再按 WebSocket 分片创建独立播放器。
|
||||
- 默认预缓冲 120ms，网络瞬时抖动时自动重新缓冲，减少断音、金属音和语速漂移。
|
||||
- 连续处理跨分片的奇数字节，并在开始、结束和断流时做短淡入淡出以避免爆音。
|
||||
- 测试报告新增播放分片数、采样数和欠载次数。
|
||||
|
||||
### 0.3.3 失败会话修复
|
||||
|
||||
- 火山 `error` 事件与连接 Promise 的异常共用一个失败清理任务，同一轮仅提交一次 `session_failed`，保留首次错误。
|
||||
- 失败清理期间禁止发送文字或重复结束；延迟到达的事件不再恢复“通话中”。失败后重新检测即可开始新一轮。
|
||||
- 新通话清空上一轮验收结果、错误和计时；失败清理的关闭不再标为正常关闭。
|
||||
- `npm test` 包含执行实际渲染器代码的并发失败回归测试，不会连接线上平台或消耗火山额度。
|
||||
- 此修复不代表火山 `internal_error: StartVoiceChat failed` 的上游原因已解决。
|
||||
|
||||
本地自动回归证明 Windows 程序、平台控制面、数据库、PCM 管道和协议状态机完整可执行。最终真实火山验收仍必须使用实际 Instance/Product/Device/Bot/License 凭据，确认火山返回真实 ASR、LLM、TTS 音频；程序不会把本机模拟网关的 PASS 冒充真实供应商验收。
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
const crypto = require("node:crypto");
|
||||
const WebSocket = require("ws");
|
||||
|
||||
const DEFAULT_GATEWAY_URL = "wss://ai-gateway.vei.volces.com/v1/realtime";
|
||||
|
||||
function requiredText(value, name, maximum = 512) {
|
||||
const normalized = String(value || "").trim();
|
||||
if (!normalized || normalized.length > maximum) throw new Error(`${name}不能为空或过长`);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function validateDeviceName(value) {
|
||||
const name = requiredText(value, "火山设备名称", 128);
|
||||
if (!/^[a-zA-Z0-9_\-.:@]{1,128}$/.test(name)) throw new Error("火山设备名称格式不正确");
|
||||
return name;
|
||||
}
|
||||
|
||||
function hmacBase64(secret, content) {
|
||||
return crypto.createHmac("sha256", secret).update(content).digest("base64");
|
||||
}
|
||||
|
||||
function websocketSignature({ authType = 1, deviceName, randomNumber, productKey, timestamp, instanceID }, deviceSecret) {
|
||||
const content = `auth_type=${authType}&device_name=${deviceName}&random_num=${randomNumber}&product_key=${productKey}×tamp=${timestamp}&instance_id=${instanceID}`;
|
||||
return hmacBase64(deviceSecret, content);
|
||||
}
|
||||
|
||||
function buildSessionPrompt(sessionConfig) {
|
||||
const sections = [
|
||||
requiredText(sessionConfig.authorized_context, "授权上下文", 4000),
|
||||
String(sessionConfig.system_prompt || "").trim(),
|
||||
String(sessionConfig.persona_tone || "").trim() ? `对话语气：${String(sessionConfig.persona_tone).trim()}` : "",
|
||||
`本次对话标识：${requiredText(sessionConfig.dialog_id, "对话标识", 200)}`,
|
||||
].filter(Boolean);
|
||||
return sections.join("\n\n");
|
||||
}
|
||||
|
||||
function buildSessionUpdate(sessionConfig, eventID, modelMode = "llm") {
|
||||
const normalizedMode = String(modelMode || "llm").trim().toLowerCase();
|
||||
if (!new Set(["llm", "s2s"]).has(normalizedMode)) throw new Error("智能体模型类型必须为 llm 或 s2s");
|
||||
const dialogID = requiredText(sessionConfig.dialog_id, "对话标识", 200);
|
||||
const systemPrompt = buildSessionPrompt(sessionConfig);
|
||||
const config = {
|
||||
ASRConfig: { TurnDetectionMode: 0 },
|
||||
};
|
||||
if (normalizedMode === "s2s") {
|
||||
config.S2SConfig = {
|
||||
ProviderParams: {
|
||||
dialog: {
|
||||
system_role: systemPrompt,
|
||||
speaking_style: String(sessionConfig.persona_tone || "").trim(),
|
||||
dialog_id: dialogID,
|
||||
},
|
||||
},
|
||||
};
|
||||
} else {
|
||||
config.LLMConfig = { SystemMessages: [systemPrompt] };
|
||||
}
|
||||
return {
|
||||
event_id: eventID,
|
||||
type: "session.update",
|
||||
session: {
|
||||
object: "realtime.session",
|
||||
model: "",
|
||||
input_audio_format: "pcm16",
|
||||
output_audio_format: "pcm16",
|
||||
config,
|
||||
agent_config: { EnableConversationStateCallback: false },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeProviderError(event) {
|
||||
const code = String(event?.error?.code || event?.error?.type || "PROVIDER_ERROR");
|
||||
const message = String(event?.error?.message || "火山硬件智能体返回错误");
|
||||
return new Error(`${code}: ${message}`);
|
||||
}
|
||||
|
||||
class HardwareAgentClient {
|
||||
constructor(options = {}) {
|
||||
this.WebSocketImpl = options.WebSocketImpl || WebSocket;
|
||||
this.gatewayURL = options.gatewayURL || DEFAULT_GATEWAY_URL;
|
||||
this.emit = typeof options.emit === "function" ? options.emit : () => {};
|
||||
this.socket = undefined;
|
||||
this.sessionCreated = undefined;
|
||||
this.providerConnectionRef = "";
|
||||
this.pendingCalls = new Map();
|
||||
this.pingTimer = undefined;
|
||||
this.connectionPromise = undefined;
|
||||
this.sessionUpdateSent = false;
|
||||
}
|
||||
|
||||
async connect(input) {
|
||||
if (this.socket) throw new Error("火山硬件智能体连接已经存在");
|
||||
const instanceID = requiredText(input.instanceID, "实例 ID", 200);
|
||||
const productKey = requiredText(input.productKey, "Product Key", 200);
|
||||
const deviceName = validateDeviceName(input.deviceName);
|
||||
const deviceSecret = requiredText(input.deviceSecret, "Device Secret", 4096);
|
||||
const hardwareID = requiredText(input.hardwareID, "Hardware ID", 200);
|
||||
const botID = requiredText(input.botID, "Bot ID", 200);
|
||||
const randomNumber = crypto.randomInt(1, 2_147_483_647);
|
||||
const timestamp = Math.floor(Date.now() / 1000);
|
||||
const signature = websocketSignature({ deviceName, randomNumber, productKey, timestamp, instanceID }, deviceSecret);
|
||||
const target = new URL(this.gatewayURL);
|
||||
target.searchParams.set("bot", botID);
|
||||
target.searchParams.set("wait_for_session_update", "true");
|
||||
const headers = {
|
||||
"X-Auth-Type": "1",
|
||||
"X-Product-Key": productKey,
|
||||
"X-Device-Name": deviceName,
|
||||
"X-Random-Num": String(randomNumber),
|
||||
"X-Timestamp": String(timestamp),
|
||||
"X-Instance-Id": instanceID,
|
||||
"X-Signature": signature,
|
||||
"X-Hardware-Id": hardwareID,
|
||||
};
|
||||
const socket = new this.WebSocketImpl(target.toString(), { headers, handshakeTimeout: 10_000, maxPayload: 8 * 1024 * 1024 });
|
||||
this.socket = socket;
|
||||
this.connectionPromise = new Promise((resolve, reject) => {
|
||||
const timeout = setTimeout(() => reject(new Error("等待火山 session.updated 超时")), 15_000);
|
||||
const finish = (callback, value) => {
|
||||
clearTimeout(timeout);
|
||||
callback(value);
|
||||
};
|
||||
socket.once("open", () => {
|
||||
this.emit({ type: "transport.open" });
|
||||
// wait_for_session_update=true makes the gateway wait for the client
|
||||
// configuration before it publishes the usable session. Sending only
|
||||
// after session.created deadlocks until the provider-side timeout.
|
||||
this.sendJSON(buildSessionUpdate(input.sessionConfig, `windows-session-update-${crypto.randomUUID()}`, input.modelMode));
|
||||
this.sessionUpdateSent = true;
|
||||
});
|
||||
socket.on("message", (data) => {
|
||||
let event;
|
||||
try { event = JSON.parse(Buffer.isBuffer(data) ? data.toString("utf8") : String(data)); }
|
||||
catch { this.emit({ type: "transport.invalid_message" }); return; }
|
||||
this.handleProviderEvent(event, input.sessionConfig, input.modelMode);
|
||||
// In the hardware gateway's wait_for_session_update mode, the initial
|
||||
// session.created event is published only after the pre-creation
|
||||
// session.update has been accepted. Some Bot versions do not emit an
|
||||
// additional session.updated event for that initial configuration.
|
||||
if (event.type === "session.updated" || (event.type === "session.created" && this.sessionUpdateSent)) {
|
||||
const connectionRef = String(event?.session?.id || this.providerConnectionRef || event.event_id || "").trim();
|
||||
if (!connectionRef) return finish(reject, new Error("火山会话确认未提供可关联的会话编号"));
|
||||
this.providerConnectionRef = connectionRef;
|
||||
finish(resolve, { providerConnectionRef: connectionRef, session: event.session || {} });
|
||||
} else if (event.type === "error" && !this.providerConnectionRef) {
|
||||
finish(reject, normalizeProviderError(event));
|
||||
}
|
||||
});
|
||||
socket.once("unexpected-response", (_request, response) => {
|
||||
finish(reject, new Error(`火山 WebSocket 鉴权失败：HTTP ${response.statusCode}`));
|
||||
});
|
||||
socket.once("error", (error) => {
|
||||
this.emit({ type: "transport.error", message: error.message });
|
||||
if (!this.providerConnectionRef) finish(reject, error);
|
||||
});
|
||||
socket.once("close", (code, reason) => {
|
||||
this.stopPing();
|
||||
this.socket = undefined;
|
||||
this.emit({ type: "transport.close", code, reason: Buffer.from(reason || "").toString("utf8") });
|
||||
if (!this.providerConnectionRef) finish(reject, new Error(`火山连接提前关闭：${code}`));
|
||||
});
|
||||
});
|
||||
try {
|
||||
const result = await this.connectionPromise;
|
||||
this.startPing();
|
||||
return result;
|
||||
} catch (error) {
|
||||
this.close(1011, "connection setup failed");
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
handleProviderEvent(event, sessionConfig, modelMode) {
|
||||
if (event.type === "session.created") {
|
||||
this.sessionCreated = event.session || {};
|
||||
this.providerConnectionRef = String(event?.session?.id || event.event_id || "").trim();
|
||||
}
|
||||
if (event.type === "conversation.item.created" && event?.item?.type === "function_call" && event.item.call_id) {
|
||||
this.pendingCalls.set(String(event.item.call_id), String(event.item.name || ""));
|
||||
}
|
||||
this.emit(event);
|
||||
}
|
||||
|
||||
sendAudio(bytes) {
|
||||
const buffer = Buffer.from(bytes);
|
||||
if (!buffer.length) return;
|
||||
if (buffer.length > 64 * 1024) throw new Error("单个音频分片不能超过64KiB");
|
||||
this.sendJSON({
|
||||
event_id: `windows-audio-${crypto.randomUUID()}`,
|
||||
type: "input_audio_buffer.append",
|
||||
audio: buffer.toString("base64"),
|
||||
});
|
||||
}
|
||||
|
||||
sendText(text, interruptMode = 1) {
|
||||
const normalized = requiredText(text, "文本问题", 2000);
|
||||
this.sendJSON({
|
||||
event_id: `windows-text-${crypto.randomUUID()}`,
|
||||
type: "conversation.item.create",
|
||||
item: { type: "message", role: "user", content: [{ type: "input_text", text: normalized }], interrupt_mode: interruptMode },
|
||||
});
|
||||
}
|
||||
|
||||
completeFunctionCall(callID, output) {
|
||||
const normalizedCallID = requiredText(callID, "函数调用编号", 200);
|
||||
this.sendJSON({
|
||||
event_id: `windows-function-${crypto.randomUUID()}`,
|
||||
type: "conversation.item.create",
|
||||
item: { type: "function_call_output", call_id: normalizedCallID, output: JSON.stringify(output) },
|
||||
});
|
||||
this.sendJSON({ event_id: `windows-response-${crypto.randomUUID()}`, type: "response.create" });
|
||||
this.pendingCalls.delete(normalizedCallID);
|
||||
}
|
||||
|
||||
sendJSON(value) {
|
||||
if (!this.socket || this.socket.readyState !== this.WebSocketImpl.OPEN) throw new Error("火山硬件智能体连接未就绪");
|
||||
this.socket.send(JSON.stringify(value));
|
||||
}
|
||||
|
||||
startPing() {
|
||||
this.stopPing();
|
||||
this.pingTimer = setInterval(() => {
|
||||
try { if (this.socket?.readyState === this.WebSocketImpl.OPEN && typeof this.socket.ping === "function") this.socket.ping(); }
|
||||
catch { /* close/error callbacks own recovery */ }
|
||||
}, 20_000);
|
||||
}
|
||||
|
||||
stopPing() {
|
||||
if (this.pingTimer) clearInterval(this.pingTimer);
|
||||
this.pingTimer = undefined;
|
||||
}
|
||||
|
||||
close(code = 1000, reason = "device session ended") {
|
||||
this.stopPing();
|
||||
const socket = this.socket;
|
||||
this.socket = undefined;
|
||||
this.providerConnectionRef = "";
|
||||
this.sessionUpdateSent = false;
|
||||
this.pendingCalls.clear();
|
||||
if (!socket) return;
|
||||
try { socket.close(code, reason); } catch { try { socket.terminate?.(); } catch { /* no-op */ } }
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
DEFAULT_GATEWAY_URL,
|
||||
HardwareAgentClient,
|
||||
buildSessionPrompt,
|
||||
buildSessionUpdate,
|
||||
websocketSignature,
|
||||
};
|
||||
@@ -0,0 +1,97 @@
|
||||
const assert = require("node:assert/strict");
|
||||
const { EventEmitter } = require("node:events");
|
||||
const test = require("node:test");
|
||||
const {
|
||||
HardwareAgentClient,
|
||||
buildSessionUpdate,
|
||||
websocketSignature,
|
||||
} = require("./hardware-agent-client.cjs");
|
||||
|
||||
test("WebSocket signatures use the documented canonical field order", () => {
|
||||
const connection = websocketSignature({ authType: 1, deviceName: "device-1", randomNumber: 42, productKey: "product-1", timestamp: 1700000000, instanceID: "instance-1" }, "device-secret");
|
||||
assert.equal(connection, "6OFE1DUZzRgNy0o5IzZSmqKw3SUP4s3GxK2YHe8phfo=");
|
||||
});
|
||||
|
||||
test("LLM session update uses the official Config/AgentConfig schema", () => {
|
||||
const update = buildSessionUpdate({ dialog_id: "conversation-1", system_prompt: "你是陪伴助手", persona_tone: "亲切", authorized_context: "只可使用已授权背景" }, "event-1");
|
||||
assert.equal(update.type, "session.update");
|
||||
assert.equal(update.session.object, "realtime.session");
|
||||
assert.equal(update.session.input_audio_format, "pcm16");
|
||||
assert.equal(update.session.output_audio_format, "pcm16");
|
||||
assert.equal(update.session.config.ASRConfig.TurnDetectionMode, 0);
|
||||
assert.match(update.session.config.LLMConfig.SystemMessages[0], /只可使用已授权背景/);
|
||||
assert.match(update.session.config.LLMConfig.SystemMessages[0], /你是陪伴助手/);
|
||||
assert.match(update.session.config.LLMConfig.SystemMessages[0], /亲切/);
|
||||
assert.equal(update.session.agent_config.EnableConversationStateCallback, false);
|
||||
assert.equal("system_prompt" in update.session, false);
|
||||
});
|
||||
|
||||
test("S2S session update maps persona and dialog id to provider dialog config", () => {
|
||||
const update = buildSessionUpdate({ dialog_id: "conversation-2", system_prompt: "你是陪伴助手", persona_tone: "亲切自然", authorized_context: "只使用授权信息" }, "event-2", "s2s");
|
||||
const dialog = update.session.config.S2SConfig.ProviderParams.dialog;
|
||||
assert.equal(dialog.dialog_id, "conversation-2");
|
||||
assert.equal(dialog.speaking_style, "亲切自然");
|
||||
assert.match(dialog.system_role, /只使用授权信息/);
|
||||
assert.match(dialog.system_role, /你是陪伴助手/);
|
||||
assert.equal("LLMConfig" in update.session.config, false);
|
||||
});
|
||||
|
||||
class FakeWebSocket extends EventEmitter {
|
||||
static OPEN = 1;
|
||||
constructor(url, options) {
|
||||
super();
|
||||
this.url = url;
|
||||
this.options = options;
|
||||
this.readyState = 0;
|
||||
this.sent = [];
|
||||
FakeWebSocket.last = this;
|
||||
setImmediate(() => {
|
||||
this.readyState = FakeWebSocket.OPEN;
|
||||
this.emit("open");
|
||||
this.emit("message", Buffer.from(JSON.stringify({ type: "session.created", event_id: "created-event", session: { id: "provider-session-1" } })));
|
||||
});
|
||||
}
|
||||
send(body) {
|
||||
const value = JSON.parse(body);
|
||||
this.sent.push(value);
|
||||
if (value.type === "session.update") {
|
||||
setImmediate(() => this.emit("message", Buffer.from(JSON.stringify({ type: "session.updated", event_id: "updated-event", session: { id: "provider-session-1" } }))));
|
||||
}
|
||||
}
|
||||
ping() {}
|
||||
close(code, reason) {
|
||||
this.readyState = 3;
|
||||
this.emit("close", code, Buffer.from(reason));
|
||||
}
|
||||
}
|
||||
|
||||
test("hardware client waits for dynamic config acknowledgement before becoming ready", async () => {
|
||||
const events = [];
|
||||
const client = new HardwareAgentClient({ WebSocketImpl: FakeWebSocket, gatewayURL: "ws://127.0.0.1:19090/v1/realtime", emit: (event) => events.push(event) });
|
||||
const connected = await client.connect({
|
||||
instanceID: "instance-1", productKey: "product-1", deviceName: "device-1",
|
||||
deviceSecret: "device-secret-with-at-least-32-characters", hardwareID: "windows-test", botID: "bot-1",
|
||||
sessionConfig: { dialog_id: "conversation-1", authorized_context: "authorized" },
|
||||
});
|
||||
assert.equal(connected.providerConnectionRef, "provider-session-1");
|
||||
assert.match(FakeWebSocket.last.url, /bot=bot-1/);
|
||||
assert.equal(FakeWebSocket.last.options.headers["X-Product-Key"], "product-1");
|
||||
assert.equal(FakeWebSocket.last.sent[0].type, "session.update");
|
||||
client.sendAudio(Buffer.from([1, 2, 3, 4]));
|
||||
client.sendText("你好");
|
||||
assert.equal(FakeWebSocket.last.sent[1].type, "input_audio_buffer.append");
|
||||
assert.equal(FakeWebSocket.last.sent[2].type, "conversation.item.create");
|
||||
assert.ok(events.some((event) => event.type === "session.created"));
|
||||
client.close();
|
||||
});
|
||||
|
||||
test("hardware client sends session.update immediately when the WebSocket opens", async () => {
|
||||
const client = new HardwareAgentClient({ WebSocketImpl: FakeWebSocket, gatewayURL: "ws://127.0.0.1:19090/v1/realtime" });
|
||||
await client.connect({
|
||||
instanceID: "instance-1", productKey: "product-1", deviceName: "device-1",
|
||||
deviceSecret: "device-secret-with-at-least-32-characters", hardwareID: "windows-test", botID: "bot-1",
|
||||
sessionConfig: { dialog_id: "conversation-1", authorized_context: "authorized" },
|
||||
});
|
||||
assert.equal(FakeWebSocket.last.sent.filter((event) => event.type === "session.update").length, 1);
|
||||
client.close();
|
||||
});
|
||||
@@ -0,0 +1,153 @@
|
||||
const assert = require("node:assert/strict");
|
||||
const test = require("node:test");
|
||||
const fs = require("node:fs");
|
||||
const path = require("node:path");
|
||||
const vm = require("node:vm");
|
||||
const ts = require("typescript");
|
||||
|
||||
// Execute the actual renderer with a small DOM/IPC fixture; no network or audio.
|
||||
const source = fs.readFileSync(path.join(__dirname, "../src/hardware.ts"), "utf8");
|
||||
const compiled = ts.transpileModule(source, {
|
||||
compilerOptions: { target: ts.ScriptTarget.ES2022, module: ts.ModuleKind.CommonJS },
|
||||
}).outputText;
|
||||
|
||||
function fixture(overrides = {}) {
|
||||
const elements = new Map();
|
||||
const element = () => ({
|
||||
style: {}, value: "", textContent: "", disabled: false, children: [],
|
||||
append(...items) { this.children.push(...items); },
|
||||
prepend(...items) { this.children.unshift(...items); },
|
||||
replaceChildren(...items) { this.children = items; },
|
||||
querySelector() { return null; }, addEventListener() {},
|
||||
});
|
||||
const get = (id) => {
|
||||
if (!elements.has(id)) elements.set(id, element());
|
||||
return elements.get(id);
|
||||
};
|
||||
const events = [];
|
||||
const closes = [];
|
||||
const texts = [];
|
||||
const created = {
|
||||
conversation_id: "test-conversation", session: {},
|
||||
runtime: { transport: "low_load", config_digest: "digest", session_nonce: "nonce", provider_session_ref: "ref", bot_id: "test-bot" },
|
||||
session_config: { authorized_context: "test", prompt_bundle_version: "test-v1" },
|
||||
};
|
||||
const ipc = {
|
||||
getRuntimeDefaults: async () => ({}), getHardwareRuntimeDefaults: async () => ({}),
|
||||
onHardwareProviderEvent: () => () => {},
|
||||
createHardwareAgentSession: async () => created,
|
||||
closeHardwareAgent: async (input) => { closes.push(input); },
|
||||
hardwareAgentEvent: async (input) => { events.push(input); return { status: "FAILED" }; },
|
||||
sendHardwareAgentText: async (input) => { texts.push(input); },
|
||||
...overrides,
|
||||
};
|
||||
const context = vm.createContext({
|
||||
exports: {}, require: () => ({}), console, URL,
|
||||
document: { querySelector: () => get("app"), getElementById: get, createElement: element },
|
||||
window: { deviceSimulator: ipc, addEventListener() {}, clearInterval() {}, setTimeout, setInterval() {} },
|
||||
});
|
||||
vm.runInContext(compiled + `\nglobalThis.renderer = { state, controls, setPhase, setCheck, resetSessionState, failHardwareConversation, stopHardwareConversation, startHardwareConversation, handleProviderEvent, sendTextQuestion, buildReport };`, context);
|
||||
return { r: context.renderer, events, closes, texts, created, get, ipc };
|
||||
}
|
||||
|
||||
test("provider error plus connect rejection sends one failure, including late duplicates", async () => {
|
||||
const f = fixture();
|
||||
const r = f.r;
|
||||
f.ipc.connectHardwareAgent = async () => {
|
||||
void r.handleProviderEvent({ type: "error", error: { code: "internal_error", message: "StartVoiceChat failed" } });
|
||||
throw new Error("wrapped connect rejection");
|
||||
};
|
||||
r.setPhase("ready", "ready");
|
||||
await r.startHardwareConversation();
|
||||
await r.failHardwareConversation("SECOND_FAILURE");
|
||||
await r.stopHardwareConversation();
|
||||
await r.handleProviderEvent({ type: "error", error: { message: "late error" } });
|
||||
assert.equal(f.events.length, 1);
|
||||
assert.equal(f.events[0].eventType, "session_failed");
|
||||
assert.equal(f.events[0].errorCode, "VOLC_HARDWARE_AGENT_ERROR");
|
||||
assert.equal(f.closes.length, 1);
|
||||
assert.equal(r.state.failure, "internal_error: StartVoiceChat failed");
|
||||
assert.equal(r.state.phase, "error");
|
||||
assert.equal(r.controls.stop.disabled, true);
|
||||
assert.equal(r.controls.check.disabled, false);
|
||||
});
|
||||
|
||||
test("concurrent failure callers share pending cleanup; late events cannot reactivate", async () => {
|
||||
let release;
|
||||
const gate = new Promise((resolve) => { release = resolve; });
|
||||
const f = fixture({ closeHardwareAgent: () => gate });
|
||||
f.r.state.platformSession = f.created;
|
||||
f.r.state.failure = "first cause";
|
||||
f.r.setPhase("active", "active");
|
||||
const first = f.r.failHardwareConversation("FIRST");
|
||||
const second = f.r.failHardwareConversation("SECOND");
|
||||
assert.equal(first, second);
|
||||
assert.equal(f.r.state.phase, "stopping");
|
||||
await f.r.handleProviderEvent({ type: "response.done" });
|
||||
assert.equal(f.r.state.phase, "stopping");
|
||||
release();
|
||||
await first;
|
||||
assert.equal(f.events.length, 1);
|
||||
assert.equal(f.events[0].errorCode, "FIRST");
|
||||
await f.r.handleProviderEvent({ type: "transport.close", code: 1011 });
|
||||
assert.equal(f.r.state.checks.get("provider_closed").status, "failed");
|
||||
f.r.controls.textQuestion.value = "should not send";
|
||||
await f.r.sendTextQuestion();
|
||||
assert.equal(f.texts.length, 0);
|
||||
});
|
||||
|
||||
test("new call clears stale checks, duration and failure while retaining device preparation", async () => {
|
||||
const f = fixture();
|
||||
f.r.state.platformSession = f.created;
|
||||
f.r.state.failure = "old failure";
|
||||
await f.r.failHardwareConversation("OLD");
|
||||
for (const id of f.r.state.checks.keys()) f.r.setCheck(id, "passed", "old");
|
||||
f.get("duration").textContent = "48:15";
|
||||
f.r.resetSessionState();
|
||||
assert.equal(f.r.state.failure, undefined);
|
||||
assert.equal(f.r.state.failureTask, undefined);
|
||||
assert.equal(f.get("duration").textContent, "00:00");
|
||||
assert.equal(f.r.state.checks.get("platform_authenticated").status, "passed");
|
||||
assert.equal(f.r.state.checks.get("credential_resolved").status, "passed");
|
||||
assert.equal(f.r.state.checks.get("platform_completed").status, "pending");
|
||||
f.r.state.platformSession = { ...f.created, conversation_id: "next-conversation" };
|
||||
await f.r.failHardwareConversation("NEW");
|
||||
assert.equal(f.events.length, 2);
|
||||
assert.equal(f.events[1].conversationID, "next-conversation");
|
||||
assert.equal(f.events[1].sequenceNo, 1);
|
||||
});
|
||||
|
||||
test("failed platform receipt does not trigger a new terminal event on hangup", async () => {
|
||||
let attempts = 0;
|
||||
const f = fixture({ hardwareAgentEvent: async () => { attempts += 1; throw new Error("network unavailable"); } });
|
||||
f.r.state.platformSession = f.created;
|
||||
await f.r.failHardwareConversation("FIRST");
|
||||
await f.r.stopHardwareConversation();
|
||||
assert.equal(attempts, 1);
|
||||
assert.equal(f.r.state.phase, "error");
|
||||
assert.equal(f.r.buildReport().checks.find((c) => c.id === "platform_completed").status, "failed");
|
||||
});
|
||||
|
||||
test("connection rejection without a provider event still reports failure once", async () => {
|
||||
const f = fixture({ connectHardwareAgent: async () => { throw new Error("connect timeout"); } });
|
||||
f.r.setPhase("ready", "ready");
|
||||
await f.r.startHardwareConversation();
|
||||
assert.equal(f.events.length, 1);
|
||||
assert.equal(f.events[0].errorCode, "WINDOWS_HARDWARE_START_FAILED");
|
||||
assert.match(f.r.state.failure, /connect timeout/);
|
||||
assert.equal(f.r.state.phase, "error");
|
||||
});
|
||||
|
||||
test("normal hangup still reports completed and closes normally", async () => {
|
||||
const f = fixture();
|
||||
f.ipc.hardwareAgentEvent = async (input) => { f.events.push(input); return { status: "COMPLETED" }; };
|
||||
f.r.state.platformSession = f.created;
|
||||
f.r.setPhase("active", "active");
|
||||
await f.r.stopHardwareConversation();
|
||||
assert.equal(f.events.length, 1);
|
||||
assert.equal(f.events[0].eventType, "session_ended");
|
||||
assert.equal(f.closes[0].code, 1000);
|
||||
assert.equal(f.r.state.phase, "completed");
|
||||
assert.equal(f.r.state.failureTask, undefined);
|
||||
assert.equal(f.r.controls.start.disabled, false);
|
||||
});
|
||||
@@ -1,14 +1,24 @@
|
||||
const { app, BrowserWindow, ipcMain, session } = require("electron");
|
||||
const { app, BrowserWindow, dialog, ipcMain, safeStorage, session } = require("electron");
|
||||
const crypto = require("node:crypto");
|
||||
const fs = require("node:fs");
|
||||
const path = require("node:path");
|
||||
const tls = require("node:tls");
|
||||
const { HardwareAgentClient } = require("./hardware-agent-client.cjs");
|
||||
|
||||
app.commandLine.appendSwitch("autoplay-policy", "no-user-gesture-required");
|
||||
const automatedUI = process.env.SIMULATOR_AUTO_TEST === "1" || process.env.SIMULATOR_REAL_E2E === "1";
|
||||
if (automatedUI) {
|
||||
app.commandLine.appendSwitch("use-fake-device-for-media-stream");
|
||||
app.commandLine.appendSwitch("use-fake-ui-for-media-stream");
|
||||
}
|
||||
app.disableHardwareAcceleration();
|
||||
|
||||
let deviceConfig = null;
|
||||
let desiredConfigRevision = 1;
|
||||
let mainWindow = null;
|
||||
let hardwareConfig = null;
|
||||
let hardwareClient = null;
|
||||
let activeHardwarePlatformSession = null;
|
||||
|
||||
function createWindow() {
|
||||
const window = new BrowserWindow({
|
||||
@@ -26,6 +36,10 @@ function createWindow() {
|
||||
sandbox: true,
|
||||
},
|
||||
});
|
||||
mainWindow = window;
|
||||
window.once("closed", () => {
|
||||
if (mainWindow === window) mainWindow = null;
|
||||
});
|
||||
|
||||
const devArgument = process.argv.find((argument) => argument.startsWith("--dev-server-url="));
|
||||
const devServerURL = devArgument ? devArgument.slice("--dev-server-url=".length) : "";
|
||||
@@ -47,7 +61,7 @@ function createWindow() {
|
||||
} catch (error) {
|
||||
console.error("renderer diagnostics failed", error);
|
||||
}
|
||||
if (process.env.SIMULATOR_AUTO_TEST === "1") {
|
||||
if (automatedUI) {
|
||||
void runAutomatedSmoke(window);
|
||||
}
|
||||
});
|
||||
@@ -62,9 +76,15 @@ function createWindow() {
|
||||
if (screenshotPath) {
|
||||
window.webContents.once("did-finish-load", () => {
|
||||
setTimeout(async () => {
|
||||
const image = await window.webContents.capturePage();
|
||||
fs.writeFileSync(screenshotPath, image.toPNG());
|
||||
app.quit();
|
||||
try {
|
||||
const image = await window.webContents.capturePage();
|
||||
fs.writeFileSync(screenshotPath, image.toPNG());
|
||||
} catch (error) {
|
||||
console.error("simulator screenshot failed", error);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
app.quit();
|
||||
}
|
||||
}, 1200);
|
||||
});
|
||||
}
|
||||
@@ -95,24 +115,54 @@ async function runAutomatedSmoke(window) {
|
||||
async function collectAutomatedConversationResult(window) {
|
||||
const delay = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||
const deadline = Date.now() + 120_000;
|
||||
let realQuestionSent = false;
|
||||
let snapshot = {};
|
||||
while (Date.now() < deadline) {
|
||||
await delay(1000);
|
||||
const encoded = await window.webContents.executeJavaScript(`JSON.stringify({
|
||||
phase: document.querySelector('#phaseBadge')?.textContent || '',
|
||||
conversation_id: document.querySelector('#conversationID')?.textContent || '',
|
||||
model_mode: document.querySelector('#modelMode')?.value || '',
|
||||
subject: [...document.querySelectorAll('.transcript-turn--subject span')].map((item) => item.textContent || ''),
|
||||
agent: [...document.querySelectorAll('.transcript-turn--agent span')].map((item) => item.textContent || ''),
|
||||
turns: [...document.querySelectorAll('.transcript-turn')].map((item) => ({speaker: item.classList.contains('transcript-turn--subject') ? 'subject' : 'agent', text: item.querySelector('span')?.textContent || ''})),
|
||||
events: [...document.querySelectorAll('#eventLog li span')].map((item) => item.textContent || '')
|
||||
events: [...document.querySelectorAll('#eventLog li span')].map((item) => item.textContent || ''),
|
||||
checks: [...document.querySelectorAll('#flowChecks .flow-check')].map((item) => ({label: item.querySelector('span')?.textContent || '', detail: item.querySelector('small')?.textContent || '', status: item.classList.contains('flow-check--passed') ? 'passed' : item.classList.contains('flow-check--failed') ? 'failed' : 'pending'}))
|
||||
})`);
|
||||
snapshot = JSON.parse(encoded);
|
||||
console.log(`[auto-test] conversation ${encoded}`);
|
||||
if (process.env.SIMULATOR_REAL_E2E === "1" && snapshot.phase === "通话中" && !realQuestionSent) {
|
||||
const question = String(process.env.SIMULATOR_REAL_E2E_QUESTION || "请用一句简短的话介绍你自己。 ").trim();
|
||||
await window.webContents.executeJavaScript(`(() => { const input = document.querySelector('#textQuestion'); if (input) input.value = ${JSON.stringify(question)}; document.querySelector('#sendTextButton')?.click(); })()`);
|
||||
realQuestionSent = true;
|
||||
console.log("[real-e2e] text question sent");
|
||||
}
|
||||
const subjectIndex = snapshot.turns?.findIndex((turn) => turn.speaker === "subject") ?? -1;
|
||||
const hasAgentReply = subjectIndex >= 0 && snapshot.turns.slice(subjectIndex + 1).some((turn) => turn.speaker === "agent");
|
||||
if (hasAgentReply) {
|
||||
await window.webContents.executeJavaScript(`document.querySelector('#stopButton')?.click()`);
|
||||
snapshot.result = "PASS";
|
||||
const hardwarePage = await window.webContents.executeJavaScript(`Boolean(document.querySelector('#flowChecks'))`);
|
||||
const waitForPlatformStop = hardwarePage && process.env.SIMULATOR_AUTO_WAIT_FOR_PLATFORM_STOP === "1";
|
||||
if (!waitForPlatformStop) await window.webContents.executeJavaScript(`document.querySelector('#stopButton')?.click()`);
|
||||
if (hardwarePage) {
|
||||
const stopDeadline = Date.now() + 30_000;
|
||||
while (Date.now() < stopDeadline) {
|
||||
await delay(500);
|
||||
const phase = await window.webContents.executeJavaScript(`document.querySelector('#phaseBadge')?.textContent || ''`);
|
||||
if (phase === "已完成" || phase === "异常") break;
|
||||
}
|
||||
const finalEncoded = await window.webContents.executeJavaScript(`JSON.stringify({
|
||||
phase: document.querySelector('#phaseBadge')?.textContent || '',
|
||||
conversation_id: document.querySelector('#conversationID')?.textContent || '',
|
||||
model_mode: document.querySelector('#modelMode')?.value || '',
|
||||
turns: [...document.querySelectorAll('.transcript-turn')].map((item) => ({speaker: item.classList.contains('transcript-turn--subject') ? 'subject' : 'agent', text: item.querySelector('span')?.textContent || ''})),
|
||||
events: [...document.querySelectorAll('#eventLog li span')].map((item) => item.textContent || ''),
|
||||
checks: [...document.querySelectorAll('#flowChecks .flow-check')].map((item) => ({label: item.querySelector('span')?.textContent || '', detail: item.querySelector('small')?.textContent || '', status: item.classList.contains('flow-check--passed') ? 'passed' : item.classList.contains('flow-check--failed') ? 'failed' : 'pending'}))
|
||||
})`);
|
||||
snapshot = JSON.parse(finalEncoded);
|
||||
snapshot.result = snapshot.phase === "已完成" ? "PASS" : "ERROR";
|
||||
} else {
|
||||
snapshot.result = "PASS";
|
||||
}
|
||||
break;
|
||||
}
|
||||
if (snapshot.phase === "异常") {
|
||||
@@ -160,7 +210,10 @@ function loadDefaultDeviceSecret() {
|
||||
const localAppData = String(process.env.LOCALAPPDATA || "").trim();
|
||||
const userProfile = String(process.env.USERPROFILE || "").trim();
|
||||
const portableDirectory = String(process.env.PORTABLE_EXECUTABLE_DIR || "").trim();
|
||||
if (localAppData) candidates.push(path.join(localAppData, "DataTerminalVoiceDevice", "server-device-secret"));
|
||||
if (localAppData) {
|
||||
candidates.push(path.join(localAppData, "DataTerminalHardwareAgent", "server-device-secret"));
|
||||
candidates.push(path.join(localAppData, "DataTerminalVoiceDevice", "server-device-secret"));
|
||||
}
|
||||
if (userProfile) candidates.push(path.join(userProfile, "AppData", "Local", "DataTerminalVoiceDevice", "server-device-secret"));
|
||||
try {
|
||||
const home = app.getPath("home");
|
||||
@@ -182,11 +235,134 @@ function loadDefaultDeviceSecret() {
|
||||
|
||||
ipcMain.handle("sim:get-runtime-defaults", () => ({
|
||||
apiURL: process.env.SIMULATOR_DEFAULT_API_URL || "https://121.41.46.238",
|
||||
deviceID: process.env.SIMULATOR_DEFAULT_DEVICE_ID || "10000000-0000-0000-0000-000000000511",
|
||||
deviceID: process.env.SIMULATOR_DEFAULT_DEVICE_ID || "10000000-0000-0000-0000-000000000512",
|
||||
deviceSecret: "",
|
||||
deviceSecretConfigured: loadDefaultDeviceSecret().length >= 32,
|
||||
}));
|
||||
|
||||
function hardwareCredentialPath() {
|
||||
return path.join(app.getPath("userData"), "volc-hardware-device-credential.json");
|
||||
}
|
||||
|
||||
function readSavedHardwareMetadata() {
|
||||
try {
|
||||
const stored = JSON.parse(fs.readFileSync(hardwareCredentialPath(), "utf8"));
|
||||
if (!stored.instanceID || !stored.productKey || !stored.deviceName || !stored.encryptedDeviceSecret) return undefined;
|
||||
return stored;
|
||||
} catch { return undefined; }
|
||||
}
|
||||
|
||||
function loadSavedHardwareCredential(identity) {
|
||||
if (!safeStorage.isEncryptionAvailable()) return undefined;
|
||||
try {
|
||||
const stored = JSON.parse(fs.readFileSync(hardwareCredentialPath(), "utf8"));
|
||||
if (stored.schemaVersion !== 3 || stored.instanceID !== identity.instanceID || stored.productKey !== identity.productKey || stored.deviceName !== identity.deviceName) return undefined;
|
||||
const secret = safeStorage.decryptString(Buffer.from(stored.encryptedDeviceSecret, "base64"));
|
||||
if (secret.length < 16) return undefined;
|
||||
return { deviceSecret: secret, hardwareID: String(stored.hardwareID || identity.hardwareID || "") };
|
||||
} catch { return undefined; }
|
||||
}
|
||||
|
||||
function saveHardwareCredential(identity, deviceSecret, metadata = {}) {
|
||||
if (!safeStorage.isEncryptionAvailable()) return false;
|
||||
const target = hardwareCredentialPath();
|
||||
fs.mkdirSync(path.dirname(target), { recursive: true });
|
||||
const payload = {
|
||||
schemaVersion: 3,
|
||||
instanceID: identity.instanceID,
|
||||
productKey: identity.productKey,
|
||||
deviceName: identity.deviceName,
|
||||
hardwareID: identity.hardwareID,
|
||||
platformDeviceID: String(deviceConfig?.deviceID || ""),
|
||||
credentialRef: String(metadata.credentialRef || ""),
|
||||
credentialVersion: Number(metadata.credentialVersion || 1),
|
||||
encryptedDeviceSecret: safeStorage.encryptString(deviceSecret).toString("base64"),
|
||||
updatedAt: new Date().toISOString(),
|
||||
};
|
||||
const temporary = `${target}.${process.pid}.tmp`;
|
||||
fs.writeFileSync(temporary, `${JSON.stringify(payload, null, 2)}\n`, { encoding: "utf8", mode: 0o600 });
|
||||
fs.renameSync(temporary, target);
|
||||
return true;
|
||||
}
|
||||
|
||||
function hardwareIdentity(input) {
|
||||
const instanceID = String(input?.instanceID || "").trim();
|
||||
const productKey = String(input?.productKey || "").trim();
|
||||
const deviceName = String(input?.deviceName || "").trim();
|
||||
const hardwareID = String(input?.hardwareID || "").trim();
|
||||
if (!instanceID || instanceID.length > 200) throw new Error("实例 ID 不能为空或过长");
|
||||
if (!productKey || productKey.length > 200) throw new Error("Product Key 不能为空或过长");
|
||||
if (!/^[a-zA-Z0-9_\-.:@]{1,128}$/.test(deviceName)) throw new Error("火山设备名称格式不正确");
|
||||
if (!hardwareID || hardwareID.length > 200) throw new Error("Hardware ID 不能为空或过长");
|
||||
return { instanceID, productKey, deviceName, hardwareID };
|
||||
}
|
||||
|
||||
ipcMain.handle("sim:get-hardware-runtime-defaults", () => {
|
||||
const candidateMetadata = readSavedHardwareMetadata();
|
||||
const legacyMock = candidateMetadata && [candidateMetadata.instanceID, candidateMetadata.productKey, candidateMetadata.deviceName, candidateMetadata.hardwareID]
|
||||
.some((value) => String(value || "").toLowerCase().includes("mock") || String(value || "").toLowerCase().includes("e2e"));
|
||||
const savedMetadata = automatedUI || !legacyMock ? candidateMetadata : undefined;
|
||||
const instanceID = String(process.env.SIMULATOR_VOLC_INSTANCE_ID || savedMetadata?.instanceID || "").trim();
|
||||
const productKey = String(process.env.SIMULATOR_VOLC_PRODUCT_KEY || savedMetadata?.productKey || "").trim();
|
||||
const deviceName = String(process.env.SIMULATOR_VOLC_DEVICE_NAME || savedMetadata?.deviceName || "windows-hardware-agent-test").trim();
|
||||
const hardwareID = String(process.env.SIMULATOR_VOLC_HARDWARE_ID || savedMetadata?.hardwareID || `windows-${crypto.createHash("sha256").update(app.getPath("userData")).digest("hex").slice(0, 24)}`).trim();
|
||||
const modelMode = String(process.env.SIMULATOR_VOLC_MODEL_MODE || "llm").trim().toLowerCase() === "s2s" ? "s2s" : "llm";
|
||||
const saved = instanceID && productKey ? loadSavedHardwareCredential({ instanceID, productKey, deviceName, hardwareID }) : undefined;
|
||||
return { instanceID, productKey, deviceName, hardwareID, modelMode, deviceCredentialConfigured: Boolean(saved) };
|
||||
});
|
||||
|
||||
ipcMain.handle("sim:configure-hardware", async (_event, input) => {
|
||||
if (!deviceConfig) throw new Error("请先连接平台");
|
||||
const requestedHardwareID = String(input?.hardwareID || "").trim();
|
||||
if (!requestedHardwareID || requestedHardwareID.length > 200) throw new Error("Hardware ID 不能为空或过长");
|
||||
const automatedTest = process.env.SIMULATOR_AUTO_TEST === "1";
|
||||
let provisioned;
|
||||
if (automatedTest) {
|
||||
provisioned = {
|
||||
instance_id: String(process.env.SIMULATOR_VOLC_INSTANCE_ID || "").trim(),
|
||||
product_key: String(process.env.SIMULATOR_VOLC_PRODUCT_KEY || "").trim(),
|
||||
device_name: String(process.env.SIMULATOR_VOLC_DEVICE_NAME || "").trim(),
|
||||
hardware_id: String(process.env.SIMULATOR_VOLC_HARDWARE_ID || requestedHardwareID).trim(),
|
||||
device_secret: String(process.env.SIMULATOR_VOLC_DEVICE_SECRET || "").trim(),
|
||||
credential_source: "isolated_test_override",
|
||||
credential_version: 1,
|
||||
credential_ref: "test-only",
|
||||
};
|
||||
} else {
|
||||
provisioned = await deviceRequest("POST", "/device/v3/hardware-agent-credentials:provision", {
|
||||
hardware_id: requestedHardwareID,
|
||||
}, `windows-hardware-provision-${nonce()}`);
|
||||
}
|
||||
const identity = hardwareIdentity({
|
||||
instanceID: provisioned?.instance_id,
|
||||
productKey: provisioned?.product_key,
|
||||
deviceName: provisioned?.device_name,
|
||||
hardwareID: provisioned?.hardware_id,
|
||||
});
|
||||
const deviceSecret = String(provisioned?.device_secret || "").trim();
|
||||
if (deviceSecret.length < 16 || deviceSecret.length > 4096) throw new Error("Device Secret 长度不正确");
|
||||
const persisted = automatedTest ? false : saveHardwareCredential(identity, deviceSecret, {
|
||||
credentialRef: provisioned?.credential_ref,
|
||||
credentialVersion: provisioned?.credential_version,
|
||||
});
|
||||
if (!automatedTest && !persisted) throw new Error("Windows 安全存储不可用，拒绝保存 Device Secret");
|
||||
hardwareConfig = { ...identity, deviceSecret };
|
||||
// 首次预配会由平台发布产品级默认 Bot/传输配置，并可能推进配置版本。
|
||||
// 设备必须重新拉取版本，后续创建会话时不能继续上报预配前的旧 revision。
|
||||
if (!automatedTest) {
|
||||
const configuration = await deviceRequest("GET", "/device/v2/config");
|
||||
desiredConfigRevision = Number(configuration?.revision || desiredConfigRevision || 1);
|
||||
}
|
||||
return {
|
||||
...identity,
|
||||
registered: String(provisioned?.credential_source || "").includes("dynamic_registration"),
|
||||
persisted,
|
||||
credentialSource: String(provisioned?.credential_source || "platform_provisioning"),
|
||||
credentialVersion: Number(provisioned?.credential_version || 1),
|
||||
configRevision: desiredConfigRevision,
|
||||
};
|
||||
});
|
||||
|
||||
function nonce() {
|
||||
return crypto.randomBytes(16).toString("hex");
|
||||
}
|
||||
@@ -235,7 +411,7 @@ ipcMain.handle("sim:configure", (_event, input) => configure(input));
|
||||
|
||||
ipcMain.handle("sim:probe", async () => {
|
||||
const query = new URLSearchParams({
|
||||
firmware_version: "0.3.0-windows-rtc",
|
||||
firmware_version: "0.3.2-windows-hardware-agent",
|
||||
board_rev: "windows-x64",
|
||||
rtc_sdk_version: "web-4.69.0-electron",
|
||||
codec: "opus",
|
||||
@@ -280,6 +456,14 @@ ipcMain.handle("sim:probe-rtc-network", async () => {
|
||||
return { endpoints };
|
||||
});
|
||||
|
||||
ipcMain.handle("sim:probe-hardware-network", async () => {
|
||||
const endpoints = await Promise.all([
|
||||
probeTLSEndpoint("ai-gateway.vei.volces.com"),
|
||||
probeTLSEndpoint("iot-cn-shanghai.iot.volces.com"),
|
||||
]);
|
||||
return { endpoints };
|
||||
});
|
||||
|
||||
ipcMain.handle("sim:get-automation-audio", async () => {
|
||||
const configuredPath = String(process.env.SIMULATOR_AUTOMATED_AUDIO_PATH || "").trim();
|
||||
if (!configuredPath) return undefined;
|
||||
@@ -309,6 +493,173 @@ ipcMain.handle("sim:create-conversation", async () => deviceRequest("POST", "/de
|
||||
requested_goal: "open_conversation",
|
||||
}, `windows-create-${nonce()}`));
|
||||
|
||||
ipcMain.handle("sim:create-hardware-agent-session", async () => deviceRequest("POST", "/device/v3/agent-sessions", {
|
||||
device_time: new Date().toISOString(),
|
||||
monotonic_ms: Math.max(0, Math.round(performance.now())),
|
||||
capabilities: {
|
||||
hardware_agent: true,
|
||||
sdk_version: "1.1.0-windows-standard-protocol",
|
||||
transports: ["low_load"],
|
||||
input_audio_format: "pcm_s16le_16000_mono",
|
||||
output_audio_format: "pcm_s16le_16000_mono",
|
||||
server_event_callback: false,
|
||||
transcript_upload: true,
|
||||
dynamic_session_config: true,
|
||||
credential_rotation: true,
|
||||
near_pcm_recording: false,
|
||||
far_pcm_recording: false,
|
||||
},
|
||||
network: { type: "windows-local" },
|
||||
requested_goal: "open_conversation",
|
||||
config_revision: desiredConfigRevision,
|
||||
}, `windows-hardware-create-${nonce()}`));
|
||||
|
||||
function emitHardwareEvent(event) {
|
||||
if (!mainWindow || mainWindow.isDestroyed()) return;
|
||||
mainWindow.webContents.send("sim:hardware-provider-event", event);
|
||||
}
|
||||
|
||||
function allowedHardwareGatewayURL() {
|
||||
const configured = String(process.env.SIMULATOR_HARDWARE_GATEWAY_URL || "wss://ai-gateway.vei.volces.com/v1/realtime").trim();
|
||||
const parsed = new URL(configured);
|
||||
const official = parsed.protocol === "wss:" && parsed.hostname === "ai-gateway.vei.volces.com";
|
||||
const local = (parsed.protocol === "ws:" || parsed.protocol === "wss:") && ["127.0.0.1", "localhost", "[::1]"].includes(parsed.hostname);
|
||||
if (!official && !local) throw new Error("硬件智能体网关只允许火山官方地址或本机测试地址");
|
||||
return parsed.toString();
|
||||
}
|
||||
|
||||
async function invokeHardwareMemoryTool(event) {
|
||||
const client = hardwareClient;
|
||||
const platformSession = activeHardwarePlatformSession;
|
||||
if (!client || !platformSession) return;
|
||||
const callID = String(event.call_id || "");
|
||||
const functionName = String(client.pendingCalls.get(callID) || "").toLowerCase();
|
||||
if (!["memory_search", "search_memory", "retrieve_memory"].includes(functionName)) return;
|
||||
let argumentsValue = {};
|
||||
try { argumentsValue = JSON.parse(String(event.arguments || "{}")); } catch { /* invalid input is handled below */ }
|
||||
const query = String(argumentsValue.query || argumentsValue.keyword || argumentsValue.text || "").trim();
|
||||
const limit = Math.max(1, Math.min(5, Number(argumentsValue.limit || 3)));
|
||||
try {
|
||||
if (!query) throw new Error("记忆工具缺少 query 参数");
|
||||
const baseURL = new URL(deviceConfig.apiURL);
|
||||
const target = new URL(platformSession.session_config.tool_url, `${baseURL.origin}/`);
|
||||
if (target.origin !== baseURL.origin) throw new Error("记忆工具地址必须与平台同源");
|
||||
const response = await fetch(target, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Accept: "application/json",
|
||||
Authorization: `Bearer ${platformSession.session_config.tool_scope_token}`,
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
body: JSON.stringify({ query, limit }),
|
||||
});
|
||||
const raw = await response.text();
|
||||
let body;
|
||||
try { body = raw ? JSON.parse(raw) : {}; } catch { body = { message: raw }; }
|
||||
if (!response.ok) throw new Error(body?.error?.message || body?.message || `HTTP ${response.status}`);
|
||||
const result = body?.data ?? body;
|
||||
client.completeFunctionCall(callID, result);
|
||||
emitHardwareEvent({ type: "platform.memory_search.completed", call_id: callID, query, result_count: result?.items?.length || 0, degraded: Boolean(result?.degraded) });
|
||||
} catch (error) {
|
||||
client.completeFunctionCall(callID, { items: [], degraded: true, error: error.message });
|
||||
emitHardwareEvent({ type: "platform.memory_search.failed", call_id: callID, message: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
ipcMain.handle("sim:connect-hardware-agent", async (_event, input) => {
|
||||
if (!deviceConfig) throw new Error("请先连接平台");
|
||||
if (!hardwareConfig) throw new Error("请先完成火山设备鉴权配置");
|
||||
if (hardwareClient) throw new Error("已有硬件智能体连接，请先结束");
|
||||
const platformSession = input?.platformSession;
|
||||
if (!platformSession?.runtime?.bot_id || !platformSession?.session_config?.dialog_id) throw new Error("平台硬件会话配置不完整");
|
||||
if (platformSession.runtime.transport !== "low_load") throw new Error("Windows 标准协议测试程序只接受设备绑定为 low_load 的会话");
|
||||
const modelMode = String(input?.modelMode || "llm").trim().toLowerCase();
|
||||
if (!new Set(["llm", "s2s"]).has(modelMode)) throw new Error("智能体模型类型必须为 llm 或 s2s");
|
||||
activeHardwarePlatformSession = platformSession;
|
||||
const client = new HardwareAgentClient({
|
||||
gatewayURL: allowedHardwareGatewayURL(),
|
||||
emit: (event) => {
|
||||
emitHardwareEvent(event);
|
||||
if (event.type === "response.function_call_arguments.done") void invokeHardwareMemoryTool(event);
|
||||
},
|
||||
});
|
||||
hardwareClient = client;
|
||||
try {
|
||||
return await client.connect({
|
||||
...hardwareConfig,
|
||||
botID: platformSession.runtime.bot_id,
|
||||
sessionConfig: platformSession.session_config,
|
||||
modelMode,
|
||||
});
|
||||
} catch (error) {
|
||||
hardwareClient = null;
|
||||
activeHardwarePlatformSession = null;
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
|
||||
ipcMain.on("sim:hardware-agent-audio", (_event, bytes) => {
|
||||
try { hardwareClient?.sendAudio(Buffer.from(bytes)); }
|
||||
catch (error) { emitHardwareEvent({ type: "transport.audio_error", message: error.message }); }
|
||||
});
|
||||
|
||||
ipcMain.handle("sim:hardware-agent-text", (_event, input) => {
|
||||
if (!hardwareClient) throw new Error("火山硬件智能体连接未就绪");
|
||||
hardwareClient.sendText(input?.text, Number(input?.interruptMode || 1));
|
||||
return { accepted: true };
|
||||
});
|
||||
|
||||
ipcMain.handle("sim:close-hardware-agent", (_event, input) => {
|
||||
const client = hardwareClient;
|
||||
hardwareClient = null;
|
||||
activeHardwarePlatformSession = null;
|
||||
client?.close(Number(input?.code || 1000), String(input?.reason || "device session ended").slice(0, 100));
|
||||
return { closed: true };
|
||||
});
|
||||
|
||||
ipcMain.handle("sim:save-hardware-report", async (_event, report) => {
|
||||
const result = await dialog.showSaveDialog(mainWindow, {
|
||||
title: "导出硬件对话智能体测试报告",
|
||||
defaultPath: `hardware-agent-test-${new Date().toISOString().replace(/[:.]/g, "-")}.json`,
|
||||
filters: [{ name: "JSON 测试报告", extensions: ["json"] }],
|
||||
});
|
||||
if (result.canceled || !result.filePath) return { saved: false };
|
||||
const redact = (value) => {
|
||||
if (Array.isArray(value)) return value.map(redact);
|
||||
if (!value || typeof value !== "object") return value;
|
||||
const output = {};
|
||||
for (const [key, item] of Object.entries(value)) {
|
||||
if (/secret|signature|authorization|tool_scope_token|systemmessages|system_role|system_prompt|authorized_context|speaking_style|instructions/i.test(key)) output[key] = "[REDACTED]";
|
||||
else output[key] = redact(item);
|
||||
}
|
||||
return output;
|
||||
};
|
||||
const safeReport = redact(JSON.parse(JSON.stringify(report)));
|
||||
fs.writeFileSync(result.filePath, `${JSON.stringify(safeReport, null, 2)}\n`, "utf8");
|
||||
return { saved: true, path: result.filePath };
|
||||
});
|
||||
|
||||
ipcMain.handle("sim:hardware-agent-event", async (_event, input) => deviceRequest(
|
||||
"POST",
|
||||
`/device/v3/agent-sessions/${encodeURIComponent(input.conversationID)}/events`,
|
||||
{
|
||||
event_id: input.eventID || `windows-hardware-event-${nonce()}`,
|
||||
sequence_no: Number(input.sequenceNo),
|
||||
event_type: input.eventType,
|
||||
occurred_at: new Date().toISOString(),
|
||||
provider_session_ref: input.providerSessionRef,
|
||||
session_nonce: input.sessionNonce,
|
||||
...(input.eventType === "session_started" ? {
|
||||
provider_connection_ref: input.providerConnectionRef,
|
||||
bot_id: input.botID,
|
||||
transport: input.transport,
|
||||
config_digest: input.configDigest,
|
||||
dynamic_config_applied: input.dynamicConfigApplied,
|
||||
} : {}),
|
||||
...(input.errorCode ? { error_code: input.errorCode } : {}),
|
||||
},
|
||||
));
|
||||
|
||||
ipcMain.handle("sim:activate-conversation", async (_event, input) => deviceRequest(
|
||||
"POST",
|
||||
`/device/v2/conversations/${encodeURIComponent(input.conversationID)}/activate`,
|
||||
@@ -338,7 +689,7 @@ ipcMain.handle("sim:heartbeat", async (_event, input) => deviceRequest(
|
||||
conversation_status: "ACTIVE",
|
||||
rtc: { joined: true, uplink: true, downlink: true },
|
||||
network: { type: "windows-local" },
|
||||
firmware_version: "0.3.0-windows-rtc",
|
||||
firmware_version: "0.3.2-windows-hardware-agent",
|
||||
reported_config_revision: desiredConfigRevision,
|
||||
rtc_credential_expires_at: input.credentialExpiresAt,
|
||||
recording_queue: { pending_segments: 0, failed_segments: 0 },
|
||||
@@ -356,7 +707,7 @@ ipcMain.handle("sim:submit-transcript-batch", async (_event, input) => {
|
||||
const digest = crypto.createHash("sha256").update(JSON.stringify(payload)).digest("hex").slice(0, 32);
|
||||
return deviceRequest(
|
||||
"POST",
|
||||
`/device/v2/conversations/${encodeURIComponent(conversationID)}/turns:batch`,
|
||||
`/${input?.hardwareAgent ? "device/v3/agent-sessions" : "device/v2/conversations"}/${encodeURIComponent(conversationID)}/turns:batch`,
|
||||
payload,
|
||||
`windows-turns-${digest}`,
|
||||
);
|
||||
@@ -433,6 +784,9 @@ app.whenReady().then(() => {
|
||||
});
|
||||
|
||||
app.on("window-all-closed", () => {
|
||||
hardwareClient?.close(1001, "application closed");
|
||||
hardwareClient = null;
|
||||
activeHardwarePlatformSession = null;
|
||||
if (process.platform !== "darwin") app.quit();
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
const assert = require("node:assert/strict");
|
||||
const test = require("node:test");
|
||||
const fs = require("node:fs");
|
||||
const path = require("node:path");
|
||||
const vm = require("node:vm");
|
||||
|
||||
const source = fs.readFileSync(path.join(__dirname, "../public/pcm-playback.worklet.js"), "utf8");
|
||||
|
||||
function fixture({ outputRate = 48000, prebufferMs = 120 } = {}) {
|
||||
const messages = [];
|
||||
let Processor;
|
||||
class AudioWorkletProcessor {
|
||||
constructor() {
|
||||
this.port = { onmessage: null, postMessage: (message) => messages.push(message) };
|
||||
}
|
||||
}
|
||||
const context = vm.createContext({
|
||||
AudioWorkletProcessor,
|
||||
Float32Array,
|
||||
ArrayBuffer,
|
||||
Math,
|
||||
sampleRate: outputRate,
|
||||
registerProcessor: (name, implementation) => {
|
||||
assert.equal(name, "pcm16-playback");
|
||||
Processor = implementation;
|
||||
},
|
||||
});
|
||||
vm.runInContext(source, context);
|
||||
const processor = new Processor({ processorOptions: { sourceSampleRate: 16000, prebufferMs } });
|
||||
const push = (values) => {
|
||||
const samples = new Float32Array(values);
|
||||
processor.port.onmessage({ data: { type: "push", samples: samples.buffer } });
|
||||
};
|
||||
const process = () => {
|
||||
const output = new Float32Array(128);
|
||||
assert.equal(processor.process([], [[output]]), true);
|
||||
return output;
|
||||
};
|
||||
return { messages, processor, push, process };
|
||||
}
|
||||
|
||||
test("waits for prebuffer then resamples continuous 16k PCM without fragment gaps", () => {
|
||||
const f = fixture({ outputRate: 48000, prebufferMs: 10 });
|
||||
f.push(new Array(120).fill(0.25));
|
||||
assert.ok(f.process().every((sample) => sample === 0), "must remain silent below the prebuffer threshold");
|
||||
f.push(new Array(400).fill(0.25));
|
||||
const first = f.process();
|
||||
const second = f.process();
|
||||
assert.ok(first.some((sample) => sample > 0.05), "buffered playback should start with a short anti-pop fade");
|
||||
assert.ok(second.every((sample) => sample > 0.12), "playback should remain continuous across source fragments");
|
||||
assert.equal(f.messages.filter((message) => message.type === "underflow").length, 0);
|
||||
});
|
||||
|
||||
test("plays 16k source at real-time duration instead of the former 1.5x speed", () => {
|
||||
const f = fixture({ outputRate: 48000, prebufferMs: 1 });
|
||||
f.push(new Array(1000).fill(0.25));
|
||||
f.process();
|
||||
assert.ok(Math.abs(f.processor.readPosition - 128 / 3) < 0.001, `consumed ${f.processor.readPosition} source samples`);
|
||||
});
|
||||
|
||||
test("reports network starvation and re-enters prebuffering instead of scheduling tiny buffers", () => {
|
||||
const f = fixture({ outputRate: 48000, prebufferMs: 0.01 });
|
||||
f.push(new Array(8).fill(0.5));
|
||||
f.process();
|
||||
assert.equal(f.messages.filter((message) => message.type === "underflow").length, 1);
|
||||
assert.ok(f.process().every((sample) => sample === 0), "starved playback should wait for a new prebuffer");
|
||||
});
|
||||
|
||||
test("drains the final response cleanly and emits a completion signal", () => {
|
||||
const f = fixture({ outputRate: 24000 });
|
||||
f.push(new Array(64).fill(-0.25));
|
||||
f.processor.port.onmessage({ data: { type: "end" } });
|
||||
f.process();
|
||||
assert.equal(f.messages.filter((message) => message.type === "drained").length, 1);
|
||||
assert.equal(f.messages.filter((message) => message.type === "underflow").length, 0);
|
||||
});
|
||||
@@ -5,8 +5,23 @@ contextBridge.exposeInMainWorld("deviceSimulator", {
|
||||
configure: (input) => ipcRenderer.invoke("sim:configure", input),
|
||||
probe: () => ipcRenderer.invoke("sim:probe"),
|
||||
probeRTCNetwork: () => ipcRenderer.invoke("sim:probe-rtc-network"),
|
||||
probeHardwareNetwork: () => ipcRenderer.invoke("sim:probe-hardware-network"),
|
||||
getHardwareRuntimeDefaults: () => ipcRenderer.invoke("sim:get-hardware-runtime-defaults"),
|
||||
configureHardware: (input) => ipcRenderer.invoke("sim:configure-hardware", input),
|
||||
getAutomationAudio: () => ipcRenderer.invoke("sim:get-automation-audio"),
|
||||
createConversation: () => ipcRenderer.invoke("sim:create-conversation"),
|
||||
createHardwareAgentSession: () => ipcRenderer.invoke("sim:create-hardware-agent-session"),
|
||||
connectHardwareAgent: (input) => ipcRenderer.invoke("sim:connect-hardware-agent", input),
|
||||
sendHardwareAgentAudio: (bytes) => ipcRenderer.send("sim:hardware-agent-audio", bytes),
|
||||
sendHardwareAgentText: (input) => ipcRenderer.invoke("sim:hardware-agent-text", input),
|
||||
closeHardwareAgent: (input) => ipcRenderer.invoke("sim:close-hardware-agent", input),
|
||||
onHardwareProviderEvent: (callback) => {
|
||||
const listener = (_event, value) => callback(value);
|
||||
ipcRenderer.on("sim:hardware-provider-event", listener);
|
||||
return () => ipcRenderer.removeListener("sim:hardware-provider-event", listener);
|
||||
},
|
||||
saveHardwareReport: (report) => ipcRenderer.invoke("sim:save-hardware-report", report),
|
||||
hardwareAgentEvent: (input) => ipcRenderer.invoke("sim:hardware-agent-event", input),
|
||||
activateConversation: (input) => ipcRenderer.invoke("sim:activate-conversation", input),
|
||||
getConversation: (conversationID) => ipcRenderer.invoke("sim:get-conversation", conversationID),
|
||||
heartbeat: (input) => ipcRenderer.invoke("sim:heartbeat", input),
|
||||
|
||||
@@ -4,10 +4,10 @@
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self' data:; style-src 'self'; connect-src 'self' data: https://121.41.46.238 https://*.rtc.volcvideo.com https://*.volcvideos.com wss: http://127.0.0.1:* http://localhost:*; media-src 'self' blob:; worker-src 'self' blob: data:; img-src 'self' data:;" />
|
||||
<title>语音终端模拟器</title>
|
||||
<title>硬件对话智能体测试程序</title>
|
||||
</head>
|
||||
<body>
|
||||
<main id="app"></main>
|
||||
<script type="module" src="/src/main.ts"></script>
|
||||
<script type="module" src="/src/hardware.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
@echo off
|
||||
setlocal
|
||||
set "SECRET_FILE=%~dp0.server-device-secret"
|
||||
set "DEVICE_EXE=%~dp0voice-device-simulator-0.1.0-windows-x64.exe"
|
||||
set "SECRET_FILE=%LOCALAPPDATA%\DataTerminalHardwareAgent\server-device-secret"
|
||||
set "DEVICE_EXE=%~dp0hardware-agent-tester-0.3.5-windows-x64.exe"
|
||||
if not exist "%DEVICE_EXE%" set "DEVICE_EXE=%~dp0release\hardware-agent-tester-0.3.5-windows-x64.exe"
|
||||
|
||||
if not exist "%SECRET_FILE%" (
|
||||
echo Cloud device credential is missing: %SECRET_FILE%
|
||||
@@ -9,12 +10,12 @@ if not exist "%SECRET_FILE%" (
|
||||
exit /b 1
|
||||
)
|
||||
if not exist "%DEVICE_EXE%" (
|
||||
echo Voice device simulator is missing: %DEVICE_EXE%
|
||||
echo Hardware agent tester is missing: %DEVICE_EXE%
|
||||
pause
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
set /p SIMULATOR_DEFAULT_DEVICE_SECRET=<"%SECRET_FILE%"
|
||||
set "SIMULATOR_DEFAULT_API_URL=https://121.41.46.238"
|
||||
set "SIMULATOR_DEFAULT_DEVICE_ID=10000000-0000-0000-0000-000000000511"
|
||||
set "SIMULATOR_DEFAULT_DEVICE_ID=10000000-0000-0000-0000-000000000512"
|
||||
start "" "%DEVICE_EXE%"
|
||||
|
||||
@@ -1,25 +1,46 @@
|
||||
[CmdletBinding()]
|
||||
param()
|
||||
param(
|
||||
[switch]$RealE2E,
|
||||
[string]$ResultPath = (Join-Path $env:TEMP "hardware-agent-real-e2e.json")
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$secretPath = Join-Path $env:LOCALAPPDATA "DataTerminalVoiceDevice\server-device-secret"
|
||||
$executablePath = Join-Path $PSScriptRoot "voice-device-simulator-0.1.0-windows-x64.exe"
|
||||
$secretPath = Join-Path $env:LOCALAPPDATA "DataTerminalHardwareAgent\server-device-secret"
|
||||
$executablePath = Join-Path $PSScriptRoot "hardware-agent-tester-0.3.5-windows-x64.exe"
|
||||
if (-not (Test-Path -LiteralPath $executablePath -PathType Leaf)) {
|
||||
$executablePath = Join-Path $PSScriptRoot "release\hardware-agent-tester-0.3.5-windows-x64.exe"
|
||||
}
|
||||
|
||||
if (-not (Test-Path -LiteralPath $secretPath -PathType Leaf)) {
|
||||
throw "Cloud device credential is missing: $secretPath"
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $executablePath -PathType Leaf)) {
|
||||
throw "Voice device simulator is missing: $executablePath"
|
||||
throw "Hardware agent tester is missing: $executablePath"
|
||||
}
|
||||
|
||||
$env:SIMULATOR_DEFAULT_API_URL = "https://121.41.46.238"
|
||||
$env:SIMULATOR_DEFAULT_DEVICE_ID = "10000000-0000-0000-0000-000000000511"
|
||||
$env:SIMULATOR_DEFAULT_DEVICE_ID = "10000000-0000-0000-0000-000000000512"
|
||||
$env:SIMULATOR_DEFAULT_DEVICE_SECRET = [IO.File]::ReadAllText($secretPath).Trim()
|
||||
try {
|
||||
Start-Process -FilePath $executablePath
|
||||
if ($RealE2E) {
|
||||
$env:SIMULATOR_REAL_E2E = "1"
|
||||
$env:SIMULATOR_AUTO_RESULT_PATH = $ResultPath
|
||||
$env:SIMULATOR_VOLC_MODEL_MODE = "llm"
|
||||
$env:SIMULATOR_REAL_E2E_QUESTION = "Please introduce yourself in one short sentence."
|
||||
$process = Start-Process -FilePath $executablePath -WindowStyle Hidden -PassThru -Wait
|
||||
if ($process.ExitCode -ne 0) { throw "Hardware agent tester exited with code $($process.ExitCode)" }
|
||||
if (-not (Test-Path -LiteralPath $ResultPath -PathType Leaf)) { throw "Real E2E report was not created: $ResultPath" }
|
||||
}
|
||||
else {
|
||||
Start-Process -FilePath $executablePath
|
||||
}
|
||||
}
|
||||
finally {
|
||||
Remove-Item Env:SIMULATOR_DEFAULT_API_URL -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:SIMULATOR_DEFAULT_DEVICE_ID -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:SIMULATOR_DEFAULT_DEVICE_SECRET -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:SIMULATOR_REAL_E2E -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:SIMULATOR_AUTO_RESULT_PATH -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:SIMULATOR_VOLC_MODEL_MODE -ErrorAction SilentlyContinue
|
||||
Remove-Item Env:SIMULATOR_REAL_E2E_QUESTION -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta http-equiv="Content-Security-Policy" content="default-src 'self'; script-src 'self' data:; style-src 'self'; connect-src 'self' data: https://121.41.46.238 https://*.rtc.volcvideo.com https://*.volcvideos.com wss: http://127.0.0.1:* http://localhost:*; media-src 'self' blob:; worker-src 'self' blob: data:; img-src 'self' data:;" />
|
||||
<title>传统 RTC 语音终端模拟器</title>
|
||||
</head>
|
||||
<body>
|
||||
<main id="app"></main>
|
||||
<script type="module" src="/src/main.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
+25
-3
@@ -1,14 +1,15 @@
|
||||
{
|
||||
"name": "data-terminal-voice-device-simulator",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.5",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "data-terminal-voice-device-simulator",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.5",
|
||||
"dependencies": {
|
||||
"@volcengine/rtc": "4.69.0"
|
||||
"@volcengine/rtc": "4.69.0",
|
||||
"ws": "8.21.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"concurrently": "^9.2.1",
|
||||
@@ -4857,6 +4858,27 @@
|
||||
"dev": true,
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/ws": {
|
||||
"version": "8.21.3",
|
||||
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
|
||||
"integrity": "sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=10.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"bufferutil": "^4.0.1",
|
||||
"utf-8-validate": ">=5.0.2"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"bufferutil": {
|
||||
"optional": true
|
||||
},
|
||||
"utf-8-validate": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/xmlbuilder": {
|
||||
"version": "15.1.1",
|
||||
"resolved": "https://registry.npmjs.org/xmlbuilder/-/xmlbuilder-15.1.1.tgz",
|
||||
|
||||
@@ -1,18 +1,20 @@
|
||||
{
|
||||
"name": "data-terminal-voice-device-simulator",
|
||||
"version": "0.1.0",
|
||||
"version": "0.3.5",
|
||||
"private": true,
|
||||
"description": "Windows real-audio device simulator for the data terminal cloud platform",
|
||||
"description": "Windows end-to-end tester for Volcengine hardware conversational agents",
|
||||
"main": "electron/main.cjs",
|
||||
"scripts": {
|
||||
"dev": "concurrently -k \"vite --host 127.0.0.1 --port 5177\" \"wait-on http://127.0.0.1:5177 && electron . --dev-server-url=http://127.0.0.1:5177\"",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"test": "node --test electron/*.test.cjs",
|
||||
"build:renderer": "vite build",
|
||||
"start": "npm run build:renderer && electron .",
|
||||
"build": "npm run typecheck && npm run build:renderer && electron-builder --win portable"
|
||||
"build": "npm test && npm run typecheck && npm run build:renderer && electron-builder --win portable"
|
||||
},
|
||||
"dependencies": {
|
||||
"@volcengine/rtc": "4.69.0"
|
||||
"@volcengine/rtc": "4.69.0",
|
||||
"ws": "8.21.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"concurrently": "^9.2.1",
|
||||
@@ -23,22 +25,25 @@
|
||||
"wait-on": "^8.0.4"
|
||||
},
|
||||
"build": {
|
||||
"appId": "cn.dataterminal.voice-device-simulator",
|
||||
"productName": "语音终端模拟器",
|
||||
"artifactName": "voice-device-simulator-${version}-windows-${arch}.${ext}",
|
||||
"appId": "cn.dataterminal.hardware-agent-tester",
|
||||
"productName": "硬件对话智能体测试程序",
|
||||
"artifactName": "hardware-agent-tester-${version}-windows-${arch}.${ext}",
|
||||
"directories": {
|
||||
"output": "release"
|
||||
},
|
||||
"files": [
|
||||
"dist/**/*",
|
||||
"electron/**/*",
|
||||
"!electron/*.test.cjs",
|
||||
"package.json"
|
||||
],
|
||||
"win": {
|
||||
"target": [
|
||||
{
|
||||
"target": "portable",
|
||||
"arch": ["x64"]
|
||||
"arch": [
|
||||
"x64"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
class PCM16CaptureProcessor extends AudioWorkletProcessor {
|
||||
constructor(options) {
|
||||
super();
|
||||
this.targetRate = Number(options.processorOptions?.targetSampleRate || 16000);
|
||||
this.frameSamples = Number(options.processorOptions?.frameSamples || 320);
|
||||
this.ratio = sampleRate / this.targetRate;
|
||||
this.source = [];
|
||||
this.position = 0;
|
||||
this.output = [];
|
||||
}
|
||||
|
||||
process(inputs) {
|
||||
const channel = inputs[0]?.[0];
|
||||
if (!channel?.length) return true;
|
||||
for (const sample of channel) this.source.push(sample);
|
||||
while (this.position + this.ratio <= this.source.length) {
|
||||
const start = Math.floor(this.position);
|
||||
const end = Math.max(start + 1, Math.floor(this.position + this.ratio));
|
||||
let sum = 0;
|
||||
for (let index = start; index < end && index < this.source.length; index += 1) sum += this.source[index];
|
||||
this.output.push(sum / Math.max(1, end - start));
|
||||
this.position += this.ratio;
|
||||
if (this.output.length >= this.frameSamples) this.flush();
|
||||
}
|
||||
const consumed = Math.floor(this.position);
|
||||
if (consumed > 0) {
|
||||
this.source.splice(0, consumed);
|
||||
this.position -= consumed;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
flush() {
|
||||
const frame = this.output.splice(0, this.frameSamples);
|
||||
const pcm = new Int16Array(frame.length);
|
||||
let energy = 0;
|
||||
for (let index = 0; index < frame.length; index += 1) {
|
||||
const sample = Math.max(-1, Math.min(1, frame[index]));
|
||||
energy += sample * sample;
|
||||
pcm[index] = sample < 0 ? Math.round(sample * 32768) : Math.round(sample * 32767);
|
||||
}
|
||||
const volume = Math.min(100, Math.round(Math.sqrt(energy / Math.max(1, frame.length)) * 280));
|
||||
this.port.postMessage({ type: "pcm", bytes: pcm.buffer, volume }, [pcm.buffer]);
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor("pcm16-capture", PCM16CaptureProcessor);
|
||||
@@ -0,0 +1,115 @@
|
||||
class PCM16PlaybackProcessor extends AudioWorkletProcessor {
|
||||
constructor(options) {
|
||||
super();
|
||||
const config = options.processorOptions || {};
|
||||
this.sourceSampleRate = Number(config.sourceSampleRate || 16000);
|
||||
this.prebufferSamples = Math.max(1, Math.round(this.sourceSampleRate * Number(config.prebufferMs || 120) / 1000));
|
||||
this.capacity = Math.max(this.sourceSampleRate * 30, this.prebufferSamples * 4);
|
||||
this.ring = new Float32Array(this.capacity);
|
||||
this.readPosition = 0;
|
||||
this.writePosition = 0;
|
||||
this.playing = false;
|
||||
this.ended = false;
|
||||
this.drained = false;
|
||||
this.lastOutput = 0;
|
||||
this.fadeInRemaining = 0;
|
||||
this.fadeFrames = Math.max(1, Math.round(sampleRate * 0.005));
|
||||
this.underflows = 0;
|
||||
this.port.onmessage = (message) => this.onMessage(message.data || {});
|
||||
}
|
||||
|
||||
onMessage(message) {
|
||||
if (message.type === "reset") {
|
||||
this.reset();
|
||||
return;
|
||||
}
|
||||
if (message.type === "end") {
|
||||
this.ended = true;
|
||||
return;
|
||||
}
|
||||
if (message.type !== "push" || !(message.samples instanceof ArrayBuffer)) return;
|
||||
const samples = new Float32Array(message.samples);
|
||||
if (this.drained) this.reset();
|
||||
this.ended = false;
|
||||
for (let index = 0; index < samples.length; index += 1) {
|
||||
if (this.writePosition - Math.floor(this.readPosition) >= this.capacity - 1) {
|
||||
this.readPosition = this.writePosition - this.capacity + 2;
|
||||
}
|
||||
this.ring[this.writePosition % this.capacity] = samples[index];
|
||||
this.writePosition += 1;
|
||||
}
|
||||
}
|
||||
|
||||
reset() {
|
||||
this.readPosition = 0;
|
||||
this.writePosition = 0;
|
||||
this.playing = false;
|
||||
this.ended = false;
|
||||
this.drained = false;
|
||||
this.lastOutput = 0;
|
||||
this.fadeInRemaining = 0;
|
||||
}
|
||||
|
||||
available() {
|
||||
return this.writePosition - this.readPosition;
|
||||
}
|
||||
|
||||
startWhenReady() {
|
||||
if (this.playing) return true;
|
||||
const available = this.available();
|
||||
if (available >= this.prebufferSamples || (this.ended && available > 1)) {
|
||||
this.playing = true;
|
||||
this.fadeInRemaining = this.fadeFrames;
|
||||
this.port.postMessage({ type: "started", bufferedSamples: Math.floor(available) });
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
finish(output, fromIndex, eventType) {
|
||||
const remaining = Math.min(this.fadeFrames, output.length - fromIndex);
|
||||
for (let index = 0; index < remaining; index += 1) {
|
||||
output[fromIndex + index] = this.lastOutput * (1 - (index + 1) / remaining);
|
||||
}
|
||||
output.fill(0, fromIndex + remaining);
|
||||
this.lastOutput = 0;
|
||||
this.playing = false;
|
||||
if (eventType === "drained") {
|
||||
if (!this.drained) this.port.postMessage({ type: "drained" });
|
||||
this.drained = true;
|
||||
} else {
|
||||
this.underflows += 1;
|
||||
this.port.postMessage({ type: "underflow", count: this.underflows });
|
||||
}
|
||||
}
|
||||
|
||||
process(_inputs, outputs) {
|
||||
const output = outputs[0]?.[0];
|
||||
if (!output) return true;
|
||||
output.fill(0);
|
||||
if (!this.startWhenReady()) return true;
|
||||
|
||||
const sourceStep = this.sourceSampleRate / sampleRate;
|
||||
for (let index = 0; index < output.length; index += 1) {
|
||||
const base = Math.floor(this.readPosition);
|
||||
if (base + 1 >= this.writePosition) {
|
||||
this.finish(output, index, this.ended ? "drained" : "underflow");
|
||||
break;
|
||||
}
|
||||
const fraction = this.readPosition - base;
|
||||
const first = this.ring[base % this.capacity];
|
||||
const second = this.ring[(base + 1) % this.capacity];
|
||||
let value = first + (second - first) * fraction;
|
||||
if (this.fadeInRemaining > 0) {
|
||||
value *= 1 - this.fadeInRemaining / this.fadeFrames;
|
||||
this.fadeInRemaining -= 1;
|
||||
}
|
||||
output[index] = value;
|
||||
this.lastOutput = value;
|
||||
this.readPosition += sourceStep;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor("pcm16-playback", PCM16PlaybackProcessor);
|
||||
+72
-1
@@ -34,6 +34,32 @@ type CreatedConversation = {
|
||||
};
|
||||
};
|
||||
|
||||
type CreatedHardwareAgentSession = {
|
||||
conversation_id: string;
|
||||
status: "SESSION_READY";
|
||||
runtime_type: "volc_hardware_agent";
|
||||
runtime: {
|
||||
provider: "volcengine";
|
||||
bot_id: string;
|
||||
transport: "high_quality" | "low_load";
|
||||
provider_session_ref: string;
|
||||
session_nonce: string;
|
||||
config_digest: string;
|
||||
wait_for_session_update: boolean;
|
||||
};
|
||||
session_config: {
|
||||
dialog_id: string;
|
||||
system_prompt?: string;
|
||||
persona_tone?: string;
|
||||
authorized_context: string;
|
||||
prompt_bundle_version: string;
|
||||
tool_url?: string;
|
||||
tool_scope_token?: string;
|
||||
};
|
||||
privacy: CreatedConversation["privacy"];
|
||||
session: CreatedConversation["session"];
|
||||
};
|
||||
|
||||
type DeviceFinalTurn = {
|
||||
turn_key: string;
|
||||
sequence_no: number;
|
||||
@@ -50,15 +76,60 @@ interface Window {
|
||||
configure(input: SimulatorConfiguration): Promise<{ apiURL: string; deviceID: string }>;
|
||||
probe(): Promise<Record<string, unknown>>;
|
||||
probeRTCNetwork(): Promise<{ endpoints: Array<{ hostname: string; ok: boolean; error?: string }> }>;
|
||||
probeHardwareNetwork(): Promise<{ endpoints: Array<{ hostname: string; ok: boolean; error?: string }> }>;
|
||||
getHardwareRuntimeDefaults(): Promise<{
|
||||
instanceID: string;
|
||||
productKey: string;
|
||||
deviceName: string;
|
||||
hardwareID: string;
|
||||
modelMode: "llm" | "s2s";
|
||||
deviceCredentialConfigured: boolean;
|
||||
}>;
|
||||
configureHardware(input: {
|
||||
hardwareID: string;
|
||||
}): Promise<{
|
||||
instanceID: string;
|
||||
productKey: string;
|
||||
deviceName: string;
|
||||
hardwareID: string;
|
||||
registered: boolean;
|
||||
persisted: boolean;
|
||||
credentialSource: string;
|
||||
credentialVersion: number;
|
||||
}>;
|
||||
getAutomationAudio(): Promise<{ data_url: string; source_name: string } | undefined>;
|
||||
createConversation(): Promise<CreatedConversation>;
|
||||
createHardwareAgentSession(): Promise<CreatedHardwareAgentSession>;
|
||||
connectHardwareAgent(input: { platformSession: CreatedHardwareAgentSession; modelMode: "llm" | "s2s" }): Promise<{
|
||||
providerConnectionRef: string;
|
||||
session: Record<string, unknown>;
|
||||
}>;
|
||||
sendHardwareAgentAudio(bytes: ArrayBuffer): void;
|
||||
sendHardwareAgentText(input: { text: string; interruptMode?: number }): Promise<{ accepted: boolean }>;
|
||||
closeHardwareAgent(input?: { code?: number; reason?: string }): Promise<{ closed: boolean }>;
|
||||
onHardwareProviderEvent(callback: (event: Record<string, unknown>) => void): () => void;
|
||||
saveHardwareReport(report: Record<string, unknown>): Promise<{ saved: boolean; path?: string }>;
|
||||
hardwareAgentEvent(input: {
|
||||
conversationID: string;
|
||||
eventID?: string;
|
||||
sequenceNo: number;
|
||||
eventType: "session_started" | "session_updated" | "session_ended" | "session_failed";
|
||||
providerSessionRef: string;
|
||||
sessionNonce: string;
|
||||
providerConnectionRef?: string;
|
||||
botID?: string;
|
||||
transport?: "high_quality" | "low_load";
|
||||
configDigest?: string;
|
||||
dynamicConfigApplied?: boolean;
|
||||
errorCode?: string;
|
||||
}): Promise<{ conversation_id: string; status: string; action: "continue" | "stop_session"; duplicate: boolean }>;
|
||||
activateConversation(input: { conversationID: string; roomID: string; userID: string }): Promise<Record<string, unknown>>;
|
||||
getConversation(conversationID: string): Promise<{ status: string; action: string; failure_code?: string }>;
|
||||
heartbeat(input: { conversationID: string; sequenceNo: number; credentialExpiresAt: string }): Promise<{
|
||||
actions?: Array<{ type: string; rtc?: RTCGrant }>;
|
||||
desired_config_revision?: number;
|
||||
}>;
|
||||
submitTranscriptBatch(input: { conversationID: string; turns: DeviceFinalTurn[] }): Promise<{
|
||||
submitTranscriptBatch(input: { conversationID: string; turns: DeviceFinalTurn[]; hardwareAgent?: boolean }): Promise<{
|
||||
conversation_id: string;
|
||||
accepted_count: number;
|
||||
duplicate_count: number;
|
||||
|
||||
@@ -0,0 +1,778 @@
|
||||
import "./style.css";
|
||||
|
||||
type HardwarePhase = "idle" | "checking" | "ready" | "connecting" | "active" | "stopping" | "completed" | "error";
|
||||
type ReportCheck = { id: string; label: string; status: "pending" | "passed" | "failed"; detail?: string; at?: string };
|
||||
type ReportEvent = { at: string; source: "app" | "platform" | "volcengine"; type: string; detail?: Record<string, unknown> };
|
||||
|
||||
const checkDefinitions: Array<[string, string]> = [
|
||||
["platform_authenticated", "平台设备 HMAC 鉴权"],
|
||||
["credential_resolved", "火山一机一密凭据"],
|
||||
["context_frozen", "人设、背景与授权上下文冻结"],
|
||||
["provider_connected", "火山硬件智能体 WebSocket 建连"],
|
||||
["dynamic_config_applied", "session.updated 动态配置确认"],
|
||||
["platform_active", "平台确认会话 ACTIVE"],
|
||||
["microphone_streaming", "16kHz PCM 麦克风实时上行"],
|
||||
["audio_playback_buffered", "16kHz PCM 连续缓冲播放"],
|
||||
["heartbeat_ack", "平台 session_updated 存活回执"],
|
||||
["user_transcript", "用户最终字幕"],
|
||||
["agent_transcript", "智能体最终字幕"],
|
||||
["transcript_uploaded", "最终话轮可靠上传"],
|
||||
["provider_closed", "火山连接正常关闭"],
|
||||
["platform_completed", "平台确认会话 COMPLETED"],
|
||||
];
|
||||
|
||||
const state: {
|
||||
phase: HardwarePhase;
|
||||
platformSession?: CreatedHardwareAgentSession;
|
||||
providerConnectionRef?: string;
|
||||
eventSequence: number;
|
||||
transcriptSequence: number;
|
||||
pendingTurns: DeviceFinalTurn[];
|
||||
transcriptUpload?: Promise<void>;
|
||||
heartbeatTimer?: number;
|
||||
durationTimer?: number;
|
||||
startedAt?: number;
|
||||
stopping: boolean;
|
||||
intentionalProviderClose: boolean;
|
||||
microphoneStream?: MediaStream;
|
||||
captureContext?: AudioContext;
|
||||
captureNode?: AudioWorkletNode;
|
||||
playbackContext?: AudioContext;
|
||||
playbackNode?: AudioWorkletNode;
|
||||
playbackReady?: Promise<AudioWorkletNode>;
|
||||
playbackPendingByte?: number;
|
||||
playbackChunks: number;
|
||||
playbackSamples: number;
|
||||
playbackUnderflows: number;
|
||||
responseTranscripts: Map<string, string>;
|
||||
speechStartedAt?: number;
|
||||
checks: Map<string, ReportCheck>;
|
||||
events: ReportEvent[];
|
||||
turns: DeviceFinalTurn[];
|
||||
failure?: string;
|
||||
failureTask?: Promise<void>;
|
||||
} = {
|
||||
phase: "idle",
|
||||
eventSequence: 0,
|
||||
transcriptSequence: 0,
|
||||
pendingTurns: [],
|
||||
stopping: false,
|
||||
intentionalProviderClose: false,
|
||||
playbackChunks: 0,
|
||||
playbackSamples: 0,
|
||||
playbackUnderflows: 0,
|
||||
responseTranscripts: new Map(),
|
||||
checks: new Map(checkDefinitions.map(([id, label]) => [id, { id, label, status: "pending" }])),
|
||||
events: [],
|
||||
turns: [],
|
||||
};
|
||||
|
||||
const app = document.querySelector<HTMLElement>("#app")!;
|
||||
app.innerHTML = `
|
||||
<section class="shell hardware-shell">
|
||||
<header class="hero">
|
||||
<div>
|
||||
<p class="eyebrow">WINDOWS · VOLC HARDWARE AGENT · LOW LOAD</p>
|
||||
<h1>硬件对话智能体测试程序</h1>
|
||||
<p class="subtitle">真实连接平台设备 v3 控制面和火山硬件对话智能体 WebSocket，验证动态人设、PCM 双向语音、字幕、记忆工具、平台停止与结束闭环。</p>
|
||||
</div>
|
||||
<div class="hero-actions"><a class="legacy-link" href="./legacy.html">传统 RTC 测试页</a><div class="overall" id="overallStatus"><strong>尚未检测</strong></div></div>
|
||||
</header>
|
||||
|
||||
<section class="status-grid" aria-label="链路状态">
|
||||
<article class="status-card"><span>平台 v3</span><strong id="platformStatus">未检测</strong><small id="platformDetail">等待设备鉴权</small></article>
|
||||
<article class="status-card"><span>火山设备凭据</span><strong id="credentialStatus">未配置</strong><small id="credentialDetail">支持动态注册</small></article>
|
||||
<article class="status-card"><span>动态会话配置</span><strong id="configStatus">未应用</strong><small id="configDetail">麦克风保持关闭</small></article>
|
||||
<article class="status-card"><span>通话时长</span><strong id="duration">00:00</strong><small id="providerStatus">网关未连接</small></article>
|
||||
</section>
|
||||
|
||||
<section class="content-grid hardware-content-grid">
|
||||
<article class="panel setup-panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">SECURE SETUP</p><h2>平台与火山设备</h2></div><span class="safe">密钥不写入报告</span></div>
|
||||
<h3>平台设备身份</h3>
|
||||
<label>平台地址<input id="apiURL" value="https://121.41.46.238" autocomplete="off" /></label>
|
||||
<label>平台设备 ID<input id="deviceID" autocomplete="off" /></label>
|
||||
<label>平台设备签名密钥<input id="deviceSecret" type="password" autocomplete="off" /></label>
|
||||
<h3>火山硬件身份</h3>
|
||||
<div class="row"><label>实例 ID（平台下发）<input id="instanceID" readonly /></label><label>Product Key（平台下发）<input id="productKey" readonly /></label></div>
|
||||
<div class="row"><label>设备名称（平台分配）<input id="volcDeviceName" readonly /></label><label>Hardware ID<input id="hardwareID" autocomplete="off" /></label></div>
|
||||
<label>智能体模型类型<select id="modelMode"><option value="llm">通用 LLM（LLMConfig.SystemMessages）</option><option value="s2s">豆包端到端语音（S2S dialog）</option></select></label>
|
||||
<label>麦克风<select id="microphone"><option value="">默认麦克风</option></select></label>
|
||||
<button class="secondary" id="checkButton">检测、鉴权并准备设备</button>
|
||||
<p class="hint">Product Secret 只保存在平台密钥服务中。平台代设备完成动态注册，设备仅接收自己的一机一密并使用 Windows Safe Storage 加密保存；默认 Bot 由平台产品配置自动分配，设备不能覆盖。</p>
|
||||
</article>
|
||||
|
||||
<article class="panel call-panel hardware-call-panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">REAL END-TO-END FLOW</p><h2>硬件智能体通话</h2></div><span class="phase-badge" id="phaseBadge">待机</span></div>
|
||||
<div class="conversation-orb" id="conversationOrb"><div class="orb-core"><span id="orbLabel">准备检测</span></div></div>
|
||||
<div class="meters">
|
||||
<div><div class="meter-label"><span>16k PCM 上行</span><strong id="micValue">0</strong></div><div class="meter"><i id="micMeter"></i></div></div>
|
||||
<div><div class="meter-label"><span>16k PCM 下行</span><strong id="agentValue">0</strong></div><div class="meter agent"><i id="agentMeter"></i></div></div>
|
||||
</div>
|
||||
<div class="call-actions"><button class="primary" id="startButton" disabled>开始硬件智能体通话</button><button class="danger" id="stopButton" disabled>结束通话</button></div>
|
||||
<div class="text-question"><input id="textQuestion" placeholder="可选：输入文字验证智能体回复" /><button class="secondary" id="sendTextButton" disabled>发送文本问题</button></div>
|
||||
<dl class="session-info hardware-session-info">
|
||||
<div><dt>平台通话编号</dt><dd id="conversationID">—</dd></div>
|
||||
<div><dt>产品默认 Bot</dt><dd id="botID">—</dd></div>
|
||||
<div><dt>传输方案</dt><dd id="transport">low_load</dd></div>
|
||||
<div><dt>火山会话编号</dt><dd id="providerConnectionRef">—</dd></div>
|
||||
<div><dt>配置摘要</dt><dd id="configDigest">—</dd></div>
|
||||
<div><dt>冻结提示版本</dt><dd id="promptVersion">—</dd></div>
|
||||
</dl>
|
||||
</article>
|
||||
</section>
|
||||
|
||||
<section class="panel flow-panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">ACCEPTANCE GATES</p><h2>完整流程验收</h2></div><button class="text-button" id="exportReport">导出测试报告</button></div>
|
||||
<ol id="flowChecks" class="flow-checks"></ol>
|
||||
</section>
|
||||
|
||||
<section class="panel transcript-panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">FINAL TRANSCRIPTS</p><h2>双方最终字幕与平台上传</h2></div><span class="phase-badge" id="transcriptStatus">0 条</span></div>
|
||||
<ol id="liveTranscript"><li class="transcript-empty">通话开始后显示火山返回的用户和智能体最终字幕。</li></ol>
|
||||
</section>
|
||||
|
||||
<section class="panel log-panel">
|
||||
<div class="panel-heading"><div><p class="eyebrow">AUDITABLE EVENT STREAM</p><h2>平台与火山事件</h2></div><button class="text-button" id="clearLog">清空显示</button></div>
|
||||
<ol id="eventLog"><li><time>系统</time><span>测试程序已启动，等待环境检测。</span></li></ol>
|
||||
</section>
|
||||
</section>
|
||||
`;
|
||||
|
||||
const $ = <T extends HTMLElement>(id: string) => document.getElementById(id) as T;
|
||||
const controls = {
|
||||
apiURL: $<HTMLInputElement>("apiURL"), deviceID: $<HTMLInputElement>("deviceID"), deviceSecret: $<HTMLInputElement>("deviceSecret"),
|
||||
instanceID: $<HTMLInputElement>("instanceID"), productKey: $<HTMLInputElement>("productKey"),
|
||||
volcDeviceName: $<HTMLInputElement>("volcDeviceName"), hardwareID: $<HTMLInputElement>("hardwareID"),
|
||||
modelMode: $<HTMLSelectElement>("modelMode"),
|
||||
microphone: $<HTMLSelectElement>("microphone"), check: $<HTMLButtonElement>("checkButton"), start: $<HTMLButtonElement>("startButton"),
|
||||
stop: $<HTMLButtonElement>("stopButton"), textQuestion: $<HTMLInputElement>("textQuestion"), sendText: $<HTMLButtonElement>("sendTextButton"),
|
||||
};
|
||||
|
||||
function setText(id: string, value: string) { $(id).textContent = value; }
|
||||
function errorMessage(error: unknown) { return error instanceof Error ? error.message : String(error); }
|
||||
|
||||
function recordEvent(source: ReportEvent["source"], type: string, detail?: Record<string, unknown>) {
|
||||
state.events.push({ at: new Date().toISOString(), source, type, detail });
|
||||
if (state.events.length > 1000) state.events.splice(0, state.events.length - 1000);
|
||||
}
|
||||
|
||||
function log(message: string, level: "info" | "success" | "error" = "info", source: ReportEvent["source"] = "app", type = "log") {
|
||||
const item = document.createElement("li");
|
||||
item.className = level;
|
||||
const time = document.createElement("time");
|
||||
time.textContent = new Date().toLocaleTimeString("zh-CN", { hour12: false });
|
||||
const content = document.createElement("span");
|
||||
content.textContent = message;
|
||||
item.append(time, content);
|
||||
$("eventLog").prepend(item);
|
||||
recordEvent(source, type, { level, message });
|
||||
}
|
||||
|
||||
function setPhase(phase: HardwarePhase, detail: string) {
|
||||
state.phase = phase;
|
||||
const labels: Record<HardwarePhase, string> = { idle: "待机", checking: "检测中", ready: "已就绪", connecting: "连接中", active: "通话中", stopping: "结束中", completed: "已完成", error: "异常" };
|
||||
setText("phaseBadge", labels[phase]);
|
||||
setText("orbLabel", detail);
|
||||
setText("overallStatus", detail);
|
||||
$("overallStatus").className = `overall phase-${phase}`;
|
||||
$("conversationOrb").className = `conversation-orb phase-${phase}`;
|
||||
controls.check.disabled = ["connecting", "active", "stopping"].includes(phase);
|
||||
controls.start.disabled = phase !== "ready" && phase !== "completed";
|
||||
controls.stop.disabled = !["active", "error"].includes(phase) || !state.platformSession || !!state.failureTask;
|
||||
controls.sendText.disabled = phase !== "active";
|
||||
}
|
||||
|
||||
function setCheck(id: string, status: ReportCheck["status"], detail?: string) {
|
||||
const check = state.checks.get(id);
|
||||
if (!check) return;
|
||||
check.status = status;
|
||||
check.detail = detail;
|
||||
check.at = new Date().toISOString();
|
||||
renderChecks();
|
||||
}
|
||||
|
||||
function resetChecks() {
|
||||
state.checks = new Map(checkDefinitions.map(([id, label]) => [id, { id, label, status: "pending" }]));
|
||||
renderChecks();
|
||||
}
|
||||
|
||||
function renderChecks() {
|
||||
const list = $("flowChecks");
|
||||
list.replaceChildren(...[...state.checks.values()].map((check) => {
|
||||
const item = document.createElement("li");
|
||||
item.className = `flow-check flow-check--${check.status}`;
|
||||
const mark = document.createElement("i");
|
||||
mark.textContent = check.status === "passed" ? "✓" : check.status === "failed" ? "!" : "·";
|
||||
const content = document.createElement("span");
|
||||
content.textContent = check.label;
|
||||
const detail = document.createElement("small");
|
||||
detail.textContent = check.detail || (check.status === "pending" ? "等待验证" : check.status === "passed" ? "已通过" : "未通过");
|
||||
item.append(mark, content, detail);
|
||||
return item;
|
||||
}));
|
||||
}
|
||||
|
||||
function updateMeter(kind: "mic" | "agent", value: number) {
|
||||
const normalized = Math.max(0, Math.min(100, Math.round(value)));
|
||||
$(`${kind}Meter`).style.width = `${normalized}%`;
|
||||
setText(`${kind}Value`, String(normalized));
|
||||
}
|
||||
|
||||
async function loadDefaults() {
|
||||
const [platform, hardware] = await Promise.all([
|
||||
window.deviceSimulator.getRuntimeDefaults(),
|
||||
window.deviceSimulator.getHardwareRuntimeDefaults(),
|
||||
]);
|
||||
controls.apiURL.value = platform.apiURL;
|
||||
controls.deviceID.value = platform.deviceID;
|
||||
if (platform.deviceSecretConfigured) controls.deviceSecret.placeholder = "已从 Windows 安全凭据文件加载";
|
||||
controls.instanceID.value = hardware.instanceID;
|
||||
controls.productKey.value = hardware.productKey;
|
||||
controls.volcDeviceName.value = hardware.deviceName;
|
||||
controls.hardwareID.value = hardware.hardwareID;
|
||||
controls.modelMode.value = hardware.modelMode;
|
||||
if (hardware.deviceCredentialConfigured) setText("credentialDetail", "本机已有平台签发的一机一密安全缓存");
|
||||
}
|
||||
|
||||
async function enumerateMicrophones() {
|
||||
const permission = await navigator.mediaDevices.getUserMedia({ audio: true });
|
||||
permission.getTracks().forEach((track) => track.stop());
|
||||
const devices = (await navigator.mediaDevices.enumerateDevices()).filter((device) => device.kind === "audioinput");
|
||||
controls.microphone.replaceChildren(new Option("默认麦克风", ""));
|
||||
devices.forEach((device, index) => controls.microphone.add(new Option(device.label || `麦克风 ${index + 1}`, device.deviceId)));
|
||||
}
|
||||
|
||||
async function configureAndProbe() {
|
||||
if (state.failureTask) await state.failureTask;
|
||||
setPhase("checking", "正在鉴权和检测");
|
||||
resetChecks();
|
||||
state.events = [];
|
||||
state.turns = [];
|
||||
state.failure = undefined;
|
||||
try {
|
||||
await window.deviceSimulator.configure({ apiURL: controls.apiURL.value, deviceID: controls.deviceID.value, deviceSecret: controls.deviceSecret.value });
|
||||
await window.deviceSimulator.probe();
|
||||
setCheck("platform_authenticated", "passed", "设备 bootstrap/config HMAC 请求成功");
|
||||
setText("platformStatus", "鉴权成功");
|
||||
setText("platformDetail", controls.apiURL.value);
|
||||
const hardware = await window.deviceSimulator.configureHardware({ hardwareID: controls.hardwareID.value });
|
||||
controls.instanceID.value = hardware.instanceID;
|
||||
controls.productKey.value = hardware.productKey;
|
||||
controls.volcDeviceName.value = hardware.deviceName;
|
||||
controls.hardwareID.value = hardware.hardwareID;
|
||||
setCheck("credential_resolved", "passed", `${hardware.credentialSource}${hardware.persisted ? "，Device Secret 已写入 Windows 安全存储" : "，仅本次内存"}`);
|
||||
setText("credentialStatus", hardware.registered ? "平台注册成功" : "平台凭据可用");
|
||||
setText("credentialDetail", `${hardware.deviceName} · ${hardware.credentialSource} · 火山 License 额度已在动态注册时由服务端消费`);
|
||||
const network = await window.deviceSimulator.probeHardwareNetwork();
|
||||
const failures = network.endpoints.filter((endpoint) => !endpoint.ok);
|
||||
if (failures.length) throw new Error(`火山网络不可达：${failures.map((item) => `${item.hostname} ${item.error || "失败"}`).join("；")}`);
|
||||
await enumerateMicrophones();
|
||||
setPhase("ready", "可以开始真实通话");
|
||||
log("平台 v3、平台代理注册、火山网关和麦克风检测通过。", "success");
|
||||
} catch (error) {
|
||||
state.failure = errorMessage(error);
|
||||
setPhase("error", "环境检测失败");
|
||||
setText("platformStatus", "检测失败");
|
||||
setText("platformDetail", state.failure);
|
||||
log(`环境检测失败：${state.failure}`, "error");
|
||||
}
|
||||
}
|
||||
|
||||
function renderPlatformSession(created: CreatedHardwareAgentSession) {
|
||||
setText("conversationID", created.conversation_id);
|
||||
setText("botID", created.runtime.bot_id);
|
||||
setText("transport", created.runtime.transport);
|
||||
setText("configDigest", created.runtime.config_digest);
|
||||
setText("promptVersion", created.session_config.prompt_bundle_version);
|
||||
setText("providerConnectionRef", "等待火山连接");
|
||||
}
|
||||
|
||||
async function startHardwareConversation() {
|
||||
if (!["ready", "completed"].includes(state.phase)) return;
|
||||
if (state.failureTask) await state.failureTask;
|
||||
setPhase("connecting", "正在冻结本次上下文");
|
||||
resetSessionState();
|
||||
try {
|
||||
const created = await window.deviceSimulator.createHardwareAgentSession();
|
||||
state.platformSession = created;
|
||||
renderPlatformSession(created);
|
||||
if (created.runtime.transport !== "low_load") throw new Error(`设备绑定传输模式为 ${created.runtime.transport}，Windows 标准协议测试要求 low_load`);
|
||||
if (!created.session_config.authorized_context || !created.runtime.config_digest || !created.runtime.session_nonce) throw new Error("平台未返回完整冻结上下文或会话凭证");
|
||||
setCheck("context_frozen", "passed", `${created.session_config.prompt_bundle_version} · ${created.runtime.config_digest.slice(0, 12)}…`);
|
||||
recordEvent("platform", "hardware.session.ready", { conversation_id: created.conversation_id, bot_id: created.runtime.bot_id, transport: created.runtime.transport });
|
||||
setText("configStatus", "等待 session.updated");
|
||||
setText("configDetail", "麦克风仍保持关闭");
|
||||
const connected = await window.deviceSimulator.connectHardwareAgent({ platformSession: created, modelMode: controls.modelMode.value as "llm" | "s2s" });
|
||||
if (state.failureTask) { await state.failureTask; return; }
|
||||
state.providerConnectionRef = connected.providerConnectionRef;
|
||||
setText("providerConnectionRef", connected.providerConnectionRef);
|
||||
setText("providerStatus", "WebSocket 已连接");
|
||||
setCheck("provider_connected", "passed", connected.providerConnectionRef);
|
||||
setCheck("dynamic_config_applied", "passed", "火山已返回 session.updated");
|
||||
setText("configStatus", "已应用");
|
||||
setText("configDetail", "允许激活麦克风");
|
||||
state.eventSequence = 1;
|
||||
const activated = await window.deviceSimulator.hardwareAgentEvent({
|
||||
conversationID: created.conversation_id, sequenceNo: state.eventSequence, eventType: "session_started",
|
||||
providerSessionRef: created.runtime.provider_session_ref, sessionNonce: created.runtime.session_nonce,
|
||||
providerConnectionRef: connected.providerConnectionRef, botID: created.runtime.bot_id, transport: created.runtime.transport,
|
||||
configDigest: created.runtime.config_digest, dynamicConfigApplied: true,
|
||||
});
|
||||
if (activated.status !== "ACTIVE" || activated.action !== "continue") throw new Error(`平台未激活硬件会话：${activated.status}/${activated.action}`);
|
||||
if (state.failureTask) { await state.failureTask; return; }
|
||||
setCheck("platform_active", "passed", "session_started 已进入 ACTIVE");
|
||||
await ensurePlaybackNode();
|
||||
await startMicrophone();
|
||||
if (state.failureTask) { await state.failureTask; await stopAudio(); return; }
|
||||
setCheck("microphone_streaming", "passed", "PCM16/16kHz/单声道/20ms 分片");
|
||||
state.startedAt = Date.now();
|
||||
setPhase("active", "请开始说话");
|
||||
startTimers(created.session.heartbeat_interval_sec || 15);
|
||||
log("硬件对话智能体已激活：动态配置确认后才开放麦克风。", "success", "platform", "hardware.session.active");
|
||||
} catch (error) {
|
||||
state.failure ||= errorMessage(error);
|
||||
log(`启动硬件智能体失败：${state.failure}`, "error");
|
||||
await failHardwareConversation("WINDOWS_HARDWARE_START_FAILED");
|
||||
}
|
||||
}
|
||||
|
||||
function resetSessionState() {
|
||||
stopTimers();
|
||||
state.platformSession = undefined;
|
||||
state.providerConnectionRef = undefined;
|
||||
state.eventSequence = 0;
|
||||
state.transcriptSequence = 0;
|
||||
state.pendingTurns = [];
|
||||
state.transcriptUpload = undefined;
|
||||
state.startedAt = undefined;
|
||||
state.stopping = false;
|
||||
state.intentionalProviderClose = false;
|
||||
state.failureTask = undefined;
|
||||
state.failure = undefined;
|
||||
state.responseTranscripts.clear();
|
||||
state.speechStartedAt = undefined;
|
||||
state.turns = [];
|
||||
state.playbackPendingByte = undefined;
|
||||
state.playbackChunks = 0;
|
||||
state.playbackSamples = 0;
|
||||
state.playbackUnderflows = 0;
|
||||
// Device preparation remains valid, but every call must earn its own checks.
|
||||
for (const [id, label] of checkDefinitions) {
|
||||
if (!["platform_authenticated", "credential_resolved"].includes(id)) {
|
||||
state.checks.set(id, { id, label, status: "pending" });
|
||||
}
|
||||
}
|
||||
renderChecks();
|
||||
setText("duration", "00:00");
|
||||
setText("providerStatus", "网关未连接");
|
||||
setText("providerConnectionRef", "—");
|
||||
setText("conversationID", "—");
|
||||
setText("configStatus", "未应用");
|
||||
setText("configDetail", "麦克风保持关闭");
|
||||
$("liveTranscript").innerHTML = '<li class="transcript-empty">通话开始后显示火山返回的用户和智能体最终字幕。</li>';
|
||||
setText("transcriptStatus", "0 条");
|
||||
updateMeter("mic", 0);
|
||||
updateMeter("agent", 0);
|
||||
}
|
||||
|
||||
async function startMicrophone() {
|
||||
const constraints: MediaTrackConstraints = { channelCount: 1, echoCancellation: true, noiseSuppression: true, autoGainControl: true };
|
||||
if (controls.microphone.value) constraints.deviceId = { exact: controls.microphone.value };
|
||||
const stream = await navigator.mediaDevices.getUserMedia({ audio: constraints });
|
||||
const context = new AudioContext({ latencyHint: "interactive" });
|
||||
await context.audioWorklet.addModule(new URL("./pcm-capture.worklet.js", window.location.href).toString());
|
||||
const source = context.createMediaStreamSource(stream);
|
||||
const node = new AudioWorkletNode(context, "pcm16-capture", { processorOptions: { targetSampleRate: 16000, frameSamples: 320 } });
|
||||
const silent = context.createGain();
|
||||
silent.gain.value = 0;
|
||||
node.port.onmessage = (message) => {
|
||||
if (state.phase !== "active" && state.phase !== "connecting") return;
|
||||
if (message.data?.type !== "pcm" || !(message.data.bytes instanceof ArrayBuffer)) return;
|
||||
updateMeter("mic", Number(message.data.volume || 0));
|
||||
window.deviceSimulator.sendHardwareAgentAudio(message.data.bytes);
|
||||
};
|
||||
source.connect(node).connect(silent).connect(context.destination);
|
||||
await context.resume();
|
||||
state.microphoneStream = stream;
|
||||
state.captureContext = context;
|
||||
state.captureNode = node;
|
||||
}
|
||||
|
||||
async function stopAudio() {
|
||||
state.microphoneStream?.getTracks().forEach((track) => track.stop());
|
||||
state.captureNode?.disconnect();
|
||||
state.playbackNode?.port.postMessage({ type: "reset" });
|
||||
state.playbackNode?.disconnect();
|
||||
if (state.captureContext) try { await state.captureContext.close(); } catch { /* no-op */ }
|
||||
if (state.playbackContext) try { await state.playbackContext.close(); } catch { /* no-op */ }
|
||||
state.microphoneStream = undefined;
|
||||
state.captureNode = undefined;
|
||||
state.captureContext = undefined;
|
||||
state.playbackNode = undefined;
|
||||
state.playbackReady = undefined;
|
||||
state.playbackContext = undefined;
|
||||
state.playbackPendingByte = undefined;
|
||||
updateMeter("mic", 0);
|
||||
updateMeter("agent", 0);
|
||||
}
|
||||
|
||||
function ensurePlaybackNode(): Promise<AudioWorkletNode> {
|
||||
if (state.playbackNode) return Promise.resolve(state.playbackNode);
|
||||
if (state.playbackReady) return state.playbackReady;
|
||||
state.playbackReady = (async () => {
|
||||
const context = new AudioContext({ latencyHint: "interactive" });
|
||||
state.playbackContext = context;
|
||||
await context.audioWorklet.addModule(new URL("./pcm-playback.worklet.js", window.location.href).toString());
|
||||
const node = new AudioWorkletNode(context, "pcm16-playback", {
|
||||
processorOptions: { sourceSampleRate: 16000, prebufferMs: 120 },
|
||||
outputChannelCount: [1],
|
||||
});
|
||||
node.port.onmessage = (message) => {
|
||||
if (message.data?.type === "underflow") {
|
||||
state.playbackUnderflows = Number(message.data.count || state.playbackUnderflows + 1);
|
||||
recordEvent("app", "audio.playback.underflow", { count: state.playbackUnderflows });
|
||||
}
|
||||
if (message.data?.type === "drained") {
|
||||
updateMeter("agent", 0);
|
||||
setCheck("audio_playback_buffered", "passed", `${state.playbackChunks} 个分片 · ${state.playbackSamples} 个采样 · 欠载 ${state.playbackUnderflows} 次`);
|
||||
}
|
||||
};
|
||||
node.connect(context.destination);
|
||||
await context.resume();
|
||||
state.playbackNode = node;
|
||||
return node;
|
||||
})().catch(async (error) => {
|
||||
const context = state.playbackContext;
|
||||
state.playbackReady = undefined;
|
||||
state.playbackContext = undefined;
|
||||
if (context) try { await context.close(); } catch { /* no-op */ }
|
||||
throw error;
|
||||
});
|
||||
return state.playbackReady;
|
||||
}
|
||||
|
||||
async function playProviderAudio(base64: string) {
|
||||
if (!base64) return;
|
||||
const binary = atob(base64);
|
||||
const bytes = new Uint8Array(binary.length);
|
||||
for (let index = 0; index < binary.length; index += 1) bytes[index] = binary.charCodeAt(index);
|
||||
const totalBytes = bytes.byteLength + (state.playbackPendingByte === undefined ? 0 : 1);
|
||||
const sampleCount = Math.floor(totalBytes / 2);
|
||||
if (sampleCount === 0) {
|
||||
if (bytes.byteLength === 1) state.playbackPendingByte = bytes[0];
|
||||
return;
|
||||
}
|
||||
const samples = new Float32Array(sampleCount);
|
||||
let energy = 0;
|
||||
let byteIndex = 0;
|
||||
let sampleIndex = 0;
|
||||
if (state.playbackPendingByte !== undefined) {
|
||||
const value = new DataView(new Uint8Array([state.playbackPendingByte, bytes[0]]).buffer).getInt16(0, true) / 32768;
|
||||
samples[sampleIndex] = value;
|
||||
energy += value * value;
|
||||
sampleIndex += 1;
|
||||
byteIndex = 1;
|
||||
state.playbackPendingByte = undefined;
|
||||
}
|
||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||
while (sampleIndex < sampleCount) {
|
||||
const value = view.getInt16(byteIndex, true) / 32768;
|
||||
samples[sampleIndex] = value;
|
||||
energy += value * value;
|
||||
sampleIndex += 1;
|
||||
byteIndex += 2;
|
||||
}
|
||||
if (byteIndex < bytes.byteLength) state.playbackPendingByte = bytes[byteIndex];
|
||||
updateMeter("agent", Math.sqrt(energy / Math.max(1, sampleCount)) * 280);
|
||||
state.playbackChunks += 1;
|
||||
state.playbackSamples += sampleCount;
|
||||
const node = await ensurePlaybackNode();
|
||||
node.port.postMessage({ type: "push", samples: samples.buffer }, [samples.buffer]);
|
||||
setCheck("audio_playback_buffered", "passed", `已接收并缓冲 ${state.playbackChunks} 个火山音频分片`);
|
||||
}
|
||||
|
||||
function sanitizeProviderEvent(event: Record<string, unknown>) {
|
||||
const redact = (value: unknown, key = ""): unknown => {
|
||||
if (/systemmessages|system_role|system_prompt|instructions|authorized_context|speaking_style/i.test(key)) return "[REDACTED]";
|
||||
if (Array.isArray(value)) return value.map((item) => redact(item));
|
||||
if (!value || typeof value !== "object") return value;
|
||||
return Object.fromEntries(Object.entries(value as Record<string, unknown>).map(([itemKey, item]) => [itemKey, redact(item, itemKey)]));
|
||||
};
|
||||
const copy = redact(event) as Record<string, unknown>;
|
||||
if (typeof copy.delta === "string" && event.type === "response.audio.delta") copy.delta = `<pcm-base64:${copy.delta.length} chars>`;
|
||||
return copy;
|
||||
}
|
||||
|
||||
async function handleProviderEvent(event: Record<string, unknown>) {
|
||||
const type = String(event.type || "unknown");
|
||||
recordEvent("volcengine", type, sanitizeProviderEvent(event));
|
||||
if (type !== "response.audio.delta") log(`火山事件：${type}`, type === "error" ? "error" : "info", "volcengine", type);
|
||||
// Teardown can still deliver queued provider events. Do not revive a failed call.
|
||||
if (state.failureTask && type !== "transport.close") return;
|
||||
switch (type) {
|
||||
case "session.created": setText("providerStatus", "会话已创建"); break;
|
||||
case "session.updated": setText("configStatus", "已确认"); break;
|
||||
case "input_audio_buffer.speech_started": state.speechStartedAt = Date.now(); setPhase("active", "检测到用户说话"); break;
|
||||
case "input_audio_buffer.speech_stopped": setPhase("active", "智能体正在思考"); break;
|
||||
case "response.created": setPhase("active", "智能体正在生成"); break;
|
||||
case "response.audio.delta": await playProviderAudio(String(event.delta || "")); break;
|
||||
case "response.audio.done": state.playbackNode?.port.postMessage({ type: "end" }); break;
|
||||
case "conversation.item.input_audio_transcription.completed":
|
||||
await appendFinalTurn("subject", String(event.transcript || ""), String(event.event_id || crypto.randomUUID()), state.speechStartedAt); state.speechStartedAt = undefined; break;
|
||||
case "response.audio_transcript.delta": {
|
||||
const responseID = String(event.response_id || "unknown");
|
||||
state.responseTranscripts.set(responseID, `${state.responseTranscripts.get(responseID) || ""}${String(event.delta || "")}`);
|
||||
break;
|
||||
}
|
||||
case "response.audio_transcript.done": {
|
||||
const responseID = String(event.response_id || "unknown");
|
||||
const transcript = String(event.transcript || state.responseTranscripts.get(responseID) || "");
|
||||
state.responseTranscripts.delete(responseID);
|
||||
await appendFinalTurn("agent", transcript, responseID); break;
|
||||
}
|
||||
case "response.done": setPhase("active", "继续聆听"); break;
|
||||
case "platform.memory_search.completed": log(`平台记忆工具返回 ${Number(event.result_count || 0)} 条已授权记忆。`, "success", "platform", type); break;
|
||||
case "platform.memory_search.failed": log(`平台记忆工具降级：${String(event.message || "未知错误")}`, "error", "platform", type); break;
|
||||
case "error": {
|
||||
const providerError = event.error as Record<string, unknown> | undefined;
|
||||
state.failure ||= `${String(providerError?.code || providerError?.type || "PROVIDER_ERROR")}: ${String(providerError?.message || "火山智能体错误")}`;
|
||||
log(`火山错误：${state.failure}`, "error", "volcengine", "provider.error");
|
||||
await failHardwareConversation("VOLC_HARDWARE_AGENT_ERROR");
|
||||
break;
|
||||
}
|
||||
case "transport.close":
|
||||
setText("providerStatus", "连接已关闭");
|
||||
if (state.failureTask) setCheck("provider_closed", "failed", `失败清理关闭：WebSocket ${Number(event.code || 1000)}`);
|
||||
else if (state.intentionalProviderClose) setCheck("provider_closed", "passed", `WebSocket ${Number(event.code || 1000)}`);
|
||||
else if (["connecting", "active"].includes(state.phase)) { state.failure = `火山连接意外关闭：${String(event.code || "")}`; await failHardwareConversation("VOLC_CONNECTION_CLOSED"); }
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
async function appendFinalTurn(speaker: "subject" | "agent", rawText: string, sourceID: string, startedAt?: number) {
|
||||
const text = rawText.trim();
|
||||
if (!text || !state.platformSession) return;
|
||||
state.transcriptSequence += 1;
|
||||
const now = Date.now();
|
||||
const turn: DeviceFinalTurn = {
|
||||
turn_key: `hardware-${speaker}-${sourceID}`.slice(0, 240), sequence_no: state.transcriptSequence, speaker,
|
||||
start_at: new Date(startedAt || now).toISOString(), end_at: new Date(now).toISOString(), final_text: text, interrupted: false,
|
||||
};
|
||||
state.pendingTurns.push(turn);
|
||||
state.turns.push(turn);
|
||||
renderTurn(turn);
|
||||
setCheck(speaker === "subject" ? "user_transcript" : "agent_transcript", "passed", text.slice(0, 50));
|
||||
await flushTurns();
|
||||
}
|
||||
|
||||
function renderTurn(turn: DeviceFinalTurn) {
|
||||
const list = $("liveTranscript");
|
||||
list.querySelector(".transcript-empty")?.remove();
|
||||
const item = document.createElement("li");
|
||||
item.className = `transcript-turn transcript-turn--${turn.speaker}`;
|
||||
const speaker = document.createElement("strong"); speaker.textContent = turn.speaker === "subject" ? "用户" : "智能体";
|
||||
const text = document.createElement("span"); text.textContent = turn.final_text;
|
||||
item.append(speaker, text); list.append(item);
|
||||
setText("transcriptStatus", `${state.turns.length} 条`);
|
||||
}
|
||||
|
||||
async function flushTurns() {
|
||||
if (state.transcriptUpload) return state.transcriptUpload;
|
||||
if (!state.platformSession || state.pendingTurns.length === 0) return;
|
||||
const batch = state.pendingTurns.slice(0, 500);
|
||||
state.transcriptUpload = (async () => {
|
||||
try {
|
||||
const result = await window.deviceSimulator.submitTranscriptBatch({ conversationID: state.platformSession!.conversation_id, turns: batch, hardwareAgent: true });
|
||||
state.pendingTurns.splice(0, batch.length);
|
||||
setCheck("transcript_uploaded", "passed", `新增 ${result.accepted_count}，重复 ${result.duplicate_count}`);
|
||||
log(`最终字幕已上传平台：新增 ${result.accepted_count}，重复 ${result.duplicate_count}。`, "success", "platform", "hardware.turns.uploaded");
|
||||
} catch (error) { log(`字幕上传失败，保留队列待重试：${errorMessage(error)}`, "error"); }
|
||||
finally { state.transcriptUpload = undefined; }
|
||||
})();
|
||||
await state.transcriptUpload;
|
||||
}
|
||||
|
||||
async function flushTurnsWithRetry(attempts: number) {
|
||||
for (let attempt = 1; attempt <= attempts && state.pendingTurns.length > 0; attempt += 1) {
|
||||
await flushTurns();
|
||||
if (state.pendingTurns.length && attempt < attempts) await new Promise((resolve) => window.setTimeout(resolve, attempt * 500));
|
||||
}
|
||||
if (state.pendingTurns.length) throw new Error(`仍有 ${state.pendingTurns.length} 条最终字幕未上传`);
|
||||
}
|
||||
|
||||
function startTimers(intervalSeconds: number) {
|
||||
stopTimers();
|
||||
state.durationTimer = window.setInterval(() => {
|
||||
if (!state.startedAt) return;
|
||||
const seconds = Math.floor((Date.now() - state.startedAt) / 1000);
|
||||
setText("duration", `${String(Math.floor(seconds / 60)).padStart(2, "0")}:${String(seconds % 60).padStart(2, "0")}`);
|
||||
}, 1000);
|
||||
state.heartbeatTimer = window.setInterval(() => void sendHardwareHeartbeat(), Math.max(5, intervalSeconds) * 1000);
|
||||
void sendHardwareHeartbeat();
|
||||
}
|
||||
|
||||
function stopTimers() {
|
||||
if (state.durationTimer) window.clearInterval(state.durationTimer);
|
||||
if (state.heartbeatTimer) window.clearInterval(state.heartbeatTimer);
|
||||
state.durationTimer = undefined;
|
||||
state.heartbeatTimer = undefined;
|
||||
}
|
||||
|
||||
async function sendHardwareHeartbeat() {
|
||||
const created = state.platformSession;
|
||||
if (!created || state.phase !== "active" || state.stopping) return;
|
||||
try {
|
||||
state.eventSequence += 1;
|
||||
const result = await window.deviceSimulator.hardwareAgentEvent({
|
||||
conversationID: created.conversation_id, sequenceNo: state.eventSequence, eventType: "session_updated",
|
||||
providerSessionRef: created.runtime.provider_session_ref, sessionNonce: created.runtime.session_nonce,
|
||||
});
|
||||
setCheck("heartbeat_ack", "passed", `序号 ${state.eventSequence} · ${result.action}`);
|
||||
if (result.action === "stop_session") await stopHardwareConversation("platform_stop_session");
|
||||
await flushTurns();
|
||||
} catch (error) { log(`硬件会话存活上报失败：${errorMessage(error)}`, "error"); }
|
||||
}
|
||||
|
||||
async function stopHardwareConversation(reason = "windows_user_hangup") {
|
||||
if (state.failureTask) return state.failureTask;
|
||||
if (!state.platformSession || state.stopping) return;
|
||||
state.stopping = true;
|
||||
setPhase("stopping", "正在可靠结束");
|
||||
stopTimers();
|
||||
const created = state.platformSession;
|
||||
try {
|
||||
await stopAudio();
|
||||
await flushTurnsWithRetry(3);
|
||||
state.intentionalProviderClose = true;
|
||||
await window.deviceSimulator.closeHardwareAgent({ code: 1000, reason });
|
||||
setCheck("provider_closed", "passed", "设备主动关闭 WebSocket");
|
||||
state.eventSequence += 1;
|
||||
const ended = await window.deviceSimulator.hardwareAgentEvent({
|
||||
conversationID: created.conversation_id, sequenceNo: state.eventSequence, eventType: "session_ended",
|
||||
providerSessionRef: created.runtime.provider_session_ref, sessionNonce: created.runtime.session_nonce,
|
||||
});
|
||||
if (ended.status !== "COMPLETED") throw new Error(`平台结束状态不是 COMPLETED：${ended.status}`);
|
||||
setCheck("platform_completed", "passed", "session_ended 已进入 COMPLETED");
|
||||
state.stopping = false;
|
||||
setPhase("completed", allRequiredChecksPassed() ? "完整流程通过" : "通话完成，部分验收项待覆盖");
|
||||
log("硬件对话智能体会话已正常结束，平台状态为 COMPLETED。", "success", "platform", "hardware.session.completed");
|
||||
} catch (error) {
|
||||
state.failure = errorMessage(error);
|
||||
state.stopping = false;
|
||||
setPhase("error", "结束流程失败");
|
||||
log(`结束硬件会话失败：${state.failure}`, "error");
|
||||
}
|
||||
}
|
||||
|
||||
function failHardwareConversation(errorCode: string): Promise<void> {
|
||||
// The provider error event and connect() rejection describe the same failure.
|
||||
// Install a shared task before any asynchronous cleanup, and retain it until
|
||||
// the next call so later errors/hangup cannot submit another terminal event.
|
||||
if (state.failureTask) return state.failureTask;
|
||||
const created = state.platformSession;
|
||||
state.failureTask = Promise.resolve().then(async () => {
|
||||
try {
|
||||
await stopAudio();
|
||||
try { await window.deviceSimulator.closeHardwareAgent({ code: 1011, reason: errorCode }); } catch { /* no-op */ }
|
||||
if (created) {
|
||||
try {
|
||||
state.eventSequence += 1;
|
||||
await window.deviceSimulator.hardwareAgentEvent({
|
||||
conversationID: created.conversation_id, sequenceNo: state.eventSequence, eventType: "session_failed",
|
||||
providerSessionRef: created.runtime.provider_session_ref, sessionNonce: created.runtime.session_nonce,
|
||||
errorCode,
|
||||
});
|
||||
} catch (error) { log(`平台失败事件回报未成功：${errorMessage(error)}`, "error"); }
|
||||
}
|
||||
} finally {
|
||||
state.stopping = false;
|
||||
setPhase("error", state.failure || "硬件会话失败");
|
||||
}
|
||||
});
|
||||
stopTimers();
|
||||
state.stopping = true;
|
||||
state.intentionalProviderClose = true;
|
||||
setPhase("stopping", "正在清理失败会话");
|
||||
setCheck("platform_completed", "failed", state.failure || errorCode);
|
||||
if (!state.providerConnectionRef) {
|
||||
setText("configStatus", "未完成");
|
||||
setCheck("dynamic_config_applied", "failed", state.failure || errorCode);
|
||||
}
|
||||
return state.failureTask;
|
||||
}
|
||||
|
||||
function allRequiredChecksPassed() {
|
||||
const required = ["platform_authenticated", "credential_resolved", "context_frozen", "provider_connected", "dynamic_config_applied", "platform_active", "microphone_streaming", "audio_playback_buffered", "heartbeat_ack", "transcript_uploaded", "provider_closed", "platform_completed"];
|
||||
return required.every((id) => state.checks.get(id)?.status === "passed");
|
||||
}
|
||||
|
||||
async function sendTextQuestion() {
|
||||
if (state.phase !== "active" || state.failureTask) return;
|
||||
const text = controls.textQuestion.value.trim();
|
||||
if (!text) return;
|
||||
try {
|
||||
await window.deviceSimulator.sendHardwareAgentText({ text, interruptMode: 1 });
|
||||
// 文本输入没有 ASR final 事件，设备本地将已被火山接受的最终输入作为用户话轮上报，
|
||||
// 这样文本验收与真实语音验收都会进入同一套平台聊天记录。
|
||||
await appendFinalTurn("subject", text, `text-${crypto.randomUUID()}`);
|
||||
controls.textQuestion.value = "";
|
||||
log(`已通过硬件标准协议发送文本问题：${text}`, "success", "app", "hardware.text.sent");
|
||||
} catch (error) { log(`发送文本问题失败：${errorMessage(error)}`, "error"); }
|
||||
}
|
||||
|
||||
function buildReport(): Record<string, unknown> {
|
||||
return {
|
||||
schema_version: "hardware-agent-windows-e2e-report-v1",
|
||||
generated_at: new Date().toISOString(),
|
||||
result: state.phase === "completed" && allRequiredChecksPassed() ? "PASS" : state.failure ? "FAIL" : "INCOMPLETE",
|
||||
platform_api: controls.apiURL.value,
|
||||
platform_device_id: controls.deviceID.value,
|
||||
volc_identity: { instance_id: controls.instanceID.value, product_key: controls.productKey.value, device_name: controls.volcDeviceName.value, hardware_id: controls.hardwareID.value },
|
||||
conversation_id: state.platformSession?.conversation_id,
|
||||
bot_id: state.platformSession?.runtime.bot_id,
|
||||
model_mode: controls.modelMode.value,
|
||||
transport: state.platformSession?.runtime.transport,
|
||||
audio_playback: {
|
||||
source_format: "pcm_s16le_16000_mono",
|
||||
mode: "audio_worklet_ring_buffer",
|
||||
prebuffer_ms: 120,
|
||||
chunks: state.playbackChunks,
|
||||
samples: state.playbackSamples,
|
||||
underflows: state.playbackUnderflows,
|
||||
},
|
||||
provider_connection_ref: state.providerConnectionRef,
|
||||
config_digest: state.platformSession?.runtime.config_digest,
|
||||
prompt_bundle_version: state.platformSession?.session_config.prompt_bundle_version,
|
||||
failure: state.failure,
|
||||
checks: [...state.checks.values()],
|
||||
turns: state.turns,
|
||||
events: state.events,
|
||||
};
|
||||
}
|
||||
|
||||
async function exportReport() {
|
||||
try {
|
||||
const result = await window.deviceSimulator.saveHardwareReport(buildReport());
|
||||
if (result.saved) log(`测试报告已保存：${result.path}`, "success");
|
||||
} catch (error) { log(`导出测试报告失败：${errorMessage(error)}`, "error"); }
|
||||
}
|
||||
|
||||
const unsubscribeProvider = window.deviceSimulator.onHardwareProviderEvent((event) => void handleProviderEvent(event));
|
||||
controls.check.addEventListener("click", () => void configureAndProbe());
|
||||
controls.start.addEventListener("click", () => void startHardwareConversation());
|
||||
controls.stop.addEventListener("click", () => void stopHardwareConversation());
|
||||
controls.sendText.addEventListener("click", () => void sendTextQuestion());
|
||||
controls.textQuestion.addEventListener("keydown", (event) => { if (event.key === "Enter") void sendTextQuestion(); });
|
||||
$("exportReport").addEventListener("click", () => void exportReport());
|
||||
$("clearLog").addEventListener("click", () => $("eventLog").replaceChildren());
|
||||
window.addEventListener("beforeunload", () => {
|
||||
unsubscribeProvider();
|
||||
stopTimers();
|
||||
void stopAudio();
|
||||
void window.deviceSimulator.closeHardwareAgent({ code: 1001, reason: "window closed" });
|
||||
});
|
||||
|
||||
renderChecks();
|
||||
setPhase("idle", "尚未检测");
|
||||
void loadDefaults().catch((error) => log(`读取默认配置失败：${errorMessage(error)}`, "error"));
|
||||
@@ -16,10 +16,15 @@ button:disabled { cursor: not-allowed; opacity: .45; }
|
||||
.eyebrow { color: #c8652a; font-size: 12px; font-weight: 800; letter-spacing: .16em; margin: 0 0 9px; }
|
||||
h1 { margin: 0; font-family: Georgia, "Microsoft YaHei UI", serif; font-size: 44px; letter-spacing: -.035em; }
|
||||
h2 { margin: 0; font-size: 23px; }
|
||||
h3 { margin: 24px 0 14px; color: #315966; font-size: 14px; }
|
||||
.subtitle { max-width: 760px; color: #66808e; line-height: 1.8; margin: 12px 0 0; }
|
||||
.overall { display: flex; align-items: center; min-width: 150px; justify-content: center; padding: 13px 18px; border-radius: 999px; background: rgba(255,255,255,.82); box-shadow: 0 12px 30px rgba(36,64,69,.1); color: #647a82; }
|
||||
.overall.phase-ready, .overall.phase-active { color: #1a7d57; }
|
||||
.overall.phase-error { color: #b63d2e; }
|
||||
.overall.phase-completed { color: #1a7d57; }
|
||||
.hero-actions { display: grid; justify-items: end; gap: 10px; }
|
||||
.legacy-link { color: #617985; font-size: 12px; text-decoration: none; border-bottom: 1px solid #b9c8c7; padding-bottom: 2px; }
|
||||
.legacy-link:hover { color: #c8652a; }
|
||||
|
||||
.status-grid { display: grid; grid-template-columns: repeat(4, 1fr); gap: 14px; margin-bottom: 16px; }
|
||||
.status-card, .panel { background: rgba(255,255,255,.9); border: 1px solid rgba(23,61,69,.08); box-shadow: 0 14px 40px rgba(34,61,63,.08); }
|
||||
@@ -77,7 +82,27 @@ input:focus, select:focus { border-color: #d6783f; box-shadow: 0 0 0 3px rgba(21
|
||||
.transcript-turn strong { color: #315966; font-size: 12px; }
|
||||
.transcript-turn span { color: #263f49; font-size: 14px; line-height: 1.55; }
|
||||
|
||||
.hardware-shell { width: min(1260px, calc(100vw - 48px)); }
|
||||
.hardware-content-grid { grid-template-columns: 1.05fr 1.2fr; }
|
||||
.hardware-call-panel { min-height: 720px; }
|
||||
.hardware-session-info { grid-template-columns: 1fr 1fr; }
|
||||
.text-question { display: grid; grid-template-columns: 1fr 150px; gap: 10px; margin-bottom: 22px; }
|
||||
.text-question input { margin-top: 0; }
|
||||
.text-question .secondary { padding: 10px; }
|
||||
.flow-panel { margin-top: 16px; }
|
||||
.flow-panel .panel-heading { margin-bottom: 14px; }
|
||||
.flow-checks { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 10px; list-style: none; padding: 0; margin: 0; }
|
||||
.flow-check { display: grid; grid-template-columns: 28px 1fr; grid-template-rows: auto auto; column-gap: 10px; align-items: center; padding: 13px; border: 1px solid #e3ebe8; border-radius: 14px; background: #fafcfb; }
|
||||
.flow-check i { grid-row: 1 / span 2; display: grid; place-items: center; width: 26px; height: 26px; border-radius: 50%; font-style: normal; font-weight: 800; color: #809399; background: #e8efed; }
|
||||
.flow-check span { min-width: 0; color: #31505e; font-size: 13px; font-weight: 700; }
|
||||
.flow-check small { min-width: 0; margin-top: 4px; color: #8b9ba1; font-size: 11px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
||||
.flow-check--passed { border-color: #cce5d7; background: #f4faf6; }
|
||||
.flow-check--passed i { color: white; background: #2f8d67; }
|
||||
.flow-check--failed { border-color: #edcbc5; background: #fff7f5; }
|
||||
.flow-check--failed i { color: white; background: #b94c3d; }
|
||||
|
||||
@media (max-width: 1050px) {
|
||||
.status-grid { grid-template-columns: repeat(2, 1fr); }
|
||||
.content-grid { grid-template-columns: 1fr; }
|
||||
.flow-checks { grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||
}
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
const crypto = require("node:crypto");
|
||||
const { WebSocketServer } = require("ws");
|
||||
const { websocketSignature } = require("../electron/hardware-agent-client.cjs");
|
||||
|
||||
const port = Number(process.env.MOCK_HARDWARE_GATEWAY_PORT || 19090);
|
||||
const expectedSecret = process.env.MOCK_VOLC_DEVICE_SECRET || "mock-volc-device-secret-at-least-32-characters";
|
||||
const server = new WebSocketServer({ host: "127.0.0.1", port, maxPayload: 1024 * 1024 });
|
||||
|
||||
function send(socket, value) { socket.send(JSON.stringify(value)); }
|
||||
async function requestPlatformStop(conversationID) {
|
||||
const baseURL = String(process.env.MOCK_PLATFORM_API_URL || "").replace(/\/$/, "");
|
||||
const deviceID = String(process.env.MOCK_PLATFORM_DEVICE_ID || "");
|
||||
const secret = String(process.env.MOCK_PLATFORM_DEVICE_SECRET || "");
|
||||
if (!baseURL || !deviceID || !secret || !conversationID) return;
|
||||
const requestPath = `/device/v2/conversations/${encodeURIComponent(conversationID)}/stop`;
|
||||
const body = JSON.stringify({ reason: "windows_e2e_platform_stop", device_time: new Date().toISOString(), last_heartbeat_sequence: 1, rtc_left: true, recording: { near_segments: 0, far_segments: 0 } });
|
||||
const timestamp = new Date().toISOString();
|
||||
const nonce = crypto.randomBytes(16).toString("hex");
|
||||
const digest = crypto.createHash("sha256").update(body).digest("hex");
|
||||
const canonical = ["POST", requestPath, timestamp, nonce, digest].join("\n");
|
||||
const signature = crypto.createHmac("sha256", secret).update(canonical).digest("base64url");
|
||||
const response = await fetch(`${baseURL}${requestPath}`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Accept: "application/json", "Content-Type": "application/json", "Idempotency-Key": `windows-platform-stop-${crypto.randomUUID()}`,
|
||||
"X-Device-Id": deviceID, "X-Device-Timestamp": timestamp, "X-Device-Nonce": nonce,
|
||||
"X-Device-Content-SHA256": digest, "X-Device-Signature": signature,
|
||||
},
|
||||
body,
|
||||
});
|
||||
if (!response.ok) throw new Error(`platform stop HTTP ${response.status}: ${await response.text()}`);
|
||||
process.stdout.write(`${JSON.stringify({ type: "platform_stop_requested", conversation_id: conversationID })}\n`);
|
||||
}
|
||||
function tonePCMBase64() {
|
||||
const samples = 2400;
|
||||
const bytes = Buffer.alloc(samples * 2);
|
||||
for (let index = 0; index < samples; index += 1) {
|
||||
const sample = Math.round(Math.sin((index / 16000) * Math.PI * 2 * 440) * 1800);
|
||||
bytes.writeInt16LE(sample, index * 2);
|
||||
}
|
||||
return bytes.toString("base64");
|
||||
}
|
||||
|
||||
server.on("listening", () => process.stdout.write(`${JSON.stringify({ type: "listening", port })}\n`));
|
||||
server.on("connection", (socket, request) => {
|
||||
const header = (name) => String(request.headers[name.toLowerCase()] || "");
|
||||
const expectedSignature = websocketSignature({
|
||||
authType: 1,
|
||||
deviceName: header("x-device-name"),
|
||||
randomNumber: Number(header("x-random-num")),
|
||||
productKey: header("x-product-key"),
|
||||
timestamp: Number(header("x-timestamp")),
|
||||
instanceID: header("x-instance-id"),
|
||||
}, expectedSecret);
|
||||
if (!crypto.timingSafeEqual(Buffer.from(header("x-signature")), Buffer.from(expectedSignature))) {
|
||||
socket.close(1008, "invalid signature");
|
||||
return;
|
||||
}
|
||||
const target = new URL(request.url, `ws://127.0.0.1:${port}`);
|
||||
if (!target.searchParams.get("bot") || target.searchParams.get("wait_for_session_update") !== "true") {
|
||||
socket.close(1008, "invalid query");
|
||||
return;
|
||||
}
|
||||
process.stdout.write(`${JSON.stringify({ type: "authenticated", device: header("x-device-name"), bot: target.searchParams.get("bot") })}\n`);
|
||||
send(socket, { type: "session.created", event_id: "mock-created", session: { id: "mock-provider-session-windows", model: "doubao", input_audio_format: "pcm16", output_audio_format: "pcm16", turn_detection: { type: "server_vad" } } });
|
||||
let replied = false;
|
||||
let dialogID = "";
|
||||
socket.on("message", (raw) => {
|
||||
const event = JSON.parse(raw.toString("utf8"));
|
||||
if (event.type === "session.update") {
|
||||
const llmPrompt = event.session?.config?.LLMConfig?.SystemMessages?.[0];
|
||||
const s2sDialog = event.session?.config?.S2SConfig?.ProviderParams?.dialog;
|
||||
const sessionDialogID = String(s2sDialog?.dialog_id || (llmPrompt ? llmPrompt.match(/本次对话标识：([^\n]+)/)?.[1] : "") || "").trim();
|
||||
const frozenPrompt = String(llmPrompt || s2sDialog?.system_role || "").trim();
|
||||
if (event.session?.object !== "realtime.session" || event.session?.config?.ASRConfig?.TurnDetectionMode !== 0 || !sessionDialogID || !frozenPrompt) {
|
||||
send(socket, { type: "error", event_id: "mock-error", error: { code: "INVALID_SESSION_CONFIG", message: "frozen context missing" } });
|
||||
return;
|
||||
}
|
||||
dialogID = sessionDialogID;
|
||||
send(socket, { type: "session.updated", event_id: "mock-updated", session: { ...event.session, id: "mock-provider-session-windows" } });
|
||||
process.stdout.write(`${JSON.stringify({ type: "dynamic_config_applied", dialog_id: dialogID })}\n`);
|
||||
return;
|
||||
}
|
||||
if (event.type === "input_audio_buffer.append" && !replied) {
|
||||
replied = true;
|
||||
send(socket, { type: "input_audio_buffer.speech_started", event_id: "mock-speech-start", audio_start_ms: 0 });
|
||||
send(socket, { type: "input_audio_buffer.speech_stopped", event_id: "mock-speech-stop", audio_end_ms: 800 });
|
||||
send(socket, { type: "conversation.item.input_audio_transcription.completed", event_id: "mock-user-final", item_id: "mock-user-item", transcript: "你好，请介绍一下今天的安排。" });
|
||||
send(socket, { type: "response.created", event_id: "mock-response-created", response: { id: "mock-response-1", status: "in_progress" } });
|
||||
send(socket, { type: "response.audio_transcript.delta", event_id: "mock-agent-delta", response_id: "mock-response-1", delta: "您好，今天可以先聊聊您最想做的事情。" });
|
||||
send(socket, { type: "response.audio.delta", event_id: "mock-audio-delta", response_id: "mock-response-1", delta: tonePCMBase64() });
|
||||
send(socket, { type: "response.audio_transcript.done", event_id: "mock-agent-final", response_id: "mock-response-1", transcript: "您好，今天可以先聊聊您最想做的事情。" });
|
||||
send(socket, { type: "response.audio.done", event_id: "mock-audio-done", response_id: "mock-response-1" });
|
||||
send(socket, { type: "response.done", event_id: "mock-response-done", response: { id: "mock-response-1", status: "completed", output: [] } });
|
||||
process.stdout.write(`${JSON.stringify({ type: "round_completed" })}\n`);
|
||||
if (process.env.MOCK_REQUEST_PLATFORM_STOP === "1") {
|
||||
setTimeout(() => void requestPlatformStop(dialogID).catch((error) => {
|
||||
process.stderr.write(`${error.stack || error.message}\n`);
|
||||
}), 250);
|
||||
}
|
||||
}
|
||||
});
|
||||
socket.on("close", (code) => process.stdout.write(`${JSON.stringify({ type: "closed", code })}\n`));
|
||||
});
|
||||
|
||||
function shutdown() { server.close(() => process.exit(0)); }
|
||||
process.on("SIGINT", shutdown);
|
||||
process.on("SIGTERM", shutdown);
|
||||
@@ -0,0 +1,120 @@
|
||||
param(
|
||||
[ValidateSet('llm', 's2s')]
|
||||
[string]$ModelMode = 'llm',
|
||||
[string]$ExecutablePath = ''
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$PSNativeCommandUseErrorActionPreference = $true
|
||||
|
||||
$simulatorRoot = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot '..')).Path
|
||||
$projectRoot = (Resolve-Path -LiteralPath (Join-Path $simulatorRoot '..')).Path
|
||||
$devicePublicID = '10000000-0000-0000-0000-000000000511'
|
||||
$platformSecret = 'local-simulated-device-secret-32-bytes-minimum'
|
||||
$volcSecret = 'mock-volc-device-secret-at-least-32-characters'
|
||||
$mockPort = 19090
|
||||
$runID = [Guid]::NewGuid().ToString('N')
|
||||
$resultPath = Join-Path $env:TEMP "hardware-agent-e2e-$runID.json"
|
||||
$mockOutput = Join-Path $env:TEMP "hardware-agent-gateway-$runID.log"
|
||||
$mockError = Join-Path $env:TEMP "hardware-agent-gateway-$runID.err.log"
|
||||
$mockProcess = $null
|
||||
$conversationID = $null
|
||||
$deviceInternalID = $null
|
||||
|
||||
Set-Location -LiteralPath $projectRoot
|
||||
try {
|
||||
$ready = Invoke-WebRequest -UseBasicParsing -Uri 'http://localhost:58080/health/ready' -TimeoutSec 10
|
||||
if ($ready.StatusCode -ne 200) { throw 'Local API is not ready' }
|
||||
$deviceInternalID = (docker compose exec -T postgres psql -U voice -d voice_platform -t -A -c "SELECT id FROM device WHERE public_id='$devicePublicID'::uuid" | Out-String).Trim()
|
||||
if ($deviceInternalID -notmatch '^[0-9a-fA-F-]{36}$') { throw 'Local simulator device is missing' }
|
||||
$bindingCount = (docker compose exec -T postgres psql -U voice -d voice_platform -t -A -c "SELECT count(*) FROM hardware_agent_device_binding WHERE device_id='$deviceInternalID'::uuid" | Out-String).Trim()
|
||||
if ($bindingCount -ne '0') { throw 'Simulator device already has a hardware binding; refusing to overwrite it' }
|
||||
"UPDATE device_twin SET desired_config=desired_config||'{`"voice_runtime`":`"volc_hardware_agent`",`"hardware_agent_transport`":`"low_load`",`"minimum_sdk_version`":`"1.0.0`"}'::jsonb WHERE device_id='$deviceInternalID'::uuid; INSERT INTO hardware_agent_device_binding(tenant_id,device_id,bot_id,transport,minimum_sdk_version,status) SELECT tenant_id,id,'mock-windows-hardware-bot','low_load','1.0.0','ACTIVE' FROM device WHERE id='$deviceInternalID'::uuid;" | docker compose exec -T postgres psql -U voice -d voice_platform -v ON_ERROR_STOP=1 -q | Out-Null
|
||||
|
||||
$mockScript = Join-Path $simulatorRoot 'test\mock-hardware-gateway.cjs'
|
||||
$env:MOCK_HARDWARE_GATEWAY_PORT = [string]$mockPort
|
||||
$env:MOCK_VOLC_DEVICE_SECRET = $volcSecret
|
||||
$env:MOCK_REQUEST_PLATFORM_STOP = '1'
|
||||
$env:MOCK_PLATFORM_API_URL = 'http://localhost:58080'
|
||||
$env:MOCK_PLATFORM_DEVICE_ID = $devicePublicID
|
||||
$env:MOCK_PLATFORM_DEVICE_SECRET = $platformSecret
|
||||
$mockProcess = Start-Process -FilePath 'node' -ArgumentList @($mockScript) -PassThru -WindowStyle Hidden -RedirectStandardOutput $mockOutput -RedirectStandardError $mockError
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds(10)
|
||||
do {
|
||||
Start-Sleep -Milliseconds 150
|
||||
$listening = Test-NetConnection 127.0.0.1 -Port $mockPort -InformationLevel Quiet -WarningAction SilentlyContinue
|
||||
if ($mockProcess.HasExited) { throw "Mock gateway exited: $(Get-Content -LiteralPath $mockError -Raw -ErrorAction SilentlyContinue)" }
|
||||
} until ($listening -or [DateTime]::UtcNow -gt $deadline)
|
||||
if (-not $listening) { throw 'Mock hardware gateway did not start' }
|
||||
|
||||
$env:SIMULATOR_AUTO_TEST = '1'
|
||||
$env:SIMULATOR_AUTO_WAIT_FOR_PLATFORM_STOP = '1'
|
||||
$env:SIMULATOR_AUTO_RESULT_PATH = $resultPath
|
||||
$env:SIMULATOR_DEFAULT_API_URL = 'http://localhost:58080'
|
||||
$env:SIMULATOR_DEFAULT_DEVICE_ID = $devicePublicID
|
||||
$env:SIMULATOR_DEFAULT_DEVICE_SECRET = $platformSecret
|
||||
$env:SIMULATOR_VOLC_INSTANCE_ID = 'mock-instance'
|
||||
$env:SIMULATOR_VOLC_PRODUCT_KEY = 'mock-product'
|
||||
$env:SIMULATOR_VOLC_DEVICE_NAME = 'windows-e2e-device'
|
||||
$env:SIMULATOR_VOLC_HARDWARE_ID = 'windows-e2e-hardware'
|
||||
$env:SIMULATOR_VOLC_DEVICE_SECRET = $volcSecret
|
||||
$env:SIMULATOR_VOLC_MODEL_MODE = $ModelMode
|
||||
$env:SIMULATOR_HARDWARE_GATEWAY_URL = "ws://127.0.0.1:$mockPort/v1/realtime"
|
||||
Set-Location -LiteralPath $simulatorRoot
|
||||
if ($ExecutablePath) {
|
||||
$resolvedExecutable = (Resolve-Path -LiteralPath $ExecutablePath).Path
|
||||
$electron = Start-Process -FilePath $resolvedExecutable -WorkingDirectory (Split-Path -Parent $resolvedExecutable) -PassThru -Wait
|
||||
} else {
|
||||
$electron = Start-Process -FilePath (Join-Path $simulatorRoot 'node_modules\electron\dist\electron.exe') -ArgumentList @('.') -WorkingDirectory $simulatorRoot -PassThru -Wait
|
||||
}
|
||||
if ($electron.ExitCode -ne 0) { throw "Electron E2E process failed with exit code $($electron.ExitCode)" }
|
||||
if (-not (Test-Path -LiteralPath $resultPath -PathType Leaf)) { throw 'Electron E2E result file is missing' }
|
||||
$result = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json
|
||||
if ($result.result -ne 'PASS' -or $result.phase -ne '已完成') { throw "Electron E2E failed: $($result | ConvertTo-Json -Depth 8 -Compress)" }
|
||||
if ($result.model_mode -ne $ModelMode) { throw "Electron E2E used model mode '$($result.model_mode)' instead of '$ModelMode'" }
|
||||
$conversationID = [string]$result.conversation_id
|
||||
if ($conversationID -notmatch '^[0-9a-fA-F-]{36}$') { throw 'Electron E2E did not return a conversation ID' }
|
||||
$databaseState = (docker compose exec -T postgres psql -U voice -d voice_platform -t -A -F '|' -c "SELECT c.runtime_type||'|'||c.status||'|'||hs.status||'|'||hs.sdk_version||'|'||(hs.dynamic_config_applied_at IS NOT NULL)::text||'|'||count(DISTINCT ct.id)||'|'||(SELECT count(*) FROM web_event we WHERE we.resource_public_id=c.public_id AND we.event_type IN('conversation.started','conversation.completed')) FROM conversation c JOIN hardware_agent_session hs ON hs.conversation_id=c.id LEFT JOIN conversation_turn ct ON ct.conversation_id=c.id WHERE c.public_id='$conversationID'::uuid GROUP BY c.runtime_type,c.status,hs.status,hs.sdk_version,hs.dynamic_config_applied_at,c.public_id" | Out-String).Trim()
|
||||
if ($databaseState -ne 'volc_hardware_agent|COMPLETED|COMPLETED|1.1.0-windows-standard-protocol|true|2|2') { throw "Unexpected platform state: $databaseState" }
|
||||
$gatewayEvents = Get-Content -LiteralPath $mockOutput -Raw
|
||||
if ($gatewayEvents -notmatch 'dynamic_config_applied' -or $gatewayEvents -notmatch 'round_completed' -or $gatewayEvents -notmatch 'platform_stop_requested') { throw "Mock gateway did not observe config, media, and platform-stop flow: $gatewayEvents" }
|
||||
[ordered]@{
|
||||
result = 'PASS'
|
||||
conversation_id = $conversationID
|
||||
platform_state = $databaseState
|
||||
transcript_turns = $result.turns.Count
|
||||
checks_passed = @($result.checks | Where-Object status -eq 'passed').Count
|
||||
model_mode = $result.model_mode
|
||||
mock_gateway_log = $mockOutput
|
||||
} | ConvertTo-Json -Compress
|
||||
}
|
||||
finally {
|
||||
Set-Location -LiteralPath $projectRoot
|
||||
if ($mockProcess -and -not $mockProcess.HasExited) { Stop-Process -Id $mockProcess.Id -Force -ErrorAction SilentlyContinue }
|
||||
if ($conversationID -match '^[0-9a-fA-F-]{36}$') {
|
||||
@"
|
||||
BEGIN;
|
||||
CREATE TEMP TABLE hc AS SELECT id,context_snapshot_id,consent_snapshot_id FROM conversation WHERE public_id='$conversationID'::uuid;
|
||||
DELETE FROM audit_event WHERE resource_id IN(SELECT id FROM hc);
|
||||
DELETE FROM web_event WHERE resource_public_id='$conversationID'::uuid;
|
||||
DELETE FROM idempotency_record WHERE resource_id IN(SELECT id FROM hc);
|
||||
DELETE FROM conversation_turn WHERE conversation_id IN(SELECT id FROM hc);
|
||||
DELETE FROM conversation_memory_use WHERE conversation_id IN(SELECT id FROM hc);
|
||||
DELETE FROM memory_tool_call WHERE conversation_id IN(SELECT id FROM hc);
|
||||
DELETE FROM hardware_agent_event WHERE conversation_id IN(SELECT id FROM hc);
|
||||
DELETE FROM hardware_agent_session WHERE conversation_id IN(SELECT id FROM hc);
|
||||
DELETE FROM conversation_usage_budget WHERE conversation_id IN(SELECT id FROM hc);
|
||||
DELETE FROM outbox_event WHERE aggregate_id IN(SELECT id FROM hc);
|
||||
DELETE FROM conversation_state_history WHERE conversation_id IN(SELECT id FROM hc);
|
||||
DELETE FROM conversation WHERE id IN(SELECT id FROM hc);
|
||||
DELETE FROM context_snapshot WHERE id IN(SELECT context_snapshot_id FROM hc);
|
||||
DELETE FROM consent_snapshot WHERE id IN(SELECT consent_snapshot_id FROM hc);
|
||||
COMMIT;
|
||||
"@ | docker compose exec -T postgres psql -U voice -d voice_platform -v ON_ERROR_STOP=1 -q | Out-Null
|
||||
}
|
||||
if ($deviceInternalID -match '^[0-9a-fA-F-]{36}$') {
|
||||
"DELETE FROM hardware_agent_device_binding WHERE device_id='$deviceInternalID'::uuid; UPDATE device_twin SET desired_config=desired_config-'voice_runtime'-'hardware_agent_transport'-'minimum_sdk_version' WHERE device_id='$deviceInternalID'::uuid;" | docker compose exec -T postgres psql -U voice -d voice_platform -v ON_ERROR_STOP=1 -q | Out-Null
|
||||
}
|
||||
@('MOCK_HARDWARE_GATEWAY_PORT','MOCK_VOLC_DEVICE_SECRET','MOCK_REQUEST_PLATFORM_STOP','MOCK_PLATFORM_API_URL','MOCK_PLATFORM_DEVICE_ID','MOCK_PLATFORM_DEVICE_SECRET','SIMULATOR_AUTO_TEST','SIMULATOR_AUTO_WAIT_FOR_PLATFORM_STOP','SIMULATOR_AUTO_RESULT_PATH','SIMULATOR_DEFAULT_API_URL','SIMULATOR_DEFAULT_DEVICE_ID','SIMULATOR_DEFAULT_DEVICE_SECRET','SIMULATOR_VOLC_INSTANCE_ID','SIMULATOR_VOLC_PRODUCT_KEY','SIMULATOR_VOLC_DEVICE_NAME','SIMULATOR_VOLC_HARDWARE_ID','SIMULATOR_VOLC_DEVICE_SECRET','SIMULATOR_VOLC_MODEL_MODE','SIMULATOR_HARDWARE_GATEWAY_URL') | ForEach-Object { Remove-Item "Env:$_" -ErrorAction SilentlyContinue }
|
||||
Remove-Item -LiteralPath $resultPath,$mockOutput,$mockError -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
import { defineConfig } from "vite";
|
||||
import { resolve } from "node:path";
|
||||
|
||||
export default defineConfig({
|
||||
base: "./",
|
||||
@@ -6,5 +7,11 @@ export default defineConfig({
|
||||
outDir: "dist",
|
||||
emptyOutDir: true,
|
||||
sourcemap: true,
|
||||
rollupOptions: {
|
||||
input: {
|
||||
hardware: resolve(__dirname, "index.html"),
|
||||
legacy: resolve(__dirname, "legacy.html"),
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
@@ -1,21 +1,24 @@
|
||||
云端语音测试设备使用说明
|
||||
云端硬件对话智能体测试程序使用说明
|
||||
|
||||
1. 将整个 ZIP 解压到普通文件夹，不要只从压缩包内直接运行。
|
||||
2. 双击“启动云端测试设备.cmd”，不要直接双击 EXE。
|
||||
3. 首次运行时允许 Windows 麦克风访问。
|
||||
4. 点击“检测平台和音频设备”，检测通过后即可开始语音聊天。
|
||||
1. 将 ZIP 完整解压到普通文件夹，不要直接在压缩包内运行。
|
||||
2. 双击“启动云端测试设备.cmd”，或运行 hardware-agent-tester-0.3.2-windows-x64.exe。
|
||||
3. 首次运行允许 Windows 麦克风访问。
|
||||
4. 填写平台设备信息和 Hardware ID；火山 Instance ID、Product Key、Device Name 由平台自动下发。
|
||||
5. Product Secret 只配置在平台密钥服务中，测试程序不会要求输入；平台代设备注册后只下发本设备的一机一密。
|
||||
6. 根据火山控制台中的 Bot 类型选择“通用 LLM”或“豆包端到端语音”。选错会导致 session.update 校验失败。
|
||||
7. 点击“检测、鉴权并准备设备”，13项验收区开始记录证据。
|
||||
8. 检测通过后点击“开始硬件智能体通话”。只有收到火山 session.updated 且平台确认 ACTIVE 后才会开放麦克风。
|
||||
9. 完成双方对话后点击结束；也可以从平台停止通话，程序会在 session_updated 回执中收到 stop_session 并自动结束。
|
||||
10. 点击“导出测试报告”，将 JSON 报告交给开发人员验收。
|
||||
|
||||
云端后台：https://121.41.46.238
|
||||
测试设备：10000000-0000-0000-0000-000000000511
|
||||
平台维护中心必须把本设备设置为 volc_hardware_agent + low_load，并绑定与火山产品关联的真实 Bot ID。公网平台尚未部署 v3 路由时会返回 404，此时不能进行真实测试。
|
||||
|
||||
网络需要允许 HTTPS 443、WSS 以及火山 RTC 所需的出站网络连接。
|
||||
网络需要允许 HTTPS 443 和 WSS 到：
|
||||
- 平台服务器
|
||||
- ai-gateway.vei.volces.com
|
||||
|
||||
如果检测平台时提示 RTC 网络检测失败，或入房超过 15 秒：
|
||||
- 关闭 VPN、系统代理和网络加速器；
|
||||
- 在 Windows 防火墙/杀毒软件中允许本程序联网；
|
||||
- 优先改用手机热点复测，以排除公司、校园或酒店网络限制；
|
||||
- 需要放行 TCP 443 到火山 RTC Web SDK 4.69.0 使用的 *.rtc.volcvideo.com 和 *.volcvideos.com。
|
||||
`iot-cn-shanghai.iot.volces.com`只需要平台服务器访问，设备无需访问动态注册地址。
|
||||
|
||||
如果手机热点、关闭VPN/代理后仍然无法入房，请在失败后点击页面底部“导出 RTC 日志”，把 Windows“下载”目录中最新生成的 RTC 日志文件交给开发人员分析。
|
||||
安全提示：Product Secret 永不到达设备。平台下发的一机一密会使用当前 Windows 用户的 Safe Storage 加密保存，导出的测试报告不包含任何密钥。
|
||||
|
||||
安全提示：本测试包包含服务器测试设备凭据，仅限受控测试使用，请勿公开上传或转发。测试完成后请删除其他电脑上的解压目录和 ZIP 文件。
|
||||
当前测试 EXE 未配置商业代码签名证书，Windows 可能提示“未知发布者”。请先用随包 SHA-256 文件核对完整性；正式外发版本应使用甲方或发布主体的代码签名证书签名。
|
||||
|
||||
Reference in New Issue
Block a user