feat: add encrypted server backup verification

README.md

@@ -332,7 +332,7 @@ scripts/         本地启动和质量检查脚本
 test/            契约、集成和端到端测试
 ```
 
-公共接口约束与监控指标见[可观测性与接口治理设计](docs/observability-and-api-governance.md)，设备签名和模拟联调见[设备接入与启动配置联调说明](docs/device-integration.md)，设备命令见[设备命令可靠下发说明](docs/device-commands.md)，固件流程见[固件制品、签名、灰度与回滚开发说明](docs/firmware-rollout.md)，A/B升级与启动失败回退见[设备A/B分区固件升级与启动确认协议](docs/device-firmware-ab-protocol.md)，设备/云端时间线对账见[固件真机联调一致性证据采集与校验说明](docs/firmware-conformance-evidence.md)，双轨录音见[双轨录音上传、校验与访问开发说明](docs/media-upload-and-access.md)，网页状态通知见[Web状态事件流开发说明](docs/web-event-stream.md)，具体分享见[具体资源分享开发说明](docs/resource-sharing.md)，运营配置见[内部运营身份与运行配置版本开发说明](docs/operations-config.md)，用量费用与额度见[用量、费用与额度开发说明](docs/usage-cost-quota.md)，告警闭环见[告警监测与运营处置开发说明](docs/alert-operations.md)，真实网关归档见[告警接收器一致性证据](docs/alert-receiver-conformance-evidence.md)，云密钥轮换见[KMS/HSM轮换一致性证据](docs/kms-hsm-rotation-evidence.md)，生产恢复见[生产等价恢复演练一致性证据](docs/production-recovery-evidence.md)，生产基础设施见[生产基础设施一致性证据](docs/production-infrastructure-evidence.md)，联调工作台及发布候选验收快照见[集成诊断工作台开发说明](docs/integration-diagnostics.md)，试点每日证据见[试点运行日报开发说明](docs/pilot-daily-operations.md)，试点周度复盘口径、外部投诉缺口、接口和轻量烟测见[试点运行周报开发说明](docs/pilot-weekly-review.md)，敏感数据提权见[运营敏感数据临时访问开发说明](docs/ops-sensitive-access.md)，外部上线阶段见[真机、真实供应商、试点与生产发布外部验收清单](docs/external-acceptance-checklist.md)，生产门禁和压测告警见[压测、告警与生产安全门禁](docs/load-alerting-and-production-security.md)，专项实测证据见[性能、恢复、安全与兼容性专项验证报告](docs/performance-recovery-security-compatibility.md)，隐私数据访问见[隐私数据请求开发说明](docs/privacy-requests.md)，私有导出见[私有数据导出开发说明](docs/data-export.md)。
+公共接口约束与监控指标见[可观测性与接口治理设计](docs/observability-and-api-governance.md)，设备签名和模拟联调见[设备接入与启动配置联调说明](docs/device-integration.md)，设备命令见[设备命令可靠下发说明](docs/device-commands.md)，固件流程见[固件制品、签名、灰度与回滚开发说明](docs/firmware-rollout.md)，A/B升级与启动失败回退见[设备A/B分区固件升级与启动确认协议](docs/device-firmware-ab-protocol.md)，设备/云端时间线对账见[固件真机联调一致性证据采集与校验说明](docs/firmware-conformance-evidence.md)，双轨录音见[双轨录音上传、校验与访问开发说明](docs/media-upload-and-access.md)，网页状态通知见[Web状态事件流开发说明](docs/web-event-stream.md)，具体分享见[具体资源分享开发说明](docs/resource-sharing.md)，运营配置见[内部运营身份与运行配置版本开发说明](docs/operations-config.md)，用量费用与额度见[用量、费用与额度开发说明](docs/usage-cost-quota.md)，告警闭环见[告警监测与运营处置开发说明](docs/alert-operations.md)，真实网关归档见[告警接收器一致性证据](docs/alert-receiver-conformance-evidence.md)，云KMS轮换见[KMS/HSM轮换一致性证据](docs/kms-hsm-rotation-evidence.md)，生产恢复见[生产等价恢复演练一致性证据](docs/production-recovery-evidence.md)，单机云端备份见[自动备份与隔离恢复手册](docs/server-backup-and-restore.md)，生产基础设施见[生产基础设施一致性证据](docs/production-infrastructure-evidence.md)，联调工作台及发布候选验收快照见[集成诊断工作台开发说明](docs/integration-diagnostics.md)，试点每日证据见[试点运行日报开发说明](docs/pilot-daily-operations.md)，试点周度复盘口径、外部投诉缺口、接口和轻量烟测见[试点运行周报开发说明](docs/pilot-weekly-review.md)，敏感数据提权见[运营敏感数据临时访问开发说明](docs/ops-sensitive-access.md)，外部上线阶段见[真机、真实供应商、试点与生产发布外部验收清单](docs/external-acceptance-checklist.md)，生产门禁和压测告警见[压测、告警与生产安全门禁](docs/load-alerting-and-production-security.md)，专项实测证据见[性能、恢复、安全与兼容性专项验证报告](docs/performance-recovery-security-compatibility.md)，隐私数据访问见[隐私数据请求开发说明](docs/privacy-requests.md)，私有导出见[私有数据导出开发说明](docs/data-export.md)。
 
 从框架搭建到生产落地的阶段任务、里程碑、人员、测试门禁、依赖和风险见[详细开发计划](docs/development-plan.md)，当前实际完成证据见[开发状态与门禁记录](docs/development-status.md)。
 

deploy/server/README.md

@@ -1,6 +1,6 @@
 # 阿里云单机演示部署
 
-该配置用于单机公网演示，不是生产验收配置。只向公网发布HTTP 80和HTTPS 443；PostgreSQL、Redis、MinIO、Keycloak内部端口以及API/Worker端口均不映射到宿主机。当前阿里云演示机使用Let's Encrypt短期IP证书，`certbot-ip-renew.timer`负责自动续期。真实RTC、模型、内容安全、云KMS、备份恢复和监控告警仍按阶段8验收清单执行。
+该配置用于单机公网演示，不是生产验收配置。只向公网发布HTTP 80和HTTPS 443；PostgreSQL、Redis、MinIO、Keycloak内部端口以及API/Worker端口均不映射到宿主机。当前阿里云演示机使用Let's Encrypt短期IP证书，`certbot-ip-renew.timer`负责自动续期。单机备份覆盖PostgreSQL、Redis、MinIO对象和当前Keycloak数据目录，并执行数据库级隔离恢复校验；跨故障域备份、真实RTC、模型、内容安全、云KMS和生产监控告警仍按阶段8验收清单执行。
 
 服务器至少需要Docker Engine 29与Compose 2.40。首次部署先生成权限为600的服务器专用配置：
 
@@ -22,3 +22,24 @@ docker compose -f compose.yaml -f deploy/server/compose.override.yaml \
 ```
 
 安全组和主机防火墙只保留TCP 80、443以及受限来源的SSH 22。HTTP 80仅用于跳转到标准HTTPS 443。正式环境仍应配置自有域名、生产OIDC和真实外部服务，并切换到Kubernetes生产覆盖层。
+
+## 自动备份与恢复校验
+
+首次安装定时任务：
+
+```bash
+chmod 0755 deploy/server/backup.sh deploy/server/verify-backup.sh deploy/server/install-backup.sh
+sudo deploy/server/install-backup.sh
+sudo systemctl start platform-backup.service
+sudo systemctl status platform-backup.service --no-pager
+```
+
+备份以AES-256-CBC、PBKDF2和独立随机口令加密，默认保存在`/var/backups/data-terminal-cloud`并保留14天。每次定时任务会实际把PostgreSQL恢复到随机隔离数据库，对比迁移状态、客户组织、账号、表和外键计数，再删除临时数据库。Redis RDB、MinIO对象和暂停后的Keycloak数据目录会进入同一加密归档并进行逐文件SHA-256校验；当前单机配置不把这些文件级检查冒充跨故障域生产恢复。
+
+手工验证已有备份：
+
+```bash
+sudo deploy/server/verify-backup.sh /var/backups/data-terminal-cloud/platform-YYYYmmddTHHMMSSZ.tar.gz.enc
+```
+
+加密口令位于`/etc/data-terminal-cloud/backup-passphrase`且仅root可读。必须把该口令和加密备份复制到不同故障域；口令丢失将无法恢复备份。

deploy/server/backup.sh

@@ -0,0 +1,145 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+
+project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current}
+backup_root=${BACKUP_ROOT:-/var/backups/data-terminal-cloud}
+passphrase_file=${BACKUP_PASSPHRASE_FILE:-/etc/data-terminal-cloud/backup-passphrase}
+retention_days=${BACKUP_RETENTION_DAYS:-14}
+verify_after_backup=${VERIFY_AFTER_BACKUP:-false}
+
+case "$backup_root" in
+  /*) ;;
+  *) echo "BACKUP_ROOT must be an absolute path" >&2; exit 2 ;;
+esac
+if [[ "$backup_root" == "/" || "$backup_root" == "/var" || "$backup_root" == "/var/backups" ]]; then
+  echo "BACKUP_ROOT is too broad" >&2
+  exit 2
+fi
+if [[ ! "$retention_days" =~ ^[1-9][0-9]*$ ]]; then
+  echo "BACKUP_RETENTION_DAYS must be a positive integer" >&2
+  exit 2
+fi
+if [[ ! -r "$passphrase_file" ]]; then
+  echo "Backup passphrase file is missing or unreadable: $passphrase_file" >&2
+  exit 2
+fi
+
+compose=(docker compose -f "$project_root/compose.yaml" -f "$project_root/deploy/server/compose.override.yaml" --env-file "$project_root/deploy/server/.env")
+mkdir -p "$backup_root"
+chmod 0700 "$backup_root"
+umask 077
+
+exec 9>"$backup_root/.backup.lock"
+if ! flock -n 9; then
+  echo "Another backup is already running" >&2
+  exit 3
+fi
+
+stamp=$(date -u +%Y%m%dT%H%M%SZ)
+staging=$(mktemp -d "$backup_root/.staging-${stamp}-XXXXXX")
+plain_archive="$backup_root/.platform-${stamp}.tar.gz"
+encrypted_archive="$backup_root/platform-${stamp}.tar.gz.enc"
+checksum_file="$encrypted_archive.sha256"
+minio_container=""
+keycloak_container=""
+keycloak_paused=false
+
+cleanup() {
+  status=$?
+  if [[ "$keycloak_paused" == true && -n "$keycloak_container" ]]; then
+    docker unpause "$keycloak_container" >/dev/null 2>&1 || true
+  fi
+  if [[ -n "$minio_container" ]]; then
+    docker exec "$minio_container" rm -rf /tmp/platform-backup /tmp/platform-mc >/dev/null 2>&1 || true
+  fi
+  rm -rf -- "$staging"
+  rm -f -- "$plain_archive"
+  exit "$status"
+}
+trap cleanup EXIT INT TERM
+
+service_container() {
+  local service=$1
+  local container
+  container=$("${compose[@]}" ps -q "$service")
+  if [[ -z "$container" ]] || [[ $(docker inspect -f '{{.State.Running}}' "$container") != true ]]; then
+    echo "Required service is not running: $service" >&2
+    exit 4
+  fi
+  printf '%s' "$container"
+}
+
+postgres_container=$(service_container postgres)
+redis_container=$(service_container redis)
+minio_container=$(service_container minio)
+keycloak_container=$(service_container keycloak)
+redis_password=$(sed -n 's/^REDIS_PASSWORD=//p' "$project_root/deploy/server/.env")
+if [[ -z "$redis_password" ]]; then
+  echo "REDIS_PASSWORD is missing from the server environment" >&2
+  exit 4
+fi
+
+docker exec "$postgres_container" pg_dump -U voice -d voice_platform -Fc --no-owner --no-acl >"$staging/postgres.dump"
+docker exec "$postgres_container" psql -U voice -d voice_platform -Atc \
+  "SELECT version||'|'||dirty||'|'||(SELECT count(*) FROM tenant)||'|'||(SELECT count(*) FROM user_account)||'|'||(SELECT count(*) FROM information_schema.tables WHERE table_schema='public')||'|'||(SELECT count(*) FROM information_schema.table_constraints WHERE constraint_type='FOREIGN KEY') FROM schema_migrations;" \
+  >"$staging/database-evidence.txt"
+
+mkdir -p "$staging/redis"
+printf '%s\n' "$redis_password" | docker exec -i "$redis_container" sh -ec '
+  IFS= read -r REDISCLI_AUTH
+  export REDISCLI_AUTH
+  redis-cli --no-auth-warning SAVE >/dev/null
+'
+docker exec "$redis_container" redis-check-rdb /data/dump.rdb >/dev/null
+docker cp "$redis_container:/data/dump.rdb" "$staging/redis/dump.rdb" >/dev/null
+
+docker exec "$minio_container" sh -ec '
+  rm -rf /tmp/platform-backup /tmp/platform-mc
+  mkdir -p /tmp/platform-backup /tmp/platform-mc
+  mc --config-dir /tmp/platform-mc alias set local http://127.0.0.1:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null
+  mc --config-dir /tmp/platform-mc mirror --overwrite --preserve local/ /tmp/platform-backup/
+'
+mkdir -p "$staging/object-store"
+docker cp "$minio_container:/tmp/platform-backup/." "$staging/object-store/" >/dev/null
+docker exec "$minio_container" rm -rf /tmp/platform-backup /tmp/platform-mc
+
+docker pause "$keycloak_container" >/dev/null
+keycloak_paused=true
+mkdir -p "$staging/keycloak"
+docker cp "$keycloak_container:/opt/keycloak/data/." "$staging/keycloak/" >/dev/null
+docker unpause "$keycloak_container" >/dev/null
+keycloak_paused=false
+
+object_files=$(find "$staging/object-store" -type f | wc -l | tr -d ' ')
+keycloak_files=$(find "$staging/keycloak" -type f | wc -l | tr -d ' ')
+cat >"$staging/manifest.properties" <<EOF
+format=platform-server-backup-v1
+created_at=${stamp}
+build_version=$(sed -n 's/^PLATFORM_BUILD_VERSION=//p' "$project_root/deploy/server/.env")
+build_commit=$(sed -n 's/^PLATFORM_BUILD_COMMIT=//p' "$project_root/deploy/server/.env")
+database_evidence=$(cat "$staging/database-evidence.txt")
+object_files=${object_files}
+keycloak_files=${keycloak_files}
+redis_snapshot=true
+keycloak_snapshot_mode=paused_filesystem_copy
+EOF
+
+(
+  cd "$staging"
+  find . -type f ! -name SHA256SUMS -print0 | sort -z | xargs -0 -r sha256sum >SHA256SUMS
+)
+tar -C "$staging" -czf "$plain_archive" .
+openssl enc -aes-256-cbc -salt -pbkdf2 -iter 200000 \
+  -pass "file:$passphrase_file" -in "$plain_archive" -out "$encrypted_archive"
+sha256sum "$encrypted_archive" >"$checksum_file"
+chmod 0600 "$encrypted_archive" "$checksum_file"
+
+while IFS= read -r -d '' expired; do
+  rm -f -- "$expired" "$expired.sha256"
+done < <(find "$backup_root" -maxdepth 1 -type f -name 'platform-*.tar.gz.enc' -mtime "+$retention_days" -print0)
+
+if [[ "$verify_after_backup" == true ]]; then
+  "$project_root/deploy/server/verify-backup.sh" "$encrypted_archive"
+fi
+
+printf '%s\n' "$encrypted_archive"

deploy/server/install-backup.sh

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+
+if [[ ${EUID:-$(id -u)} -ne 0 ]]; then
+  echo "Run this installer as root" >&2
+  exit 2
+fi
+
+project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current}
+install -d -m 0700 /etc/data-terminal-cloud /var/backups/data-terminal-cloud
+if [[ ! -s /etc/data-terminal-cloud/backup-passphrase ]]; then
+  umask 077
+  openssl rand -base64 48 >/etc/data-terminal-cloud/backup-passphrase
+fi
+chmod 0600 /etc/data-terminal-cloud/backup-passphrase
+install -m 0644 "$project_root/deploy/server/platform-backup.service" /etc/systemd/system/platform-backup.service
+install -m 0644 "$project_root/deploy/server/platform-backup.timer" /etc/systemd/system/platform-backup.timer
+systemctl daemon-reload
+systemctl enable --now platform-backup.timer
+systemctl is-active --quiet platform-backup.timer
+echo "platform-backup.timer enabled"

deploy/server/platform-backup.service

@@ -0,0 +1,19 @@
+[Unit]
+Description=Encrypted data terminal cloud backup and restore verification
+Requires=docker.service
+After=docker.service network-online.target
+
+[Service]
+Type=oneshot
+User=root
+Group=root
+UMask=0077
+Environment=PLATFORM_ROOT=/opt/data-terminal-cloud/current
+Environment=BACKUP_ROOT=/var/backups/data-terminal-cloud
+Environment=BACKUP_RETENTION_DAYS=14
+Environment=VERIFY_AFTER_BACKUP=true
+ExecStart=/opt/data-terminal-cloud/current/deploy/server/backup.sh
+Nice=10
+IOSchedulingClass=best-effort
+IOSchedulingPriority=7
+TimeoutStartSec=2h

deploy/server/platform-backup.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Run encrypted data terminal cloud backup daily
+
+[Timer]
+OnCalendar=*-*-* 02:40:00
+RandomizedDelaySec=45m
+Persistent=true
+Unit=platform-backup.service
+
+[Install]
+WantedBy=timers.target

deploy/server/verify-backup.sh

@@ -0,0 +1,106 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+
+project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current}
+backup_root=${BACKUP_ROOT:-/var/backups/data-terminal-cloud}
+passphrase_file=${BACKUP_PASSPHRASE_FILE:-/etc/data-terminal-cloud/backup-passphrase}
+archive=${1:-}
+
+if [[ -z "$archive" ]]; then
+  archive=$(find "$backup_root" -maxdepth 1 -type f -name 'platform-*.tar.gz.enc' -printf '%T@ %p\n' | sort -nr | head -n1 | cut -d' ' -f2-)
+fi
+if [[ -z "$archive" || ! -f "$archive" ]]; then
+  echo "Backup archive was not found" >&2
+  exit 2
+fi
+archive=$(realpath "$archive")
+backup_root=$(realpath "$backup_root")
+case "$archive" in
+  "$backup_root"/platform-*.tar.gz.enc) ;;
+  *) echo "Backup archive is outside the configured backup directory" >&2; exit 2 ;;
+esac
+if [[ ! -r "$passphrase_file" ]]; then
+  echo "Backup passphrase file is missing or unreadable" >&2
+  exit 2
+fi
+
+compose=(docker compose -f "$project_root/compose.yaml" -f "$project_root/deploy/server/compose.override.yaml" --env-file "$project_root/deploy/server/.env")
+postgres_container=$("${compose[@]}" ps -q postgres)
+if [[ -z "$postgres_container" ]] || [[ $(docker inspect -f '{{.State.Running}}' "$postgres_container") != true ]]; then
+  echo "PostgreSQL service is not running" >&2
+  exit 3
+fi
+redis_container=$("${compose[@]}" ps -q redis)
+if [[ -z "$redis_container" ]] || [[ $(docker inspect -f '{{.State.Running}}' "$redis_container") != true ]]; then
+  echo "Redis service is not running" >&2
+  exit 3
+fi
+
+umask 077
+work=$(mktemp -d "$backup_root/.verify-XXXXXX")
+plain="$work/backup.tar.gz"
+verify_token=$(openssl rand -hex 12)
+restore_database="voice_restore_verify_$verify_token"
+redis_verify_path="/tmp/platform-backup-$verify_token.rdb"
+database_created=false
+
+cleanup() {
+  status=$?
+  if [[ "$database_created" == true ]]; then
+    docker exec "$postgres_container" dropdb -U voice --if-exists "$restore_database" >/dev/null 2>&1 || true
+  fi
+  docker exec "$redis_container" rm -f -- "$redis_verify_path" >/dev/null 2>&1 || true
+  rm -rf -- "$work"
+  exit "$status"
+}
+trap cleanup EXIT INT TERM
+
+if [[ -f "$archive.sha256" ]]; then
+  (cd "$(dirname "$archive")" && sha256sum -c "$(basename "$archive").sha256") >/dev/null
+fi
+openssl enc -d -aes-256-cbc -pbkdf2 -iter 200000 \
+  -pass "file:$passphrase_file" -in "$archive" -out "$plain"
+tar -tzf "$plain" >"$work/archive-paths.txt"
+if grep -Eq '(^/|(^|/)\.\.(/|$))' "$work/archive-paths.txt"; then
+  echo "Backup contains an unsafe archive path" >&2
+  exit 4
+fi
+mkdir -p "$work/extracted"
+tar -xzf "$plain" --no-same-owner -C "$work/extracted"
+(
+  cd "$work/extracted"
+  sha256sum -c SHA256SUMS >/dev/null
+)
+
+expected=$(tr -d '\r\n' <"$work/extracted/database-evidence.txt")
+docker exec "$postgres_container" createdb -U voice "$restore_database"
+database_created=true
+docker exec -i "$postgres_container" pg_restore -U voice -d "$restore_database" --no-owner --no-acl <"$work/extracted/postgres.dump"
+actual=$(docker exec "$postgres_container" psql -U voice -d "$restore_database" -Atc \
+  "SELECT version||'|'||dirty||'|'||(SELECT count(*) FROM tenant)||'|'||(SELECT count(*) FROM user_account)||'|'||(SELECT count(*) FROM information_schema.tables WHERE table_schema='public')||'|'||(SELECT count(*) FROM information_schema.table_constraints WHERE constraint_type='FOREIGN KEY') FROM schema_migrations;")
+if [[ "$actual" != "$expected" ]]; then
+  echo "Restored database evidence does not match the backup: expected=$expected actual=$actual" >&2
+  exit 5
+fi
+
+[[ -s "$work/extracted/redis/dump.rdb" ]] || { echo "Redis snapshot is missing" >&2; exit 6; }
+[[ -d "$work/extracted/object-store" ]] || { echo "Object-store snapshot is missing" >&2; exit 6; }
+[[ -d "$work/extracted/keycloak" ]] || { echo "Keycloak snapshot is missing" >&2; exit 6; }
+expected_object_files=$(sed -n 's/^object_files=//p' "$work/extracted/manifest.properties")
+expected_keycloak_files=$(sed -n 's/^keycloak_files=//p' "$work/extracted/manifest.properties")
+actual_object_files=$(find "$work/extracted/object-store" -type f | wc -l | tr -d ' ')
+actual_keycloak_files=$(find "$work/extracted/keycloak" -type f | wc -l | tr -d ' ')
+if [[ ! "$expected_object_files" =~ ^[0-9]+$ || "$actual_object_files" != "$expected_object_files" ]]; then
+  echo "Object-store file count does not match the manifest" >&2
+  exit 6
+fi
+if [[ ! "$expected_keycloak_files" =~ ^[1-9][0-9]*$ || "$actual_keycloak_files" != "$expected_keycloak_files" ]]; then
+  echo "Keycloak file count does not match the manifest" >&2
+  exit 6
+fi
+docker cp "$work/extracted/redis/dump.rdb" "$redis_container:$redis_verify_path" >/dev/null
+docker exec "$redis_container" redis-check-rdb "$redis_verify_path" >/dev/null
+docker exec "$redis_container" rm -f -- "$redis_verify_path"
+
+printf 'backup_restore_verified archive=%s database_evidence=%s object_files=%s keycloak_files=%s redis_rdb=valid\n' \
+  "$archive" "$actual" "$actual_object_files" "$actual_keycloak_files"

docs/development-status.md

@@ -1,6 +1,6 @@
 # 开发状态与门禁记录
 
-> 更新日期：2026-08-25  
+> 更新日期：2026-08-26
 > 当前接口契约版本：0.64.0  
 > 当前主里程碑：M8目标环境联调、试点与上线验收（外部条件待提供）
 
@@ -180,6 +180,7 @@
 | 媒体清单容量 | 10批×1000分片准确落库和关联；批量SQL优化后平均0.12秒、p95/最大0.16秒，API容器内存峰值15.8MiB |
 | 隐私删除并发 | 100个并发受理全部成功、HTTP失败率0%、提交p95 385.15ms；11.58秒内600个子任务、100个墓碑和100个主体冻结全部对账 |
 | PostgreSQL隔离恢复 | 实际`pg_dump`/`pg_restore`到随机隔离库，迁移`22|false`、66张表、190个外键和关键计数一致；本地约5.14秒，临时库/备份已清理 |
+| 阿里云单机加密备份与恢复 | 目标服务器已安装每日systemd备份任务，覆盖PostgreSQL、认证Redis RDB、MinIO全部桶和暂停后的Keycloak数据目录；归档使用AES-256-CBC/PBKDF2加密、逐文件SHA-256并保留14天。首次实际归档成功，数据库从V38备份恢复到随机隔离库后`38|false|2|3|73|215`六项证据一致，归档内Redis RDB格式有效、Keycloak 3个文件一致、临时恢复库为0；同机备份尚不作为跨故障域生产恢复证据 |
 | 进程与Redis恢复 | API/Callbacks/Worker重启恢复；停机期间Outbox仅执行一次；Redis 14个热状态键清空后由签名设备请求重建，PostgreSQL权威计数不变 |
 | 供应商故障恢复 | V23持久化按租户/供应商的可用状态；连续临时失败暂停新启动但停止任务继续尝试；演练确认暂停后新外呼0、停止尝试2次、历史接口200、CRITICAL告警出现，恢复后全部收敛；V23单版本回退/重升通过 |
 | 安全专项 | Go 1.26.6容器`govulncheck`可达漏洞0、npm高危审计0、最终镜像HIGH/CRITICAL已修复漏洞0、Gitleaks无泄漏；HTTP认证/授权/注入式输入/1MiB正文/回调签名/Nonce重放/安全头/日志过滤烟测通过 |

docs/requirements-traceability.md

@@ -90,6 +90,6 @@
 | P0414 | 首期需求、验收用例、缺陷和风险形成对应关系 | 本文完成首期功能到代码、接口、数据和验证的基线；V30问题清单由V36补充严重级别、根因、修复版本、回归证据及未关闭HIGH/CRITICAL发布阻断汇总；OpenAPI 0.50.0—0.55.0可按候选构建导出六项技术门禁及脱敏阻断告警定位快照，绑定候选版本/构建摘要后按业务时区归档日/周运行证据。`cmd/release-evidence`现使用v3清单，把同一候选的快照、接口、迁移、部署、SBOM和十类外部状态形成逐文件摘要；远程CI供应链、真机固件、真实终端RTC、真实Agent模型、真实告警网关、云KMS/HSM轮换、生产等价恢复、生产基础设施和Safari兼容分别要求结构化报告，并把14项CI检查/镜像/SBOM、设备固件场景、语音方式与恢复场景、生成/Embedding/内容安全质量、路由演练、轮换模式、恢复类型、基础设施控制项及Safari矩阵覆盖与缺口写入清单。OpenAPI 0.64.0、V38及九个离线校验器把云端状态与远程构建、外部设备、RTC/语音供应商、模型提供器、网关、密钥提供器、目标环境和浏览器证据绑定；`content_safety_tls`把策略服务网络/证书状态纳入指标、告警、运营页面和候选快照。缺检查、缺制品、缺场景、缺演练、缺质量类别、缺控制项、`PENDING`和本地模拟证据均不会被误报为外部验收完成，整体发布仍必须基于真实联调、灰度和审批记录确认。 |
 | P0445 | 需求编号关联代码、测试、接口或配置变更 | 本文逐项建立关联；后续功能变更必须同步更新本矩阵和`development-status.md`。 |
 
-当前仍不能在本地宣告完成的部分集中为：真实火山RTC与豆包三模式联调、真实方舟内容与Embedding质量、真实听筒的长者交流体验、量产设备/KMS注入、生产对象存储与备份、真机异常恢复、真实账单和生产级内容安全。它们属于阶段8外部验收，不应通过增加本地低价值测试伪造完成证据。
+当前仍不能宣告完成的部分集中为：真实火山RTC与豆包三模式联调、真实方舟内容与Embedding质量、真实听筒的长者交流体验、量产设备/KMS注入、跨故障域生产对象存储与备份、真机异常恢复、真实账单和生产级内容安全。阿里云单机环境已经形成加密全组件归档、数据库隔离恢复和Redis RDB校验，但同机归档不能替代托管PITR、异地副本和生产恢复演练。这些剩余项属于阶段8外部验收，不应通过增加本地低价值测试伪造完成证据。
 
 下一批本地开发应优先从矩阵中出现“进行中”的条目选择；若没有本地功能缺口，则只继续完善能够直接降低阶段8联调风险的模拟器、诊断和交付工具，不以覆盖率数字替代业务功能。

docs/server-backup-and-restore.md

@@ -0,0 +1,101 @@
+# 单机云端备份与隔离恢复手册
+
+## 1. 适用范围
+
+本手册适用于`deploy/server/compose.override.yaml`部署的单机公网演示环境。目标是在不暴露数据库、Redis、对象存储和身份服务端口的前提下，形成可自动执行、可校验、可追溯的加密备份，并证明PostgreSQL备份能够恢复到独立数据库。
+
+该能力不是跨可用区生产灾备。单机磁盘故障仍可能同时损失在线数据和本机备份，因此加密归档和口令必须另行复制到不同故障域。
+
+## 2. 备份范围
+
+| 组件 | 备份方式 | 当前恢复验证 |
+|---|---|---|
+| PostgreSQL | `pg_dump`自定义格式，不包含所有者和ACL | 实际恢复到随机隔离数据库，并核对迁移、客户组织、账号、表和外键计数 |
+| Redis | 认证执行`SAVE`后复制RDB | `redis-check-rdb`验证归档内RDB格式，不覆盖在线Redis |
+| MinIO | 使用容器内`mc mirror`复制全部桶和对象 | 逐文件SHA-256和对象文件数量一致性检查 |
+| Keycloak | 暂停容器后复制`/opt/keycloak/data`，随后立即恢复 | 逐文件SHA-256和文件数量检查；当前开发模式H2数据不作为生产OIDC方案 |
+
+备份归档使用`AES-256-CBC + PBKDF2`加密，迭代次数为200000。每个归档旁边保存加密文件自身的SHA-256；归档内部另有逐文件`SHA256SUMS`和不含凭据的`manifest.properties`。
+
+## 3. 自动执行流程
+
+```text
+获取独占锁
+  → PostgreSQL逻辑转储和证据快照
+  → Redis认证SAVE及RDB校验
+  → MinIO全桶对象镜像
+  → 短暂暂停Keycloak并复制一致性数据目录
+  → 逐文件SHA-256
+  → 压缩、加密、生成归档摘要
+  → PostgreSQL随机隔离库实际恢复
+  → Redis RDB及文件清单复核
+  → 删除临时数据库和临时明文
+  → 按保留期清理旧加密归档
+```
+
+任何步骤失败都会令systemd服务失败；未完成的明文暂存目录由退出陷阱删除。备份使用`flock`避免重叠执行，Keycloak无论成功或异常退出都会尝试解除暂停。
+
+## 4. 安装与运行
+
+```bash
+cd /opt/data-terminal-cloud/current
+chmod 0755 deploy/server/backup.sh deploy/server/verify-backup.sh deploy/server/install-backup.sh
+deploy/server/install-backup.sh
+systemctl start platform-backup.service
+systemctl show platform-backup.service -p Result -p ExecMainStatus
+systemctl list-timers platform-backup.timer --no-pager
+```
+
+默认每天02:40执行，并附加最长45分钟随机延迟，避免固定时刻资源竞争；默认保留14天。定时任务设置`Persistent=true`，服务器在计划时刻关机时会在恢复后补跑。
+
+可通过环境变量覆盖：
+
+| 变量 | 默认值 | 说明 |
+|---|---|---|
+| `PLATFORM_ROOT` | `/opt/data-terminal-cloud/current` | 当前发布目录 |
+| `BACKUP_ROOT` | `/var/backups/data-terminal-cloud` | 加密归档目录，必须为安全的绝对路径 |
+| `BACKUP_PASSPHRASE_FILE` | `/etc/data-terminal-cloud/backup-passphrase` | 仅root可读的独立随机口令 |
+| `BACKUP_RETENTION_DAYS` | `14` | 本机备份保留天数 |
+| `VERIFY_AFTER_BACKUP` | `false`；systemd中为`true` | 备份后是否立即执行隔离恢复校验 |
+
+## 5. 手工校验与故障定位
+
+校验最新备份：
+
+```bash
+/opt/data-terminal-cloud/current/deploy/server/verify-backup.sh
+```
+
+校验指定备份：
+
+```bash
+/opt/data-terminal-cloud/current/deploy/server/verify-backup.sh \
+  /var/backups/data-terminal-cloud/platform-YYYYmmddTHHMMSSZ.tar.gz.enc
+```
+
+查看任务结果：
+
+```bash
+systemctl show platform-backup.service -p Result -p ExecMainStatus -p ActiveState
+journalctl -u platform-backup.service -n 100 --no-pager
+find /var/backups/data-terminal-cloud -maxdepth 1 -type f -name 'platform-*.enc' -ls
+```
+
+一次性服务成功结束后显示`inactive (dead)`是正常状态，应结合`Result=success`和`ExecMainStatus=0`判断。校验脚本不会停止在线业务、覆盖在线数据库或载入Redis快照；随机恢复数据库在成功和失败路径都会清理。
+
+## 6. 安全与恢复边界
+
+- 备份口令不写入仓库、归档、日志或Compose环境文件；默认权限为`0600 root:root`。
+- 备份目录默认权限为`0700`，归档和摘要为`0600`。
+- 明文转储只存在于同一受控暂存目录，归档加密后立即删除。
+- 当前归档留在同一云服务器只提供误操作恢复能力，不提供主机或地域故障保护。
+- 生产环境应使用托管PostgreSQL持续备份/PITR、版本化对象存储、外部OIDC、跨故障域副本、KMS托管密钥和受控恢复演练。
+- 不能仅凭定时任务成功声称满足某个RPO/RTO；必须结合备份频率、异地复制、真实数据量和受控恢复耗时单独验收。
+
+## 7. 上线前检查
+
+1. 把加密归档复制到不同故障域，并独立验证复制后的SHA-256。
+2. 把口令保存到云KMS、密码库或离线托管介质，不与归档放在同一位置。
+3. 配置备份失败和长时间未成功告警。
+4. 在预生产环境执行PostgreSQL、Redis、对象存储和OIDC的完整恢复演练。
+5. 记录实际备份大小、持续时间、恢复时间和业务抽查结果，再确定RPO/RTO。