diff --git a/README.md b/README.md index 646254e..5deb201 100644 --- a/README.md +++ b/README.md @@ -332,7 +332,7 @@ scripts/ 本地启动和质量检查脚本 test/ 契约、集成和端到端测试 ``` -公共接口约束与监控指标见[可观测性与接口治理设计](docs/observability-and-api-governance.md),设备签名和模拟联调见[设备接入与启动配置联调说明](docs/device-integration.md),设备命令见[设备命令可靠下发说明](docs/device-commands.md),固件流程见[固件制品、签名、灰度与回滚开发说明](docs/firmware-rollout.md),A/B升级与启动失败回退见[设备A/B分区固件升级与启动确认协议](docs/device-firmware-ab-protocol.md),设备/云端时间线对账见[固件真机联调一致性证据采集与校验说明](docs/firmware-conformance-evidence.md),双轨录音见[双轨录音上传、校验与访问开发说明](docs/media-upload-and-access.md),网页状态通知见[Web状态事件流开发说明](docs/web-event-stream.md),具体分享见[具体资源分享开发说明](docs/resource-sharing.md),运营配置见[内部运营身份与运行配置版本开发说明](docs/operations-config.md),用量费用与额度见[用量、费用与额度开发说明](docs/usage-cost-quota.md),告警闭环见[告警监测与运营处置开发说明](docs/alert-operations.md),真实网关归档见[告警接收器一致性证据](docs/alert-receiver-conformance-evidence.md),云密钥轮换见[KMS/HSM轮换一致性证据](docs/kms-hsm-rotation-evidence.md),生产恢复见[生产等价恢复演练一致性证据](docs/production-recovery-evidence.md),生产基础设施见[生产基础设施一致性证据](docs/production-infrastructure-evidence.md),联调工作台及发布候选验收快照见[集成诊断工作台开发说明](docs/integration-diagnostics.md),试点每日证据见[试点运行日报开发说明](docs/pilot-daily-operations.md),试点周度复盘口径、外部投诉缺口、接口和轻量烟测见[试点运行周报开发说明](docs/pilot-weekly-review.md),敏感数据提权见[运营敏感数据临时访问开发说明](docs/ops-sensitive-access.md),外部上线阶段见[真机、真实供应商、试点与生产发布外部验收清单](docs/external-acceptance-checklist.md),生产门禁和压测告警见[压测、告警与生产安全门禁](docs/load-alerting-and-production-security.md),专项实测证据见[性能、恢复、安全与兼容性专项验证报告](docs/performance-recovery-security-compatibility.md),隐私数据访问见[隐私数据请求开发说明](docs/privacy-requests.md),私有导出见[私有数据导出开发说明](docs/data-export.md)。 +公共接口约束与监控指标见[可观测性与接口治理设计](docs/observability-and-api-governance.md),设备签名和模拟联调见[设备接入与启动配置联调说明](docs/device-integration.md),设备命令见[设备命令可靠下发说明](docs/device-commands.md),固件流程见[固件制品、签名、灰度与回滚开发说明](docs/firmware-rollout.md),A/B升级与启动失败回退见[设备A/B分区固件升级与启动确认协议](docs/device-firmware-ab-protocol.md),设备/云端时间线对账见[固件真机联调一致性证据采集与校验说明](docs/firmware-conformance-evidence.md),双轨录音见[双轨录音上传、校验与访问开发说明](docs/media-upload-and-access.md),网页状态通知见[Web状态事件流开发说明](docs/web-event-stream.md),具体分享见[具体资源分享开发说明](docs/resource-sharing.md),运营配置见[内部运营身份与运行配置版本开发说明](docs/operations-config.md),用量费用与额度见[用量、费用与额度开发说明](docs/usage-cost-quota.md),告警闭环见[告警监测与运营处置开发说明](docs/alert-operations.md),真实网关归档见[告警接收器一致性证据](docs/alert-receiver-conformance-evidence.md),云KMS轮换见[KMS/HSM轮换一致性证据](docs/kms-hsm-rotation-evidence.md),生产恢复见[生产等价恢复演练一致性证据](docs/production-recovery-evidence.md),单机云端备份见[自动备份与隔离恢复手册](docs/server-backup-and-restore.md),生产基础设施见[生产基础设施一致性证据](docs/production-infrastructure-evidence.md),联调工作台及发布候选验收快照见[集成诊断工作台开发说明](docs/integration-diagnostics.md),试点每日证据见[试点运行日报开发说明](docs/pilot-daily-operations.md),试点周度复盘口径、外部投诉缺口、接口和轻量烟测见[试点运行周报开发说明](docs/pilot-weekly-review.md),敏感数据提权见[运营敏感数据临时访问开发说明](docs/ops-sensitive-access.md),外部上线阶段见[真机、真实供应商、试点与生产发布外部验收清单](docs/external-acceptance-checklist.md),生产门禁和压测告警见[压测、告警与生产安全门禁](docs/load-alerting-and-production-security.md),专项实测证据见[性能、恢复、安全与兼容性专项验证报告](docs/performance-recovery-security-compatibility.md),隐私数据访问见[隐私数据请求开发说明](docs/privacy-requests.md),私有导出见[私有数据导出开发说明](docs/data-export.md)。 从框架搭建到生产落地的阶段任务、里程碑、人员、测试门禁、依赖和风险见[详细开发计划](docs/development-plan.md),当前实际完成证据见[开发状态与门禁记录](docs/development-status.md)。 diff --git a/deploy/server/README.md b/deploy/server/README.md index 1b3df1e..667d097 100644 --- a/deploy/server/README.md +++ b/deploy/server/README.md @@ -1,6 +1,6 @@ # 阿里云单机演示部署 -该配置用于单机公网演示,不是生产验收配置。只向公网发布HTTP 80和HTTPS 443;PostgreSQL、Redis、MinIO、Keycloak内部端口以及API/Worker端口均不映射到宿主机。当前阿里云演示机使用Let's Encrypt短期IP证书,`certbot-ip-renew.timer`负责自动续期。真实RTC、模型、内容安全、云KMS、备份恢复和监控告警仍按阶段8验收清单执行。 +该配置用于单机公网演示,不是生产验收配置。只向公网发布HTTP 80和HTTPS 443;PostgreSQL、Redis、MinIO、Keycloak内部端口以及API/Worker端口均不映射到宿主机。当前阿里云演示机使用Let's Encrypt短期IP证书,`certbot-ip-renew.timer`负责自动续期。单机备份覆盖PostgreSQL、Redis、MinIO对象和当前Keycloak数据目录,并执行数据库级隔离恢复校验;跨故障域备份、真实RTC、模型、内容安全、云KMS和生产监控告警仍按阶段8验收清单执行。 服务器至少需要Docker Engine 29与Compose 2.40。首次部署先生成权限为600的服务器专用配置: @@ -22,3 +22,24 @@ docker compose -f compose.yaml -f deploy/server/compose.override.yaml \ ``` 安全组和主机防火墙只保留TCP 80、443以及受限来源的SSH 22。HTTP 80仅用于跳转到标准HTTPS 443。正式环境仍应配置自有域名、生产OIDC和真实外部服务,并切换到Kubernetes生产覆盖层。 + +## 自动备份与恢复校验 + +首次安装定时任务: + +```bash +chmod 0755 deploy/server/backup.sh deploy/server/verify-backup.sh deploy/server/install-backup.sh +sudo deploy/server/install-backup.sh +sudo systemctl start platform-backup.service +sudo systemctl status platform-backup.service --no-pager +``` + +备份以AES-256-CBC、PBKDF2和独立随机口令加密,默认保存在`/var/backups/data-terminal-cloud`并保留14天。每次定时任务会实际把PostgreSQL恢复到随机隔离数据库,对比迁移状态、客户组织、账号、表和外键计数,再删除临时数据库。Redis RDB、MinIO对象和暂停后的Keycloak数据目录会进入同一加密归档并进行逐文件SHA-256校验;当前单机配置不把这些文件级检查冒充跨故障域生产恢复。 + +手工验证已有备份: + +```bash +sudo deploy/server/verify-backup.sh /var/backups/data-terminal-cloud/platform-YYYYmmddTHHMMSSZ.tar.gz.enc +``` + +加密口令位于`/etc/data-terminal-cloud/backup-passphrase`且仅root可读。必须把该口令和加密备份复制到不同故障域;口令丢失将无法恢复备份。 diff --git a/deploy/server/backup.sh b/deploy/server/backup.sh new file mode 100755 index 0000000..1409d93 --- /dev/null +++ b/deploy/server/backup.sh @@ -0,0 +1,145 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current} +backup_root=${BACKUP_ROOT:-/var/backups/data-terminal-cloud} +passphrase_file=${BACKUP_PASSPHRASE_FILE:-/etc/data-terminal-cloud/backup-passphrase} +retention_days=${BACKUP_RETENTION_DAYS:-14} +verify_after_backup=${VERIFY_AFTER_BACKUP:-false} + +case "$backup_root" in + /*) ;; + *) echo "BACKUP_ROOT must be an absolute path" >&2; exit 2 ;; +esac +if [[ "$backup_root" == "/" || "$backup_root" == "/var" || "$backup_root" == "/var/backups" ]]; then + echo "BACKUP_ROOT is too broad" >&2 + exit 2 +fi +if [[ ! "$retention_days" =~ ^[1-9][0-9]*$ ]]; then + echo "BACKUP_RETENTION_DAYS must be a positive integer" >&2 + exit 2 +fi +if [[ ! -r "$passphrase_file" ]]; then + echo "Backup passphrase file is missing or unreadable: $passphrase_file" >&2 + exit 2 +fi + +compose=(docker compose -f "$project_root/compose.yaml" -f "$project_root/deploy/server/compose.override.yaml" --env-file "$project_root/deploy/server/.env") +mkdir -p "$backup_root" +chmod 0700 "$backup_root" +umask 077 + +exec 9>"$backup_root/.backup.lock" +if ! flock -n 9; then + echo "Another backup is already running" >&2 + exit 3 +fi + +stamp=$(date -u +%Y%m%dT%H%M%SZ) +staging=$(mktemp -d "$backup_root/.staging-${stamp}-XXXXXX") +plain_archive="$backup_root/.platform-${stamp}.tar.gz" +encrypted_archive="$backup_root/platform-${stamp}.tar.gz.enc" +checksum_file="$encrypted_archive.sha256" +minio_container="" +keycloak_container="" +keycloak_paused=false + +cleanup() { + status=$? + if [[ "$keycloak_paused" == true && -n "$keycloak_container" ]]; then + docker unpause "$keycloak_container" >/dev/null 2>&1 || true + fi + if [[ -n "$minio_container" ]]; then + docker exec "$minio_container" rm -rf /tmp/platform-backup /tmp/platform-mc >/dev/null 2>&1 || true + fi + rm -rf -- "$staging" + rm -f -- "$plain_archive" + exit "$status" +} +trap cleanup EXIT INT TERM + +service_container() { + local service=$1 + local container + container=$("${compose[@]}" ps -q "$service") + if [[ -z "$container" ]] || [[ $(docker inspect -f '{{.State.Running}}' "$container") != true ]]; then + echo "Required service is not running: $service" >&2 + exit 4 + fi + printf '%s' "$container" +} + +postgres_container=$(service_container postgres) +redis_container=$(service_container redis) +minio_container=$(service_container minio) +keycloak_container=$(service_container keycloak) +redis_password=$(sed -n 's/^REDIS_PASSWORD=//p' "$project_root/deploy/server/.env") +if [[ -z "$redis_password" ]]; then + echo "REDIS_PASSWORD is missing from the server environment" >&2 + exit 4 +fi + +docker exec "$postgres_container" pg_dump -U voice -d voice_platform -Fc --no-owner --no-acl >"$staging/postgres.dump" +docker exec "$postgres_container" psql -U voice -d voice_platform -Atc \ + "SELECT version||'|'||dirty||'|'||(SELECT count(*) FROM tenant)||'|'||(SELECT count(*) FROM user_account)||'|'||(SELECT count(*) FROM information_schema.tables WHERE table_schema='public')||'|'||(SELECT count(*) FROM information_schema.table_constraints WHERE constraint_type='FOREIGN KEY') FROM schema_migrations;" \ + >"$staging/database-evidence.txt" + +mkdir -p "$staging/redis" +printf '%s\n' "$redis_password" | docker exec -i "$redis_container" sh -ec ' + IFS= read -r REDISCLI_AUTH + export REDISCLI_AUTH + redis-cli --no-auth-warning SAVE >/dev/null +' +docker exec "$redis_container" redis-check-rdb /data/dump.rdb >/dev/null +docker cp "$redis_container:/data/dump.rdb" "$staging/redis/dump.rdb" >/dev/null + +docker exec "$minio_container" sh -ec ' + rm -rf /tmp/platform-backup /tmp/platform-mc + mkdir -p /tmp/platform-backup /tmp/platform-mc + mc --config-dir /tmp/platform-mc alias set local http://127.0.0.1:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" >/dev/null + mc --config-dir /tmp/platform-mc mirror --overwrite --preserve local/ /tmp/platform-backup/ +' +mkdir -p "$staging/object-store" +docker cp "$minio_container:/tmp/platform-backup/." "$staging/object-store/" >/dev/null +docker exec "$minio_container" rm -rf /tmp/platform-backup /tmp/platform-mc + +docker pause "$keycloak_container" >/dev/null +keycloak_paused=true +mkdir -p "$staging/keycloak" +docker cp "$keycloak_container:/opt/keycloak/data/." "$staging/keycloak/" >/dev/null +docker unpause "$keycloak_container" >/dev/null +keycloak_paused=false + +object_files=$(find "$staging/object-store" -type f | wc -l | tr -d ' ') +keycloak_files=$(find "$staging/keycloak" -type f | wc -l | tr -d ' ') +cat >"$staging/manifest.properties" <SHA256SUMS +) +tar -C "$staging" -czf "$plain_archive" . +openssl enc -aes-256-cbc -salt -pbkdf2 -iter 200000 \ + -pass "file:$passphrase_file" -in "$plain_archive" -out "$encrypted_archive" +sha256sum "$encrypted_archive" >"$checksum_file" +chmod 0600 "$encrypted_archive" "$checksum_file" + +while IFS= read -r -d '' expired; do + rm -f -- "$expired" "$expired.sha256" +done < <(find "$backup_root" -maxdepth 1 -type f -name 'platform-*.tar.gz.enc' -mtime "+$retention_days" -print0) + +if [[ "$verify_after_backup" == true ]]; then + "$project_root/deploy/server/verify-backup.sh" "$encrypted_archive" +fi + +printf '%s\n' "$encrypted_archive" diff --git a/deploy/server/install-backup.sh b/deploy/server/install-backup.sh new file mode 100755 index 0000000..d0143b2 --- /dev/null +++ b/deploy/server/install-backup.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +if [[ ${EUID:-$(id -u)} -ne 0 ]]; then + echo "Run this installer as root" >&2 + exit 2 +fi + +project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current} +install -d -m 0700 /etc/data-terminal-cloud /var/backups/data-terminal-cloud +if [[ ! -s /etc/data-terminal-cloud/backup-passphrase ]]; then + umask 077 + openssl rand -base64 48 >/etc/data-terminal-cloud/backup-passphrase +fi +chmod 0600 /etc/data-terminal-cloud/backup-passphrase +install -m 0644 "$project_root/deploy/server/platform-backup.service" /etc/systemd/system/platform-backup.service +install -m 0644 "$project_root/deploy/server/platform-backup.timer" /etc/systemd/system/platform-backup.timer +systemctl daemon-reload +systemctl enable --now platform-backup.timer +systemctl is-active --quiet platform-backup.timer +echo "platform-backup.timer enabled" diff --git a/deploy/server/platform-backup.service b/deploy/server/platform-backup.service new file mode 100644 index 0000000..cadd571 --- /dev/null +++ b/deploy/server/platform-backup.service @@ -0,0 +1,19 @@ +[Unit] +Description=Encrypted data terminal cloud backup and restore verification +Requires=docker.service +After=docker.service network-online.target + +[Service] +Type=oneshot +User=root +Group=root +UMask=0077 +Environment=PLATFORM_ROOT=/opt/data-terminal-cloud/current +Environment=BACKUP_ROOT=/var/backups/data-terminal-cloud +Environment=BACKUP_RETENTION_DAYS=14 +Environment=VERIFY_AFTER_BACKUP=true +ExecStart=/opt/data-terminal-cloud/current/deploy/server/backup.sh +Nice=10 +IOSchedulingClass=best-effort +IOSchedulingPriority=7 +TimeoutStartSec=2h diff --git a/deploy/server/platform-backup.timer b/deploy/server/platform-backup.timer new file mode 100644 index 0000000..ff1d92a --- /dev/null +++ b/deploy/server/platform-backup.timer @@ -0,0 +1,11 @@ +[Unit] +Description=Run encrypted data terminal cloud backup daily + +[Timer] +OnCalendar=*-*-* 02:40:00 +RandomizedDelaySec=45m +Persistent=true +Unit=platform-backup.service + +[Install] +WantedBy=timers.target diff --git a/deploy/server/verify-backup.sh b/deploy/server/verify-backup.sh new file mode 100755 index 0000000..b461490 --- /dev/null +++ b/deploy/server/verify-backup.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current} +backup_root=${BACKUP_ROOT:-/var/backups/data-terminal-cloud} +passphrase_file=${BACKUP_PASSPHRASE_FILE:-/etc/data-terminal-cloud/backup-passphrase} +archive=${1:-} + +if [[ -z "$archive" ]]; then + archive=$(find "$backup_root" -maxdepth 1 -type f -name 'platform-*.tar.gz.enc' -printf '%T@ %p\n' | sort -nr | head -n1 | cut -d' ' -f2-) +fi +if [[ -z "$archive" || ! -f "$archive" ]]; then + echo "Backup archive was not found" >&2 + exit 2 +fi +archive=$(realpath "$archive") +backup_root=$(realpath "$backup_root") +case "$archive" in + "$backup_root"/platform-*.tar.gz.enc) ;; + *) echo "Backup archive is outside the configured backup directory" >&2; exit 2 ;; +esac +if [[ ! -r "$passphrase_file" ]]; then + echo "Backup passphrase file is missing or unreadable" >&2 + exit 2 +fi + +compose=(docker compose -f "$project_root/compose.yaml" -f "$project_root/deploy/server/compose.override.yaml" --env-file "$project_root/deploy/server/.env") +postgres_container=$("${compose[@]}" ps -q postgres) +if [[ -z "$postgres_container" ]] || [[ $(docker inspect -f '{{.State.Running}}' "$postgres_container") != true ]]; then + echo "PostgreSQL service is not running" >&2 + exit 3 +fi +redis_container=$("${compose[@]}" ps -q redis) +if [[ -z "$redis_container" ]] || [[ $(docker inspect -f '{{.State.Running}}' "$redis_container") != true ]]; then + echo "Redis service is not running" >&2 + exit 3 +fi + +umask 077 +work=$(mktemp -d "$backup_root/.verify-XXXXXX") +plain="$work/backup.tar.gz" +verify_token=$(openssl rand -hex 12) +restore_database="voice_restore_verify_$verify_token" +redis_verify_path="/tmp/platform-backup-$verify_token.rdb" +database_created=false + +cleanup() { + status=$? + if [[ "$database_created" == true ]]; then + docker exec "$postgres_container" dropdb -U voice --if-exists "$restore_database" >/dev/null 2>&1 || true + fi + docker exec "$redis_container" rm -f -- "$redis_verify_path" >/dev/null 2>&1 || true + rm -rf -- "$work" + exit "$status" +} +trap cleanup EXIT INT TERM + +if [[ -f "$archive.sha256" ]]; then + (cd "$(dirname "$archive")" && sha256sum -c "$(basename "$archive").sha256") >/dev/null +fi +openssl enc -d -aes-256-cbc -pbkdf2 -iter 200000 \ + -pass "file:$passphrase_file" -in "$archive" -out "$plain" +tar -tzf "$plain" >"$work/archive-paths.txt" +if grep -Eq '(^/|(^|/)\.\.(/|$))' "$work/archive-paths.txt"; then + echo "Backup contains an unsafe archive path" >&2 + exit 4 +fi +mkdir -p "$work/extracted" +tar -xzf "$plain" --no-same-owner -C "$work/extracted" +( + cd "$work/extracted" + sha256sum -c SHA256SUMS >/dev/null +) + +expected=$(tr -d '\r\n' <"$work/extracted/database-evidence.txt") +docker exec "$postgres_container" createdb -U voice "$restore_database" +database_created=true +docker exec -i "$postgres_container" pg_restore -U voice -d "$restore_database" --no-owner --no-acl <"$work/extracted/postgres.dump" +actual=$(docker exec "$postgres_container" psql -U voice -d "$restore_database" -Atc \ + "SELECT version||'|'||dirty||'|'||(SELECT count(*) FROM tenant)||'|'||(SELECT count(*) FROM user_account)||'|'||(SELECT count(*) FROM information_schema.tables WHERE table_schema='public')||'|'||(SELECT count(*) FROM information_schema.table_constraints WHERE constraint_type='FOREIGN KEY') FROM schema_migrations;") +if [[ "$actual" != "$expected" ]]; then + echo "Restored database evidence does not match the backup: expected=$expected actual=$actual" >&2 + exit 5 +fi + +[[ -s "$work/extracted/redis/dump.rdb" ]] || { echo "Redis snapshot is missing" >&2; exit 6; } +[[ -d "$work/extracted/object-store" ]] || { echo "Object-store snapshot is missing" >&2; exit 6; } +[[ -d "$work/extracted/keycloak" ]] || { echo "Keycloak snapshot is missing" >&2; exit 6; } +expected_object_files=$(sed -n 's/^object_files=//p' "$work/extracted/manifest.properties") +expected_keycloak_files=$(sed -n 's/^keycloak_files=//p' "$work/extracted/manifest.properties") +actual_object_files=$(find "$work/extracted/object-store" -type f | wc -l | tr -d ' ') +actual_keycloak_files=$(find "$work/extracted/keycloak" -type f | wc -l | tr -d ' ') +if [[ ! "$expected_object_files" =~ ^[0-9]+$ || "$actual_object_files" != "$expected_object_files" ]]; then + echo "Object-store file count does not match the manifest" >&2 + exit 6 +fi +if [[ ! "$expected_keycloak_files" =~ ^[1-9][0-9]*$ || "$actual_keycloak_files" != "$expected_keycloak_files" ]]; then + echo "Keycloak file count does not match the manifest" >&2 + exit 6 +fi +docker cp "$work/extracted/redis/dump.rdb" "$redis_container:$redis_verify_path" >/dev/null +docker exec "$redis_container" redis-check-rdb "$redis_verify_path" >/dev/null +docker exec "$redis_container" rm -f -- "$redis_verify_path" + +printf 'backup_restore_verified archive=%s database_evidence=%s object_files=%s keycloak_files=%s redis_rdb=valid\n' \ + "$archive" "$actual" "$actual_object_files" "$actual_keycloak_files" diff --git a/docs/development-status.md b/docs/development-status.md index 80478a4..24c9db0 100644 --- a/docs/development-status.md +++ b/docs/development-status.md @@ -1,6 +1,6 @@ # 开发状态与门禁记录 -> 更新日期:2026-08-25 +> 更新日期:2026-08-26 > 当前接口契约版本:0.64.0 > 当前主里程碑:M8目标环境联调、试点与上线验收(外部条件待提供) @@ -180,6 +180,7 @@ | 媒体清单容量 | 10批×1000分片准确落库和关联;批量SQL优化后平均0.12秒、p95/最大0.16秒,API容器内存峰值15.8MiB | | 隐私删除并发 | 100个并发受理全部成功、HTTP失败率0%、提交p95 385.15ms;11.58秒内600个子任务、100个墓碑和100个主体冻结全部对账 | | PostgreSQL隔离恢复 | 实际`pg_dump`/`pg_restore`到随机隔离库,迁移`22|false`、66张表、190个外键和关键计数一致;本地约5.14秒,临时库/备份已清理 | +| 阿里云单机加密备份与恢复 | 目标服务器已安装每日systemd备份任务,覆盖PostgreSQL、认证Redis RDB、MinIO全部桶和暂停后的Keycloak数据目录;归档使用AES-256-CBC/PBKDF2加密、逐文件SHA-256并保留14天。首次实际归档成功,数据库从V38备份恢复到随机隔离库后`38|false|2|3|73|215`六项证据一致,归档内Redis RDB格式有效、Keycloak 3个文件一致、临时恢复库为0;同机备份尚不作为跨故障域生产恢复证据 | | 进程与Redis恢复 | API/Callbacks/Worker重启恢复;停机期间Outbox仅执行一次;Redis 14个热状态键清空后由签名设备请求重建,PostgreSQL权威计数不变 | | 供应商故障恢复 | V23持久化按租户/供应商的可用状态;连续临时失败暂停新启动但停止任务继续尝试;演练确认暂停后新外呼0、停止尝试2次、历史接口200、CRITICAL告警出现,恢复后全部收敛;V23单版本回退/重升通过 | | 安全专项 | Go 1.26.6容器`govulncheck`可达漏洞0、npm高危审计0、最终镜像HIGH/CRITICAL已修复漏洞0、Gitleaks无泄漏;HTTP认证/授权/注入式输入/1MiB正文/回调签名/Nonce重放/安全头/日志过滤烟测通过 | diff --git a/docs/requirements-traceability.md b/docs/requirements-traceability.md index b14f959..ad2f731 100644 --- a/docs/requirements-traceability.md +++ b/docs/requirements-traceability.md @@ -90,6 +90,6 @@ | P0414 | 首期需求、验收用例、缺陷和风险形成对应关系 | 本文完成首期功能到代码、接口、数据和验证的基线;V30问题清单由V36补充严重级别、根因、修复版本、回归证据及未关闭HIGH/CRITICAL发布阻断汇总;OpenAPI 0.50.0—0.55.0可按候选构建导出六项技术门禁及脱敏阻断告警定位快照,绑定候选版本/构建摘要后按业务时区归档日/周运行证据。`cmd/release-evidence`现使用v3清单,把同一候选的快照、接口、迁移、部署、SBOM和十类外部状态形成逐文件摘要;远程CI供应链、真机固件、真实终端RTC、真实Agent模型、真实告警网关、云KMS/HSM轮换、生产等价恢复、生产基础设施和Safari兼容分别要求结构化报告,并把14项CI检查/镜像/SBOM、设备固件场景、语音方式与恢复场景、生成/Embedding/内容安全质量、路由演练、轮换模式、恢复类型、基础设施控制项及Safari矩阵覆盖与缺口写入清单。OpenAPI 0.64.0、V38及九个离线校验器把云端状态与远程构建、外部设备、RTC/语音供应商、模型提供器、网关、密钥提供器、目标环境和浏览器证据绑定;`content_safety_tls`把策略服务网络/证书状态纳入指标、告警、运营页面和候选快照。缺检查、缺制品、缺场景、缺演练、缺质量类别、缺控制项、`PENDING`和本地模拟证据均不会被误报为外部验收完成,整体发布仍必须基于真实联调、灰度和审批记录确认。 | | P0445 | 需求编号关联代码、测试、接口或配置变更 | 本文逐项建立关联;后续功能变更必须同步更新本矩阵和`development-status.md`。 | -当前仍不能在本地宣告完成的部分集中为:真实火山RTC与豆包三模式联调、真实方舟内容与Embedding质量、真实听筒的长者交流体验、量产设备/KMS注入、生产对象存储与备份、真机异常恢复、真实账单和生产级内容安全。它们属于阶段8外部验收,不应通过增加本地低价值测试伪造完成证据。 +当前仍不能宣告完成的部分集中为:真实火山RTC与豆包三模式联调、真实方舟内容与Embedding质量、真实听筒的长者交流体验、量产设备/KMS注入、跨故障域生产对象存储与备份、真机异常恢复、真实账单和生产级内容安全。阿里云单机环境已经形成加密全组件归档、数据库隔离恢复和Redis RDB校验,但同机归档不能替代托管PITR、异地副本和生产恢复演练。这些剩余项属于阶段8外部验收,不应通过增加本地低价值测试伪造完成证据。 下一批本地开发应优先从矩阵中出现“进行中”的条目选择;若没有本地功能缺口,则只继续完善能够直接降低阶段8联调风险的模拟器、诊断和交付工具,不以覆盖率数字替代业务功能。 diff --git a/docs/server-backup-and-restore.md b/docs/server-backup-and-restore.md new file mode 100644 index 0000000..42ccfaf --- /dev/null +++ b/docs/server-backup-and-restore.md @@ -0,0 +1,101 @@ +# 单机云端备份与隔离恢复手册 + +## 1. 适用范围 + +本手册适用于`deploy/server/compose.override.yaml`部署的单机公网演示环境。目标是在不暴露数据库、Redis、对象存储和身份服务端口的前提下,形成可自动执行、可校验、可追溯的加密备份,并证明PostgreSQL备份能够恢复到独立数据库。 + +该能力不是跨可用区生产灾备。单机磁盘故障仍可能同时损失在线数据和本机备份,因此加密归档和口令必须另行复制到不同故障域。 + +## 2. 备份范围 + +| 组件 | 备份方式 | 当前恢复验证 | +|---|---|---| +| PostgreSQL | `pg_dump`自定义格式,不包含所有者和ACL | 实际恢复到随机隔离数据库,并核对迁移、客户组织、账号、表和外键计数 | +| Redis | 认证执行`SAVE`后复制RDB | `redis-check-rdb`验证归档内RDB格式,不覆盖在线Redis | +| MinIO | 使用容器内`mc mirror`复制全部桶和对象 | 逐文件SHA-256和对象文件数量一致性检查 | +| Keycloak | 暂停容器后复制`/opt/keycloak/data`,随后立即恢复 | 逐文件SHA-256和文件数量检查;当前开发模式H2数据不作为生产OIDC方案 | + +备份归档使用`AES-256-CBC + PBKDF2`加密,迭代次数为200000。每个归档旁边保存加密文件自身的SHA-256;归档内部另有逐文件`SHA256SUMS`和不含凭据的`manifest.properties`。 + +## 3. 自动执行流程 + +```text +获取独占锁 + → PostgreSQL逻辑转储和证据快照 + → Redis认证SAVE及RDB校验 + → MinIO全桶对象镜像 + → 短暂暂停Keycloak并复制一致性数据目录 + → 逐文件SHA-256 + → 压缩、加密、生成归档摘要 + → PostgreSQL随机隔离库实际恢复 + → Redis RDB及文件清单复核 + → 删除临时数据库和临时明文 + → 按保留期清理旧加密归档 +``` + +任何步骤失败都会令systemd服务失败;未完成的明文暂存目录由退出陷阱删除。备份使用`flock`避免重叠执行,Keycloak无论成功或异常退出都会尝试解除暂停。 + +## 4. 安装与运行 + +```bash +cd /opt/data-terminal-cloud/current +chmod 0755 deploy/server/backup.sh deploy/server/verify-backup.sh deploy/server/install-backup.sh +deploy/server/install-backup.sh +systemctl start platform-backup.service +systemctl show platform-backup.service -p Result -p ExecMainStatus +systemctl list-timers platform-backup.timer --no-pager +``` + +默认每天02:40执行,并附加最长45分钟随机延迟,避免固定时刻资源竞争;默认保留14天。定时任务设置`Persistent=true`,服务器在计划时刻关机时会在恢复后补跑。 + +可通过环境变量覆盖: + +| 变量 | 默认值 | 说明 | +|---|---|---| +| `PLATFORM_ROOT` | `/opt/data-terminal-cloud/current` | 当前发布目录 | +| `BACKUP_ROOT` | `/var/backups/data-terminal-cloud` | 加密归档目录,必须为安全的绝对路径 | +| `BACKUP_PASSPHRASE_FILE` | `/etc/data-terminal-cloud/backup-passphrase` | 仅root可读的独立随机口令 | +| `BACKUP_RETENTION_DAYS` | `14` | 本机备份保留天数 | +| `VERIFY_AFTER_BACKUP` | `false`;systemd中为`true` | 备份后是否立即执行隔离恢复校验 | + +## 5. 手工校验与故障定位 + +校验最新备份: + +```bash +/opt/data-terminal-cloud/current/deploy/server/verify-backup.sh +``` + +校验指定备份: + +```bash +/opt/data-terminal-cloud/current/deploy/server/verify-backup.sh \ + /var/backups/data-terminal-cloud/platform-YYYYmmddTHHMMSSZ.tar.gz.enc +``` + +查看任务结果: + +```bash +systemctl show platform-backup.service -p Result -p ExecMainStatus -p ActiveState +journalctl -u platform-backup.service -n 100 --no-pager +find /var/backups/data-terminal-cloud -maxdepth 1 -type f -name 'platform-*.enc' -ls +``` + +一次性服务成功结束后显示`inactive (dead)`是正常状态,应结合`Result=success`和`ExecMainStatus=0`判断。校验脚本不会停止在线业务、覆盖在线数据库或载入Redis快照;随机恢复数据库在成功和失败路径都会清理。 + +## 6. 安全与恢复边界 + +- 备份口令不写入仓库、归档、日志或Compose环境文件;默认权限为`0600 root:root`。 +- 备份目录默认权限为`0700`,归档和摘要为`0600`。 +- 明文转储只存在于同一受控暂存目录,归档加密后立即删除。 +- 当前归档留在同一云服务器只提供误操作恢复能力,不提供主机或地域故障保护。 +- 生产环境应使用托管PostgreSQL持续备份/PITR、版本化对象存储、外部OIDC、跨故障域副本、KMS托管密钥和受控恢复演练。 +- 不能仅凭定时任务成功声称满足某个RPO/RTO;必须结合备份频率、异地复制、真实数据量和受控恢复耗时单独验收。 + +## 7. 上线前检查 + +1. 把加密归档复制到不同故障域,并独立验证复制后的SHA-256。 +2. 把口令保存到云KMS、密码库或离线托管介质,不与归档放在同一位置。 +3. 配置备份失败和长时间未成功告警。 +4. 在预生产环境执行PostgreSQL、Redis、对象存储和OIDC的完整恢复演练。 +5. 记录实际备份大小、持续时间、恢复时间和业务抽查结果,再确定RPO/RTO。