From 1e12d329dc5505775686f834819c6c69c78ce99f Mon Sep 17 00:00:00 2001 From: Codex Local Date: Wed, 26 Aug 2026 20:55:35 +0800 Subject: [PATCH] feat: add OSS offsite backup gate --- README.md | 2 +- deploy/server/README.md | 11 ++ deploy/server/health-monitor.sh | 41 +++++- deploy/server/install-offsite-backup.sh | 28 ++++ deploy/server/install-ossutil.sh | 36 +++++ deploy/server/platform-offsite-backup.service | 18 +++ deploy/server/platform-offsite-backup.timer | 11 ++ deploy/server/sync-backup-offsite.sh | 128 ++++++++++++++++++ docs/development-status.md | 3 +- docs/offsite-backup.md | 110 +++++++++++++++ 10 files changed, 384 insertions(+), 4 deletions(-) create mode 100755 deploy/server/install-offsite-backup.sh create mode 100755 deploy/server/install-ossutil.sh create mode 100644 deploy/server/platform-offsite-backup.service create mode 100644 deploy/server/platform-offsite-backup.timer create mode 100755 deploy/server/sync-backup-offsite.sh create mode 100644 docs/offsite-backup.md diff --git a/README.md b/README.md index f6ba717..8636b06 100644 --- a/README.md +++ b/README.md @@ -332,7 +332,7 @@ scripts/ 本地启动和质量检查脚本 test/ 契约、集成和端到端测试 ``` -公共接口约束与监控指标见[可观测性与接口治理设计](docs/observability-and-api-governance.md)，设备签名和模拟联调见[设备接入与启动配置联调说明](docs/device-integration.md)，设备命令见[设备命令可靠下发说明](docs/device-commands.md)，固件流程见[固件制品、签名、灰度与回滚开发说明](docs/firmware-rollout.md)，A/B升级与启动失败回退见[设备A/B分区固件升级与启动确认协议](docs/device-firmware-ab-protocol.md)，设备/云端时间线对账见[固件真机联调一致性证据采集与校验说明](docs/firmware-conformance-evidence.md)，双轨录音见[双轨录音上传、校验与访问开发说明](docs/media-upload-and-access.md)，网页状态通知见[Web状态事件流开发说明](docs/web-event-stream.md)，具体分享见[具体资源分享开发说明](docs/resource-sharing.md)，运营配置见[内部运营身份与运行配置版本开发说明](docs/operations-config.md)，用量费用与额度见[用量、费用与额度开发说明](docs/usage-cost-quota.md)，告警闭环见[告警监测与运营处置开发说明](docs/alert-operations.md)，真实网关归档见[告警接收器一致性证据](docs/alert-receiver-conformance-evidence.md)，云KMS轮换见[KMS/HSM轮换一致性证据](docs/kms-hsm-rotation-evidence.md)，生产恢复见[生产等价恢复演练一致性证据](docs/production-recovery-evidence.md)，单机云端备份见[自动备份与隔离恢复手册](docs/server-backup-and-restore.md)，服务器健康检查见[单机云端健康监测与告警手册](docs/server-health-monitoring.md)，生产基础设施见[生产基础设施一致性证据](docs/production-infrastructure-evidence.md)，联调工作台及发布候选验收快照见[集成诊断工作台开发说明](docs/integration-diagnostics.md)，试点每日证据见[试点运行日报开发说明](docs/pilot-daily-operations.md)，试点周度复盘口径、外部投诉缺口、接口和轻量烟测见[试点运行周报开发说明](docs/pilot-weekly-review.md)，敏感数据提权见[运营敏感数据临时访问开发说明](docs/ops-sensitive-access.md)，外部上线阶段见[真机、真实供应商、试点与生产发布外部验收清单](docs/external-acceptance-checklist.md)，生产门禁和压测告警见[压测、告警与生产安全门禁](docs/load-alerting-and-production-security.md)，专项实测证据见[性能、恢复、安全与兼容性专项验证报告](docs/performance-recovery-security-compatibility.md)，隐私数据访问见[隐私数据请求开发说明](docs/privacy-requests.md)，私有导出见[私有数据导出说明](docs/data-export.md)。 +公共接口约束与监控指标见[可观测性与接口治理设计](docs/observability-and-api-governance.md)，设备签名和模拟联调见[设备接入与启动配置联调说明](docs/device-integration.md)，设备命令见[设备命令可靠下发说明](docs/device-commands.md)，固件流程见[固件制品、签名、灰度与回滚开发说明](docs/firmware-rollout.md)，A/B升级与启动失败回退见[设备A/B分区固件升级与启动确认协议](docs/device-firmware-ab-protocol.md)，设备/云端时间线对账见[固件真机联调一致性证据采集与校验说明](docs/firmware-conformance-evidence.md)，双轨录音见[双轨录音上传、校验与访问开发说明](docs/media-upload-and-access.md)，网页状态通知见[Web状态事件流开发说明](docs/web-event-stream.md)，具体分享见[具体资源分享开发说明](docs/resource-sharing.md)，运营配置见[内部运营身份与运行配置版本开发说明](docs/operations-config.md)，用量费用与额度见[用量、费用与额度开发说明](docs/usage-cost-quota.md)，告警闭环见[告警监测与运营处置开发说明](docs/alert-operations.md)，真实网关归档见[告警接收器一致性证据](docs/alert-receiver-conformance-evidence.md)，云KMS轮换见[KMS/HSM轮换一致性证据](docs/kms-hsm-rotation-evidence.md)，生产恢复见[生产等价恢复演练一致性证据](docs/production-recovery-evidence.md)，单机云端备份见[自动备份与隔离恢复手册](docs/server-backup-and-restore.md)，OSS异地备份见[阿里云OSS异地备份适配与门禁](docs/offsite-backup.md)，服务器健康检查见[单机云端健康监测与告警手册](docs/server-health-monitoring.md)，生产基础设施见[生产基础设施一致性证据](docs/production-infrastructure-evidence.md)，联调工作台及发布候选验收快照见[集成诊断工作台开发说明](docs/integration-diagnostics.md)，试点每日证据见[试点运行日报开发说明](docs/pilot-daily-operations.md)，试点周度复盘口径、外部投诉缺口、接口和轻量烟测见[试点运行周报开发说明](docs/pilot-weekly-review.md)，敏感数据提权见[运营敏感数据临时访问开发说明](docs/ops-sensitive-access.md)，外部上线阶段见[真机、真实供应商、试点与生产发布外部验收清单](docs/external-acceptance-checklist.md)，生产门禁和压测告警见[压测、告警与生产安全门禁](docs/load-alerting-and-production-security.md)，专项实测证据见[性能、恢复、安全与兼容性专项验证报告](docs/performance-recovery-security-compatibility.md)，隐私数据访问见[隐私数据请求开发说明](docs/privacy-requests.md)，私有导出见[私有数据导出开发说明](docs/data-export.md)。 从框架搭建到生产落地的阶段任务、里程碑、人员、测试门禁、依赖和风险见[详细开发计划](docs/development-plan.md)，当前实际完成证据见[开发状态与门禁记录](docs/development-status.md)。 diff --git a/deploy/server/README.md b/deploy/server/README.md index 0c4687b..71e12ba 100644 --- a/deploy/server/README.md +++ b/deploy/server/README.md @@ -55,3 +55,14 @@ python3 -m json.tool /var/lib/data-terminal-cloud-monitor/status.json ``` 监测任务每五分钟执行，正常状态为`healthy`。默认只记录结构化状态和systemd日志；在`/etc/data-terminal-cloud/monitor.env`配置`ALERT_WEBHOOK_URL`后，状态变化会发送带时间戳和HMAC-SHA256签名的JSON通知。完整检查项、签名规则、阈值和失败分支见[《单机云端健康监测与告警手册》](../../docs/server-health-monitoring.md)。 + +## OSS异地备份适配 + +```bash +chmod 0755 deploy/server/install-ossutil.sh \ + deploy/server/install-offsite-backup.sh \ + deploy/server/sync-backup-offsite.sh +sudo deploy/server/install-offsite-backup.sh +``` + +安装后默认为禁用状态，不会访问OSS。创建私有Bucket并绑定最小权限ECS RAM角色后，在`/etc/data-terminal-cloud/offsite-backup.env`填写Bucket和前缀、设置`OFFSITE_BACKUP_ENABLED=true`与`OFFSITE_BACKUP_REQUIRED=true`，再手工启动一次`platform-offsite-backup.service`。任务会上传加密归档及摘要、重新下载并复算SHA-256；健康检查在启用后强制要求最新本机归档已有对应远端验证。配置和真实验收步骤见[《阿里云OSS异地备份适配与门禁》](../../docs/offsite-backup.md)。 diff --git a/deploy/server/health-monitor.sh b/deploy/server/health-monitor.sh index 56c7a79..c7d594d 100755 --- a/deploy/server/health-monitor.sh +++ b/deploy/server/health-monitor.sh @@ -5,6 +5,7 @@ project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current} backup_root=${BACKUP_ROOT:-/var/backups/data-terminal-cloud} state_dir=${MONITOR_STATE_DIR:-/var/lib/data-terminal-cloud-monitor} monitor_config=${MONITOR_CONFIG_FILE:-/etc/data-terminal-cloud/monitor.env} +offsite_config_file=${OFFSITE_CONFIG_FILE:-/etc/data-terminal-cloud/offsite-backup.env} backup_max_age_seconds=${BACKUP_MAX_AGE_SECONDS:-129600} certificate_min_seconds=${CERTIFICATE_MIN_SECONDS:-129600} disk_warning_percent=${DISK_WARNING_PERCENT:-80} @@ -17,8 +18,16 @@ if [[ -f "$monitor_config" ]]; then fi alert_webhook_url=${ALERT_WEBHOOK_URL:-} alert_webhook_secret_file=${ALERT_WEBHOOK_SECRET_FILE:-/etc/data-terminal-cloud/monitor-webhook-secret} +if [[ -f "$offsite_config_file" ]]; then + # shellcheck disable=SC1090 + source "$offsite_config_file" +fi +offsite_enabled=${OFFSITE_BACKUP_ENABLED:-false} +offsite_required=${OFFSITE_BACKUP_REQUIRED:-false} +offsite_state_file=${OFFSITE_STATE_FILE:-/var/lib/data-terminal-cloud-offsite/status.json} +offsite_max_age_seconds=${OFFSITE_MAX_AGE_SECONDS:-129600} -for numeric in "$backup_max_age_seconds" "$certificate_min_seconds" "$disk_warning_percent" "$disk_critical_percent"; do +for numeric in "$backup_max_age_seconds" "$certificate_min_seconds" "$disk_warning_percent" "$disk_critical_percent" "$offsite_max_age_seconds"; do [[ "$numeric" =~ ^[0-9]+$ ]] || { echo "Monitor thresholds must be non-negative integers" >&2; exit 2; } done if (( disk_warning_percent >= disk_critical_percent || disk_critical_percent > 100 )); then @@ -117,6 +126,30 @@ else fi fi +if [[ "$offsite_enabled" == true || "$offsite_required" == true ]]; then + if [[ ! -r "$offsite_state_file" ]]; then + record_check offsite_backup critical status_missing + else + offsite_evidence=$(python3 -c 'import json,sys +try: + d=json.load(open(sys.argv[1], encoding="utf-8")); print("|".join([str(d.get("status", "")), str(d.get("verified_epoch", 0)), str(d.get("source_archive", ""))])) +except Exception: print("invalid|0|")' "$offsite_state_file") + IFS='|' read -r offsite_status offsite_epoch offsite_source <<<"$offsite_evidence" + offsite_age=$(($(date +%s) - offsite_epoch)) + if [[ "$offsite_status" != verified ]]; then + record_check offsite_backup critical "status=${offsite_status:-invalid}" + elif [[ "$offsite_source" != "$(basename "${latest_backup:-missing}")" ]]; then + record_check offsite_backup critical source_mismatch + elif (( offsite_epoch <= 0 || offsite_age > offsite_max_age_seconds )); then + record_check offsite_backup critical "age_seconds=$offsite_age" + else + record_check offsite_backup healthy "age_seconds=$offsite_age" + fi + fi +else + record_check offsite_backup healthy not_required +fi + backup_result=$(systemctl show platform-backup.service -p Result --value 2>/dev/null || true) backup_exit=$(systemctl show platform-backup.service -p ExecMainStatus --value 2>/dev/null || true) if [[ "$backup_result" == success && "$backup_exit" == 0 ]]; then @@ -125,7 +158,11 @@ else record_check backup_service critical "result=${backup_result:-unknown}/exit=${backup_exit:-unknown}" fi -for timer in platform-backup.timer certbot-ip-renew.timer; do +timers=(platform-backup.timer certbot-ip-renew.timer) +if [[ -f "$offsite_config_file" ]]; then + timers+=(platform-offsite-backup.timer) +fi +for timer in "${timers[@]}"; do timer_enabled=$(systemctl is-enabled "$timer" 2>/dev/null || true) timer_active=$(systemctl is-active "$timer" 2>/dev/null || true) if [[ "$timer_enabled" == enabled && "$timer_active" == active ]]; then diff --git a/deploy/server/install-offsite-backup.sh b/deploy/server/install-offsite-backup.sh new file mode 100755 index 0000000..00277fa --- /dev/null +++ b/deploy/server/install-offsite-backup.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +if [[ ${EUID:-$(id -u)} -ne 0 ]]; then + echo "Run this installer as root" >&2 + exit 2 +fi + +project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current} +"$project_root/deploy/server/install-ossutil.sh" +install -d -m 0700 /etc/data-terminal-cloud +install -d -m 0750 /var/lib/data-terminal-cloud-offsite +if [[ ! -e /etc/data-terminal-cloud/offsite-backup.env ]]; then + cat >/etc/data-terminal-cloud/offsite-backup.env <<'EOF' +OFFSITE_BACKUP_ENABLED=false +OFFSITE_BACKUP_REQUIRED=false +OFFSITE_OSS_BUCKET= +OFFSITE_OSS_PREFIX=data-terminal-cloud/121.41.46.238 +OFFSITE_OSS_CONFIG_FILE=/etc/data-terminal-cloud/ossutilconfig +EOF +fi +chmod 0600 /etc/data-terminal-cloud/offsite-backup.env +install -m 0644 "$project_root/deploy/server/platform-offsite-backup.service" /etc/systemd/system/platform-offsite-backup.service +install -m 0644 "$project_root/deploy/server/platform-offsite-backup.timer" /etc/systemd/system/platform-offsite-backup.timer +systemctl daemon-reload +systemctl enable --now platform-offsite-backup.timer +systemctl is-active --quiet platform-offsite-backup.timer +echo "platform-offsite-backup.timer enabled; upload remains disabled until OSS identity and bucket are configured" diff --git a/deploy/server/install-ossutil.sh b/deploy/server/install-ossutil.sh new file mode 100755 index 0000000..3cc2d02 --- /dev/null +++ b/deploy/server/install-ossutil.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +if [[ ${EUID:-$(id -u)} -ne 0 ]]; then + echo "Run this installer as root" >&2 + exit 2 +fi + +version=2.3.0 +archive=ossutil-${version}-linux-amd64.zip +expected_sha256=3ae4d9fc85a7a6e9f5654d1599766f1a3a42a3692870887b5ae9338d582ef65a +download_url="https://gosspublic.alicdn.com/ossutil/v2/${version}/${archive}" + +if command -v ossutil >/dev/null 2>&1 && [[ $(ossutil version 2>/dev/null) == "$version" ]]; then + echo "ossutil $version already installed" + exit 0 +fi +install_dir=$(mktemp -d /var/tmp/ossutil-install-XXXXXX) +case "$install_dir" in /var/tmp/ossutil-install-*) ;; *) echo "Unsafe install directory" >&2; exit 9;; esac +trap 'rm -rf -- "$install_dir"' EXIT +curl -fsSL --max-time 120 -o "$install_dir/$archive" "$download_url" +printf '%s %s\n' "$expected_sha256" "$install_dir/$archive" | sha256sum -c - +mkdir -p "$install_dir/unpacked" +if command -v unzip >/dev/null 2>&1; then + unzip -q "$install_dir/$archive" -d "$install_dir/unpacked" +elif command -v python3 >/dev/null 2>&1; then + python3 -m zipfile -e "$install_dir/$archive" "$install_dir/unpacked" +else + echo "Neither unzip nor python3 is available to extract ossutil" >&2 + exit 3 +fi +binary=$(find "$install_dir/unpacked" -type f -name ossutil -print -quit) +[[ -n "$binary" ]] || { echo "ossutil binary is missing from the verified archive" >&2; exit 4; } +install -m 0755 "$binary" /usr/local/bin/ossutil +[[ $(/usr/local/bin/ossutil version) == "$version" ]] +echo "ossutil $version installed" diff --git a/deploy/server/platform-offsite-backup.service b/deploy/server/platform-offsite-backup.service new file mode 100644 index 0000000..6ac6f90 --- /dev/null +++ b/deploy/server/platform-offsite-backup.service @@ -0,0 +1,18 @@ +[Unit] +Description=Copy encrypted data terminal cloud backup to OSS +Requires=network-online.target +After=network-online.target platform-backup.service + +[Service] +Type=oneshot +User=root +Group=root +UMask=0077 +Environment=BACKUP_ROOT=/var/backups/data-terminal-cloud +Environment=OFFSITE_STATE_DIR=/var/lib/data-terminal-cloud-offsite +Environment=OFFSITE_CONFIG_FILE=/etc/data-terminal-cloud/offsite-backup.env +ExecStart=/opt/data-terminal-cloud/current/deploy/server/sync-backup-offsite.sh +Nice=10 +NoNewPrivileges=true +PrivateTmp=true +TimeoutStartSec=4h diff --git a/deploy/server/platform-offsite-backup.timer b/deploy/server/platform-offsite-backup.timer new file mode 100644 index 0000000..d9b8e95 --- /dev/null +++ b/deploy/server/platform-offsite-backup.timer @@ -0,0 +1,11 @@ +[Unit] +Description=Copy encrypted data terminal cloud backup to OSS daily + +[Timer] +OnCalendar=*-*-* 04:10:00 +RandomizedDelaySec=30m +Persistent=true +Unit=platform-offsite-backup.service + +[Install] +WantedBy=timers.target diff --git a/deploy/server/sync-backup-offsite.sh b/deploy/server/sync-backup-offsite.sh new file mode 100755 index 0000000..16bf9a4 --- /dev/null +++ b/deploy/server/sync-backup-offsite.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +backup_root=${BACKUP_ROOT:-/var/backups/data-terminal-cloud} +state_dir=${OFFSITE_STATE_DIR:-/var/lib/data-terminal-cloud-offsite} +offsite_config_file=${OFFSITE_CONFIG_FILE:-/etc/data-terminal-cloud/offsite-backup.env} +ossutil_bin=${OSSUTIL_BIN:-/usr/local/bin/ossutil} + +mkdir -p "$state_dir" +chmod 0750 "$state_dir" +umask 077 +status_file="$state_dir/status.json" +status_tmp=$(mktemp "$state_dir/.status-XXXXXX") +transfer_dir=$(mktemp -d "$state_dir/.transfer-XXXXXX") + +cleanup() { + rm -f -- "$status_tmp" + rm -rf -- "$transfer_dir" +} +trap cleanup EXIT INT TERM + +write_status() { + local status=$1 + local reason=$2 + local source_archive=${3:-} + local object_ref=${4:-} + local sha256=${5:-} + local size_bytes=${6:-0} + local verified_epoch=0 + if [[ "$status" == verified ]]; then + verified_epoch=$(date +%s) + fi + python3 - "$status_tmp" "$status" "$reason" "$source_archive" "$object_ref" "$sha256" "$size_bytes" "$verified_epoch" <<'PY' +import datetime +import json +import sys + +payload = { + "schema": "platform-offsite-backup-v1", + "status": sys.argv[2], + "reason": sys.argv[3], + "source_archive": sys.argv[4], + "object_ref": sys.argv[5], + "sha256": sys.argv[6], + "size_bytes": int(sys.argv[7]), + "verified_epoch": int(sys.argv[8]), + "observed_at": datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z"), +} +with open(sys.argv[1], "w", encoding="utf-8") as target: + json.dump(payload, target, ensure_ascii=False, separators=(",", ":")) + target.write("\n") +PY + chmod 0644 "$status_tmp" + mv -f "$status_tmp" "$status_file" +} + +fail_sync() { + local reason=$1 + local code=${2:-1} + write_status failed "$reason" "${source_name:-}" "${remote_archive:-}" "${source_sha:-}" "${source_size:-0}" + echo "offsite_backup status=failed reason=$reason" >&2 + exit "$code" +} + +if [[ ! -f "$offsite_config_file" ]]; then + write_status disabled config_missing + echo "offsite_backup status=disabled reason=config_missing" + exit 0 +fi +# The installer creates this file as root:root 0600. +# shellcheck disable=SC1090 +source "$offsite_config_file" + +offsite_enabled=${OFFSITE_BACKUP_ENABLED:-false} +bucket=${OFFSITE_OSS_BUCKET:-} +prefix=${OFFSITE_OSS_PREFIX:-data-terminal-cloud} +oss_config=${OFFSITE_OSS_CONFIG_FILE:-/etc/data-terminal-cloud/ossutilconfig} + +if [[ "$offsite_enabled" != true ]]; then + write_status disabled not_enabled + echo "offsite_backup status=disabled reason=not_enabled" + exit 0 +fi +[[ "$bucket" =~ ^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$ ]] || fail_sync invalid_bucket 2 +[[ "$prefix" =~ ^[A-Za-z0-9._/-]+$ && "$prefix" != /* && "$prefix" != *..* ]] || fail_sync invalid_prefix 2 +[[ -x "$ossutil_bin" ]] || fail_sync ossutil_missing 3 +[[ -r "$oss_config" ]] || fail_sync oss_config_missing 4 + +latest_backup=$(find "$backup_root" -maxdepth 1 -type f -name 'platform-*.tar.gz.enc' -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n1 | cut -d' ' -f2- || true) +[[ -n "$latest_backup" && -f "$latest_backup" ]] || fail_sync local_backup_missing 5 +[[ -f "$latest_backup.sha256" ]] || fail_sync local_checksum_missing 5 +if ! (cd "$backup_root" && sha256sum -c "$(basename "$latest_backup").sha256" >/dev/null 2>&1); then + fail_sync local_checksum_failed 5 +fi + +source_name=$(basename "$latest_backup") +source_sha=$(sha256sum "$latest_backup" | awk '{print $1}') +source_size=$(stat -c %s "$latest_backup") +prefix=${prefix%/} +remote_archive="oss://$bucket/$prefix/$source_name" +remote_checksum="$remote_archive.sha256" +checkpoint_dir="$state_dir/checkpoints" +mkdir -p "$checkpoint_dir" + +common=(-f -c "$oss_config" --job 1 --parallel 1 --checkpoint-dir "$checkpoint_dir" --output-format json) +if ! "$ossutil_bin" cp "$latest_backup" "$remote_archive" "${common[@]}" --metadata "sha256=$source_sha" >/dev/null; then + fail_sync archive_upload_failed 6 +fi +if ! "$ossutil_bin" cp "$latest_backup.sha256" "$remote_checksum" "${common[@]}" >/dev/null; then + fail_sync checksum_upload_failed 6 +fi + +downloaded="$transfer_dir/$source_name" +downloaded_checksum="$downloaded.sha256" +if ! "$ossutil_bin" cp "$remote_archive" "$downloaded" "${common[@]}" >/dev/null; then + fail_sync archive_download_verification_failed 7 +fi +if ! "$ossutil_bin" cp "$remote_checksum" "$downloaded_checksum" "${common[@]}" >/dev/null; then + fail_sync checksum_download_verification_failed 7 +fi +downloaded_sha=$(sha256sum "$downloaded" | awk '{print $1}') +checksum_sha=$(awk 'NF >= 1 {print $1; exit}' "$downloaded_checksum") +if [[ "$downloaded_sha" != "$source_sha" || "$checksum_sha" != "$source_sha" ]]; then + fail_sync remote_checksum_mismatch 8 +fi + +write_status verified upload_and_download_verified "$source_name" "$remote_archive" "$source_sha" "$source_size" +printf 'offsite_backup status=verified source=%s object=%s size_bytes=%s\n' "$source_name" "$remote_archive" "$source_size" diff --git a/docs/development-status.md b/docs/development-status.md index 1fcb97a..521ac77 100644 --- a/docs/development-status.md +++ b/docs/development-status.md @@ -181,7 +181,8 @@ | 隐私删除并发 | 100个并发受理全部成功、HTTP失败率0%、提交p95 385.15ms；11.58秒内600个子任务、100个墓碑和100个主体冻结全部对账 | | PostgreSQL隔离恢复 | 实际`pg_dump`/`pg_restore`到随机隔离库，迁移`22|false`、66张表、190个外键和关键计数一致；本地约5.14秒，临时库/备份已清理 | | 阿里云单机加密备份与恢复 | 目标服务器已安装每日systemd备份任务，覆盖PostgreSQL、认证Redis RDB、MinIO全部桶和暂停后的Keycloak数据目录；归档使用AES-256-CBC/PBKDF2加密、逐文件SHA-256并保留14天。首次实际归档成功，数据库从V38备份恢复到随机隔离库后`38|false|2|3|73|215`六项证据一致，归档内Redis RDB格式有效、Keycloak 3个文件一致、临时恢复库为0；同机备份尚不作为跨故障域生产恢复证据 | -| 阿里云单机健康监测 | 目标服务器已安装五分钟systemd健康检查，结构化状态覆盖HTTPS、OIDC签发地址、11个常驻容器、备份新鲜度与摘要、备份执行结果、备份/证书timer、短期IP证书剩余时间和磁盘阈值。实际正常路径20项全部healthy；把备份最大年龄临时设为0秒后仅`backup_freshness`进入critical并退出1，测试目录已清理且线上容器未停止。可选Webhook仅在状态变化时发送带时间戳和HMAC-SHA256的脱敏状态；真实接收器尚未配置，因此不作为值班通知送达证据 | +| 阿里云单机健康监测 | 目标服务器已安装五分钟systemd健康检查，结构化状态覆盖HTTPS、OIDC签发地址、11个常驻容器、本机/异地备份状态与摘要、备份执行结果、三个运行timer、短期IP证书剩余时间和磁盘阈值。异地适配安装后的实际正常路径22项全部healthy；把本机备份最大年龄临时设为0秒后仅`backup_freshness`进入critical并退出1，测试目录已清理且线上容器未停止。可选Webhook仅在状态变化时发送带时间戳和HMAC-SHA256的脱敏状态；真实接收器尚未配置，因此不作为值班通知送达证据 | +| 阿里云OSS异地备份适配 | 目标服务器已按官方SHA-256安装ossutil 2.3.0，新增每日OSS同步任务、加密归档及摘要上传、远端回读复算、原子状态和健康强制门禁。隔离兼容目标验证上传/回读2个文件并得到`verified`；缺身份配置退出4并记录`failed/oss_config_missing`；设为必需但缺状态时健康检查退出1并记录`critical/status_missing`。当前实例无RAM角色、Bucket和OSS配置，正式任务保持`disabled/not_enabled`且健康状态为`not_required`，不能作为真实异地备份证据 | | 进程与Redis恢复 | API/Callbacks/Worker重启恢复；停机期间Outbox仅执行一次；Redis 14个热状态键清空后由签名设备请求重建，PostgreSQL权威计数不变 | | 供应商故障恢复 | V23持久化按租户/供应商的可用状态；连续临时失败暂停新启动但停止任务继续尝试；演练确认暂停后新外呼0、停止尝试2次、历史接口200、CRITICAL告警出现，恢复后全部收敛；V23单版本回退/重升通过 | | 安全专项 | Go 1.26.6容器`govulncheck`可达漏洞0、npm高危审计0、最终镜像HIGH/CRITICAL已修复漏洞0、Gitleaks无泄漏；HTTP认证/授权/注入式输入/1MiB正文/回调签名/Nonce重放/安全头/日志过滤烟测通过 | diff --git a/docs/offsite-backup.md b/docs/offsite-backup.md new file mode 100644 index 0000000..3acff4c --- /dev/null +++ b/docs/offsite-backup.md @@ -0,0 +1,110 @@ +# 阿里云OSS异地备份适配与门禁 + +## 1. 目标 + +本能力把`platform-backup.service`生成并校验过的加密归档复制到阿里云OSS，上传后重新下载到受控临时目录并复算SHA-256，只有源归档、远端回读文件和远端摘要文件三者一致才记录`verified`。 + +本地备份已经加密，OSS仍应使用私有Bucket、最小权限RAM角色、服务端加密、版本控制、生命周期和必要的跨区域复制。仓库不保存AccessKey、STS令牌或ossutil配置正文。 + +## 2. 当前环境状态 + +目标ECS已安装并校验`ossutil 2.3.0`，客户端压缩包使用阿里云文档公布的Linux x86_64 SHA-256验证。当前实例没有绑定RAM角色，也没有配置OSS Bucket，因此： + +- `OFFSITE_BACKUP_ENABLED=false`； +- 定时器已安装并保持活动，但任务只记录`disabled/not_enabled`； +- 健康监测记录`offsite_backup=healthy/not_required`； +- 不把当前状态声明为异地备份完成。 + +## 3. 上传与验证流程 + +```text +选择最新本机加密归档 + → 验证本机外层SHA-256 + → 上传归档到OSS私有路径 + → 上传SHA-256旁路文件 + → 从OSS重新下载归档与摘要 + → 重新计算下载文件SHA-256 + → 三方摘要一致 + → 原子写入platform-offsite-backup-v1状态 +``` + +上传和下载均固定为单任务、单文件并发，避免在当前2核小规格服务器上争抢资源。大于ossutil阈值的文件使用断点续传，检查点保存在`/var/lib/data-terminal-cloud-offsite/checkpoints`。 + +状态文件位于`/var/lib/data-terminal-cloud-offsite/status.json`，只包含源文件名、OSS对象引用、SHA-256、大小、验证时间和稳定原因，不包含凭据。 + +## 4. 安装 + +```bash +cd /opt/data-terminal-cloud/current +chmod 0755 deploy/server/install-ossutil.sh \ + deploy/server/install-offsite-backup.sh \ + deploy/server/sync-backup-offsite.sh +deploy/server/install-offsite-backup.sh +systemctl list-timers platform-offsite-backup.timer --no-pager +``` + +安装器固定下载`ossutil 2.3.0`并校验官方SHA-256；已有同版本时不会重复下载。若服务器没有`unzip`，使用Python标准库解压。 + +## 5. OSS准备 + +业务方或云管理员需要在阿里云控制台完成： + +1. 创建私有OSS Bucket，确定地域、版本控制、服务端加密和生命周期策略； +2. 优先给ECS绑定最小权限实例RAM角色，避免长期AccessKey落盘； +3. 授权该角色仅能向指定Bucket前缀上传、读取和查询备份对象； +4. 如需地域级灾备，为目标Bucket配置跨区域复制； +5. 使用ossutil交互配置或RAM角色模式生成`/etc/data-terminal-cloud/ossutilconfig`，权限必须为`0600 root:root`。 + +当前脚本需要上传后回读，因此至少要具备指定前缀的写入和读取权限。Bucket创建、RAM授权和跨区域复制属于云资源变更，不能通过服务器root账号替代完成。 + +## 6. 启用 + +编辑`/etc/data-terminal-cloud/offsite-backup.env`： + +```bash +OFFSITE_BACKUP_ENABLED=true +OFFSITE_BACKUP_REQUIRED=true +OFFSITE_OSS_BUCKET=your-private-backup-bucket +OFFSITE_OSS_PREFIX=data-terminal-cloud/121.41.46.238 +OFFSITE_OSS_CONFIG_FILE=/etc/data-terminal-cloud/ossutilconfig +``` + +然后执行： + +```bash +systemctl start platform-offsite-backup.service +systemctl show platform-offsite-backup.service -p Result -p ExecMainStatus +python3 -m json.tool /var/lib/data-terminal-cloud-offsite/status.json +systemctl start platform-health-monitor.service +``` + +期望异地状态为`verified`，健康状态中的`offsite_backup`为`healthy`。启用或设为必需后，缺状态、上传失败、回读失败、摘要不一致、源归档不匹配或超过36小时都会令健康检查进入critical。 + +## 7. 定时与保留 + +本机备份默认在02:40执行并带最多45分钟随机延迟；OSS同步默认在04:10执行并带最多30分钟随机延迟。OSS侧历史保留应由Bucket生命周期和版本控制管理，脚本不会主动删除远端对象，避免错误配置扩大为批量删除。 + +## 8. 失败语义 + +| 原因 | 含义 | +|---|---| +| `ossutil_missing` | 官方客户端未安装 | +| `oss_config_missing` | 身份配置不可读 | +| `local_backup_missing` | 没有本机加密备份 | +| `local_checksum_failed` | 本机归档已损坏 | +| `archive_upload_failed` | OSS归档上传失败 | +| `checksum_upload_failed` | OSS摘要上传失败 | +| `archive_download_verification_failed` | 远端归档无法回读 | +| `checksum_download_verification_failed` | 远端摘要无法回读 | +| `remote_checksum_mismatch` | 回读内容与源归档不一致 | + +失败状态会原子写入状态文件并使systemd任务失败；本机已验证备份不会被删除。 + +## 9. 已执行验证 + +- 禁用状态：任务成功记录`disabled/not_enabled`，不访问OSS； +- 隔离兼容目标：上传归档和摘要、回读并复算，状态为`verified`且远端有2个文件； +- 缺身份配置：退出4并记录`failed/oss_config_missing`； +- 健康门禁：设为必需但缺远端状态时退出1并记录`offsite_backup=critical/status_missing`。 + +隔离兼容目标只验证脚本控制流与摘要闭环，不是阿里云真实Bucket证据。真实门禁必须在配置RAM角色和私有Bucket后重新执行。