From 0bf375155efa3d3c1fcf1a4b5cc4d9cec79175cb Mon Sep 17 00:00:00 2001 From: Codex Local Date: Wed, 26 Aug 2026 20:41:26 +0800 Subject: [PATCH] feat: monitor cloud server health --- README.md | 2 +- deploy/server/README.md | 14 +- deploy/server/health-monitor.sh | 232 ++++++++++++++++++ deploy/server/install-monitoring.sh | 29 +++ deploy/server/platform-health-monitor.service | 18 ++ deploy/server/platform-health-monitor.timer | 12 + docs/development-status.md | 1 + docs/server-health-monitoring.md | 123 ++++++++++ 8 files changed, 429 insertions(+), 2 deletions(-) create mode 100755 deploy/server/health-monitor.sh create mode 100755 deploy/server/install-monitoring.sh create mode 100644 deploy/server/platform-health-monitor.service create mode 100644 deploy/server/platform-health-monitor.timer create mode 100644 docs/server-health-monitoring.md diff --git a/README.md b/README.md index 5deb201..f6ba717 100644 --- a/README.md +++ b/README.md @@ -332,7 +332,7 @@ scripts/ 本地启动和质量检查脚本 test/ 契约、集成和端到端测试 ``` -公共接口约束与监控指标见[可观测性与接口治理设计](docs/observability-and-api-governance.md)，设备签名和模拟联调见[设备接入与启动配置联调说明](docs/device-integration.md)，设备命令见[设备命令可靠下发说明](docs/device-commands.md)，固件流程见[固件制品、签名、灰度与回滚开发说明](docs/firmware-rollout.md)，A/B升级与启动失败回退见[设备A/B分区固件升级与启动确认协议](docs/device-firmware-ab-protocol.md)，设备/云端时间线对账见[固件真机联调一致性证据采集与校验说明](docs/firmware-conformance-evidence.md)，双轨录音见[双轨录音上传、校验与访问开发说明](docs/media-upload-and-access.md)，网页状态通知见[Web状态事件流开发说明](docs/web-event-stream.md)，具体分享见[具体资源分享开发说明](docs/resource-sharing.md)，运营配置见[内部运营身份与运行配置版本开发说明](docs/operations-config.md)，用量费用与额度见[用量、费用与额度开发说明](docs/usage-cost-quota.md)，告警闭环见[告警监测与运营处置开发说明](docs/alert-operations.md)，真实网关归档见[告警接收器一致性证据](docs/alert-receiver-conformance-evidence.md)，云KMS轮换见[KMS/HSM轮换一致性证据](docs/kms-hsm-rotation-evidence.md)，生产恢复见[生产等价恢复演练一致性证据](docs/production-recovery-evidence.md)，单机云端备份见[自动备份与隔离恢复手册](docs/server-backup-and-restore.md)，生产基础设施见[生产基础设施一致性证据](docs/production-infrastructure-evidence.md)，联调工作台及发布候选验收快照见[集成诊断工作台开发说明](docs/integration-diagnostics.md)，试点每日证据见[试点运行日报开发说明](docs/pilot-daily-operations.md)，试点周度复盘口径、外部投诉缺口、接口和轻量烟测见[试点运行周报开发说明](docs/pilot-weekly-review.md)，敏感数据提权见[运营敏感数据临时访问开发说明](docs/ops-sensitive-access.md)，外部上线阶段见[真机、真实供应商、试点与生产发布外部验收清单](docs/external-acceptance-checklist.md)，生产门禁和压测告警见[压测、告警与生产安全门禁](docs/load-alerting-and-production-security.md)，专项实测证据见[性能、恢复、安全与兼容性专项验证报告](docs/performance-recovery-security-compatibility.md)，隐私数据访问见[隐私数据请求开发说明](docs/privacy-requests.md)，私有导出见[私有数据导出开发说明](docs/data-export.md)。 +公共接口约束与监控指标见[可观测性与接口治理设计](docs/observability-and-api-governance.md)，设备签名和模拟联调见[设备接入与启动配置联调说明](docs/device-integration.md)，设备命令见[设备命令可靠下发说明](docs/device-commands.md)，固件流程见[固件制品、签名、灰度与回滚开发说明](docs/firmware-rollout.md)，A/B升级与启动失败回退见[设备A/B分区固件升级与启动确认协议](docs/device-firmware-ab-protocol.md)，设备/云端时间线对账见[固件真机联调一致性证据采集与校验说明](docs/firmware-conformance-evidence.md)，双轨录音见[双轨录音上传、校验与访问开发说明](docs/media-upload-and-access.md)，网页状态通知见[Web状态事件流开发说明](docs/web-event-stream.md)，具体分享见[具体资源分享开发说明](docs/resource-sharing.md)，运营配置见[内部运营身份与运行配置版本开发说明](docs/operations-config.md)，用量费用与额度见[用量、费用与额度开发说明](docs/usage-cost-quota.md)，告警闭环见[告警监测与运营处置开发说明](docs/alert-operations.md)，真实网关归档见[告警接收器一致性证据](docs/alert-receiver-conformance-evidence.md)，云KMS轮换见[KMS/HSM轮换一致性证据](docs/kms-hsm-rotation-evidence.md)，生产恢复见[生产等价恢复演练一致性证据](docs/production-recovery-evidence.md)，单机云端备份见[自动备份与隔离恢复手册](docs/server-backup-and-restore.md)，服务器健康检查见[单机云端健康监测与告警手册](docs/server-health-monitoring.md)，生产基础设施见[生产基础设施一致性证据](docs/production-infrastructure-evidence.md)，联调工作台及发布候选验收快照见[集成诊断工作台开发说明](docs/integration-diagnostics.md)，试点每日证据见[试点运行日报开发说明](docs/pilot-daily-operations.md)，试点周度复盘口径、外部投诉缺口、接口和轻量烟测见[试点运行周报开发说明](docs/pilot-weekly-review.md)，敏感数据提权见[运营敏感数据临时访问开发说明](docs/ops-sensitive-access.md)，外部上线阶段见[真机、真实供应商、试点与生产发布外部验收清单](docs/external-acceptance-checklist.md)，生产门禁和压测告警见[压测、告警与生产安全门禁](docs/load-alerting-and-production-security.md)，专项实测证据见[性能、恢复、安全与兼容性专项验证报告](docs/performance-recovery-security-compatibility.md)，隐私数据访问见[隐私数据请求开发说明](docs/privacy-requests.md)，私有导出见[私有数据导出说明](docs/data-export.md)。 从框架搭建到生产落地的阶段任务、里程碑、人员、测试门禁、依赖和风险见[详细开发计划](docs/development-plan.md)，当前实际完成证据见[开发状态与门禁记录](docs/development-status.md)。 diff --git a/deploy/server/README.md b/deploy/server/README.md index 667d097..0c4687b 100644 --- a/deploy/server/README.md +++ b/deploy/server/README.md @@ -1,6 +1,6 @@ # 阿里云单机演示部署 -该配置用于单机公网演示，不是生产验收配置。只向公网发布HTTP 80和HTTPS 443；PostgreSQL、Redis、MinIO、Keycloak内部端口以及API/Worker端口均不映射到宿主机。当前阿里云演示机使用Let's Encrypt短期IP证书，`certbot-ip-renew.timer`负责自动续期。单机备份覆盖PostgreSQL、Redis、MinIO对象和当前Keycloak数据目录，并执行数据库级隔离恢复校验；跨故障域备份、真实RTC、模型、内容安全、云KMS和生产监控告警仍按阶段8验收清单执行。 +该配置用于单机公网演示，不是生产验收配置。只向公网发布HTTP 80和HTTPS 443；PostgreSQL、Redis、MinIO、Keycloak内部端口以及API/Worker端口均不映射到宿主机。当前阿里云演示机使用Let's Encrypt短期IP证书，`certbot-ip-renew.timer`负责自动续期。单机备份覆盖PostgreSQL、Redis、MinIO对象和当前Keycloak数据目录，并执行数据库级隔离恢复校验；五分钟健康检查覆盖公网入口、OIDC、容器、备份、证书、定时任务和磁盘。跨故障域备份、真实告警接收器、真实RTC、模型、内容安全和云KMS仍按阶段8验收清单执行。 服务器至少需要Docker Engine 29与Compose 2.40。首次部署先生成权限为600的服务器专用配置： @@ -43,3 +43,15 @@ sudo deploy/server/verify-backup.sh /var/backups/data-terminal-cloud/platform-YY ``` 加密口令位于`/etc/data-terminal-cloud/backup-passphrase`且仅root可读。必须把该口令和加密备份复制到不同故障域；口令丢失将无法恢复备份。 + +## 健康监测与告警 + +```bash +chmod 0755 deploy/server/health-monitor.sh deploy/server/install-monitoring.sh +sudo deploy/server/install-monitoring.sh +sudo systemctl start platform-health-monitor.service +sudo systemctl show platform-health-monitor.service -p Result -p ExecMainStatus +python3 -m json.tool /var/lib/data-terminal-cloud-monitor/status.json +``` + +监测任务每五分钟执行，正常状态为`healthy`。默认只记录结构化状态和systemd日志；在`/etc/data-terminal-cloud/monitor.env`配置`ALERT_WEBHOOK_URL`后，状态变化会发送带时间戳和HMAC-SHA256签名的JSON通知。完整检查项、签名规则、阈值和失败分支见[《单机云端健康监测与告警手册》](../../docs/server-health-monitoring.md)。 diff --git a/deploy/server/health-monitor.sh b/deploy/server/health-monitor.sh new file mode 100755 index 0000000..56c7a79 --- /dev/null +++ b/deploy/server/health-monitor.sh @@ -0,0 +1,232 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current} +backup_root=${BACKUP_ROOT:-/var/backups/data-terminal-cloud} +state_dir=${MONITOR_STATE_DIR:-/var/lib/data-terminal-cloud-monitor} +monitor_config=${MONITOR_CONFIG_FILE:-/etc/data-terminal-cloud/monitor.env} +backup_max_age_seconds=${BACKUP_MAX_AGE_SECONDS:-129600} +certificate_min_seconds=${CERTIFICATE_MIN_SECONDS:-129600} +disk_warning_percent=${DISK_WARNING_PERCENT:-80} +disk_critical_percent=${DISK_CRITICAL_PERCENT:-90} + +if [[ -f "$monitor_config" ]]; then + # The installer creates this file as root:root 0600. + # shellcheck disable=SC1090 + source "$monitor_config" +fi +alert_webhook_url=${ALERT_WEBHOOK_URL:-} +alert_webhook_secret_file=${ALERT_WEBHOOK_SECRET_FILE:-/etc/data-terminal-cloud/monitor-webhook-secret} + +for numeric in "$backup_max_age_seconds" "$certificate_min_seconds" "$disk_warning_percent" "$disk_critical_percent"; do + [[ "$numeric" =~ ^[0-9]+$ ]] || { echo "Monitor thresholds must be non-negative integers" >&2; exit 2; } +done +if (( disk_warning_percent >= disk_critical_percent || disk_critical_percent > 100 )); then + echo "Disk thresholds are invalid" >&2 + exit 2 +fi + +compose=(docker compose -f "$project_root/compose.yaml" -f "$project_root/deploy/server/compose.override.yaml" --env-file "$project_root/deploy/server/.env") +mkdir -p "$state_dir" +chmod 0750 "$state_dir" +umask 077 +checks_file=$(mktemp "$state_dir/.checks-XXXXXX") +status_tmp=$(mktemp "$state_dir/.status-XXXXXX") +oidc_tmp=$(mktemp "$state_dir/.oidc-XXXXXX") +curl_config=$(mktemp "$state_dir/.curl-XXXXXX") +status_file="$state_dir/status.json" +critical_count=0 +warning_count=0 + +cleanup() { + rm -f -- "$checks_file" "$status_tmp" "$oidc_tmp" "$curl_config" +} +trap cleanup EXIT INT TERM + +record_check() { + local name=$1 + local status=$2 + local value=${3:-} + value=${value//$'\t'/ } + value=${value//$'\r'/ } + value=${value//$'\n'/ } + printf '%s\t%s\t%s\n' "$name" "$status" "$value" >>"$checks_file" + case "$status" in + critical) critical_count=$((critical_count + 1)) ;; + warning) warning_count=$((warning_count + 1)) ;; + esac +} + +public_origin=$(sed -n 's/^PUBLIC_ORIGIN=//p' "$project_root/deploy/server/.env") +if [[ "$public_origin" == https://* ]]; then + record_check public_origin healthy "$public_origin" +else + record_check public_origin critical invalid_https_origin +fi + +if curl --fail --silent --show-error --max-time 10 "$public_origin/api/v1/version" >/dev/null; then + record_check public_https healthy http_200 +else + record_check public_https critical request_failed +fi + +expected_issuer="$public_origin/realms/voice-local" +if curl --fail --silent --show-error --max-time 10 \ + "$expected_issuer/.well-known/openid-configuration" -o "$oidc_tmp"; then + actual_issuer=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1], encoding="utf-8")).get("issuer", ""))' "$oidc_tmp" 2>/dev/null || true) + if [[ "$actual_issuer" == "$expected_issuer" ]]; then + record_check oidc_issuer healthy issuer_matches + else + record_check oidc_issuer critical issuer_mismatch + fi +else + record_check oidc_issuer critical request_failed +fi + +required_services=(postgres redis minio keycloak api callbacks worker web gateway mock-provider mock-alert-receiver) +for service in "${required_services[@]}"; do + container=$("${compose[@]}" ps -q "$service") + if [[ -z "$container" ]]; then + record_check "container_$service" critical missing + continue + fi + state=$(docker inspect -f '{{.State.Status}}' "$container") + health=$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$container") + if [[ "$state" != running ]]; then + record_check "container_$service" critical "$state" + elif [[ "$health" != none && "$health" != healthy ]]; then + record_check "container_$service" critical "$health" + else + record_check "container_$service" healthy "$state/$health" + fi +done + +latest_backup=$(find "$backup_root" -maxdepth 1 -type f -name 'platform-*.tar.gz.enc' -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n1 | cut -d' ' -f2- || true) +if [[ -z "$latest_backup" || ! -f "$latest_backup" ]]; then + record_check backup_freshness critical missing +else + now_epoch=$(date +%s) + backup_epoch=$(stat -c %Y "$latest_backup") + backup_age=$((now_epoch - backup_epoch)) + if (( backup_age > backup_max_age_seconds )); then + record_check backup_freshness critical "age_seconds=$backup_age" + elif [[ ! -f "$latest_backup.sha256" ]] || ! (cd "$backup_root" && sha256sum -c "$(basename "$latest_backup").sha256" >/dev/null 2>&1); then + record_check backup_freshness critical checksum_failed + else + record_check backup_freshness healthy "age_seconds=$backup_age" + fi +fi + +backup_result=$(systemctl show platform-backup.service -p Result --value 2>/dev/null || true) +backup_exit=$(systemctl show platform-backup.service -p ExecMainStatus --value 2>/dev/null || true) +if [[ "$backup_result" == success && "$backup_exit" == 0 ]]; then + record_check backup_service healthy success +else + record_check backup_service critical "result=${backup_result:-unknown}/exit=${backup_exit:-unknown}" +fi + +for timer in platform-backup.timer certbot-ip-renew.timer; do + timer_enabled=$(systemctl is-enabled "$timer" 2>/dev/null || true) + timer_active=$(systemctl is-active "$timer" 2>/dev/null || true) + if [[ "$timer_enabled" == enabled && "$timer_active" == active ]]; then + record_check "$timer" healthy enabled/active + else + record_check "$timer" critical "${timer_enabled:-unknown}/${timer_active:-unknown}" + fi +done + +public_hostname=$(python3 -c 'import sys,urllib.parse; print(urllib.parse.urlsplit(sys.argv[1]).hostname or "")' "$public_origin") +certificate_file=${TLS_CERTIFICATE_FILE:-/etc/letsencrypt/live/$public_hostname/fullchain.pem} +if [[ ! -r "$certificate_file" ]]; then + record_check certificate critical missing +else + certificate_end=$(openssl x509 -in "$certificate_file" -noout -enddate | cut -d= -f2-) + certificate_end_epoch=$(date -d "$certificate_end" +%s) + certificate_seconds=$((certificate_end_epoch - $(date +%s))) + if (( certificate_seconds < certificate_min_seconds )); then + record_check certificate critical "remaining_seconds=$certificate_seconds" + else + record_check certificate healthy "remaining_seconds=$certificate_seconds" + fi +fi + +disk_used=$(df -P "$backup_root" | awk 'NR==2 {gsub(/%/, "", $5); print $5}') +if [[ ! "$disk_used" =~ ^[0-9]+$ ]]; then + record_check disk_usage critical unreadable +elif (( disk_used >= disk_critical_percent )); then + record_check disk_usage critical "used_percent=$disk_used" +elif (( disk_used >= disk_warning_percent )); then + record_check disk_usage warning "used_percent=$disk_used" +else + record_check disk_usage healthy "used_percent=$disk_used" +fi + +if (( critical_count > 0 )); then + overall=critical +elif (( warning_count > 0 )); then + overall=warning +else + overall=healthy +fi +observed_at=$(date -u +%Y-%m-%dT%H:%M:%SZ) +previous_status=$(python3 -c 'import json,sys +try: print(json.load(open(sys.argv[1], encoding="utf-8")).get("status", "unknown")) +except Exception: print("unknown")' "$status_file" 2>/dev/null || true) + +python3 - "$checks_file" "$status_tmp" "$overall" "$observed_at" <<'PY' +import json +import sys + +checks = [] +with open(sys.argv[1], encoding="utf-8") as source: + for line in source: + name, status, value = line.rstrip("\n").split("\t", 2) + checks.append({"name": name, "status": status, "value": value}) +payload = { + "schema": "platform-server-health-v1", + "status": sys.argv[3], + "observed_at": sys.argv[4], + "checks": checks, +} +with open(sys.argv[2], "w", encoding="utf-8") as target: + json.dump(payload, target, ensure_ascii=False, separators=(",", ":")) + target.write("\n") +PY +chmod 0644 "$status_tmp" +mv -f "$status_tmp" "$status_file" + +if [[ -n "$alert_webhook_url" && "$previous_status" != "$overall" ]]; then + if [[ ! -r "$alert_webhook_secret_file" ]]; then + record_check alert_delivery critical secret_missing + echo "Alert webhook is configured but its HMAC secret is unreadable" >&2 + exit 5 + fi + timestamp=$(date +%s) + signature=$(python3 - "$alert_webhook_secret_file" "$status_file" "$timestamp" <<'PY' +import hashlib +import hmac +import sys + +secret = open(sys.argv[1], "rb").read().rstrip(b"\r\n") +body = open(sys.argv[2], "rb").read() +print(hmac.new(secret, sys.argv[3].encode() + b"." + body, hashlib.sha256).hexdigest()) +PY +) + python3 - "$alert_webhook_url" "$curl_config" <<'PY' +import sys + +url = sys.argv[1].replace("\\", "\\\\").replace('"', '\\"') +open(sys.argv[2], "w", encoding="utf-8").write(f'url = "{url}"\n') +PY + curl --fail --silent --show-error --max-time 10 --config "$curl_config" \ + -H 'Content-Type: application/json' \ + -H "X-Platform-Timestamp: $timestamp" \ + -H "X-Platform-Signature: sha256=$signature" \ + --data-binary "@$status_file" >/dev/null +fi + +printf 'platform_health status=%s critical=%s warning=%s status_file=%s\n' \ + "$overall" "$critical_count" "$warning_count" "$status_file" +if (( critical_count > 0 )); then + exit 1 +fi diff --git a/deploy/server/install-monitoring.sh b/deploy/server/install-monitoring.sh new file mode 100755 index 0000000..1fbff24 --- /dev/null +++ b/deploy/server/install-monitoring.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +if [[ ${EUID:-$(id -u)} -ne 0 ]]; then + echo "Run this installer as root" >&2 + exit 2 +fi + +project_root=${PLATFORM_ROOT:-/opt/data-terminal-cloud/current} +install -d -m 0700 /etc/data-terminal-cloud +install -d -m 0750 /var/lib/data-terminal-cloud-monitor +if [[ ! -s /etc/data-terminal-cloud/monitor-webhook-secret ]]; then + umask 077 + openssl rand -base64 48 >/etc/data-terminal-cloud/monitor-webhook-secret +fi +chmod 0600 /etc/data-terminal-cloud/monitor-webhook-secret +if [[ ! -e /etc/data-terminal-cloud/monitor.env ]]; then + cat >/etc/data-terminal-cloud/monitor.env <<'EOF' +ALERT_WEBHOOK_URL= +ALERT_WEBHOOK_SECRET_FILE=/etc/data-terminal-cloud/monitor-webhook-secret +EOF +fi +chmod 0600 /etc/data-terminal-cloud/monitor.env +install -m 0644 "$project_root/deploy/server/platform-health-monitor.service" /etc/systemd/system/platform-health-monitor.service +install -m 0644 "$project_root/deploy/server/platform-health-monitor.timer" /etc/systemd/system/platform-health-monitor.timer +systemctl daemon-reload +systemctl enable --now platform-health-monitor.timer +systemctl is-active --quiet platform-health-monitor.timer +echo "platform-health-monitor.timer enabled" diff --git a/deploy/server/platform-health-monitor.service b/deploy/server/platform-health-monitor.service new file mode 100644 index 0000000..97034d1 --- /dev/null +++ b/deploy/server/platform-health-monitor.service @@ -0,0 +1,18 @@ +[Unit] +Description=Check data terminal cloud server health +Requires=docker.service +After=docker.service network-online.target + +[Service] +Type=oneshot +User=root +Group=root +UMask=0077 +Environment=PLATFORM_ROOT=/opt/data-terminal-cloud/current +Environment=BACKUP_ROOT=/var/backups/data-terminal-cloud +Environment=MONITOR_STATE_DIR=/var/lib/data-terminal-cloud-monitor +ExecStart=/opt/data-terminal-cloud/current/deploy/server/health-monitor.sh +Nice=10 +NoNewPrivileges=true +PrivateTmp=true +TimeoutStartSec=2m diff --git a/deploy/server/platform-health-monitor.timer b/deploy/server/platform-health-monitor.timer new file mode 100644 index 0000000..2a30c47 --- /dev/null +++ b/deploy/server/platform-health-monitor.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Check data terminal cloud server health every five minutes + +[Timer] +OnBootSec=2m +OnUnitActiveSec=5m +RandomizedDelaySec=30s +Persistent=true +Unit=platform-health-monitor.service + +[Install] +WantedBy=timers.target diff --git a/docs/development-status.md b/docs/development-status.md index 24c9db0..1fcb97a 100644 --- a/docs/development-status.md +++ b/docs/development-status.md @@ -181,6 +181,7 @@ | 隐私删除并发 | 100个并发受理全部成功、HTTP失败率0%、提交p95 385.15ms；11.58秒内600个子任务、100个墓碑和100个主体冻结全部对账 | | PostgreSQL隔离恢复 | 实际`pg_dump`/`pg_restore`到随机隔离库，迁移`22|false`、66张表、190个外键和关键计数一致；本地约5.14秒，临时库/备份已清理 | | 阿里云单机加密备份与恢复 | 目标服务器已安装每日systemd备份任务，覆盖PostgreSQL、认证Redis RDB、MinIO全部桶和暂停后的Keycloak数据目录；归档使用AES-256-CBC/PBKDF2加密、逐文件SHA-256并保留14天。首次实际归档成功，数据库从V38备份恢复到随机隔离库后`38|false|2|3|73|215`六项证据一致，归档内Redis RDB格式有效、Keycloak 3个文件一致、临时恢复库为0；同机备份尚不作为跨故障域生产恢复证据 | +| 阿里云单机健康监测 | 目标服务器已安装五分钟systemd健康检查，结构化状态覆盖HTTPS、OIDC签发地址、11个常驻容器、备份新鲜度与摘要、备份执行结果、备份/证书timer、短期IP证书剩余时间和磁盘阈值。实际正常路径20项全部healthy；把备份最大年龄临时设为0秒后仅`backup_freshness`进入critical并退出1，测试目录已清理且线上容器未停止。可选Webhook仅在状态变化时发送带时间戳和HMAC-SHA256的脱敏状态；真实接收器尚未配置，因此不作为值班通知送达证据 | | 进程与Redis恢复 | API/Callbacks/Worker重启恢复；停机期间Outbox仅执行一次；Redis 14个热状态键清空后由签名设备请求重建，PostgreSQL权威计数不变 | | 供应商故障恢复 | V23持久化按租户/供应商的可用状态；连续临时失败暂停新启动但停止任务继续尝试；演练确认暂停后新外呼0、停止尝试2次、历史接口200、CRITICAL告警出现，恢复后全部收敛；V23单版本回退/重升通过 | | 安全专项 | Go 1.26.6容器`govulncheck`可达漏洞0、npm高危审计0、最终镜像HIGH/CRITICAL已修复漏洞0、Gitleaks无泄漏；HTTP认证/授权/注入式输入/1MiB正文/回调签名/Nonce重放/安全头/日志过滤烟测通过 | diff --git a/docs/server-health-monitoring.md b/docs/server-health-monitoring.md new file mode 100644 index 0000000..e03eb47 --- /dev/null +++ b/docs/server-health-monitoring.md @@ -0,0 +1,123 @@ +# 单机云端健康监测与告警手册 + +## 1. 目标与边界 + +`deploy/server/health-monitor.sh`为单机阿里云演示环境提供五分钟一次的轻量健康检查。它用于发现“公网入口不可用、身份签发地址漂移、容器退出、备份过期或损坏、证书接近到期、定时任务停止、磁盘空间不足”等直接影响可用性的问题。 + +该脚本不替代Prometheus、Grafana、云监控或真实值班系统。默认只写结构化状态文件和systemd日志；配置外部接收器后，状态发生变化时才发送一次HMAC签名Webhook，避免每五分钟重复通知。 + +## 2. 检查项 + +| 检查 | 正常条件 | 失败状态 | +|---|---|---| +| 公网地址 | `PUBLIC_ORIGIN`为HTTPS | `invalid_https_origin` | +| HTTPS入口 | `/api/v1/version`在10秒内返回成功 | `request_failed` | +| OIDC | Discovery中的`issuer`与公网地址完全一致 | `issuer_mismatch`或`request_failed` | +| 容器 | PostgreSQL、Redis、MinIO、Keycloak、API、Callbacks、Worker、Web、Gateway和两个模拟服务均为运行态；有健康检查的容器必须为healthy | `missing`、容器状态或健康状态 | +| 备份 | 最新加密归档不超过36小时且外层SHA-256有效 | `missing`、`age_seconds=...`或`checksum_failed` | +| 备份任务 | 最近一次`platform-backup.service`结果成功且退出码为0 | `result=.../exit=...` | +| 定时任务 | 备份和证书续期timer均为enabled/active | 当前启用/活动状态 | +| 证书 | 本地活动证书剩余时间不少于36小时 | `missing`或`remaining_seconds=...` | +| 磁盘 | 低于80%为健康，80%—89%为warning，90%及以上为critical | `used_percent=...` | + +状态文件写入`/var/lib/data-terminal-cloud-monitor/status.json`，格式为`platform-server-health-v1`。每个检查只有稳定名称、状态和脱敏值，不包含数据库密码、Webhook地址、证书正文、业务数据或令牌。 + +## 3. 状态模型 + +```text +全部检查健康 → healthy，退出0 +没有critical但存在warning → warning，退出0 +任一检查critical → critical，退出1 +``` + +systemd oneshot任务成功结束后显示`inactive (dead)`属于正常行为，应使用以下字段判断： + +```bash +systemctl show platform-health-monitor.service -p Result -p ExecMainStatus -p ActiveState +``` + +期望结果为`Result=success`、`ExecMainStatus=0`。状态为critical时服务会显示失败，timer仍会在下一个周期继续执行。 + +## 4. 安装与运行 + +```bash +cd /opt/data-terminal-cloud/current +chmod 0755 deploy/server/health-monitor.sh deploy/server/install-monitoring.sh +deploy/server/install-monitoring.sh +systemctl start platform-health-monitor.service +systemctl list-timers platform-health-monitor.timer --no-pager +python3 -m json.tool /var/lib/data-terminal-cloud-monitor/status.json +``` + +定时任务在开机两分钟后首次执行，之后每五分钟执行并附加最多30秒随机延迟。`Persistent=true`确保错过计划时间后补跑。 + +## 5. 可选Webhook通知 + +安装脚本创建： + +- `/etc/data-terminal-cloud/monitor.env`：`0600 root:root`； +- `/etc/data-terminal-cloud/monitor-webhook-secret`：独立随机HMAC密钥，`0600 root:root`。 + +编辑配置： + +```bash +ALERT_WEBHOOK_URL=https://receiver.example.com/platform-health +ALERT_WEBHOOK_SECRET_FILE=/etc/data-terminal-cloud/monitor-webhook-secret +``` + +只有总体状态发生变化时才发送JSON状态文件。请求包含： + +- `Content-Type: application/json`； +- `X-Platform-Timestamp`：Unix秒； +- `X-Platform-Signature: sha256=`； +- 签名正文：`HMAC-SHA256(secret, timestamp + "." + raw_body)`。 + +接收端应限制时间偏差、按时间戳和签名去重并使用恒定时间比较。Webhook地址和密钥不会进入状态文件或日志。正式配置前应先在接收端完成验签联通测试。 + +## 6. 参数 + +| 环境变量 | 默认值 | 说明 | +|---|---|---| +| `BACKUP_MAX_AGE_SECONDS` | `129600` | 最新备份最大年龄，36小时 | +| `CERTIFICATE_MIN_SECONDS` | `129600` | 证书最小剩余时间，36小时 | +| `DISK_WARNING_PERCENT` | `80` | 磁盘警告阈值 | +| `DISK_CRITICAL_PERCENT` | `90` | 磁盘严重阈值 | +| `MONITOR_STATE_DIR` | `/var/lib/data-terminal-cloud-monitor` | 原子状态文件目录 | +| `MONITOR_CONFIG_FILE` | `/etc/data-terminal-cloud/monitor.env` | 可选通知配置 | +| `TLS_CERTIFICATE_FILE` | 根据`PUBLIC_ORIGIN`主机名推导 | 当前活动证书文件 | + +阈值只能使用非负整数，磁盘警告阈值必须小于严重阈值。普通调整无需重建镜像，但应记录变更理由并重新执行一次手工检查。 + +## 7. 验收与排障 + +正常路径： + +```bash +/opt/data-terminal-cloud/current/deploy/server/health-monitor.sh +``` + +不改变线上状态的备份过期失败分支： + +```bash +test_dir=$(mktemp -d /var/tmp/platform-monitor-test-XXXXXX) +MONITOR_STATE_DIR="$test_dir" MONITOR_CONFIG_FILE=/dev/null \ + BACKUP_MAX_AGE_SECONDS=0 \ + /opt/data-terminal-cloud/current/deploy/server/health-monitor.sh +``` + +期望退出1，状态为critical，唯一人工注入的严重检查为`backup_freshness`。测试目录验证后应精确删除。 + +常用排障命令： + +```bash +journalctl -u platform-health-monitor.service -n 100 --no-pager +systemctl list-timers platform-health-monitor.timer platform-backup.timer certbot-ip-renew.timer --no-pager +docker compose -f compose.yaml -f deploy/server/compose.override.yaml --env-file deploy/server/.env ps -a +``` + +## 8. 下一步生产化 + +1. 配置真实告警网关、主值班组和升级组，并完成HMAC验签联通测试。 +2. 将状态接入云监控或Prometheus长期存储，建立可用率和告警送达趋势。 +3. 将加密备份复制到不同故障域，并对“异地复制最后成功时间”增加检查。 +4. 使用外部OIDC、托管数据库/Redis/对象存储后，把资源级高可用、PITR和容量指标纳入同一状态模型。