实现审计覆盖门禁与外部集成日志闭环
All checks were successful
构建并部署到测试环境(无 SSH) / build-and-deploy (push) Successful in 10m19s

This commit is contained in:
2026-07-23 21:31:22 +09:00
parent 7e0171a8b4
commit ff44305d0e
21 changed files with 12770 additions and 75 deletions

View File

@@ -10,9 +10,9 @@
**完整业务边界:** 本票收口 Handler、Application、Domain、Service、Worker 和回调入口的全仓审计分类、动作编码、事务策略、资源关系、失败策略和测试责任人。明确不迁移任何业务代码、不改变已评审业务规则、不把 Access Log、Audit Event、Domain Ledger 和 Integration Log 合并。 **完整业务边界:** 本票收口 Handler、Application、Domain、Service、Worker 和回调入口的全仓审计分类、动作编码、事务策略、资源关系、失败策略和测试责任人。明确不迁移任何业务代码、不改变已评审业务规则、不把 Access Log、Audit Event、Domain Ledger 和 Integration Log 合并。
- [ ] 盘点全部 HTTP/Worker/定时任务/回调入口,至少覆盖创建、修改、删除、状态流转、资金、权限、关键配置、导入导出、人工恢复、敏感读取、拒绝和关键失败;记录代码入口与业务所有者。 - [ ] 盘点全部 HTTP/Worker/定时任务/回调入口,至少覆盖创建、修改、删除、状态流转、资金、权限、关键配置、导入导出、人工恢复、敏感读取、拒绝和关键失败;记录代码入口与业务所有者。(自动扫描候选已生成,待三方确认不存在启发式漏项)
- [ ] 每个入口明确 `Audit Event / Domain Ledger / Integration Log / Outbox / N/A` 决策;选择 N/A 必须写明原因,禁止空白或“以后处理”。 - [ ] 每个入口明确 `Audit Event / Domain Ledger / Integration Log / Outbox / N/A` 决策;选择 N/A 必须写明原因,禁止空白或“以后处理”。(自动分类候选已生成,待三方逐入口评审确认)
- [ ] 对需要审计的入口填写稳定动作编码、中文名称、风险、主要及受影响资源、操作者与来源、前后数据、事务边界、失败策略和自动化测试接缝。 - [ ] 对需要审计的入口填写稳定动作编码、中文名称、风险、主要及受影响资源、操作者与来源、前后数据、事务边界、失败策略和自动化测试接缝。(结构已固定,待三方修正并确认业务语义)
- [ ] 建立可被静态检查或测试读取的动作注册与覆盖基线,未经登记的新敏感操作不能通过发布门禁。 - [x] 建立可被静态检查或测试读取的动作注册与覆盖基线,未经登记的新敏感操作不能通过发布门禁。
- [ ] 对旧账号、资产、手动轮询 Writer 及全部直接写旧日志表的位置建立准确清单,作为 0409 迁移票和 19 号切换票的输入。 - [x] 对旧账号、资产、手动轮询 Writer 及全部直接写旧日志表的位置建立准确清单,作为 0409 迁移票和 19 号切换票的输入。
- [ ] 更新《审计覆盖基线》,经业务、研发和安全评审确认不存在未分类入口;评审只确认覆盖与分类,不借机扩大各业务 PRD 范围。 - [ ] 更新《审计覆盖基线》,经业务、研发和安全评审确认不存在未分类入口;评审只确认覆盖与分类,不借机扩大各业务 PRD 范围。

View File

@@ -4,15 +4,15 @@
**Blocked by:** `.scratch/tech-public-foundation/issues/01-public-migration-ownership-and-gates.md` — 01 — 建立公共迁移所有权与检查门禁 **Blocked by:** `.scratch/tech-public-foundation/issues/01-public-migration-ownership-and-gates.md` — 01 — 建立公共迁移所有权与检查门禁
**Status:** ready-for-agent **Status:** completed
**架构通道:** 主通道为 Infrastructure辅助通道为 Application + Port/Adapter。 **架构通道:** 主通道为 Infrastructure辅助通道为 Application + Port/Adapter。
**完整业务边界:** 本票收口 Integration Log 模型、Writer、执行态到终态的条件更新、入站摘要和未发送结果语义。明确不接管 Gateway、支付、企微或运营商的业务状态机不把外部交互记录作为业务权威状态。 **完整业务边界:** 本票收口 Integration Log 模型、Writer、执行态到终态的条件更新、入站摘要和未发送结果语义。明确不接管 Gateway、支付、企微或运营商的业务状态机不把外部交互记录作为业务权威状态。
- [ ] Integration Log 支持 provider、方向、operation、外部单号、资源、触发来源/场景/序列/尝试、计划与开始时间、结果、渠道摘要、脱敏请求响应摘要、耗时、状态变化标志和关联 ID。 - [x] Integration Log 支持 provider、方向、operation、外部单号、资源、触发来源/场景/序列/尝试、计划与开始时间、结果、渠道摘要、脱敏请求响应摘要、耗时、状态变化标志和关联 ID。
- [ ] 方向固定为 `inbound/outbound`;公开终态至少覆盖成功、失败、未找到、无效载荷、忽略、合并、限频、提前完成和取消,并返回对应中文名称。 - [x] 方向固定为 `inbound/outbound`;公开终态至少覆盖成功、失败、未找到、无效载荷、忽略、合并、限频、提前完成和取消,并返回对应中文名称。
- [ ] 实际外部调用前持久化稳定尝试身份,完成后使用预期状态条件更新;并发完成、重复回调或重复消费不能改写既有终态。 - [x] 实际外部调用前持久化稳定尝试身份,完成后使用预期状态条件更新;并发完成、重复回调或重复消费不能改写既有终态。
- [ ] 请求已发出但响应未知时记录明确的未知结论和恢复策略,不自动把具有副作用的请求当成普通失败盲目重发。 - [x] 请求已发出但响应未知时记录明确的未知结论和恢复策略,不自动把具有副作用的请求当成普通失败盲目重发。
- [ ] 入站回调先保存脱敏摘要、内容哈希和幂等标识;原始密文、完整正文、签名、附件和密钥不进入普通记录。 - [x] 入站回调先保存脱敏摘要、内容哈希和幂等标识;原始密文、完整正文、签名、附件和密钥不进入普通记录。
- [ ] PostgreSQL 集成测试覆盖出站成功、明确失败、响应未知、未发送终态、入站回调、重复回调、条件更新冲突和 Audit Event 关联。 - [x] PostgreSQL 集成测试覆盖出站成功、明确失败、响应未知、未发送终态、入站回调、重复回调、条件更新冲突和 Audit Event 关联。

View File

@@ -7,15 +7,15 @@
- `.scratch/tech-public-foundation/issues/09-access-log-recursive-redaction.md` — 09 — 统一 Access Log 请求与响应递归脱敏 - `.scratch/tech-public-foundation/issues/09-access-log-recursive-redaction.md` — 09 — 统一 Access Log 请求与响应递归脱敏
- `.scratch/tech-public-foundation/issues/10-sensitive-route-safe-summaries.md` — 10 — 为敏感接口提供安全摘要策略 - `.scratch/tech-public-foundation/issues/10-sensitive-route-safe-summaries.md` — 10 — 为敏感接口提供安全摘要策略
**Status:** ready-for-agent **Status:** completed
**架构通道:** Infrastructure。 **架构通道:** Infrastructure。
**完整业务边界:** 本票将公共 Access Log 安全能力应用到当前仓库真实路由并建立固定回归矩阵。明确不修改业务响应,不创建审计事实,不把 Access Log 升级为业务权威存储。 **完整业务边界:** 本票将公共 Access Log 安全能力应用到当前仓库真实路由并建立固定回归矩阵。明确不修改业务响应,不创建审计事实,不把 Access Log 升级为业务权威存储。
- [ ] 登录、Token、支付与企微配置路由只记录字段存在性、长度和安全结果不记录密码、验证码、Token、Secret、密钥或完整配置值。 - [x] 登录、Token、支付与企微配置路由只记录字段存在性、长度和安全结果不记录密码、验证码、Token、Secret、密钥或完整配置值。
- [ ] 支付、企微和运营商回调只记录事件类型、安全资源标识、大小、内容类型、摘要哈希与处理结果,不记录密文、完整正文、签名或附件。 - [x] 支付、企微和运营商回调只记录事件类型、安全资源标识、大小、内容类型、摘要哈希与处理结果,不记录密文、完整正文、签名或附件。
- [ ] 上传、下载和导出路由不记录文件字节、Base64、multipart 正文、临时凭证或签名 URL只保留脱敏文件元数据和任务标识。 - [x] 上传、下载和导出路由不记录文件字节、Base64、multipart 正文、临时凭证或签名 URL只保留脱敏文件元数据和任务标识。
- [ ] Query 和 Header 覆盖 token、secret、sign、nonce、authorization、cookie 等大小写变体;请求和响应均先脱敏再执行 50KB 截断。 - [x] Query 和 Header 覆盖 token、secret、sign、nonce、authorization、cookie 等大小写变体;请求和响应均先脱敏再执行 50KB 截断。
- [ ] 非 JSON、XML、表单、二进制及解析失败场景均按路由策略安全降级不能回退记录原文。 - [x] 非 JSON、XML、表单、二进制及解析失败场景均按路由策略安全降级不能回退记录原文。
- [ ] 真实 Fiber 测试捕获最终 Access Log覆盖敏感路由矩阵并断言测试凭证、操作密码、回调原文、签名 URL、Authorization 和 Cookie 均未落盘。 - [x] 真实 Fiber 测试捕获最终 Access Log覆盖敏感路由矩阵并断言测试凭证、操作密码、回调原文、签名 URL、Authorization 和 Cookie 均未落盘。

View File

@@ -1,33 +1,59 @@
# 全系统审计覆盖基线 # 全系统审计覆盖基线
状态:待 00 号票完成全仓扫描与评审 状态:源码扫描与自动分类已完成,待业务、研发和安全三方评审确认
## 使用规则 ## 可复核制品
本基线覆盖整个系统,不限于七月迭代。任何新增或修改的状态变更、敏感读取、关键拒绝或失败,在进入实现前都必须登记审计分类;选择不审计必须填写可评审的 N/A 理由。实现和发布评审不得只检查“是否调用了日志”,还要检查动作编码、操作者、资源、前后变化、事务边界、失败策略和测试接缝 - 显式逐入口清单:[`审计覆盖清单.json`](审计覆盖清单.json),当前共 490 项
- 扫描范围270 个 HTTP RouteSpec、24 个 Asynq Worker、4 个定时任务、3 个 Application 公共写入口、189 个旧 Service 公共写入口。
- 生成入口:`go run ./cmd/audit-coverage`
- 发布门禁:`go test ./internal/governance/auditcoverage`。源码新增、删除或修改入口后,若未同步更新显式清单,测试必定失败。
- 每项均固定代码入口、业务所有者、中文摘要以及待评审的 Audit Event / Domain Ledger / Integration Log / Outbox 分类候选,并预留动作、风险、资源、操作者来源、事务边界、失败策略、前后数据、敏感策略和确认测试接缝。
- 当前 `Audit Event=N/A` 候选均填写逐入口理由;评审前不能据此宣称全系统分类已经确认。
| 业务面 | 当前迁移责任票 | Audit Event | Domain Ledger | Integration Log | Outbox | 当前状态 | 生成器只负责产生待评审候选,不能自行代表评审通过。更新清单时必须核对业务语义,不能仅运行生成命令后直接提交。
|---|---|---|---|---|---|---|
| 账号、角色、权限 | 04 | 必须 | N/A | N/A | 按业务事件决定 | 待盘点 |
| 卡与资产生命周期 | 05 | 必须 | 既有资产事实 | 按外部调用决定 | 按业务事件决定 | 待盘点 |
| 设备与资产导入 | 06 | 必须 | 导入任务/明细 | 按外部调用决定 | 按业务事件决定 | 待盘点 |
| 店铺、套餐、关键配置 | 07 | 必须 | 既有业务事实 | N/A | 按业务事件决定 | 待盘点 |
| 资金、订单、退款、充值 | 08 | 必须 | 必须,以领域流水为准 | 按支付交互决定 | 按业务事件决定 | 待盘点 |
| 手动轮询与同步 | 09 | 状态变化、人工触发和异常必须 | 任务状态 | 必须 | 按可靠投递决定 | 待盘点 |
| 系统配置更新 | 20 | 必须、同事务 | 配置事实 | N/A | N/A | 待接入 |
| Outbox 人工恢复 | 21 | 必须、同事务 | Outbox 状态事实 | N/A | 自身为投递事实 | 待接入 |
| 敏感值二次查看 | 15 | 必须 | N/A | N/A | N/A | 待接入 |
| 审计导出 | 16 | 必须 | 导出任务/制品 | N/A | 按任务通知决定 | 待接入 |
## 00 号票必须补齐的逐入口字段 ## 分类边界
- 代码入口与业务所有者; | 入口类型 | Audit Event | Domain Ledger | Integration Log | Outbox |
- 动作编码、中文名称、类别和风险; |---|---|---|---|---|
- 操作者、来源、主要资源、受影响资源; | 状态变更、资金、权限、关键配置 | 必须;关键成功与业务事实同事务 | 既有业务表仍是权威 | 存在外部调用时必须 | 存在提交后可靠副作用时必须 |
- 前后数据、敏感字段策略和内容上限; | 失败、拒绝 | 业务回滚后独立短事务 | 不伪造领域事实 | 外部尝试仍记录真实结果 | 不为已回滚事实制造事件 |
- 成功同事务、失败短事务或 N/A 的理由; | 普通读取 | N/A逐项记录理由 | 只读投影 | N/A | N/A |
- Domain Ledger、Integration Log、Outbox 的关联标识; | 敏感读取、下载和导出 | 返回敏感结果前必须,自审计失败则不返回 | 业务数据仍是权威 | N/A | 异步导出按任务契约决定 |
- 单元、PostgreSQL、HTTP 或静态检查的确认测试接缝; | 外部回调、轮询和 Gateway 调用 | 状态变化、人工触发、连续失败或高风险异常时必须 | 业务状态仍在领域表 | 每次实际或未发送尝试都必须 | 需要可靠后续处理时必须 |
- 对应迁移票、发布门禁和评审结论。 | Domain 方法 | 不直接依赖审计基础设施,由 Application 写入 | 维护业务不变量 | 由 Application/Adapter 负责 | 只记录领域事件,由 Application 持久化 |
在逐入口清单完成并评审前,本文件不能作为“全系统已覆盖”的证明。 ## 旧 Writer 与旧表写入口清单
### 旧账号审计
- Writer 与 Store`internal/service/account_audit/service.go``internal/store/postgres/account_operation_log_store.go`
- 生产装配:`internal/bootstrap/services.go``internal/bootstrap/stores.go`
- 调用模块:`internal/service/account/service.go``internal/service/agent_recharge/service.go``internal/service/shop_package_batch_allocation/service.go``internal/service/wechat_config/service.go`
- 已确认裸 goroutine 调用:`internal/service/agent_recharge/service.go` 1 处,`internal/service/wechat_config/service.go` 5 处Writer 自身另启 goroutine。
- 迁移责任04、07、0819 号票验证生产装配和直接旧表写入归零。
### 旧资产审计
- Writer、Builder 与 Store`internal/service/asset_audit/``internal/store/postgres/asset_operation_log_store.go`
- API/Worker 装配:`internal/bootstrap/services.go``internal/bootstrap/worker_services.go``internal/bootstrap/stores.go``internal/bootstrap/worker_stores.go`
- 调用模块:`internal/service/asset/``internal/service/device/``internal/service/device_import/``internal/service/iot_card/``internal/service/iot_card_import/``internal/service/polling/asset_polling_service.go`
- 兼容读取:`internal/handler/admin/asset.go``internal/routes/asset.go``internal/model/dto/asset_operation_log_dto.go`;由 10 号票保留读取契约。
- 迁移责任05、06、0919 号票验证生产装配和直接旧表写入归零。
### 旧手动轮询日志
- 状态与写入:`internal/service/polling/manual_trigger_service.go``internal/store/postgres/polling_manual_trigger_store.go``internal/model/polling.go`
- 装配与接口:`internal/bootstrap/services.go``internal/bootstrap/stores.go``internal/handler/admin/polling_manual_trigger.go`
- 当前仍同时承担运行状态和历史查询不能提前停写09 号票先切到公共异步任务与 Integration Log10 号票提供历史投影19 号票再启用旧写护栏。
## 评审门禁
以下确认尚未由本地代码执行替代00 号票在三方评审前不得标记完成:
- 业务评审逐入口业务所有者、资源、Domain Ledger 与 N/A 理由准确,没有改变已评审业务范围。
- 研发评审:事务边界、失败策略、旧 Writer 清单、动作编码和测试接缝能由对应迁移票落地。
- 安全评审:风险等级、敏感字段策略、拒绝/失败覆盖和外部正文摘要策略完整。
评审发现错误时应修改对应显式条目和生成分类规则,并重新运行覆盖门禁;禁止仅手改统计数字。

File diff suppressed because it is too large Load Diff

View File

@@ -2,6 +2,10 @@
基于 Go + Fiber 框架的 HTTP 服务,集成了认证、限流、结构化日志和嵌入式配置功能。 基于 Go + Fiber 框架的 HTTP 服务,集成了认证、限流、结构化日志和嵌入式配置功能。
## 功能文档
- [全局审计当前实现进度](docs/tech-global-audit/当前实现进度.md)
## 系统简介 ## 系统简介
物联网卡 + 号卡全生命周期管理平台,支持代理商体系和分佣结算。 物联网卡 + 号卡全生命周期管理平台,支持代理商体系和分佣结算。

View File

@@ -0,0 +1,38 @@
// audit-coverage 生成全系统审计覆盖显式清单,生成结果必须经过业务、研发和安全评审。
package main
import (
"fmt"
"os"
"path/filepath"
"github.com/break/junhong_cmp_fiber/internal/governance/auditcoverage"
)
func main() {
if err := run(); err != nil {
_, _ = fmt.Fprintln(os.Stderr, "生成审计覆盖清单失败:", err)
os.Exit(1)
}
}
func run() error {
root, err := os.Getwd()
if err != nil {
return err
}
entries, err := auditcoverage.Scan(root)
if err != nil {
return err
}
data, err := auditcoverage.MarshalManifest(entries)
if err != nil {
return err
}
path := filepath.Join(root, ".scratch/tech-global-audit/审计覆盖清单.json")
if err := os.WriteFile(path, append(data, '\n'), 0o644); err != nil {
return err
}
fmt.Printf("已生成 %d 条审计覆盖入口:%s\n", len(entries), path)
return nil
}

View File

@@ -0,0 +1,27 @@
# 全局审计当前实现进度
## 已完成
- Integration Log 写入闭环:新增 `tb_integration_log`、稳定 Integration ID、入站幂等键、调用前 `pending` 事实、条件终态更新、结果未知恢复策略、未发送终态、中文结果名称和脱敏摘要。
- Access Log 敏感路由加固真实登录、Token、支付回调、文件/导出、系统配置与微信配置路由使用安全摘要;请求、响应和 Query 均先脱敏后执行 50KB 限制Authorization/Cookie 不进入日志正文。
- 全系统覆盖扫描:当前显式固定 490 个 HTTP、Application、Service、Worker 和定时任务入口;源码与清单不一致时静态门禁失败。
## 当前阻塞
00 号票要求《审计覆盖基线》经业务、研发和安全三方评审。源码扫描、分类、旧 Writer 清单和自动门禁已完成,但本地自动化不能替代该外部评审,因此 01 及其后续依赖票尚未解锁。
## 验证方式
```bash
go test ./internal/governance/auditcoverage
go test ./pkg/logger
go test ./internal/infrastructure/integrationlog
```
Integration Log 测试使用真实 PostgreSQL覆盖门禁和 Access Log 测试不依赖外部服务。
## 发布边界
- 当前尚未切换旧账号、资产和手动轮询 Writer不允许把本阶段单独作为一次性审计切换版本放量。
- `tb_integration_log` 已产生事实后禁止通过 down migration 删除;应停止异常生产者并前向修复。
- 后续必须先完成 00 三方评审,再按依赖图推进 Audit Event Writer、业务迁移、查询与 19 号停机门禁。

View File

@@ -0,0 +1,46 @@
package auditcoverage_test
import (
"path/filepath"
"reflect"
"runtime"
"testing"
"github.com/break/junhong_cmp_fiber/internal/governance/auditcoverage"
)
func TestReviewedAuditCoverageManifestMatchesAllRegisteredEntrypoints(t *testing.T) {
_, currentFile, _, ok := runtime.Caller(0)
if !ok {
t.Fatal("无法定位覆盖门禁测试文件")
}
root := filepath.Clean(filepath.Join(filepath.Dir(currentFile), "../../../"))
actual, err := auditcoverage.Scan(root)
if err != nil {
t.Fatalf("扫描全系统入口失败:%v", err)
}
expected, err := auditcoverage.LoadManifest(filepath.Join(root, ".scratch/tech-global-audit/审计覆盖清单.json"))
if err != nil {
t.Fatalf("读取经评审审计覆盖清单失败:%v", err)
}
if len(actual) == 0 {
t.Fatal("入口扫描结果为空,覆盖门禁不可用")
}
if !reflect.DeepEqual(expected, actual) {
t.Fatalf("源码入口与审计覆盖清单不一致:清单 %d 项,源码 %d 项;请重新分类并完成评审后更新清单", len(expected), len(actual))
}
for _, entry := range expected {
if entry.AuditEvent == "" || entry.DomainLedger == "" || entry.IntegrationLog == "" || entry.Outbox == "" ||
entry.Transaction == "" || entry.FailureStrategy == "" || entry.SensitivePolicy == "" ||
entry.BeforeAfterPolicy == "" || entry.TestSeam == "" {
t.Fatalf("入口 %s 存在未分类字段", entry.Key)
}
if entry.AuditEvent == "N/A" && entry.NAReason == "" {
t.Fatalf("入口 %s 的 Audit Event 为 N/A 但缺少理由", entry.Key)
}
if entry.AuditEvent != "N/A" && (entry.ActionCode == "" || entry.ActionName == "" || entry.Category == "" ||
entry.Risk == "" || entry.PrimaryResource == "") {
t.Fatalf("入口 %s 缺少动作、风险或主要资源", entry.Key)
}
}
}

View File

@@ -0,0 +1,448 @@
// Package auditcoverage 提供全系统入口的可复核审计覆盖扫描。
package auditcoverage
import (
"fmt"
"go/ast"
"go/parser"
"go/token"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"github.com/bytedance/sonic"
)
// Entry 是一个必须经过审计分类的 HTTP、Worker 或定时任务入口。
type Entry struct {
Key string `json:"key"`
Kind string `json:"kind"`
CodeEntry string `json:"code_entry"`
Owner string `json:"owner"`
Method string `json:"method,omitempty"`
Path string `json:"path,omitempty"`
Summary string `json:"summary"`
AuditEvent string `json:"audit_event"`
DomainLedger string `json:"domain_ledger"`
IntegrationLog string `json:"integration_log"`
Outbox string `json:"outbox"`
ActionCode string `json:"action_code,omitempty"`
ActionName string `json:"action_name,omitempty"`
Category string `json:"category,omitempty"`
Risk string `json:"risk,omitempty"`
PrimaryResource string `json:"primary_resource,omitempty"`
AffectedResource string `json:"affected_resource,omitempty"`
ActorSource string `json:"actor_source"`
Transaction string `json:"transaction"`
FailureStrategy string `json:"failure_strategy"`
SensitivePolicy string `json:"sensitive_policy"`
BeforeAfterPolicy string `json:"before_after_policy"`
TestSeam string `json:"test_seam"`
NAReason string `json:"na_reason,omitempty"`
}
// Scan 扫描当前仓库中对外 HTTP、Asynq Worker 和定时任务注册入口。
func Scan(root string) ([]Entry, error) {
var entries []Entry
files := []string{"internal/routes", "internal/application", "internal/domain", "internal/service", "pkg/queue", "cmd/worker"}
for _, directory := range files {
err := filepath.Walk(filepath.Join(root, directory), func(path string, info os.FileInfo, walkErr error) error {
if walkErr != nil {
return walkErr
}
if info.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
return nil
}
found, err := scanFile(root, path)
if err != nil {
return err
}
entries = append(entries, found...)
return nil
})
if err != nil {
return nil, err
}
}
sort.Slice(entries, func(i, j int) bool { return entries[i].Key < entries[j].Key })
return entries, nil
}
// LoadManifest 读取经评审的显式覆盖快照。
func LoadManifest(path string) ([]Entry, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, err
}
var entries []Entry
if err := sonic.Unmarshal(data, &entries); err != nil {
return nil, err
}
return entries, nil
}
// MarshalManifest 将扫描结果输出为稳定、便于评审的 JSON。
func MarshalManifest(entries []Entry) ([]byte, error) {
return sonic.ConfigStd.MarshalIndent(entries, "", " ")
}
func scanFile(root, path string) ([]Entry, error) {
set := token.NewFileSet()
file, err := parser.ParseFile(set, path, nil, 0)
if err != nil {
return nil, err
}
relative, err := filepath.Rel(root, path)
if err != nil {
return nil, err
}
var entries []Entry
ast.Inspect(file, func(node ast.Node) bool {
call, ok := node.(*ast.CallExpr)
if !ok {
return true
}
position := set.Position(call.Pos())
if identifier, ok := call.Fun.(*ast.Ident); ok && identifier.Name == "Register" && len(call.Args) >= 7 {
method, methodOK := stringLiteral(call.Args[3])
pathSuffix, pathOK := stringLiteral(call.Args[4])
if methodOK && pathOK {
entry := classifyHTTP(relative, position.Line, method, pathSuffix, expression(call.Args[5]), routeSummary(call.Args[6]))
entries = append(entries, entry)
}
return true
}
selector, ok := call.Fun.(*ast.SelectorExpr)
if !ok {
return true
}
switch selector.Sel.Name {
case "HandleFunc":
if len(call.Args) >= 2 {
entries = append(entries, classifyWorker(relative, position.Line, expression(call.Args[0]), expression(call.Args[1])))
}
case "Register":
if strings.HasPrefix(relative, "cmd/worker/") {
if taskType, schedule, ok := scheduledTask(call); ok {
entries = append(entries, classifySchedule(relative, position.Line, taskType, schedule))
}
}
}
return true
})
if strings.HasPrefix(relative, "internal/application/") || strings.HasPrefix(relative, "internal/domain/") ||
strings.HasPrefix(relative, "internal/service/") {
for _, declaration := range file.Decls {
function, ok := declaration.(*ast.FuncDecl)
if !ok || function.Recv == nil || !isBusinessMethod(function.Name.Name) {
continue
}
position := set.Position(function.Pos())
entries = append(entries, classifyBusinessMethod(relative, position.Line, function.Name.Name))
}
}
return entries, nil
}
func classifyHTTP(file string, line int, method, path, handler, summary string) Entry {
owner := strings.TrimSuffix(filepath.Base(file), ".go")
if summary == "" {
summary = handler
}
entry := Entry{
Key: fmt.Sprintf("http:%s:%d:%s:%s", file, line, method, path), Kind: "http",
CodeEntry: fmt.Sprintf("%s:%d %s", file, line, handler), Owner: owner,
Method: method, Path: path, Summary: summary, ActorSource: httpActorSource(file),
DomainLedger: ledgerDecision(owner), IntegrationLog: integrationDecision(file, path),
Outbox: "按用例是否存在提交后可靠副作用决定;无可靠副作用时 N/A",
SensitivePolicy: "禁止字段删除手机号、IP、ICCID、金额和第三方单号按权限脱敏单字段 16KB 上限",
BeforeAfterPolicy: "写操作保存脱敏后的直接字段变化;批量命令保存摘要和权威明细引用",
TestSeam: "真实 Fiber + Application/Service 公共用例 + PostgreSQL 事实;覆盖门禁静态比对本入口",
}
if method == "GET" && !isSensitiveRead(file, path, summary) {
entry.AuditEvent = "N/A"
entry.Transaction = "N/A"
entry.FailureStrategy = "Access Log 记录统一错误;普通读取不创建业务审计"
entry.NAReason = "普通只读查询,不改变业务事实且不返回需二次授权的完整敏感值"
return entry
}
entry.AuditEvent = "必须"
entry.ActionCode = actionCode(owner, handler)
entry.ActionName = summary
entry.Category = categoryFor(owner)
entry.Risk = riskFor(owner, path, summary)
entry.PrimaryResource = owner
entry.AffectedResource = "由对应 Application/Service 用例按直接影响资源显式填写,禁止递归扩展"
entry.Transaction = "成功事件与关键业务事实同一 GORM 事务;敏感读取在返回前写入"
entry.FailureStrategy = "业务回滚后的 failed/denied 使用独立短事务;二次失败保留业务错误并记录 critical"
return entry
}
func classifyWorker(file string, line int, taskType, handler string) Entry {
owner := workerOwner(taskType)
entry := Entry{
Key: fmt.Sprintf("worker:%s:%d:%s", file, line, taskType), Kind: "worker",
CodeEntry: fmt.Sprintf("%s:%d %s", file, line, handler), Owner: owner,
Summary: "处理异步任务 " + taskType, AuditEvent: "按状态变化、人工触发、连续失败或高风险异常决定",
DomainLedger: ledgerDecision(owner), IntegrationLog: workerIntegrationDecision(taskType),
Outbox: "任务来源 Outbox/业务任务事实;消费端按稳定事件或任务 ID 幂等",
ActionCode: actionCode(owner, handler), ActionName: "处理异步任务(" + taskType + "",
Category: categoryFor(owner), Risk: riskFor(owner, taskType, handler),
PrimaryResource: owner, AffectedResource: "任务载荷定位的直接业务资源",
ActorSource: "system_task/asynq", Transaction: "业务状态变化、领域流水和 Audit Event 按用例原子提交",
FailureStrategy: "Worker 返回错误由公共重试恢复;终态失败保存中文安全摘要,禁止裸 goroutine 审计",
SensitivePolicy: "不记录完整任务载荷、文件内容、外部正文、凭证或签名 URL",
BeforeAfterPolicy: "状态变化保存直接前后值;无业务变化时仅保留 Integration Log",
TestSeam: "公开 Asynq Handler + PostgreSQL/Redis 事实 + 重复消费测试;覆盖门禁静态比对本入口",
}
return entry
}
func classifySchedule(file string, line int, taskType, schedule string) Entry {
return Entry{
Key: fmt.Sprintf("schedule:%s:%d:%s", file, line, taskType), Kind: "scheduled_job",
CodeEntry: fmt.Sprintf("%s:%d", file, line), Owner: workerOwner(taskType), Summary: "按 " + schedule + " 调度 " + taskType,
AuditEvent: "N/A", DomainLedger: "N/A", IntegrationLog: "N/A", Outbox: "N/A",
ActorSource: "system_task/scheduled_job", Transaction: "N/A",
FailureStrategy: "调度注册失败阻止 Worker 启动;执行结果由对应 Worker 入口负责",
SensitivePolicy: "调度日志仅记录任务类型与安全时间信息",
BeforeAfterPolicy: "N/A调度入口不修改业务事实",
TestSeam: "调度注册公开函数 + 覆盖门禁静态比对本入口",
NAReason: "本入口只产生调度信号,不直接读取或修改业务事实;审计责任位于对应 Worker",
}
}
func classifyBusinessMethod(file string, line int, method string) Entry {
parts := strings.Split(file, "/")
layer := parts[1]
owner := strings.TrimSuffix(filepath.Base(filepath.Dir(file)), ".go")
if owner == "service" || owner == "application" || owner == "domain" {
owner = strings.TrimSuffix(filepath.Base(file), ".go")
}
entry := Entry{
Key: fmt.Sprintf("%s:%s:%d:%s", layer, file, line, method), Kind: layer,
CodeEntry: fmt.Sprintf("%s:%d %s", file, line, method), Owner: owner,
Summary: "业务方法 " + method, DomainLedger: ledgerDecision(owner),
IntegrationLog: businessIntegrationDecision(file, method),
Outbox: "存在提交后可靠副作用时必须在同一事务追加;否则 N/A",
ActionCode: actionCode(owner, method), Risk: riskFor(owner, file, method),
PrimaryResource: owner, AffectedResource: "完整用例直接修改或引用的资源",
ActorSource: "由调用入口传入操作者与来源快照",
SensitivePolicy: "禁止字段删除;受控字段脱敏;批量明细留在领域任务或制品",
BeforeAfterPolicy: "完整用例保存脱敏后的直接业务变化Domain 方法由 Application 投影",
TestSeam: "Application/Service 公共方法 + PostgreSQL 事实Domain 使用纯领域测试;覆盖门禁静态比对本入口",
}
if layer == "domain" {
entry.AuditEvent = "N/A"
entry.Transaction = "由 Application 组合根负责"
entry.FailureStrategy = "返回领域错误,由 Application 在回滚后裁决 failed/denied 审计"
entry.NAReason = "Domain 只维护业务不变量和领域事实不依赖审计基础设施Audit Event 由 Application 写入"
entry.ActionCode = ""
entry.ActionName = ""
entry.Category = ""
entry.Risk = ""
entry.PrimaryResource = ""
return entry
}
entry.AuditEvent = "必须"
entry.ActionName = "执行业务方法(" + method + ""
entry.Category = categoryFor(owner)
entry.Transaction = "关键成功事件与业务事实同一 GORM 事务"
entry.FailureStrategy = "业务回滚后的 failed/denied 使用独立短事务;审计二次失败记录 critical"
return entry
}
func routeSummary(expr ast.Expr) string {
composite, ok := expr.(*ast.CompositeLit)
if !ok {
return ""
}
for _, element := range composite.Elts {
pair, ok := element.(*ast.KeyValueExpr)
if !ok || expression(pair.Key) != "Summary" {
continue
}
value, _ := stringLiteral(pair.Value)
return value
}
return ""
}
func scheduledTask(call *ast.CallExpr) (string, string, bool) {
if len(call.Args) < 2 {
return "", "", false
}
schedule, _ := stringLiteral(call.Args[0])
taskCall, ok := call.Args[1].(*ast.CallExpr)
if !ok {
return "", "", false
}
selector, ok := taskCall.Fun.(*ast.SelectorExpr)
if !ok || selector.Sel.Name != "NewTask" || len(taskCall.Args) == 0 {
return "", "", false
}
return expression(taskCall.Args[0]), schedule, true
}
func stringLiteral(expr ast.Expr) (string, bool) {
literal, ok := expr.(*ast.BasicLit)
if !ok || literal.Kind != token.STRING {
return "", false
}
value, err := strconv.Unquote(literal.Value)
return value, err == nil
}
func expression(expr ast.Expr) string {
switch value := expr.(type) {
case *ast.Ident:
return value.Name
case *ast.SelectorExpr:
return expression(value.X) + "." + value.Sel.Name
case *ast.BasicLit:
return value.Value
case *ast.CallExpr:
return expression(value.Fun)
default:
return fmt.Sprintf("%T", expr)
}
}
func httpActorSource(file string) string {
switch {
case strings.HasSuffix(file, "personal.go"):
return "personal_customer/personal_api"
case strings.HasSuffix(file, "order.go"):
return "按路由分为 admin_user/admin_api 或外部回调入口"
default:
return "登录账号快照/admin_api"
}
}
func isSensitiveRead(file, path, summary string) bool {
text := strings.ToLower(file + " " + path + " " + summary)
for _, marker := range []string{"download", "export", "realname-link", "实名", "敏感", "完整", "operation-password"} {
if strings.Contains(text, marker) {
return true
}
}
return false
}
func integrationDecision(file, path string) string {
text := strings.ToLower(file + " " + path)
if strings.Contains(text, "callback") || strings.HasSuffix(file, "order.go") &&
(strings.Contains(path, "pay") || strings.Contains(path, "alipay")) {
return "必须:业务处理前保存入站安全摘要与幂等标识"
}
return "无外部交互时 N/A用例调用 Gateway、支付、企微或运营商时必须"
}
func workerIntegrationDecision(taskType string) string {
text := strings.ToLower(taskType)
if strings.Contains(text, "polling") {
return "必须:每次实际请求或未发送裁决均记录"
}
return "Worker 调用外部系统时必须;纯本地处理 N/A"
}
func businessIntegrationDecision(file, method string) string {
text := strings.ToLower(file + " " + method)
for _, marker := range []string{"polling", "gateway", "payment", "wechat", "wecom", "carrier", "sms"} {
if strings.Contains(text, marker) {
return "调用外部系统或处理回调时必须;纯本地分支 N/A"
}
}
return "N/A当前方法按代码位置属于本地业务用例后续新增外部调用必须重新分类"
}
func ledgerDecision(owner string) string {
for _, marker := range []string{"order", "recharge", "refund", "commission", "wallet"} {
if strings.Contains(owner, marker) {
return "必须:订单、充值、退款、钱包流水等既有业务表是领域权威"
}
}
if strings.Contains(owner, "import") || strings.Contains(owner, "export") {
return "业务任务及明细表是批量结果权威"
}
return "既有业务表是状态事实Audit Event 不替代业务模型"
}
func riskFor(owner, path, summary string) string {
text := strings.ToLower(owner + " " + path + " " + summary)
for _, marker := range []string{"wallet", "refund", "recharge", "permission", "role", "password", "config", "权限", "资金", "退款", "充值"} {
if strings.Contains(text, marker) {
return "high"
}
}
return "normal"
}
func categoryFor(owner string) string {
text := strings.ToLower(owner)
for _, marker := range []string{"wallet", "refund", "recharge", "commission", "order"} {
if strings.Contains(text, marker) {
return "finance"
}
}
for _, marker := range []string{"account", "role", "permission", "auth"} {
if strings.Contains(text, marker) {
return "security"
}
}
for _, marker := range []string{"card", "device", "asset", "polling"} {
if strings.Contains(text, marker) {
return "asset"
}
}
return "business"
}
func actionCode(owner, handler string) string {
method := handler
if index := strings.LastIndex(method, "."); index >= 0 {
method = method[index+1:]
}
return normalize(owner) + "." + normalize(method)
}
func workerOwner(taskType string) string {
return normalize(strings.TrimPrefix(taskType, "constants.TaskType"))
}
func isBusinessMethod(name string) bool {
for _, prefix := range []string{
"Create", "Update", "Delete", "Set", "Assign", "Remove", "Cancel", "Reject", "Approve",
"Import", "Allocate", "Recall", "Stop", "Resume", "Bind", "Unbind", "Reset", "Activate",
"Deactivate", "Trigger", "Handle", "Process", "Execute", "Replay", "Release", "Change", "Pay",
"Refund", "Recharge", "Withdraw", "Grant", "Revoke", "Deduct", "Credit", "Debit", "Freeze",
"Unfreeze", "Resolve", "Expire", "Invalidate", "Archive", "Cleanup", "Adjust", "Add", "Batch",
"Enable", "Disable", "Login", "Logout", "Refresh", "Upload", "Download", "Save", "Restore",
"Submit", "Sync", "Migrate", "Send",
} {
if strings.HasPrefix(name, prefix) {
return true
}
}
return false
}
func normalize(value string) string {
value = strings.Trim(value, "\"")
var output []rune
for index, current := range []rune(value) {
if current >= 'A' && current <= 'Z' {
if index > 0 {
output = append(output, '_')
}
current += 'a' - 'A'
}
if current == '-' || current == ':' || current == '/' {
current = '_'
}
output = append(output, current)
}
return strings.Trim(strings.ReplaceAll(string(output), "__", "_"), "_")
}

View File

@@ -0,0 +1,284 @@
// Package integrationlog 提供外部交互尝试的可靠持久化能力。
package integrationlog
import (
"context"
"crypto/sha256"
"encoding/hex"
"strings"
"time"
"github.com/google/uuid"
"gorm.io/datatypes"
"gorm.io/gorm"
"gorm.io/gorm/clause"
"github.com/break/junhong_cmp_fiber/internal/model"
"github.com/break/junhong_cmp_fiber/pkg/constants"
pkgerrors "github.com/break/junhong_cmp_fiber/pkg/errors"
"github.com/break/junhong_cmp_fiber/pkg/sanitizer"
)
// Attempt 描述一次外部调用前必须持久化的稳定事实。
type Attempt struct {
IntegrationID string
Provider string
Direction string
Operation string
ExternalID *string
ResourceType string
ResourceID *string
ResourceKey *string
TriggerSource *string
TriggerScene *string
TriggerSeries *string
ScheduledAt *time.Time
StartedAt *time.Time
Attempt int
RequestSummary any
Metadata any
RequestID *string
CorrelationID *string
AuditEventID *uint
InitialResult string
RecoveryStrategy *string
}
// Completion 描述外部尝试从待处理状态进入终态的结果。
type Completion struct {
Result string
HTTPStatus int
ProviderCode string
ProviderMessage string
ResponseSummary any
DurationMS int64
StateChanged bool
AuditEventID *uint
RecoveryStrategy string
}
// InboundAttempt 描述业务处理前必须保存的入站回调安全事实。
type InboundAttempt struct {
IntegrationID string
IdempotencyKey string
Provider string
Operation string
ExternalID string
ResourceType string
ResourceID *string
ResourceKey *string
RawPayload []byte
ContentType string
RequestID *string
CorrelationID *string
}
// Repository 负责创建稳定尝试及受控地进入终态。
type Repository struct {
db *gorm.DB
now func() time.Time
}
// NewRepository 创建 Integration Log Repository。
func NewRepository(db *gorm.DB) *Repository {
return &Repository{db: db, now: time.Now}
}
// Start 在实际调用外部系统前持久化尝试事实。
func (r *Repository) Start(ctx context.Context, input Attempt) (*model.IntegrationLog, error) {
if r == nil || r.db == nil {
return nil, pkgerrors.New(pkgerrors.CodeInvalidStatus, "Integration Log 数据库未配置")
}
if err := validateAttempt(input); err != nil {
return nil, err
}
requestSummary, err := marshalSummary(input.RequestSummary)
if err != nil {
return nil, pkgerrors.Wrap(pkgerrors.CodeInvalidParam, err, "Integration Log 请求摘要无效")
}
metadata, err := marshalSummary(input.Metadata)
if err != nil {
return nil, pkgerrors.Wrap(pkgerrors.CodeInvalidParam, err, "Integration Log 元数据无效")
}
if input.IntegrationID == "" {
input.IntegrationID = uuid.NewString()
}
if input.Attempt <= 0 {
input.Attempt = 1
}
if input.StartedAt == nil {
startedAt := r.now().UTC()
input.StartedAt = &startedAt
}
result := input.InitialResult
if result == "" {
result = constants.IntegrationResultPending
}
resourceType := optionalString(input.ResourceType)
log := &model.IntegrationLog{
IntegrationID: input.IntegrationID, Provider: input.Provider, Direction: input.Direction,
Operation: input.Operation, ExternalID: input.ExternalID, ResourceType: resourceType,
ResourceID: input.ResourceID, ResourceKey: input.ResourceKey, TriggerSource: input.TriggerSource,
TriggerScene: input.TriggerScene, TriggerSeries: input.TriggerSeries, ScheduledAt: input.ScheduledAt,
StartedAt: input.StartedAt, Attempt: input.Attempt, Result: result,
RequestSummary: requestSummary, Metadata: metadata, RequestID: input.RequestID,
CorrelationID: input.CorrelationID, AuditEventID: input.AuditEventID,
RecoveryStrategy: input.RecoveryStrategy,
}
if err := r.db.WithContext(ctx).Create(log).Error; err != nil {
return nil, pkgerrors.Wrap(pkgerrors.CodeDatabaseError, err, "写入 Integration Log 失败")
}
return log, nil
}
// Complete 仅允许把待处理尝试条件更新为一个公开终态。
func (r *Repository) Complete(ctx context.Context, integrationID string, completion Completion) (*model.IntegrationLog, error) {
if r == nil || r.db == nil {
return nil, pkgerrors.New(pkgerrors.CodeInvalidStatus, "Integration Log 数据库未配置")
}
if integrationID == "" || !isTerminalResult(completion.Result) {
return nil, pkgerrors.New(pkgerrors.CodeInvalidParam, "Integration Log 终态参数无效")
}
if completion.Result == constants.IntegrationResultUnknown && strings.TrimSpace(completion.RecoveryStrategy) == "" {
return nil, pkgerrors.New(pkgerrors.CodeInvalidParam, "结果未知必须记录明确恢复策略")
}
responseSummary, err := marshalSummary(completion.ResponseSummary)
if err != nil {
return nil, pkgerrors.Wrap(pkgerrors.CodeInvalidParam, err, "Integration Log 响应摘要无效")
}
updates := map[string]any{
"result": completion.Result, "duration_ms": completion.DurationMS,
"state_changed": completion.StateChanged, "response_summary": responseSummary,
"updated_at": r.now().UTC(),
}
if completion.HTTPStatus != 0 {
updates["http_status"] = completion.HTTPStatus
}
if completion.ProviderCode != "" {
updates["provider_code"] = completion.ProviderCode
}
if completion.ProviderMessage != "" {
updates["provider_message"] = sanitizer.TextSummary(completion.ProviderMessage)
}
if completion.AuditEventID != nil {
updates["audit_event_id"] = completion.AuditEventID
}
if completion.RecoveryStrategy != "" {
updates["recovery_strategy"] = completion.RecoveryStrategy
}
result := r.db.WithContext(ctx).Model(&model.IntegrationLog{}).
Where("integration_id = ? AND result = ?", integrationID, constants.IntegrationResultPending).
Updates(updates)
if result.Error != nil {
return nil, pkgerrors.Wrap(pkgerrors.CodeDatabaseError, result.Error, "终结 Integration Log 失败")
}
if result.RowsAffected != 1 {
return nil, pkgerrors.New(pkgerrors.CodeConflict, "Integration Log 已进入终态或不存在")
}
var saved model.IntegrationLog
if err := r.db.WithContext(ctx).Where("integration_id = ?", integrationID).First(&saved).Error; err != nil {
return nil, pkgerrors.Wrap(pkgerrors.CodeDatabaseError, err, "读取 Integration Log 终态失败")
}
return &saved, nil
}
// RecordInbound 在业务处理前幂等保存入站回调的安全摘要。
func (r *Repository) RecordInbound(ctx context.Context, input InboundAttempt) (*model.IntegrationLog, bool, error) {
if r == nil || r.db == nil {
return nil, false, pkgerrors.New(pkgerrors.CodeInvalidStatus, "Integration Log 数据库未配置")
}
if input.Provider == "" || input.Operation == "" || input.IdempotencyKey == "" || len(input.RawPayload) == 0 {
return nil, false, pkgerrors.New(pkgerrors.CodeInvalidParam, "入站 Integration Log 参数无效")
}
if input.IntegrationID == "" {
input.IntegrationID = uuid.NewString()
}
hash := sha256.Sum256(input.RawPayload)
summary, err := marshalSummary(map[string]any{
"content_type": input.ContentType,
"payload_bytes": len(input.RawPayload),
"content_hash": hex.EncodeToString(hash[:]),
})
if err != nil {
return nil, false, pkgerrors.Wrap(pkgerrors.CodeInvalidParam, err, "入站 Integration Log 摘要无效")
}
now := r.now().UTC()
log := &model.IntegrationLog{
IntegrationID: input.IntegrationID, IdempotencyKey: &input.IdempotencyKey,
Provider: input.Provider, Direction: constants.IntegrationDirectionInbound, Operation: input.Operation,
ExternalID: optionalString(input.ExternalID), ResourceType: optionalString(input.ResourceType),
ResourceID: input.ResourceID, ResourceKey: input.ResourceKey, StartedAt: &now, Attempt: 1,
Result: constants.IntegrationResultPending, RequestSummary: summary,
ContentHash: hex.EncodeToString(hash[:]), RequestID: input.RequestID, CorrelationID: input.CorrelationID,
}
result := r.db.WithContext(ctx).Clauses(clause.OnConflict{
Columns: []clause.Column{{Name: "provider"}, {Name: "operation"}, {Name: "idempotency_key"}},
DoNothing: true,
}).Create(log)
if result.Error != nil {
return nil, false, pkgerrors.Wrap(pkgerrors.CodeDatabaseError, result.Error, "写入入站 Integration Log 失败")
}
if result.RowsAffected == 1 {
return log, true, nil
}
var existing model.IntegrationLog
if err := r.db.WithContext(ctx).Where(
"provider = ? AND operation = ? AND idempotency_key = ?", input.Provider, input.Operation, input.IdempotencyKey,
).First(&existing).Error; err != nil {
return nil, false, pkgerrors.Wrap(pkgerrors.CodeDatabaseError, err, "读取重复入站 Integration Log 失败")
}
if existing.ContentHash != hex.EncodeToString(hash[:]) {
return nil, false, pkgerrors.New(pkgerrors.CodeConflict, "入站幂等标识对应的载荷不一致")
}
return &existing, false, nil
}
func validateAttempt(input Attempt) error {
if input.Provider == "" || input.Operation == "" {
return pkgerrors.New(pkgerrors.CodeInvalidParam, "Integration Log 提供方和操作不能为空")
}
if input.Direction != constants.IntegrationDirectionInbound && input.Direction != constants.IntegrationDirectionOutbound {
return pkgerrors.New(pkgerrors.CodeInvalidParam, "Integration Log 方向无效")
}
if input.InitialResult != "" && input.InitialResult != constants.IntegrationResultPending && !isUnsentResult(input.InitialResult) {
return pkgerrors.New(pkgerrors.CodeInvalidParam, "Integration Log 初始结果只能是待处理或未发送终态")
}
return nil
}
func isTerminalResult(result string) bool {
switch result {
case constants.IntegrationResultSuccess, constants.IntegrationResultFailed, constants.IntegrationResultUnknown,
constants.IntegrationResultNotFound, constants.IntegrationResultInvalidPayload, constants.IntegrationResultIgnored,
constants.IntegrationResultMerged, constants.IntegrationResultRateLimited, constants.IntegrationResultCompleted,
constants.IntegrationResultCancelled:
return true
default:
return false
}
}
func isUnsentResult(result string) bool {
switch result {
case constants.IntegrationResultIgnored, constants.IntegrationResultMerged, constants.IntegrationResultRateLimited,
constants.IntegrationResultCompleted, constants.IntegrationResultCancelled:
return true
default:
return false
}
}
func marshalSummary(value any) (datatypes.JSON, error) {
if value == nil {
return nil, nil
}
encoded, err := sanitizer.MarshalSummary(value)
return datatypes.JSON(encoded), err
}
func optionalString(value string) *string {
if value == "" {
return nil
}
return &value
}

View File

@@ -0,0 +1,277 @@
package integrationlog_test
import (
"context"
"fmt"
"strings"
"sync"
"testing"
"gorm.io/gorm"
"github.com/break/junhong_cmp_fiber/internal/infrastructure/integrationlog"
"github.com/break/junhong_cmp_fiber/internal/model"
"github.com/break/junhong_cmp_fiber/internal/testutil"
"github.com/break/junhong_cmp_fiber/pkg/constants"
)
func TestOutboundAttemptPersistsBeforeConditionalCompletion(t *testing.T) {
db := testutil.NewPostgresTransaction(t)
createTemporaryIntegrationLogTable(t, db)
repository := integrationlog.NewRepository(db)
attempt, err := repository.Start(context.Background(), integrationlog.Attempt{
IntegrationID: "integration-outbound-1",
Provider: "gateway",
Direction: constants.IntegrationDirectionOutbound,
Operation: "query_card_status",
ResourceType: "iot_card",
ResourceID: testutil.StringPointer("1001"),
RequestSummary: map[string]any{
"iccid": "8986001234567890123",
"access_token": "must-not-persist",
"credential": "must-not-persist",
"private_url": "https://must-not-persist.example",
},
})
if err != nil {
t.Fatalf("持久化外部尝试失败:%v", err)
}
if attempt.Result != constants.IntegrationResultPending {
t.Fatalf("调用前必须是待处理状态,得到 %q", attempt.Result)
}
if strings.Contains(string(attempt.RequestSummary), "must-not-persist") {
t.Fatalf("请求摘要泄露禁止字段:%s", attempt.RequestSummary)
}
auditEventID := uint(77)
completed, err := repository.Complete(context.Background(), attempt.IntegrationID, integrationlog.Completion{
Result: constants.IntegrationResultSuccess,
HTTPStatus: 200,
ProviderCode: "0",
ProviderMessage: "查询成功",
StateChanged: true,
AuditEventID: &auditEventID,
ResponseSummary: map[string]any{"status": "active", "secret": "must-not-persist"},
})
if err != nil {
t.Fatalf("终结外部尝试失败:%v", err)
}
if completed.Result != constants.IntegrationResultSuccess || !completed.StateChanged ||
completed.AuditEventID == nil || *completed.AuditEventID != auditEventID {
t.Fatalf("外部尝试终态错误:%+v", completed)
}
if completed.ProviderMessage == nil || strings.Contains(*completed.ProviderMessage, "查询成功") {
t.Fatalf("不可信渠道消息必须转换为不可逆摘要:%+v", completed.ProviderMessage)
}
if _, err := repository.Complete(context.Background(), attempt.IntegrationID, integrationlog.Completion{
Result: constants.IntegrationResultFailed,
}); err == nil {
t.Fatal("既有终态不得被重复完成改写")
}
}
func TestIntegrationResultNamesCoverEveryPublicTerminalState(t *testing.T) {
for _, result := range []string{
constants.IntegrationResultSuccess, constants.IntegrationResultFailed, constants.IntegrationResultUnknown,
constants.IntegrationResultNotFound, constants.IntegrationResultInvalidPayload, constants.IntegrationResultIgnored,
constants.IntegrationResultMerged, constants.IntegrationResultRateLimited, constants.IntegrationResultCompleted,
constants.IntegrationResultCancelled,
} {
if constants.IntegrationResultName(result) == "" {
t.Fatalf("公开终态 %q 缺少中文名称", result)
}
}
}
func TestInboundAttemptIsIdempotentAndNeverPersistsRawPayload(t *testing.T) {
db := testutil.NewPostgresTransaction(t)
createTemporaryIntegrationLogTable(t, db)
repository := integrationlog.NewRepository(db)
input := integrationlog.InboundAttempt{
IntegrationID: "integration-inbound-1",
IdempotencyKey: "wechat:callback:transaction-1",
Provider: "wechat",
Operation: "payment_callback",
ExternalID: "transaction-1",
RawPayload: []byte(`{"sign":"raw-signature","ciphertext":"raw-ciphertext"}`),
ContentType: "application/json",
}
first, created, err := repository.RecordInbound(context.Background(), input)
if err != nil || !created {
t.Fatalf("首次保存入站尝试失败created=%v err=%v", created, err)
}
second, created, err := repository.RecordInbound(context.Background(), input)
if err != nil || created {
t.Fatalf("重复入站尝试应返回既有事实created=%v err=%v", created, err)
}
if first.ID != second.ID || first.ContentHash == "" {
t.Fatalf("重复回调未复用稳定事实或缺少内容哈希first=%+v second=%+v", first, second)
}
serialized := string(first.RequestSummary)
if strings.Contains(serialized, "raw-signature") || strings.Contains(serialized, "raw-ciphertext") {
t.Fatalf("入站摘要泄露原始载荷:%s", serialized)
}
conflict := input
conflict.IntegrationID = "integration-inbound-conflict"
conflict.RawPayload = []byte(`{"different":true}`)
if _, _, err := repository.RecordInbound(context.Background(), conflict); err == nil {
t.Fatal("相同入站幂等标识对应不同载荷时必须拒绝")
}
}
func TestUnknownResultRequiresExplicitRecoveryAndUnsentAttemptIsTerminal(t *testing.T) {
db := testutil.NewPostgresTransaction(t)
createTemporaryIntegrationLogTable(t, db)
repository := integrationlog.NewRepository(db)
pending, err := repository.Start(context.Background(), integrationlog.Attempt{
IntegrationID: "integration-unknown-1", Provider: "wecom",
Direction: constants.IntegrationDirectionOutbound, Operation: "submit_approval",
})
if err != nil {
t.Fatalf("准备结果未知尝试失败:%v", err)
}
if _, err := repository.Complete(context.Background(), pending.IntegrationID, integrationlog.Completion{
Result: constants.IntegrationResultUnknown,
}); err == nil {
t.Fatal("结果未知必须记录明确恢复策略")
}
unknown, err := repository.Complete(context.Background(), pending.IntegrationID, integrationlog.Completion{
Result: constants.IntegrationResultUnknown, RecoveryStrategy: "按原外部单号查询结果,禁止盲目重发",
})
if err != nil || unknown.RecoveryStrategy == nil {
t.Fatalf("保存结果未知及恢复策略失败log=%+v err=%v", unknown, err)
}
for index, terminal := range []string{
constants.IntegrationResultIgnored, constants.IntegrationResultMerged, constants.IntegrationResultRateLimited,
constants.IntegrationResultCompleted, constants.IntegrationResultCancelled,
} {
unsent, err := repository.Start(context.Background(), integrationlog.Attempt{
IntegrationID: fmt.Sprintf("integration-unsent-%d", index), Provider: "gateway",
Direction: constants.IntegrationDirectionOutbound, Operation: "query_card_status", InitialResult: terminal,
})
if err != nil || unsent.Result != terminal || unsent.HTTPStatus != nil {
t.Fatalf("未发送终态不得伪造 HTTP 结果log=%+v err=%v", unsent, err)
}
if _, err := repository.Complete(context.Background(), unsent.IntegrationID, integrationlog.Completion{
Result: constants.IntegrationResultSuccess,
}); err == nil {
t.Fatal("未发送终态不得被后续完成改写")
}
}
if _, err := repository.Start(context.Background(), integrationlog.Attempt{
Provider: "gateway", Direction: constants.IntegrationDirectionOutbound,
Operation: "query_card_status", InitialResult: constants.IntegrationResultSuccess,
}); err == nil {
t.Fatal("实际调用结果不得绕过 pending 直接写终态")
}
}
func TestExplicitFailureAndConcurrentCompletionKeepFirstTerminalFact(t *testing.T) {
db := testutil.NewPostgresDatabase(t)
integrationIDs := []string{"integration-failed-1", "integration-concurrent-1"}
t.Cleanup(func() {
if err := db.Where("integration_id IN ?", integrationIDs).Delete(&model.IntegrationLog{}).Error; err != nil {
t.Errorf("清理 Integration Log 并发测试数据失败:%v", err)
}
})
if err := db.Where("integration_id IN ?", integrationIDs).Delete(&model.IntegrationLog{}).Error; err != nil {
t.Fatalf("准备 Integration Log 并发测试隔离数据失败:%v", err)
}
repository := integrationlog.NewRepository(db)
failedAttempt, err := repository.Start(context.Background(), integrationlog.Attempt{
IntegrationID: "integration-failed-1", Provider: "gateway",
Direction: constants.IntegrationDirectionOutbound, Operation: "query_card_status",
})
if err != nil {
t.Fatalf("准备明确失败尝试失败:%v", err)
}
failed, err := repository.Complete(context.Background(), failedAttempt.IntegrationID, integrationlog.Completion{
Result: constants.IntegrationResultFailed, ProviderCode: "UPSTREAM_REJECTED", ProviderMessage: "上游明确拒绝",
})
if err != nil || failed.Result != constants.IntegrationResultFailed {
t.Fatalf("明确失败未保存为失败终态log=%+v err=%v", failed, err)
}
attempt, err := repository.Start(context.Background(), integrationlog.Attempt{
IntegrationID: "integration-concurrent-1", Provider: "gateway",
Direction: constants.IntegrationDirectionOutbound, Operation: "query_card_status",
})
if err != nil {
t.Fatalf("准备并发终结尝试失败:%v", err)
}
results := make(chan error, 2)
var group sync.WaitGroup
for _, terminal := range []string{constants.IntegrationResultSuccess, constants.IntegrationResultFailed} {
group.Add(1)
go func(result string) {
defer group.Done()
_, completeErr := repository.Complete(context.Background(), attempt.IntegrationID, integrationlog.Completion{Result: result})
results <- completeErr
}(terminal)
}
group.Wait()
close(results)
successes := 0
for completeErr := range results {
if completeErr == nil {
successes++
}
}
if successes != 1 {
t.Fatalf("并发终结必须且只能一个成功,实际成功 %d 次", successes)
}
var saved model.IntegrationLog
if err := db.Where("integration_id = ?", attempt.IntegrationID).First(&saved).Error; err != nil {
t.Fatalf("读取并发终态失败:%v", err)
}
if saved.Result != constants.IntegrationResultSuccess && saved.Result != constants.IntegrationResultFailed {
t.Fatalf("并发终结未保存明确成功或失败:%+v", saved)
}
}
func createTemporaryIntegrationLogTable(t *testing.T, db *gorm.DB) {
t.Helper()
if err := db.Exec(`CREATE TEMP TABLE tb_integration_log (
id bigserial PRIMARY KEY,
integration_id varchar(64) NOT NULL UNIQUE,
idempotency_key varchar(160),
provider varchar(32) NOT NULL,
direction varchar(16) NOT NULL,
operation varchar(64) NOT NULL,
external_id varchar(128),
resource_type varchar(64),
resource_id varchar(128),
resource_key varchar(128),
trigger_source varchar(32),
trigger_scene varchar(128),
trigger_series varchar(64),
scheduled_at timestamptz,
started_at timestamptz,
attempt integer NOT NULL DEFAULT 1,
result varchar(20) NOT NULL,
http_status integer,
provider_code varchar(64),
provider_message varchar(500),
request_summary jsonb,
response_summary jsonb,
content_hash varchar(64),
duration_ms bigint NOT NULL DEFAULT 0,
state_changed boolean NOT NULL DEFAULT false,
metadata jsonb,
recovery_strategy varchar(255),
request_id varchar(64),
correlation_id varchar(64),
audit_event_id bigint,
created_at timestamptz NOT NULL DEFAULT NOW(),
updated_at timestamptz NOT NULL DEFAULT NOW(),
CONSTRAINT uq_test_integration_idempotency UNIQUE (provider, operation, idempotency_key)
) ON COMMIT DROP`).Error; err != nil {
t.Fatalf("创建 Integration Log 测试表失败:%v", err)
}
}

View File

@@ -0,0 +1,48 @@
package model
import (
"time"
"gorm.io/datatypes"
)
// IntegrationLog 是外部交互及未发送尝试的权威持久化模型。
type IntegrationLog struct {
ID uint `gorm:"column:id;primaryKey;autoIncrement" json:"id"`
IntegrationID string `gorm:"column:integration_id;type:varchar(64);not null;uniqueIndex" json:"integration_id"`
IdempotencyKey *string `gorm:"column:idempotency_key;type:varchar(160)" json:"idempotency_key,omitempty"`
Provider string `gorm:"column:provider;type:varchar(32);not null" json:"provider"`
Direction string `gorm:"column:direction;type:varchar(16);not null" json:"direction"`
Operation string `gorm:"column:operation;type:varchar(64);not null" json:"operation"`
ExternalID *string `gorm:"column:external_id;type:varchar(128)" json:"external_id,omitempty"`
ResourceType *string `gorm:"column:resource_type;type:varchar(64)" json:"resource_type,omitempty"`
ResourceID *string `gorm:"column:resource_id;type:varchar(128)" json:"resource_id,omitempty"`
ResourceKey *string `gorm:"column:resource_key;type:varchar(128)" json:"resource_key,omitempty"`
TriggerSource *string `gorm:"column:trigger_source;type:varchar(32)" json:"trigger_source,omitempty"`
TriggerScene *string `gorm:"column:trigger_scene;type:varchar(128)" json:"trigger_scene,omitempty"`
TriggerSeries *string `gorm:"column:trigger_series;type:varchar(64)" json:"trigger_series,omitempty"`
ScheduledAt *time.Time `gorm:"column:scheduled_at;type:timestamptz" json:"scheduled_at,omitempty"`
StartedAt *time.Time `gorm:"column:started_at;type:timestamptz" json:"started_at,omitempty"`
Attempt int `gorm:"column:attempt;type:int;not null;default:1" json:"attempt"`
Result string `gorm:"column:result;type:varchar(20);not null" json:"result"`
HTTPStatus *int `gorm:"column:http_status" json:"http_status,omitempty"`
ProviderCode *string `gorm:"column:provider_code;type:varchar(64)" json:"provider_code,omitempty"`
ProviderMessage *string `gorm:"column:provider_message;type:varchar(500)" json:"provider_message,omitempty"`
RequestSummary datatypes.JSON `gorm:"column:request_summary;type:jsonb" json:"request_summary,omitempty"`
ResponseSummary datatypes.JSON `gorm:"column:response_summary;type:jsonb" json:"response_summary,omitempty"`
ContentHash string `gorm:"column:content_hash;type:varchar(64);not null;default:''" json:"content_hash,omitempty"`
DurationMS int64 `gorm:"column:duration_ms;not null;default:0" json:"duration_ms"`
StateChanged bool `gorm:"column:state_changed;not null;default:false" json:"state_changed"`
Metadata datatypes.JSON `gorm:"column:metadata;type:jsonb" json:"metadata,omitempty"`
RecoveryStrategy *string `gorm:"column:recovery_strategy;type:varchar(255)" json:"recovery_strategy,omitempty"`
RequestID *string `gorm:"column:request_id;type:varchar(64)" json:"request_id,omitempty"`
CorrelationID *string `gorm:"column:correlation_id;type:varchar(64)" json:"correlation_id,omitempty"`
AuditEventID *uint `gorm:"column:audit_event_id" json:"audit_event_id,omitempty"`
CreatedAt time.Time `gorm:"column:created_at;type:timestamptz;not null;autoCreateTime" json:"created_at"`
UpdatedAt time.Time `gorm:"column:updated_at;type:timestamptz;not null;autoUpdateTime" json:"updated_at"`
}
// TableName 返回外部集成日志表名。
func (IntegrationLog) TableName() string {
return "tb_integration_log"
}

View File

@@ -15,6 +15,19 @@ import (
// NewPostgresTransaction 创建自动回滚的 PostgreSQL 测试事务。 // NewPostgresTransaction 创建自动回滚的 PostgreSQL 测试事务。
func NewPostgresTransaction(t *testing.T) *gorm.DB { func NewPostgresTransaction(t *testing.T) *gorm.DB {
t.Helper()
db := NewPostgresDatabase(t)
tx := db.Begin()
if tx.Error != nil {
t.Fatalf("开启测试事务失败:%v", tx.Error)
}
t.Cleanup(func() { _ = tx.Rollback().Error })
return tx
}
// NewPostgresDatabase 创建自动关闭、可使用多个连接的 PostgreSQL 测试数据库。
// 仅在需要验证真实并发条件更新时使用;普通集成测试仍优先使用自动回滚事务。
func NewPostgresDatabase(t *testing.T) *gorm.DB {
t.Helper() t.Helper()
if os.Getenv("JUNHONG_DATABASE_HOST") == "" { if os.Getenv("JUNHONG_DATABASE_HOST") == "" {
t.Skip("未加载 .env.local跳过依赖真实 PostgreSQL 的集成测试") t.Skip("未加载 .env.local跳过依赖真实 PostgreSQL 的集成测试")
@@ -27,17 +40,12 @@ func NewPostgresTransaction(t *testing.T) *gorm.DB {
if err != nil { if err != nil {
t.Fatalf("连接 PostgreSQL 失败:%v", err) t.Fatalf("连接 PostgreSQL 失败:%v", err)
} }
tx := db.Begin()
if tx.Error != nil {
t.Fatalf("开启测试事务失败:%v", tx.Error)
}
t.Cleanup(func() { t.Cleanup(func() {
_ = tx.Rollback().Error
if sqlDB, dbErr := db.DB(); dbErr == nil { if sqlDB, dbErr := db.DB(); dbErr == nil {
_ = sqlDB.Close() _ = sqlDB.Close()
} }
}) })
return tx return db
} }
// NewRedisClient 创建真实 Redis 测试客户端并在测试结束时关闭。 // NewRedisClient 创建真实 Redis 测试客户端并在测试结束时关闭。

View File

@@ -0,0 +1,11 @@
-- 已产生外部交互事实后禁止通过降级删除,避免链路历史丢失。
DO $$
BEGIN
IF to_regclass('tb_integration_log') IS NOT NULL
AND EXISTS (SELECT 1 FROM tb_integration_log LIMIT 1) THEN
RAISE EXCEPTION 'tb_integration_log 已存在外部交互事实,禁止删表回滚,请停止生产者后向前修复';
END IF;
END
$$;
DROP TABLE IF EXISTS tb_integration_log;

View File

@@ -0,0 +1,55 @@
-- 创建外部集成尝试权威记录;不建立外键,业务关系通过稳定 ID 显式维护。
CREATE TABLE tb_integration_log (
id bigserial PRIMARY KEY,
integration_id varchar(64) NOT NULL,
idempotency_key varchar(160),
provider varchar(32) NOT NULL,
direction varchar(16) NOT NULL,
operation varchar(64) NOT NULL,
external_id varchar(128),
resource_type varchar(64),
resource_id varchar(128),
resource_key varchar(128),
trigger_source varchar(32),
trigger_scene varchar(128),
trigger_series varchar(64),
scheduled_at timestamptz,
started_at timestamptz,
attempt integer NOT NULL DEFAULT 1,
result varchar(20) NOT NULL,
http_status integer,
provider_code varchar(64),
provider_message varchar(500),
request_summary jsonb,
response_summary jsonb,
content_hash varchar(64) NOT NULL DEFAULT '',
duration_ms bigint NOT NULL DEFAULT 0,
state_changed boolean NOT NULL DEFAULT false,
metadata jsonb,
recovery_strategy varchar(255),
request_id varchar(64),
correlation_id varchar(64),
audit_event_id bigint,
created_at timestamptz NOT NULL DEFAULT NOW(),
updated_at timestamptz NOT NULL DEFAULT NOW(),
CONSTRAINT uq_integration_log_id UNIQUE (integration_id),
CONSTRAINT uq_integration_log_idempotency UNIQUE (provider, operation, idempotency_key),
CONSTRAINT ck_integration_log_direction CHECK (direction IN ('inbound', 'outbound')),
CONSTRAINT ck_integration_log_result CHECK (result IN (
'pending', 'success', 'failed', 'unknown', 'not_found', 'invalid_payload',
'ignored', 'merged', 'rate_limited', 'completed', 'cancelled'
)),
CONSTRAINT ck_integration_log_attempt CHECK (attempt > 0),
CONSTRAINT ck_integration_log_duration CHECK (duration_ms >= 0)
);
CREATE INDEX idx_integration_log_time ON tb_integration_log (created_at DESC, id DESC);
CREATE INDEX idx_integration_log_provider ON tb_integration_log (provider, operation, result, created_at DESC);
CREATE INDEX idx_integration_log_resource ON tb_integration_log (resource_type, resource_id, created_at DESC);
CREATE INDEX idx_integration_log_resource_key ON tb_integration_log (resource_type, resource_key, created_at DESC);
CREATE INDEX idx_integration_log_trigger ON tb_integration_log (trigger_series, attempt);
CREATE INDEX idx_integration_log_request ON tb_integration_log (request_id, created_at ASC) WHERE request_id IS NOT NULL;
CREATE INDEX idx_integration_log_correlation ON tb_integration_log (correlation_id, created_at ASC) WHERE correlation_id IS NOT NULL;
COMMENT ON TABLE tb_integration_log IS '外部集成调用、回调及未发送尝试记录';
COMMENT ON COLUMN tb_integration_log.result IS '尝试结果pending 仅为内部执行态,其余为公开终态';

View File

@@ -0,0 +1,51 @@
package constants
const (
// IntegrationDirectionInbound 表示外部系统调用本系统。
IntegrationDirectionInbound = "inbound"
// IntegrationDirectionOutbound 表示本系统调用外部系统。
IntegrationDirectionOutbound = "outbound"
)
const (
// IntegrationResultPending 表示外部尝试已建立但尚未终结。
IntegrationResultPending = "pending"
// IntegrationResultSuccess 表示外部尝试成功。
IntegrationResultSuccess = "success"
// IntegrationResultFailed 表示外部尝试明确失败。
IntegrationResultFailed = "failed"
// IntegrationResultUnknown 表示请求已发出但结果未知,需要按记录的策略恢复。
IntegrationResultUnknown = "unknown"
// IntegrationResultNotFound 表示外部资源不存在。
IntegrationResultNotFound = "not_found"
// IntegrationResultInvalidPayload 表示入站载荷无效。
IntegrationResultInvalidPayload = "invalid_payload"
// IntegrationResultIgnored 表示外部尝试被安全忽略。
IntegrationResultIgnored = "ignored"
// IntegrationResultMerged 表示尝试被合并且未发送请求。
IntegrationResultMerged = "merged"
// IntegrationResultRateLimited 表示尝试因限频未发送请求。
IntegrationResultRateLimited = "rate_limited"
// IntegrationResultCompleted 表示业务已达预期,尝试提前完成且未发送请求。
IntegrationResultCompleted = "completed"
// IntegrationResultCancelled 表示尝试已取消。
IntegrationResultCancelled = "cancelled"
)
// IntegrationResultName 返回外部尝试结果的中文名称。
func IntegrationResultName(result string) string {
names := map[string]string{
IntegrationResultPending: "待处理",
IntegrationResultSuccess: "成功",
IntegrationResultFailed: "失败",
IntegrationResultUnknown: "结果未知",
IntegrationResultNotFound: "未找到",
IntegrationResultInvalidPayload: "无效载荷",
IntegrationResultIgnored: "已忽略",
IntegrationResultMerged: "已合并",
IntegrationResultRateLimited: "已限频",
IntegrationResultCompleted: "已提前完成",
IntegrationResultCancelled: "已取消",
}
return names[result]
}

View File

@@ -1,6 +1,7 @@
package logger package logger
import ( import (
"fmt"
"net/url" "net/url"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -10,25 +11,34 @@ import (
// AccessPolicy 是敏感路由的安全摘要策略。 // AccessPolicy 是敏感路由的安全摘要策略。
type AccessPolicy struct { type AccessPolicy struct {
Name string Name string
Sensitive bool Sensitive bool
SafeFields map[string]struct{} PresenceOnly bool
FileFields map[string]struct{} SafeFields map[string]struct{}
ResultFields map[string]struct{}
FileFields map[string]struct{}
} }
var ( var (
loginPolicy = AccessPolicy{ loginPolicy = AccessPolicy{
Name: "login_token", Sensitive: true, Name: "login_token", Sensitive: true, PresenceOnly: true,
SafeFields: fieldSet("username", "user_id", "account_id", "success", "result_code"), SafeFields: fieldSet("username", "user_id", "account_id", "success", "result_code"),
ResultFields: fieldSet("success", "result_code", "status"),
} }
paymentPolicy = AccessPolicy{ paymentPolicy = AccessPolicy{
Name: "payment", Sensitive: true, Name: "payment", Sensitive: true, PresenceOnly: true,
SafeFields: fieldSet("order_no", "payment_no", "channel", "payment_method", "result_code", "amount", "status"), SafeFields: fieldSet("order_no", "payment_no", "channel", "payment_method", "result_code", "amount", "status"),
ResultFields: fieldSet("result_code", "status"),
} }
wecomPolicy = AccessPolicy{ wecomPolicy = AccessPolicy{
Name: "wecom_callback", Sensitive: true, Name: "wecom_callback", Sensitive: true,
SafeFields: fieldSet("event_type", "resource_type", "resource_id", "result_code", "status"), SafeFields: fieldSet("event_type", "resource_type", "resource_id", "result_code", "status"),
} }
configPolicy = AccessPolicy{
Name: "sensitive_config", Sensitive: true, PresenceOnly: true,
SafeFields: fieldSet("config_key", "value_type", "module", "readonly", "sensitive", "configured", "status", "result_code"),
ResultFields: fieldSet("readonly", "sensitive", "configured", "status", "result_code"),
}
filePolicy = AccessPolicy{ filePolicy = AccessPolicy{
Name: "file_export", Sensitive: true, Name: "file_export", Sensitive: true,
SafeFields: fieldSet("content_type", "file_size", "size", "count", "task_id", "status", "result_code"), SafeFields: fieldSet("content_type", "file_size", "size", "count", "task_id", "status", "result_code"),
@@ -50,6 +60,8 @@ func policyForPath(path string) AccessPolicy {
switch { switch {
case strings.Contains(normalized, "/login"), strings.Contains(normalized, "token"): case strings.Contains(normalized, "/login"), strings.Contains(normalized, "token"):
return loginPolicy return loginPolicy
case strings.Contains(normalized, "/system-configs"), strings.Contains(normalized, "/wechat-configs"):
return configPolicy
case strings.Contains(normalized, "/callback") && (strings.Contains(normalized, "wecom") || strings.Contains(normalized, "wework")): case strings.Contains(normalized, "/callback") && (strings.Contains(normalized, "wecom") || strings.Contains(normalized, "wework")):
return wecomPolicy return wecomPolicy
case strings.Contains(normalized, "payment"), strings.Contains(normalized, "wechat-pay"), case strings.Contains(normalized, "payment"), strings.Contains(normalized, "wechat-pay"),
@@ -102,7 +114,13 @@ func collectSafeFields(value any, policy AccessPolicy, output map[string]any) {
case string: case string:
collectSafeScalar(key, scalar, policy, output) collectSafeScalar(key, scalar, policy, output)
case float64, bool: case float64, bool:
if _, allowed := policy.SafeFields[strings.ToLower(key)]; allowed { if policy.PresenceOnly {
if _, isResult := policy.ResultFields[strings.ToLower(key)]; isResult {
output[key] = scalar
} else {
output[key] = map[string]any{"present": true, "length": len(fmt.Sprint(scalar))}
}
} else if _, allowed := policy.SafeFields[strings.ToLower(key)]; allowed {
output[key] = scalar output[key] = scalar
} }
default: default:
@@ -118,6 +136,14 @@ func collectSafeFields(value any, policy AccessPolicy, output map[string]any) {
func collectSafeScalar(key, value string, policy AccessPolicy, output map[string]any) { func collectSafeScalar(key, value string, policy AccessPolicy, output map[string]any) {
normalized := strings.ToLower(key) normalized := strings.ToLower(key)
if policy.PresenceOnly {
if _, isResult := policy.ResultFields[normalized]; isResult {
output[key] = value
return
}
output[key] = map[string]any{"present": true, "length": len(value)}
return
}
if _, isFileName := policy.FileFields[normalized]; isFileName { if _, isFileName := policy.FileFields[normalized]; isFileName {
base := filepath.Base(value) base := filepath.Base(value)
hash := digest([]byte(base)) hash := digest([]byte(base))

View File

@@ -5,10 +5,10 @@ import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"net/url" "net/url"
"strings"
"time" "time"
"github.com/break/junhong_cmp_fiber/pkg/constants" "github.com/break/junhong_cmp_fiber/pkg/constants"
"github.com/break/junhong_cmp_fiber/pkg/sanitizer"
"github.com/bytedance/sonic" "github.com/bytedance/sonic"
"github.com/gofiber/fiber/v2" "github.com/gofiber/fiber/v2"
"go.uber.org/zap" "go.uber.org/zap"
@@ -135,20 +135,7 @@ func sanitizeJSONValue(value any) {
// shouldMaskField 判断字段名是否属于访问日志敏感字段 // shouldMaskField 判断字段名是否属于访问日志敏感字段
func shouldMaskField(key string) bool { func shouldMaskField(key string) bool {
normalized := strings.ToLower(key) return sanitizer.IsForbiddenField(key)
return strings.Contains(normalized, "password") ||
strings.Contains(normalized, "passwd") ||
strings.Contains(normalized, "credential") ||
strings.Contains(normalized, "authorization") ||
strings.Contains(normalized, "cookie") ||
strings.Contains(normalized, "key") ||
strings.Contains(normalized, "url") ||
strings.Contains(normalized, "qr_content") ||
strings.Contains(normalized, "verification_code") ||
strings.Contains(normalized, "sign") ||
strings.Contains(normalized, "nonce") ||
strings.Contains(normalized, "token") ||
strings.Contains(normalized, "secret")
} }
// Middleware 创建 Fiber 日志中间件 // Middleware 创建 Fiber 日志中间件

View File

@@ -92,6 +92,8 @@ func TestSensitiveRouteMatrixNeverLogsRawPayload(t *testing.T) {
}) })
request := httptest.NewRequest("POST", tc.path+"?signature=query-secret", strings.NewReader(tc.body)) request := httptest.NewRequest("POST", tc.path+"?signature=query-secret", strings.NewReader(tc.body))
request.Header.Set("Content-Type", tc.contentType) request.Header.Set("Content-Type", tc.contentType)
request.Header.Set("Authorization", "Bearer header-secret")
request.Header.Set("Cookie", "session=cookie-secret")
response, err := app.Test(request) response, err := app.Test(request)
if err != nil { if err != nil {
t.Fatalf("执行敏感路由请求失败:%v", err) t.Fatalf("执行敏感路由请求失败:%v", err)
@@ -99,7 +101,7 @@ func TestSensitiveRouteMatrixNeverLogsRawPayload(t *testing.T) {
_, _ = io.Copy(io.Discard, response.Body) _, _ = io.Copy(io.Discard, response.Body)
_ = response.Body.Close() _ = response.Body.Close()
logged := output.String() logged := output.String()
for _, secret := range []string{tc.secret, "query-secret", "response-secret"} { for _, secret := range []string{tc.secret, "query-secret", "response-secret", "header-secret", "cookie-secret"} {
if strings.Contains(logged, secret) { if strings.Contains(logged, secret) {
t.Fatalf("敏感路由日志泄露 %q%s", secret, logged) t.Fatalf("敏感路由日志泄露 %q%s", secret, logged)
} }
@@ -131,6 +133,92 @@ func TestSensitiveRouteLongBodyRecordsTruncationWithoutRawSecret(t *testing.T) {
} }
} }
func TestRealConfigurationRoutesOnlyLogSafeSummary(t *testing.T) {
for _, path := range []string{
"/api/admin/system-configs/payment.private_key",
"/api/admin/wechat-configs/1",
} {
t.Run(path, func(t *testing.T) {
var output bytes.Buffer
log := zap.New(zapcore.NewCore(zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()), zapcore.AddSync(&output), zapcore.InfoLevel))
app := fiber.New()
app.Use(MiddlewareWithLogger(log))
app.Put(path, func(c *fiber.Ctx) error {
return c.JSON(fiber.Map{"value": "response-private-material", "status": "ok"})
})
request := httptest.NewRequest("PUT", path, strings.NewReader(`{"value":"request-private-material","config_key":"payment.private_key"}`))
request.Header.Set("Content-Type", "application/json")
response, err := app.Test(request)
if err != nil {
t.Fatalf("执行配置路由请求失败:%v", err)
}
_ = response.Body.Close()
logged := output.String()
for _, secret := range []string{"request-private-material", "response-private-material"} {
if strings.Contains(logged, secret) {
t.Fatalf("配置路由日志泄露 %q%s", secret, logged)
}
}
if !strings.Contains(logged, `"body_policy":"sensitive_config"`) {
t.Fatalf("配置路由未应用安全摘要策略:%s", logged)
}
if !strings.Contains(logged, `\"present\":true`) || !strings.Contains(logged, `\"length\":`) {
t.Fatalf("配置路由未记录字段存在性和长度:%s", logged)
}
})
}
}
func TestLoginRouteDoesNotLogUsernameAndOnlyKeepsPresenceLength(t *testing.T) {
var output bytes.Buffer
log := zap.New(zapcore.NewCore(zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()), zapcore.AddSync(&output), zapcore.InfoLevel))
app := fiber.New()
app.Use(MiddlewareWithLogger(log))
app.Post("/api/auth/login", func(c *fiber.Ctx) error {
return c.JSON(fiber.Map{"username": "visible-user", "success": true})
})
request := httptest.NewRequest("POST", "/api/auth/login", strings.NewReader(`{"username":"visible-user","password":"login-secret"}`))
request.Header.Set("Content-Type", "application/json")
response, err := app.Test(request)
if err != nil {
t.Fatalf("执行登录请求失败:%v", err)
}
_ = response.Body.Close()
logged := output.String()
if strings.Contains(logged, "visible-user") || strings.Contains(logged, "login-secret") {
t.Fatalf("登录路由泄露账号或凭证:%s", logged)
}
if !strings.Contains(logged, `\"present\":true`) || !strings.Contains(logged, `\"length\":`) {
t.Fatalf("登录路由未记录字段存在性和长度:%s", logged)
}
}
func TestPaymentRouteOnlyLogsPresenceLengthAndSafeResult(t *testing.T) {
var output bytes.Buffer
log := zap.New(zapcore.NewCore(zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()), zapcore.AddSync(&output), zapcore.InfoLevel))
app := fiber.New()
app.Use(MiddlewareWithLogger(log))
app.Post("/api/callback/alipay", func(c *fiber.Ctx) error {
return c.JSON(fiber.Map{"payment_no": "PAY-SECRET-1", "amount": 12345, "status": "success"})
})
request := httptest.NewRequest("POST", "/api/callback/alipay", strings.NewReader("order_no=ORDER-SECRET-1&amount=12345&sign=signature-secret"))
request.Header.Set("Content-Type", "application/x-www-form-urlencoded")
response, err := app.Test(request)
if err != nil {
t.Fatalf("执行支付路由请求失败:%v", err)
}
_ = response.Body.Close()
logged := output.String()
for _, value := range []string{"PAY-SECRET-1", "ORDER-SECRET-1", "signature-secret"} {
if strings.Contains(logged, value) {
t.Fatalf("支付路由泄露原值 %q%s", value, logged)
}
}
if !strings.Contains(logged, `\"present\":true`) || !strings.Contains(logged, `\"status\":\"success\"`) {
t.Fatalf("支付路由缺少存在性摘要或安全结果:%s", logged)
}
}
func TestSanitizeInvalidJSONNeverFallsBackToRawBody(t *testing.T) { func TestSanitizeInvalidJSONNeverFallsBackToRawBody(t *testing.T) {
t.Parallel() t.Parallel()

View File

@@ -0,0 +1,76 @@
// Package sanitizer 提供 Access、Audit 与 Integration 共用的敏感字段清理能力。
package sanitizer
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"
"github.com/bytedance/sonic"
)
var forbiddenFragments = []string{
"password", "passwd", "credential", "operation_password", "verification_code", "captcha",
"access_token", "refresh_token", "authorization", "cookie", "secret", "private_key", "public_key",
"encoding_aes_key", "callback_token", "signature", "sign", "nonce", "media_id", "signed_url",
"private_url", "qr_content", "id_card", "identity_number",
}
// IsForbiddenField 判断字段是否禁止进入普通日志、审计或外部交互摘要。
func IsForbiddenField(key string) bool {
normalized := strings.ToLower(strings.NewReplacer("-", "_", ".", "_").Replace(key))
for _, fragment := range forbiddenFragments {
if strings.Contains(normalized, fragment) {
return true
}
}
if strings.HasSuffix(normalized, "_key") || normalized == "key" || strings.HasSuffix(normalized, "_url") || normalized == "url" {
return true
}
return false
}
// MarshalSummary 递归删除禁止字段并返回 sonic 编码的安全 JSON。
func MarshalSummary(value any) ([]byte, error) {
if value == nil {
return nil, nil
}
encoded, err := sonic.Marshal(value)
if err != nil {
return nil, err
}
var normalized any
if err := sonic.Unmarshal(encoded, &normalized); err != nil {
return nil, err
}
RemoveForbiddenFields(normalized)
return sonic.Marshal(normalized)
}
// RemoveForbiddenFields 原地递归删除 Map 或数组中的禁止字段。
func RemoveForbiddenFields(value any) {
switch typed := value.(type) {
case map[string]any:
for key, item := range typed {
if IsForbiddenField(key) {
delete(typed, key)
continue
}
RemoveForbiddenFields(item)
}
case []any:
for _, item := range typed {
RemoveForbiddenFields(item)
}
}
}
// TextSummary 将不可信外部文本转换为不可逆大小和哈希摘要。
func TextSummary(value string) string {
if value == "" {
return ""
}
sum := sha256.Sum256([]byte(value))
return fmt.Sprintf("外部文本摘要 bytes=%d sha256=%s", len(value), hex.EncodeToString(sum[:8]))
}