// Command audit-retention-simulate 在测试环境演练逐日归档与清理边界。 package main import ( "context" "crypto/sha256" "fmt" "os" "path/filepath" "strings" "time" "github.com/bytedance/sonic" "go.uber.org/zap" "gorm.io/datatypes" "gorm.io/gorm" auditarchive "github.com/break/junhong_cmp_fiber/internal/application/auditarchive" "github.com/break/junhong_cmp_fiber/internal/model" "github.com/break/junhong_cmp_fiber/pkg/config" "github.com/break/junhong_cmp_fiber/pkg/constants" "github.com/break/junhong_cmp_fiber/pkg/database" logpkg "github.com/break/junhong_cmp_fiber/pkg/logger" "github.com/break/junhong_cmp_fiber/pkg/storage" ) const ( simulationMonth = "2001-02" simulationInstance = "retention-simulation-2001-02" simulationPrefix = "retention-simulation-2001-02" ) type simulationSummary struct { Month string `json:"month"` Days int `json:"days"` ArchiveRuns int64 `json:"archive_runs"` SuccessfulRuns int64 `json:"successful_runs"` FinalIntegrationRuns int64 `json:"final_integration_runs"` MaxRevision int `json:"max_revision"` MaxAttemptCount int `json:"max_attempt_count"` CompressionRatio float64 `json:"compression_ratio"` DryRunEventCount int64 `json:"dry_run_event_count"` DryRunResourceCount int64 `json:"dry_run_resource_count"` DryRunIntegrationCount int64 `json:"dry_run_integration_count"` EstimatedCleanupBatches int64 `json:"estimated_cleanup_batches"` CleanupMarkersBefore int64 `json:"cleanup_markers_before"` TargetRowsAfterCleanup int64 `json:"target_rows_after_cleanup"` BoundaryRowsAfterCleanup int64 `json:"boundary_rows_after_cleanup"` CleanupMarkersAfter int64 `json:"cleanup_markers_after"` RetentionAuditRecorded bool `json:"retention_audit_recorded"` } func main() { if err := run(context.Background()); err != nil { fmt.Fprintln(os.Stderr, "归档留存仿真失败:", err) os.Exit(1) } } func run(ctx context.Context) error { cfg, err := config.Load() if err != nil { return err } if !strings.Contains(strings.ToLower(cfg.Database.DBName), "test") || os.Getenv("JUNHONG_AUDIT_RETENTION_SIMULATION_CONFIRM") != cfg.Database.DBName { return fmt.Errorf("仅允许显式确认的测试数据库,当前数据库为 %q", cfg.Database.DBName) } logger := zap.NewNop() if err := os.MkdirAll(filepath.Dir(cfg.Logging.RetentionLog.Filename), 0755); err != nil { return err } retentionLogger, syncRetentionLogger := logpkg.NewRetentionLogger(cfg.Logging.Level, logpkg.LogRotationConfig{Filename: cfg.Logging.RetentionLog.Filename, MaxSize: cfg.Logging.RetentionLog.MaxSize, MaxBackups: cfg.Logging.RetentionLog.MaxBackups, MaxAge: cfg.Logging.RetentionLog.MaxAge, Compress: cfg.Logging.RetentionLog.Compress}) defer func() { _ = syncRetentionLogger() }() db, err := database.InitPostgreSQL(&cfg.Database, logger) if err != nil { return err } sqlDB, err := db.DB() if err != nil { return err } defer sqlDB.Close() provider, err := storage.NewS3Provider(&cfg.Storage) if err != nil { return err } service, err := auditarchive.NewService(db, provider, simulationInstance) if err != nil { return err } location, err := time.LoadLocation(constants.AuditArchiveTimezone) if err != nil { return err } monthStart, _ := time.ParseInLocation("2006-01", simulationMonth, location) monthEnd := monthStart.AddDate(0, 1, 0) if os.Getenv("JUNHONG_AUDIT_RETENTION_SIMULATION_RESUME") == "true" { if err := assertReadyToResume(ctx, db, monthStart, monthEnd); err != nil { return err } } else { if err := assertEmptyTarget(ctx, db, monthStart, monthEnd); err != nil { return err } if err := seedSimulation(ctx, db, monthStart, monthEnd); err != nil { return err } if err := archiveMonth(ctx, db, service, monthStart, monthEnd); err != nil { return err } } var dryRun auditarchive.RetentionResult for date := monthStart; date.Before(monthEnd); date = date.AddDate(0, 0, 1) { result, retainErr := service.RetainDate(ctx, date, false) if retainErr != nil { return fmt.Errorf("逐日只读演练失败: %w", retainErr) } retentionLogger.Info("日留存仿真只读校验通过", zap.String("archive_date", result.ArchiveDate)) dryRun.EventCount += result.EventCount dryRun.ResourceCount += result.ResourceCount dryRun.IntegrationCount += result.IntegrationCount dryRun.EstimatedBatches += result.EstimatedBatches } summary, err := collectBeforeCleanup(ctx, db, monthStart, monthEnd, dryRun) if err != nil { return err } if summary.CleanupMarkersBefore != 0 { return fmt.Errorf("只归档模式写入了 %d 个清理断点", summary.CleanupMarkersBefore) } for date := monthStart; date.Before(monthEnd); date = date.AddDate(0, 0, 1) { result, retainErr := service.RetainDate(ctx, date, true) if retainErr != nil { return fmt.Errorf("隔离测试库逐日物理清理演练失败: %w", retainErr) } retentionLogger.Info("日留存仿真物理清理完成", zap.String("archive_date", result.ArchiveDate)) } if err := collectAfterCleanup(ctx, db, monthStart, monthEnd, &summary); err != nil { return err } if summary.TargetRowsAfterCleanup != 0 || summary.BoundaryRowsAfterCleanup != 6 || summary.CleanupMarkersAfter != int64(summary.Days*2) { return fmt.Errorf("清理范围复核失败: target=%d boundary=%d markers=%d", summary.TargetRowsAfterCleanup, summary.BoundaryRowsAfterCleanup, summary.CleanupMarkersAfter) } if err := syncRetentionLogger(); err != nil { return fmt.Errorf("刷新日留存仿真日志失败: %w", err) } if _, err := os.Stat(cfg.Logging.RetentionLog.Filename); err != nil { return fmt.Errorf("独立日留存日志未生成: %w", err) } encoded, _ := sonic.MarshalIndent(summary, "", " ") fmt.Println(string(encoded)) return nil } func assertReadyToResume(ctx context.Context, db *gorm.DB, start, end time.Time) error { var events, integrations, runs, successful, final int64 if err := db.WithContext(ctx).Model(&model.AuditEvent{}).Where("created_at >= ? AND created_at < ?", start, end).Count(&events).Error; err != nil { return err } if err := db.WithContext(ctx).Model(&model.IntegrationLog{}).Where("created_at >= ? AND created_at < ?", start, end).Count(&integrations).Error; err != nil { return err } query := db.WithContext(ctx).Model(&model.LogArchiveRun{}). Where("archive_date >= ? AND archive_date < ? AND instance_id = ?", start.Format(time.DateOnly), end.Format(time.DateOnly), simulationInstance) if err := query.Count(&runs).Error; err != nil { return err } if err := query.Where("status = ?", constants.ArchiveStatusSuccess).Count(&successful).Error; err != nil { return err } if err := db.WithContext(ctx).Model(&model.LogArchiveRun{}). Where("archive_date >= ? AND archive_date < ? AND instance_id = ? AND source = ? AND is_final", start.Format(time.DateOnly), end.Format(time.DateOnly), simulationInstance, constants.IntegrationArchiveSource). Count(&final).Error; err != nil { return err } days := int64(end.Sub(start).Hours() / 24) if events != days || integrations != days || runs != days*2 || successful != runs || final != days { return fmt.Errorf("现有仿真状态不可安全续跑: events=%d integrations=%d runs=%d successful=%d final=%d", events, integrations, runs, successful, final) } return nil } func assertEmptyTarget(ctx context.Context, db *gorm.DB, start, end time.Time) error { var events, integrations, runs int64 if err := db.WithContext(ctx).Model(&model.AuditEvent{}).Where("created_at >= ? AND created_at < ?", start, end).Count(&events).Error; err != nil { return err } if err := db.WithContext(ctx).Model(&model.IntegrationLog{}).Where("created_at >= ? AND created_at < ?", start, end).Count(&integrations).Error; err != nil { return err } if err := db.WithContext(ctx).Model(&model.LogArchiveRun{}).Where("archive_date >= ? AND archive_date < ?", start.Format(time.DateOnly), end.Format(time.DateOnly)).Count(&runs).Error; err != nil { return err } if events != 0 || integrations != 0 || runs != 0 { return fmt.Errorf("仿真目标月已有数据,拒绝清理: events=%d integrations=%d runs=%d", events, integrations, runs) } return nil } func seedSimulation(ctx context.Context, db *gorm.DB, start, end time.Time) error { return db.WithContext(ctx).Transaction(func(tx *gorm.DB) error { for date := start; date.Before(end); date = date.AddDate(0, 0, 1) { result := constants.IntegrationResultSuccess if date.Equal(start) { result = constants.IntegrationResultPending } if err := seedFact(tx, date.Add(12*time.Hour), date.Format("20060102"), result); err != nil { return err } } if err := seedFact(tx, start.Add(-time.Second), "before", constants.IntegrationResultSuccess); err != nil { return err } return seedFact(tx, end, "after", constants.IntegrationResultSuccess) }) } func seedFact(tx *gorm.DB, at time.Time, suffix, integrationResult string) error { marker := simulationPrefix + "-" + suffix hash := fmt.Sprintf("%x", sha256.Sum256([]byte(marker))) event := model.AuditEvent{ EventID: "evt_" + marker, OccurredAt: at, Category: constants.AuditCategoryReliability, ActionCode: constants.AuditActionLogRetentionCleanup, ActionName: "归档留存仿真", Summary: "归档留存边界仿真数据", ActorKind: constants.AuditActorSystemTask, ActorID: simulationInstance, ActorName: "归档留存仿真", Source: constants.AuditSourceWorker, ScopeType: constants.AuditScopePlatform, Result: constants.AuditResultSuccess, RiskLevel: constants.AuditRiskLow, Metadata: datatypes.JSON([]byte(`{"simulation":true}`)), ContentHash: hash, CreatedAt: at, } if err := tx.Create(&event).Error; err != nil { return err } resourceID := marker resource := model.AuditEventResource{ AuditEventID: event.ID, ResourceType: constants.AuditResourceLogArchiveMonth, ResourceID: &resourceID, ResourceKey: marker, DisplayName: marker, Relation: constants.AuditResourceRelationPrimary, Role: constants.AuditResourceRoleRetentionMonth, IdentitySnapshot: datatypes.JSON([]byte(`{"simulation":true}`)), BeforeData: datatypes.JSON([]byte(`{}`)), AfterData: datatypes.JSON([]byte(`{}`)), SubjectVisibility: constants.AuditSubjectInternalOnly, SubjectData: datatypes.JSON([]byte(`{}`)), CreatedAt: at, } if err := tx.Create(&resource).Error; err != nil { return err } resourceType, resourceKey := constants.AuditResourceLogArchiveMonth, marker integration := model.IntegrationLog{ IntegrationID: "int_" + marker, Provider: "retention_simulation", Direction: "outbound", Operation: "archive_boundary", ResourceType: &resourceType, ResourceID: &resourceID, ResourceKey: &resourceKey, Attempt: 1, Result: integrationResult, ContentHash: hash, RequestSummary: datatypes.JSON([]byte(`{"simulation":true}`)), ResponseSummary: datatypes.JSON([]byte(`{}`)), Metadata: datatypes.JSON([]byte(`{"simulation":true}`)), CreatedAt: at, UpdatedAt: at, } return tx.Create(&integration).Error } func archiveMonth(ctx context.Context, db *gorm.DB, service *auditarchive.Service, start, end time.Time) error { for date := start; date.Before(end); date = date.AddDate(0, 0, 1) { if err := service.ArchiveDate(ctx, date); err != nil { return err } if err := service.ArchiveIntegrationDate(ctx, date); err != nil { return err } } if err := db.WithContext(ctx).Model(&model.LogArchiveRun{}). Where("source = ? AND archive_date = ? AND instance_id = ?", constants.AuditArchiveSource, start.Format(time.DateOnly), simulationInstance). Update("sha256", strings.Repeat("0", 64)).Error; err != nil { return err } if err := service.ArchiveDate(ctx, start); err != nil { return fmt.Errorf("Audit 故障重试演练失败: %w", err) } if err := db.WithContext(ctx).Model(&model.IntegrationLog{}). Where("integration_id = ?", "int_"+simulationPrefix+"-"+start.Format("20060102")). Updates(map[string]any{"result": constants.IntegrationResultSuccess, "updated_at": time.Now()}).Error; err != nil { return err } for date := start; date.Before(end); date = date.AddDate(0, 0, 1) { if err := service.FinalizeIntegrationDate(ctx, date); err != nil { return err } } return nil } func collectBeforeCleanup(ctx context.Context, db *gorm.DB, start, end time.Time, dryRun auditarchive.RetentionResult) (simulationSummary, error) { summary := simulationSummary{ Month: simulationMonth, Days: int(end.Sub(start).Hours() / 24), DryRunEventCount: dryRun.EventCount, DryRunResourceCount: dryRun.ResourceCount, DryRunIntegrationCount: dryRun.IntegrationCount, EstimatedCleanupBatches: dryRun.EstimatedBatches, } var compressed, uncompressed int64 err := db.WithContext(ctx).Model(&model.LogArchiveRun{}). Where("archive_date >= ? AND archive_date < ? AND instance_id = ?", start.Format(time.DateOnly), end.Format(time.DateOnly), simulationInstance). Select("COUNT(*) AS archive_runs, COUNT(*) FILTER (WHERE status = 'success') AS successful_runs, COUNT(*) FILTER (WHERE source = 'integration' AND is_final) AS final_integration_runs, COALESCE(MAX(revision), 0) AS max_revision, COALESCE(MAX(attempt_count), 0) AS max_attempt_count, COALESCE(SUM(compressed_bytes), 0) AS compressed, COALESCE(SUM(uncompressed_bytes), 0) AS uncompressed, COUNT(*) FILTER (WHERE cleanup_started_at IS NOT NULL OR cleaned_at IS NOT NULL) AS cleanup_markers_before"). Row().Scan(&summary.ArchiveRuns, &summary.SuccessfulRuns, &summary.FinalIntegrationRuns, &summary.MaxRevision, &summary.MaxAttemptCount, &compressed, &uncompressed, &summary.CleanupMarkersBefore) if err != nil { return summary, err } if uncompressed > 0 { summary.CompressionRatio = float64(compressed) / float64(uncompressed) } return summary, nil } func collectAfterCleanup(ctx context.Context, db *gorm.DB, start, end time.Time, summary *simulationSummary) error { var events, resources, integrations int64 if err := db.WithContext(ctx).Model(&model.AuditEvent{}).Where("created_at >= ? AND created_at < ?", start, end).Count(&events).Error; err != nil { return err } if err := db.WithContext(ctx).Model(&model.AuditEventResource{}).Joins("JOIN tb_audit_event e ON e.id = tb_audit_event_resource.audit_event_id").Where("e.created_at >= ? AND e.created_at < ?", start, end).Count(&resources).Error; err != nil { return err } if err := db.WithContext(ctx).Model(&model.IntegrationLog{}).Where("created_at >= ? AND created_at < ?", start, end).Count(&integrations).Error; err != nil { return err } summary.TargetRowsAfterCleanup = events + resources + integrations if err := db.WithContext(ctx).Model(&model.AuditEvent{}).Where("event_id IN ?", []string{"evt_" + simulationPrefix + "-before", "evt_" + simulationPrefix + "-after"}).Count(&events).Error; err != nil { return err } if err := db.WithContext(ctx).Model(&model.AuditEventResource{}).Where("resource_key IN ?", []string{simulationPrefix + "-before", simulationPrefix + "-after"}).Count(&resources).Error; err != nil { return err } if err := db.WithContext(ctx).Model(&model.IntegrationLog{}).Where("integration_id IN ?", []string{"int_" + simulationPrefix + "-before", "int_" + simulationPrefix + "-after"}).Count(&integrations).Error; err != nil { return err } summary.BoundaryRowsAfterCleanup = events + resources + integrations if err := db.WithContext(ctx).Model(&model.LogArchiveRun{}). Where("archive_date >= ? AND archive_date < ? AND instance_id = ? AND cleanup_started_at IS NOT NULL AND cleaned_at IS NOT NULL", start.Format(time.DateOnly), end.Format(time.DateOnly), simulationInstance). Count(&summary.CleanupMarkersAfter).Error; err != nil { return err } return nil }