feat: 轮询系统重构(分片队列 + 停复机统一 + Handler 拆分)
All checks were successful
构建并部署到测试环境(无 SSH) / build-and-deploy (push) Successful in 9m46s

【核心变更】

1. 停复机逻辑统一(StopResumeService)
   - 新增 EvaluateAndAct 统一入口,封装三条件停复机判断
   - 停机条件:无套餐(no_package) / 流量耗尽(traffic_exhausted) / 未实名(not_realname)
   - 复机条件:stop_reason 合规 + 有套餐且未耗尽 + 已实名或行业卡
   - 修复设备套餐 Bug:hasValidPackage 按 device_id 查套餐,而非仅 iot_card_id
   - 设备维度停复机加幂等锁(Redis SetNX,TTL 30s),防止多卡并发重复调 Gateway

2. Redis 分片队列(PollingQueueManager)
   - 新建 queue_manager.go,封装所有轮询 Redis 操作
   - 16 分片 Sorted Set,Key 格式:polling:shard:{shardID}:queue:{taskType}
   - Lua 脚本原子出队(ZRANGEBYSCORE + 分批 ZREM),消除竞态窗口
   - 新增背压检测:队列深度超 50 万时 Scheduler 跳过该分片
   - RemoveFromAllQueues 覆盖 4 种任务类型(含 protect)

3. Handler 拆分(polling_handler.go 1360行 → 5个专注文件)
   - polling_base.go:共享基类(并发控制/卡缓存/重入队)
   - polling_realname_handler.go:实名采集,实名 0→1 时立即触发复机
   - polling_carddata_handler.go:流量采集,保留跨月边界检测逻辑
   - polling_package_handler.go:套餐采集,委托 EvaluateAndAct 决策
   - polling_protect_handler.go:保护期一致性检查,保护期内强制修正

4. 配置管理(PollingConfigManager)
   - 新建 config_manager.go,从 scheduler.go 提取配置职责
   - 内存缓存 + 5 分钟定时刷新,刷新失败保留原缓存
   - 修复 getCardCondition:停机卡返回 suspended,不再错配 activated 配置

5. 渐进式初始化(CardInitializer)
   - 新建 initializer.go,分批加载(每批 10 万),批次间 sleep 500ms
   - 过滤 enable_polling=false 的卡,初始化完成前 Scheduler 不出队

6. 卡生命周期服务(PollingLifecycleService)
   - 新建 lifecycle_service.go,替代已删除的 callbacks.go 和 api_callback.go
   - OnCardCreated/OnCardEnabled/OnCardStatusChanged 入队前检查 enable_polling

7. Scheduler 精简(1000+行 → 227行)
   - 保留纯调度循环:scheduleLoop + processShardSchedule + enqueueBatch
   - 保留每 10 秒触发套餐过期检测和流量重置
   - 移除所有 DB 操作、配置加载、卡初始化逻辑

8. 轮询管控 API(enable_polling)
   - 新增 PUT /api/admin/assets/:id/polling-status 接口
   - 支持对设备/卡维度开关轮询,关闭后从所有分片队列移除

9. 数据库迁移
   - 000103:tb_device 新增 enable_polling 字段(boolean, NOT NULL, DEFAULT true)
   - 000104:新增 suspended 轮询配置,为 activated 配置补全 protect_check_interval

【文件统计】
- 新增:19 个文件(handler × 5、polling 组件 × 4、迁移 × 3 等)
- 修改:20 个文件(bootstrap 注入、store 接口、monitoring 适配分片等)
- 删除:3 个文件(polling_handler.go、callbacks.go、api_callback.go)

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
This commit is contained in:
2026-04-07 12:27:04 +08:00
parent 10fcc0b3c9
commit 434a8b0349
62 changed files with 7496 additions and 3023 deletions

View File

@@ -2,144 +2,81 @@ package polling
import (
"context"
"encoding/json"
"sync"
"sync/atomic"
"time"
"github.com/hibiken/asynq"
"github.com/redis/go-redis/v9"
"go.uber.org/zap"
"gorm.io/gorm"
"github.com/break/junhong_cmp_fiber/internal/model"
packagepkg "github.com/break/junhong_cmp_fiber/internal/service/package"
"github.com/break/junhong_cmp_fiber/internal/store/postgres"
"github.com/break/junhong_cmp_fiber/pkg/constants"
)
// InitProgress 已迁移到 initializer.go
// Scheduler 轮询调度器
// 负责管理 IoT 卡的定期检查任务(实名、流量、套餐)
// 职责:读取分片 Sorted Set 中到期的卡,生成 Asynq 任务
// 不再负责:配置加载、卡初始化(分别由 PollingConfigManager、PollingInitializer 负责)
type Scheduler struct {
db *gorm.DB
redis *redis.Client
queueClient *asynq.Client
logger *zap.Logger
configStore *postgres.PollingConfigStore
iotCardStore *postgres.IotCardStore
concurrencyStore *postgres.PollingConcurrencyConfigStore
redis *redis.Client
queueClient *asynq.Client
logger *zap.Logger
queueMgr *PollingQueueManager
configMgr *PollingConfigManager
cfg *SchedulerConfig // 启动时固定,避免每次调度重新创建
// 任务 19: 套餐激活检查处理器
packageActivationHandler *PackageActivationHandler
// 任务 20: 流量重置调度处理器
dataResetHandler *DataResetHandler
dataResetHandler *DataResetHandler
// 配置缓存
configCache []*model.PollingConfig
configCacheLock sync.RWMutex
configCacheTime time.Time
// 初始化状态
initProgress *InitProgress
initCompleted atomic.Bool
// 控制信号
stopChan chan struct{}
wg sync.WaitGroup
}
// InitProgress 初始化进度
type InitProgress struct {
mu sync.RWMutex
TotalCards int64 `json:"total_cards"` // 总卡数
LoadedCards int64 `json:"loaded_cards"` // 已加载卡数
StartTime time.Time `json:"start_time"` // 开始时间
LastBatchTime time.Time `json:"last_batch_time"` // 最后一批处理时间
Status string `json:"status"` // 状态: pending, running, completed, failed
ErrorMessage string `json:"error_message"` // 错误信息
}
// SchedulerConfig 调度器配置
// 设计目标:支持一亿张卡规模
type SchedulerConfig struct {
ScheduleInterval time.Duration // 调度循环间隔(默认 1 秒,支持高吞吐)
InitBatchSize int // 初始化每批加载数量(默认 100000
InitBatchSleepDuration time.Duration // 初始化批次间休眠时间(默认 500ms
ConfigCacheTTL time.Duration // 配置缓存 TTL默认 5 分钟)
CardCacheTTL time.Duration // 卡信息缓存 TTL默认 7 天)
ScheduleBatchSize int // 每次调度取出的卡数(默认 50000
MaxManualBatchSize int // 手动触发每次处理数量(默认 1000
ScheduleInterval time.Duration
MaxManualBatchSize int
ScheduleBatchSize int
}
// DefaultSchedulerConfig 默认调度器配置
// 单 Worker 设计吞吐50000 张/秒,支持多 Worker 水平扩展
func DefaultSchedulerConfig() *SchedulerConfig {
return &SchedulerConfig{
ScheduleInterval: 1 * time.Second, // 1秒调度一次提高响应速度
InitBatchSize: 100000, // 10万张/批初始化
InitBatchSleepDuration: 500 * time.Millisecond, // 500ms 间隔,加快初始化
ConfigCacheTTL: 5 * time.Minute,
CardCacheTTL: 7 * 24 * time.Hour,
ScheduleBatchSize: 50000, // 每次取 5 万张,每秒可调度 5 万张
MaxManualBatchSize: 1000, // 手动触发每次处理 1000 张
ScheduleInterval: 1 * time.Second,
MaxManualBatchSize: 1000,
ScheduleBatchSize: constants.PollingDequeueMaxBatchSize,
}
}
// NewScheduler 创建调度器实例
// NewScheduler 创建调度器
func NewScheduler(
db *gorm.DB,
redisClient *redis.Client,
queueClient *asynq.Client,
queueMgr *PollingQueueManager,
configMgr *PollingConfigManager,
logger *zap.Logger,
packageActivationHandler *PackageActivationHandler,
dataResetHandler *DataResetHandler,
) *Scheduler {
return &Scheduler{
db: db,
redis: redisClient,
queueClient: queueClient,
queueMgr: queueMgr,
configMgr: configMgr,
logger: logger,
configStore: postgres.NewPollingConfigStore(db),
iotCardStore: postgres.NewIotCardStore(db, redisClient),
concurrencyStore: postgres.NewPollingConcurrencyConfigStore(db),
packageActivationHandler: NewPackageActivationHandler(db, redisClient, queueClient, nil, nil, logger),
dataResetHandler: NewDataResetHandler(nil, logger), // ResetService 需要通过 SetResetService 注入
initProgress: &InitProgress{
Status: "pending",
},
stopChan: make(chan struct{}),
cfg: DefaultSchedulerConfig(),
stopChan: make(chan struct{}),
packageActivationHandler: packageActivationHandler,
dataResetHandler: dataResetHandler,
}
}
// Start 启动调度
// 快速启动10秒内完成配置加载和调度器启动
// Start 启动调度循环(快速启动,配置加载和初始化由外部完成)
func (s *Scheduler) Start(ctx context.Context) error {
startTime := time.Now()
s.logger.Info("轮询调度器启动中...")
// 1. 加载轮询配置到缓存
if err := s.loadConfigs(ctx); err != nil {
s.logger.Error("加载轮询配置失败", zap.Error(err))
return err
}
s.logger.Info("轮询配置已加载", zap.Int("config_count", len(s.configCache)))
// 2. 初始化并发控制配置
if err := s.initConcurrencyConfigs(ctx); err != nil {
s.logger.Warn("初始化并发控制配置失败,使用默认值", zap.Error(err))
}
// 3. 启动调度循环(非阻塞)
s.wg.Add(1)
go s.scheduleLoop(ctx)
// 4. 启动后台渐进式初始化(非阻塞)
s.wg.Add(1)
go s.progressiveInit(ctx)
elapsed := time.Since(startTime)
s.logger.Info("轮询调度器已启动",
zap.Duration("startup_time", elapsed),
zap.Bool("fast_start", elapsed < 10*time.Second))
s.logger.Info("轮询调度器已启动")
return nil
}
@@ -151,115 +88,116 @@ func (s *Scheduler) Stop() {
s.logger.Info("轮询调度器已停止")
}
// loadConfigs 加载轮询配置缓存
func (s *Scheduler) loadConfigs(ctx context.Context) error {
configs, err := s.configStore.ListEnabled(ctx)
if err != nil {
return err
// RefreshConfigs 刷新配置缓存
func (s *Scheduler) RefreshConfigs(ctx context.Context) error {
if s.configMgr == nil {
return nil
}
s.configCacheLock.Lock()
s.configCache = configs
s.configCacheTime = time.Now()
s.configCacheLock.Unlock()
// 同步到 Redis 缓存
return s.syncConfigsToRedis(ctx, configs)
return s.configMgr.Load(ctx)
}
// syncConfigsToRedis 同步配置到 Redis
func (s *Scheduler) syncConfigsToRedis(ctx context.Context, configs []*model.PollingConfig) error {
key := constants.RedisPollingConfigsCacheKey()
// 序列化配置列表为 JSON
configData := make([]interface{}, 0, len(configs)*2)
for _, cfg := range configs {
jsonData, err := json.Marshal(cfg)
if err != nil {
s.logger.Warn("序列化轮询配置失败", zap.Uint("config_id", cfg.ID), zap.Error(err))
continue
}
configData = append(configData, cfg.ID, string(jsonData))
// SetStopResumeCallback 注入停复机回调(在 Start 前调用)
func (s *Scheduler) SetStopResumeCallback(callback packagepkg.StopResumeCallback) {
if s.packageActivationHandler != nil {
s.packageActivationHandler.stopResumeCallback = callback
}
if len(configData) > 0 {
pipe := s.redis.Pipeline()
pipe.Del(ctx, key)
// 使用 HSET 存储配置
pipe.HSet(ctx, key, configData...)
pipe.Expire(ctx, key, 24*time.Hour)
_, err := pipe.Exec(ctx)
return err
}
return nil
}
// initConcurrencyConfigs 初始化并发控制配置到 Redis
func (s *Scheduler) initConcurrencyConfigs(ctx context.Context) error {
configs, err := s.concurrencyStore.List(ctx)
if err != nil {
return err
}
for _, cfg := range configs {
key := constants.RedisPollingConcurrencyConfigKey(cfg.TaskType)
if err := s.redis.Set(ctx, key, cfg.MaxConcurrency, 0).Err(); err != nil {
s.logger.Warn("设置并发配置失败",
zap.String("task_type", cfg.TaskType),
zap.Error(err))
}
}
return nil
}
// scheduleLoop 调度循环
// 每 10 秒执行一次,从 Redis Sorted Set 获取到期的卡,生成 Asynq 任务
func (s *Scheduler) scheduleLoop(ctx context.Context) {
defer s.wg.Done()
config := DefaultSchedulerConfig()
ticker := time.NewTicker(config.ScheduleInterval)
ticker := time.NewTicker(s.cfg.ScheduleInterval)
activationTicker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
defer activationTicker.Stop()
s.logger.Info("调度循环已启动", zap.Duration("interval", config.ScheduleInterval))
s.logger.Info("调度循环已启动", zap.Duration("interval", s.cfg.ScheduleInterval))
for {
select {
case <-s.stopChan:
s.logger.Info("调度循环收到停止信号")
return
case <-ctx.Done():
s.logger.Info("调度循环收到 ctx 取消信号")
return
case <-ticker.C:
s.processSchedule(ctx)
s.processShardSchedule(ctx)
case <-activationTicker.C:
s.processActivationTasks(ctx)
}
}
}
// processSchedule 处理一次调度
func (s *Scheduler) processSchedule(ctx context.Context) {
now := time.Now().Unix()
// processShardSchedule 处理手动队列和分片定时队列(每 1 秒触发)
// 使用 90% 的 tick 间隔作为超时,确保单分片 Redis 挂起时不阻塞下一个 tick
func (s *Scheduler) processShardSchedule(ctx context.Context) {
for _, taskType := range allTaskTypes {
s.processManualQueue(ctx, taskType, s.cfg.MaxManualBatchSize)
}
// 1. 优先处理手动触发队列
s.processManualQueue(ctx, constants.TaskTypePollingRealname)
s.processManualQueue(ctx, constants.TaskTypePollingCarddata)
s.processManualQueue(ctx, constants.TaskTypePollingPackage)
s.processManualQueue(ctx, constants.TaskTypePollingProtect)
if s.queueMgr == nil {
return
}
// 2. 处理定时队列
s.processTimedQueue(ctx, constants.RedisPollingQueueRealnameKey(), constants.TaskTypePollingRealname, now)
s.processTimedQueue(ctx, constants.RedisPollingQueueCarddataKey(), constants.TaskTypePollingCarddata, now)
s.processTimedQueue(ctx, constants.RedisPollingQueuePackageKey(), constants.TaskTypePollingPackage, now)
s.processTimedQueue(ctx, constants.RedisPollingQueueProtectKey(), constants.TaskTypePollingProtect, now)
timeout := s.cfg.ScheduleInterval * 9 / 10
tickCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
// 任务 19.6: 套餐激活检查(每次调度都执行,内部会限流)
var wg sync.WaitGroup
for shardID := 0; shardID < s.queueMgr.shardCount; shardID++ {
wg.Add(1)
go func(sid int) {
defer wg.Done()
defer func() {
if r := recover(); r != nil {
s.logger.Error("分片处理 panic已恢复",
zap.Int("shard_id", sid), zap.Any("panic", r))
}
}()
s.processOneShard(tickCtx, sid)
}(shardID)
}
wg.Wait()
}
// processOneShard 处理单个分片的所有任务类型出队并推入 Asynq
func (s *Scheduler) processOneShard(ctx context.Context, shardID int) {
for _, taskType := range allTaskTypes {
depth, err := s.queueMgr.GetQueueDepth(ctx, shardID, taskType)
if err != nil {
s.logger.Warn("获取分片队列深度失败",
zap.Int("shard_id", shardID), zap.String("task_type", taskType), zap.Error(err))
} else if depth > constants.PollingBackpressureThreshold {
s.logger.Debug("背压:分片队列积压过深,跳过本轮出队",
zap.Int("shard_id", shardID), zap.String("task_type", taskType), zap.Int64("depth", depth))
continue
}
entries, err := s.queueMgr.DequeueReady(ctx, shardID, taskType, s.cfg.ScheduleBatchSize)
if err != nil {
s.logger.Error("分片出队失败",
zap.Int("shard_id", shardID), zap.String("task_type", taskType), zap.Error(err))
continue
}
if len(entries) > 0 {
cardIDs := make([]string, len(entries))
for i, e := range entries {
cardIDs[i] = formatUint(e.CardID)
}
s.enqueueBatch(ctx, taskType, cardIDs)
}
}
}
// processActivationTasks 套餐激活检查和流量重置调度(每 10 秒触发)
func (s *Scheduler) processActivationTasks(ctx context.Context) {
if s.packageActivationHandler != nil {
if err := s.packageActivationHandler.HandlePackageActivationCheck(ctx); err != nil {
s.logger.Warn("套餐激活检查失败", zap.Error(err))
}
}
// 任务 20.6: 流量重置调度(每次调度都执行,内部会限流)
if s.dataResetHandler != nil {
if err := s.dataResetHandler.HandleDataReset(ctx); err != nil {
s.logger.Warn("流量重置调度失败", zap.Error(err))
@@ -268,497 +206,43 @@ func (s *Scheduler) processSchedule(ctx context.Context) {
}
// processManualQueue 处理手动触发队列
// 优化:批量读取和提交,提高吞吐
func (s *Scheduler) processManualQueue(ctx context.Context, taskType string) {
config := DefaultSchedulerConfig()
func (s *Scheduler) processManualQueue(ctx context.Context, taskType string, maxBatch int) {
key := constants.RedisPollingManualQueueKey(taskType)
// 批量读取手动触发任务
cardIDs := make([]string, 0, config.MaxManualBatchSize)
for i := 0; i < config.MaxManualBatchSize; i++ {
cardIDStr, err := s.redis.LPop(ctx, key).Result()
if err != nil {
if err != redis.Nil {
s.logger.Error("读取手动触发队列失败",
zap.String("task_type", taskType),
zap.Error(err))
}
break
}
cardIDs = append(cardIDs, cardIDStr)
}
// 批量提交任务
if len(cardIDs) > 0 {
s.enqueueBatch(ctx, taskType, cardIDs, true)
cardIDs, err := s.redis.LPopCount(ctx, key, maxBatch).Result()
if err != nil || len(cardIDs) == 0 {
return
}
s.enqueueBatch(ctx, taskType, cardIDs)
}
// processTimedQueue 处理定时队列
// 优化:支持大批量处理,每次最多取 ScheduleBatchSize 张卡
func (s *Scheduler) processTimedQueue(ctx context.Context, queueKey, taskType string, now int64) {
config := DefaultSchedulerConfig()
// 获取所有到期的卡score <= now
// 使用 ZRANGEBYSCORE 获取,每次最多取 ScheduleBatchSize 张
cardIDs, err := s.redis.ZRangeByScore(ctx, queueKey, &redis.ZRangeBy{
Min: "-inf",
Max: formatInt64(now),
Count: int64(config.ScheduleBatchSize),
}).Result()
if err != nil {
if err != redis.Nil {
s.logger.Error("读取定时队列失败",
zap.String("queue_key", queueKey),
zap.Error(err))
// enqueueBatch 批量提交任务到 Asynq 队列;入队失败时回退至分片队列防止卡永久丢失
func (s *Scheduler) enqueueBatch(ctx context.Context, taskType string, cardIDs []string) {
for _, cardID := range cardIDs {
payload := map[string]interface{}{
"card_id": cardID,
"is_manual": false,
"timestamp": time.Now().Unix(),
}
return
}
if len(cardIDs) == 0 {
return
}
// 只在数量较大时打印日志,避免日志过多
if len(cardIDs) >= 1000 {
s.logger.Info("处理定时队列",
zap.String("task_type", taskType),
zap.Int("card_count", len(cardIDs)))
}
// 移除已取出的卡(使用最后一个卡的 score 作为边界,更精确)
if err := s.redis.ZRemRangeByScore(ctx, queueKey, "-inf", formatInt64(now)).Err(); err != nil {
s.logger.Error("移除已处理的卡失败", zap.Error(err))
}
// 批量提交任务(使用 goroutine 并行提交,提高吞吐)
s.enqueueBatch(ctx, taskType, cardIDs, false)
}
// enqueueBatch 批量提交任务到 Asynq 队列
// 使用多 goroutine 并行提交,提高吞吐量
func (s *Scheduler) enqueueBatch(ctx context.Context, taskType string, cardIDs []string, isManual bool) {
if len(cardIDs) == 0 {
return
}
// 分批并行提交,每批 1000 个,最多 10 个并行
batchSize := 1000
maxParallel := 10
sem := make(chan struct{}, maxParallel)
var wg sync.WaitGroup
for i := 0; i < len(cardIDs); i += batchSize {
end := i + batchSize
if end > len(cardIDs) {
end = len(cardIDs)
payloadBytes, marshalErr := marshalJSON(payload)
if marshalErr != nil {
s.logger.Error("序列化任务载荷失败,跳过该卡",
zap.String("task_type", taskType), zap.String("card_id", cardID), zap.Error(marshalErr))
continue
}
batch := cardIDs[i:end]
wg.Add(1)
sem <- struct{}{} // 获取信号量
go func(batch []string) {
defer wg.Done()
defer func() { <-sem }() // 释放信号量
for _, cardID := range batch {
if err := s.enqueueTask(ctx, taskType, cardID, isManual); err != nil {
s.logger.Warn("提交任务失败",
zap.String("task_type", taskType),
zap.String("card_id", cardID),
zap.Error(err))
task := asynq.NewTask(taskType, payloadBytes,
asynq.MaxRetry(0),
asynq.Timeout(30*time.Second),
asynq.Queue(constants.QueueDefault),
)
if _, err := s.queueClient.Enqueue(task); err != nil {
s.logger.Error("提交任务失败,回退至分片队列防止卡永久丢失",
zap.String("task_type", taskType), zap.String("card_id", cardID), zap.Error(err))
if id, parseErr := parseUint(cardID); parseErr == nil {
if reqErr := s.queueMgr.Requeue(ctx, id, taskType, time.Now()); reqErr != nil {
s.logger.Error("回退入队失败,卡可能永久丢失",
zap.String("card_id", cardID), zap.Error(reqErr))
}
}
}(batch)
}
wg.Wait()
}
// enqueueTask 提交任务到 Asynq 队列
func (s *Scheduler) enqueueTask(ctx context.Context, taskType, cardID string, isManual bool) error {
payload := map[string]interface{}{
"card_id": cardID,
"is_manual": isManual,
"timestamp": time.Now().Unix(),
}
task := asynq.NewTask(taskType, mustMarshal(payload),
asynq.MaxRetry(0), // 不重试,失败后重新入队
asynq.Timeout(30*time.Second), // 30秒超时
asynq.Queue(constants.QueueDefault),
)
_, err := s.queueClient.Enqueue(task)
return err
}
// progressiveInit 渐进式初始化
// 分批加载卡数据到 Redis每批 10 万张sleep 1 秒
func (s *Scheduler) progressiveInit(ctx context.Context) {
defer s.wg.Done()
config := DefaultSchedulerConfig()
s.initProgress.mu.Lock()
s.initProgress.Status = "running"
s.initProgress.StartTime = time.Now()
s.initProgress.mu.Unlock()
s.logger.Info("开始渐进式初始化...")
// 获取总卡数
var totalCards int64
if err := s.db.Model(&model.IotCard{}).Count(&totalCards).Error; err != nil {
s.logger.Error("获取卡总数失败", zap.Error(err))
s.setInitError(err.Error())
return
}
s.initProgress.mu.Lock()
s.initProgress.TotalCards = totalCards
s.initProgress.mu.Unlock()
s.logger.Info("开始加载卡数据", zap.Int64("total_cards", totalCards))
// 使用游标分批加载
var lastID uint = 0
batchCount := 0
for {
select {
case <-s.stopChan:
s.logger.Info("渐进式初始化被中断")
return
default:
}
// 加载一批卡
var cards []*model.IotCard
err := s.db.WithContext(ctx).
Where("id > ?", lastID).
Order("id ASC").
Limit(config.InitBatchSize).
Find(&cards).Error
if err != nil {
s.logger.Error("加载卡数据失败", zap.Error(err))
s.setInitError(err.Error())
return
}
if len(cards) == 0 {
break
}
// 批量处理这批卡(使用 Pipeline 提高性能)
if err := s.initCardsBatch(ctx, cards); err != nil {
s.logger.Warn("批量初始化卡轮询失败", zap.Error(err))
}
lastID = cards[len(cards)-1].ID
batchCount++
s.initProgress.mu.Lock()
s.initProgress.LoadedCards += int64(len(cards))
s.initProgress.LastBatchTime = time.Now()
s.initProgress.mu.Unlock()
s.logger.Info("完成一批卡初始化",
zap.Int("batch", batchCount),
zap.Int("batch_size", len(cards)),
zap.Int64("loaded", s.initProgress.LoadedCards),
zap.Int64("total", totalCards))
// 批次间休眠,避免打爆数据库
time.Sleep(config.InitBatchSleepDuration)
}
s.initProgress.mu.Lock()
s.initProgress.Status = "completed"
s.initProgress.mu.Unlock()
s.initCompleted.Store(true)
s.logger.Info("渐进式初始化完成",
zap.Int64("total_loaded", s.initProgress.LoadedCards),
zap.Duration("duration", time.Since(s.initProgress.StartTime)))
}
// initCardsBatch 批量初始化卡的轮询
// 使用 Redis Pipeline 批量写入,大幅提高初始化性能
// 10万张卡从 ~60秒 优化到 ~5秒
func (s *Scheduler) initCardsBatch(ctx context.Context, cards []*model.IotCard) error {
if len(cards) == 0 {
return nil
}
config := DefaultSchedulerConfig()
now := time.Now()
pipe := s.redis.Pipeline()
for _, card := range cards {
// 匹配配置
cfg := s.MatchConfig(card)
if cfg == nil {
continue // 无匹配配置,不需要轮询
}
// 添加到相应的轮询队列
if cfg.RealnameCheckInterval != nil && *cfg.RealnameCheckInterval > 0 {
nextCheck := s.calculateNextCheckTime(card.LastRealNameCheckAt, *cfg.RealnameCheckInterval)
pipe.ZAdd(ctx, constants.RedisPollingQueueRealnameKey(), redis.Z{
Score: float64(nextCheck.Unix()),
Member: card.ID,
})
}
if cfg.CarddataCheckInterval != nil && *cfg.CarddataCheckInterval > 0 {
nextCheck := s.calculateNextCheckTime(card.LastDataCheckAt, *cfg.CarddataCheckInterval)
pipe.ZAdd(ctx, constants.RedisPollingQueueCarddataKey(), redis.Z{
Score: float64(nextCheck.Unix()),
Member: card.ID,
})
}
if cfg.PackageCheckInterval != nil && *cfg.PackageCheckInterval > 0 {
nextCheck := s.calculateNextCheckTime(card.LastDataCheckAt, *cfg.PackageCheckInterval)
pipe.ZAdd(ctx, constants.RedisPollingQueuePackageKey(), redis.Z{
Score: float64(nextCheck.Unix()),
Member: card.ID,
})
}
if cfg.ProtectCheckInterval != nil && *cfg.ProtectCheckInterval > 0 {
nextCheck := s.calculateNextCheckTime(card.LastProtectCheckAt, *cfg.ProtectCheckInterval)
pipe.ZAdd(ctx, constants.RedisPollingQueueProtectKey(), redis.Z{
Score: float64(nextCheck.Unix()),
Member: card.ID,
})
}
// 缓存卡信息到 Redis
cacheKey := constants.RedisPollingCardInfoKey(card.ID)
cacheData := map[string]interface{}{
"id": card.ID,
"iccid": card.ICCID,
"card_category": card.CardCategory,
"real_name_status": card.RealNameStatus,
"network_status": card.NetworkStatus,
"carrier_id": card.CarrierID,
"stop_reason": card.StopReason,
"cached_at": now.Unix(),
}
pipe.HSet(ctx, cacheKey, cacheData)
pipe.Expire(ctx, cacheKey, config.CardCacheTTL)
}
// 执行 Pipeline
_, err := pipe.Exec(ctx)
return err
}
// initCardPolling 初始化单张卡的轮询(保留用于懒加载场景)
func (s *Scheduler) initCardPolling(ctx context.Context, card *model.IotCard) error {
// 匹配配置
config := s.MatchConfig(card)
if config == nil {
return nil // 无匹配配置,不需要轮询
}
now := time.Now()
// 添加到相应的轮询队列
if config.RealnameCheckInterval != nil && *config.RealnameCheckInterval > 0 {
nextCheck := s.calculateNextCheckTime(card.LastRealNameCheckAt, *config.RealnameCheckInterval)
if err := s.addToQueue(ctx, constants.RedisPollingQueueRealnameKey(), card.ID, nextCheck); err != nil {
return err
}
}
if config.CarddataCheckInterval != nil && *config.CarddataCheckInterval > 0 {
nextCheck := s.calculateNextCheckTime(card.LastDataCheckAt, *config.CarddataCheckInterval)
if err := s.addToQueue(ctx, constants.RedisPollingQueueCarddataKey(), card.ID, nextCheck); err != nil {
return err
}
}
if config.PackageCheckInterval != nil && *config.PackageCheckInterval > 0 {
// 套餐检查使用流量检查时间作为参考
nextCheck := s.calculateNextCheckTime(card.LastDataCheckAt, *config.PackageCheckInterval)
if err := s.addToQueue(ctx, constants.RedisPollingQueuePackageKey(), card.ID, nextCheck); err != nil {
return err
}
}
if config.ProtectCheckInterval != nil && *config.ProtectCheckInterval > 0 {
nextCheck := s.calculateNextCheckTime(card.LastProtectCheckAt, *config.ProtectCheckInterval)
if err := s.addToQueue(ctx, constants.RedisPollingQueueProtectKey(), card.ID, nextCheck); err != nil {
return err
}
}
// 缓存卡信息到 Redis
return s.cacheCardInfo(ctx, card, now)
}
// MatchConfig 匹配轮询配置
// 按优先级返回第一个匹配的配置
func (s *Scheduler) MatchConfig(card *model.IotCard) *model.PollingConfig {
s.configCacheLock.RLock()
defer s.configCacheLock.RUnlock()
for _, cfg := range s.configCache {
if s.matchConfigConditions(cfg, card) {
return cfg
}
}
return nil
}
// matchConfigConditions 检查卡是否匹配配置条件
func (s *Scheduler) matchConfigConditions(cfg *model.PollingConfig, card *model.IotCard) bool {
// 检查卡状态条件
if cfg.CardCondition != "" {
cardCondition := s.getCardCondition(card)
if cfg.CardCondition != cardCondition {
return false
}
}
// 检查卡业务类型
if cfg.CardCategory != "" {
if cfg.CardCategory != card.CardCategory {
return false
}
}
// 检查运营商
if cfg.CarrierID != nil {
if *cfg.CarrierID != card.CarrierID {
return false
}
}
return true
}
// getCardCondition 获取卡的状态条件
func (s *Scheduler) getCardCondition(card *model.IotCard) string {
if card.RealNameStatus != constants.RealNameStatusVerified {
return "not_real_name"
}
if card.NetworkStatus == 1 {
return "activated"
}
return "real_name"
}
// calculateNextCheckTime 计算下次检查时间
func (s *Scheduler) calculateNextCheckTime(lastCheckAt *time.Time, intervalSeconds int) time.Time {
now := time.Now()
if lastCheckAt == nil {
// 首次检查,立即执行(加上随机抖动避免集中)
jitter := time.Duration(now.UnixNano()%int64(intervalSeconds)) * time.Second / 10
return now.Add(jitter)
}
// 计算下次检查时间
nextCheck := lastCheckAt.Add(time.Duration(intervalSeconds) * time.Second)
if nextCheck.Before(now) {
// 如果已过期,立即执行
return now
}
return nextCheck
}
// addToQueue 添加卡到轮询队列
func (s *Scheduler) addToQueue(ctx context.Context, queueKey string, cardID uint, nextCheck time.Time) error {
score := float64(nextCheck.Unix())
member := formatUint(cardID)
return s.redis.ZAdd(ctx, queueKey, redis.Z{
Score: score,
Member: member,
}).Err()
}
// cacheCardInfo 缓存卡信息到 Redis
func (s *Scheduler) cacheCardInfo(ctx context.Context, card *model.IotCard, cachedAt time.Time) error {
key := constants.RedisPollingCardInfoKey(card.ID)
config := DefaultSchedulerConfig()
data := map[string]interface{}{
"id": card.ID,
"iccid": card.ICCID,
"card_category": card.CardCategory,
"real_name_status": card.RealNameStatus,
"network_status": card.NetworkStatus,
"carrier_id": card.CarrierID,
"stop_reason": card.StopReason,
"cached_at": cachedAt.Unix(),
}
pipe := s.redis.Pipeline()
pipe.HSet(ctx, key, data)
pipe.Expire(ctx, key, config.CardCacheTTL)
_, err := pipe.Exec(ctx)
return err
}
// setInitError 设置初始化错误
func (s *Scheduler) setInitError(msg string) {
s.initProgress.mu.Lock()
s.initProgress.Status = "failed"
s.initProgress.ErrorMessage = msg
s.initProgress.mu.Unlock()
}
// GetInitProgress 获取初始化进度
func (s *Scheduler) GetInitProgress() InitProgress {
s.initProgress.mu.RLock()
defer s.initProgress.mu.RUnlock()
return InitProgress{
TotalCards: s.initProgress.TotalCards,
LoadedCards: s.initProgress.LoadedCards,
StartTime: s.initProgress.StartTime,
LastBatchTime: s.initProgress.LastBatchTime,
Status: s.initProgress.Status,
ErrorMessage: s.initProgress.ErrorMessage,
}
}
// IsInitCompleted 检查初始化是否完成
func (s *Scheduler) IsInitCompleted() bool {
return s.initCompleted.Load()
}
// RefreshConfigs 刷新配置缓存
func (s *Scheduler) RefreshConfigs(ctx context.Context) error {
return s.loadConfigs(ctx)
}
// SetResetService 设置流量重置服务(用于依赖注入)
func (s *Scheduler) SetResetService(resetService interface{}) {
if rs, ok := resetService.(*DataResetHandler); ok {
s.dataResetHandler = rs
}
}
// SetActivationService 设置套餐激活服务(用于依赖注入)
func (s *Scheduler) SetActivationService(activationHandler *PackageActivationHandler) {
s.packageActivationHandler = activationHandler
}
// SetStopResumeCallback 注入停复机回调(用于套餐过期后主动触发停机)
func (s *Scheduler) SetStopResumeCallback(callback packagepkg.StopResumeCallback) {
if s.packageActivationHandler != nil {
s.packageActivationHandler.stopResumeCallback = callback
}
}