feat: 轮询系统重构(分片队列 + 停复机统一 + Handler 拆分)
All checks were successful
构建并部署到测试环境(无 SSH) / build-and-deploy (push) Successful in 9m46s

【核心变更】

1. 停复机逻辑统一(StopResumeService)
   - 新增 EvaluateAndAct 统一入口,封装三条件停复机判断
   - 停机条件:无套餐(no_package) / 流量耗尽(traffic_exhausted) / 未实名(not_realname)
   - 复机条件:stop_reason 合规 + 有套餐且未耗尽 + 已实名或行业卡
   - 修复设备套餐 Bug:hasValidPackage 按 device_id 查套餐,而非仅 iot_card_id
   - 设备维度停复机加幂等锁(Redis SetNX,TTL 30s),防止多卡并发重复调 Gateway

2. Redis 分片队列(PollingQueueManager)
   - 新建 queue_manager.go,封装所有轮询 Redis 操作
   - 16 分片 Sorted Set,Key 格式:polling:shard:{shardID}:queue:{taskType}
   - Lua 脚本原子出队(ZRANGEBYSCORE + 分批 ZREM),消除竞态窗口
   - 新增背压检测:队列深度超 50 万时 Scheduler 跳过该分片
   - RemoveFromAllQueues 覆盖 4 种任务类型(含 protect)

3. Handler 拆分(polling_handler.go 1360行 → 5个专注文件)
   - polling_base.go:共享基类(并发控制/卡缓存/重入队)
   - polling_realname_handler.go:实名采集,实名 0→1 时立即触发复机
   - polling_carddata_handler.go:流量采集,保留跨月边界检测逻辑
   - polling_package_handler.go:套餐采集,委托 EvaluateAndAct 决策
   - polling_protect_handler.go:保护期一致性检查,保护期内强制修正

4. 配置管理(PollingConfigManager)
   - 新建 config_manager.go,从 scheduler.go 提取配置职责
   - 内存缓存 + 5 分钟定时刷新,刷新失败保留原缓存
   - 修复 getCardCondition:停机卡返回 suspended,不再错配 activated 配置

5. 渐进式初始化(CardInitializer)
   - 新建 initializer.go,分批加载(每批 10 万),批次间 sleep 500ms
   - 过滤 enable_polling=false 的卡,初始化完成前 Scheduler 不出队

6. 卡生命周期服务(PollingLifecycleService)
   - 新建 lifecycle_service.go,替代已删除的 callbacks.go 和 api_callback.go
   - OnCardCreated/OnCardEnabled/OnCardStatusChanged 入队前检查 enable_polling

7. Scheduler 精简(1000+行 → 227行)
   - 保留纯调度循环:scheduleLoop + processShardSchedule + enqueueBatch
   - 保留每 10 秒触发套餐过期检测和流量重置
   - 移除所有 DB 操作、配置加载、卡初始化逻辑

8. 轮询管控 API(enable_polling)
   - 新增 PUT /api/admin/assets/:id/polling-status 接口
   - 支持对设备/卡维度开关轮询,关闭后从所有分片队列移除

9. 数据库迁移
   - 000103:tb_device 新增 enable_polling 字段(boolean, NOT NULL, DEFAULT true)
   - 000104:新增 suspended 轮询配置,为 activated 配置补全 protect_check_interval

【文件统计】
- 新增:19 个文件(handler × 5、polling 组件 × 4、迁移 × 3 等)
- 修改:20 个文件(bootstrap 注入、store 接口、monitoring 适配分片等)
- 删除:3 个文件(polling_handler.go、callbacks.go、api_callback.go)

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
This commit is contained in:
2026-04-07 12:27:04 +08:00
parent 10fcc0b3c9
commit 434a8b0349
62 changed files with 7496 additions and 3023 deletions

View File

@@ -0,0 +1,89 @@
package polling
import (
"context"
"go.uber.org/zap"
"github.com/break/junhong_cmp_fiber/internal/polling"
iotCardSvc "github.com/break/junhong_cmp_fiber/internal/service/iot_card"
"github.com/break/junhong_cmp_fiber/internal/store/postgres"
"github.com/break/junhong_cmp_fiber/pkg/constants"
"github.com/break/junhong_cmp_fiber/pkg/errors"
)
// AssetPollingService 资产轮询管控服务
// 管理 IoT 卡和设备的轮询启用状态
// S2 修复card 类型委托给 IotCardService含 DB 写入 + callback 通知),避免绕过生命周期
type AssetPollingService struct {
deviceStore *postgres.DeviceStore
deviceBindingStore *postgres.DeviceSimBindingStore
iotCardService *iotCardSvc.Service
queueMgr *polling.PollingQueueManager
logger *zap.Logger
}
// NewAssetPollingService 创建资产轮询管控服务
func NewAssetPollingService(
deviceStore *postgres.DeviceStore,
deviceBindingStore *postgres.DeviceSimBindingStore,
iotCardService *iotCardSvc.Service,
queueMgr *polling.PollingQueueManager,
logger *zap.Logger,
) *AssetPollingService {
return &AssetPollingService{
deviceStore: deviceStore,
deviceBindingStore: deviceBindingStore,
iotCardService: iotCardService,
queueMgr: queueMgr,
logger: logger,
}
}
// UpdatePollingStatus 更新资产轮询状态
// assetType: "card" 或 "device"
// assetID: 资产ID
// enablePolling: 是否启用轮询
func (s *AssetPollingService) UpdatePollingStatus(ctx context.Context, assetType string, assetID uint, enablePolling bool) error {
switch assetType {
case constants.AssetTypeIotCard:
// S2 修复:委托给 IotCardService确保 DB 写入 + PollingCallback 回调一并触发
return s.iotCardService.UpdatePollingStatus(ctx, assetID, enablePolling)
case constants.AssetTypeDevice:
// 1. 更新设备的 enable_polling 字段
if err := s.deviceStore.UpdatePollingStatus(ctx, assetID, enablePolling); err != nil {
return err
}
bindings, err := s.deviceBindingStore.ListByDeviceID(ctx, assetID)
if err != nil {
s.logger.Warn("查询设备绑定卡失败,绑定卡轮询状态同步可能不完整",
zap.Uint("device_id", assetID), zap.Error(err))
return nil
}
if len(bindings) == 0 {
return nil
}
cardIDs := make([]uint, len(bindings))
for i, b := range bindings {
cardIDs[i] = b.IotCardID
}
// 2. M3 修复:级联同步绑定卡的 enable_polling防止生命周期事件绕过设备级设置
if syncErr := s.iotCardService.BatchUpdatePollingStatus(ctx, cardIDs, enablePolling); syncErr != nil {
s.logger.Warn("批量同步绑定卡轮询状态失败",
zap.Uint("device_id", assetID), zap.Error(syncErr))
}
if !enablePolling {
for _, b := range bindings {
if rmErr := s.queueMgr.RemoveFromAllQueues(ctx, b.IotCardID); rmErr != nil {
s.logger.Warn("从队列移除卡失败",
zap.Uint("card_id", b.IotCardID), zap.Error(rmErr))
}
}
}
return nil
default:
return errors.New(errors.CodeInvalidParam, "资产类型无效,支持 card 或 device")
}
}

View File

@@ -5,19 +5,34 @@ import (
"time"
"github.com/redis/go-redis/v9"
"go.uber.org/zap"
"github.com/break/junhong_cmp_fiber/internal/polling"
"github.com/break/junhong_cmp_fiber/pkg/constants"
"github.com/break/junhong_cmp_fiber/pkg/errors"
)
// MonitoringService 轮询监控服务
type MonitoringService struct {
redis *redis.Client
redis *redis.Client
queueMgr *polling.PollingQueueManager
}
// NewMonitoringService 创建轮询监控服务实例
func NewMonitoringService(redis *redis.Client) *MonitoringService {
return &MonitoringService{redis: redis}
// queueMgr 可选nil 时降级到旧非分片 Key 兼容模式)
func NewMonitoringService(redis *redis.Client, opts ...interface{}) *MonitoringService {
svc := &MonitoringService{redis: redis}
for _, opt := range opts {
if qm, ok := opt.(*polling.PollingQueueManager); ok {
svc.queueMgr = qm
}
}
return svc
}
// NewMonitoringServiceWithQueueMgr 创建注入了 PollingQueueManager 的监控服务(推荐)
func NewMonitoringServiceWithQueueMgr(redis *redis.Client, queueMgr *polling.PollingQueueManager, _ *zap.Logger) *MonitoringService {
return &MonitoringService{redis: redis, queueMgr: queueMgr}
}
// OverviewStats 总览统计
@@ -84,10 +99,15 @@ func (s *MonitoringService) GetOverview(ctx context.Context) (*OverviewStats, er
}
stats.IsInitializing = stats.InitializedCards < stats.TotalCards && stats.TotalCards > 0
// 获取队列大小
stats.RealnameQueueSize, _ = s.redis.ZCard(ctx, constants.RedisPollingQueueRealnameKey()).Result()
stats.CarddataQueueSize, _ = s.redis.ZCard(ctx, constants.RedisPollingQueueCarddataKey()).Result()
stats.PackageQueueSize, _ = s.redis.ZCard(ctx, constants.RedisPollingQueuePackageKey()).Result()
if s.queueMgr != nil {
stats.RealnameQueueSize, _ = s.queueMgr.GetTotalQueueDepth(ctx, constants.TaskTypePollingRealname)
stats.CarddataQueueSize, _ = s.queueMgr.GetTotalQueueDepth(ctx, constants.TaskTypePollingCarddata)
stats.PackageQueueSize, _ = s.queueMgr.GetTotalQueueDepth(ctx, constants.TaskTypePollingPackage)
} else {
stats.RealnameQueueSize, _ = s.redis.ZCard(ctx, constants.RedisPollingQueueRealnameKey()).Result()
stats.CarddataQueueSize, _ = s.redis.ZCard(ctx, constants.RedisPollingQueueCarddataKey()).Result()
stats.PackageQueueSize, _ = s.redis.ZCard(ctx, constants.RedisPollingQueuePackageKey()).Result()
}
return stats, nil
}
@@ -120,8 +140,11 @@ func (s *MonitoringService) GetQueueStatuses(ctx context.Context) ([]*QueueStatu
queueKey = constants.RedisPollingQueuePackageKey()
}
// 获取队列大小
status.QueueSize, _ = s.redis.ZCard(ctx, queueKey).Result()
if s.queueMgr != nil {
status.QueueSize, _ = s.queueMgr.GetTotalQueueDepth(ctx, taskType)
} else {
status.QueueSize, _ = s.redis.ZCard(ctx, queueKey).Result()
}
// 获取到期数量score <= now
status.DueCount, _ = s.redis.ZCount(ctx, queueKey, "-inf", formatInt64(now)).Result()