更新一版

This commit is contained in:
Break
2026-06-12 18:10:22 +08:00
parent e139f5e227
commit 1c492fe8db
10 changed files with 1735 additions and 258 deletions

View File

@@ -2,13 +2,13 @@
> 详细方案见 [`docs/脚本/奇成数据迁移方案.md`](../../docs/脚本/奇成数据迁移方案.md)。 > 详细方案见 [`docs/脚本/奇成数据迁移方案.md`](../../docs/脚本/奇成数据迁移方案.md)。
## 1. 四步走 ## 1. 两条执行路径
``` ```
业务方 我们 运维 业务方 我们 运维
│ │ │ │ │ │
│ cards.csv │ scan_legacy.py → mapping.yaml │ │ cards.csv │ scan_legacy.py → mapping.yaml │
│ devices.csv │ ↓ 人工填 target_* │ │ devices.csv? │ ↓ 人工填 target_* │
├─────────────▶│ migrate_assets.py → step1_*.sql │ ├─────────────▶│ migrate_assets.py → step1_*.sql │
│ │ migrate_runtime.py → step2_*.sql │ │ │ migrate_runtime.py → step2_*.sql │
│ ├────────────────────────────────────▶│ psql -1 -f step*.sql │ ├────────────────────────────────────▶│ psql -1 -f step*.sql
@@ -16,6 +16,11 @@
│ │ │ 重建轮询队列/缓存 │ │ │ 重建轮询队列/缓存
``` ```
`devices.csv` 是可选文件:
- **只有卡**:只准备 `cards.csv`,不要放 `devices.csv`;脚本按独立卡迁移,不会生成设备绑定关系。
- **卡 + 设备**:同时准备 `cards.csv``devices.csv`;设备上的卡跟随设备归属,设备套餐挂到设备资产,旧套餐数据从 `package_source_slot` 对应绑定卡读取。
## 2. 文件清单 ## 2. 文件清单
``` ```
@@ -28,10 +33,10 @@ scripts/migration/
├── lib/ ← 公共库 ├── lib/ ← 公共库
├── resources/ ├── resources/
│ ├── README.md │ ├── README.md
│ ├── cards.csv.example ← 业务方提供:iccid + 可选覆盖列 │ ├── cards.csv.example ← 业务方提供:ICCID + is_industry
│ └── devices.csv.example ← 业务方提供:设备 + 卡槽 │ └── devices.csv.example ← 可选:设备 + 卡槽 + 可选槽位覆盖
├── config/ ├── config/
│ ├── mapping.yaml.example ← 唯一映射配置(全局 + carrier + package + series + agent) │ ├── mapping.yaml.example ← 唯一决策配置(归属/槽位/套餐 + carrier/package/agent)
│ └── legacy_dsn.yaml.example← 奇成 MySQL 只读 DSN │ └── legacy_dsn.yaml.example← 奇成 MySQL 只读 DSN
└── output/ ← 生成的 SQL 与日志 └── output/ ← 生成的 SQL 与日志
``` ```
@@ -40,14 +45,14 @@ scripts/migration/
### 3.1 业务方提供数据 ### 3.1 业务方提供数据
两个 CSV 放到 `resources/`(命名为 `cards.csv` / `devices.csv`): 把 CSV 放到 `resources/` 下:
- **cards.csv**:`iccid_19`/`iccid_20` 至少填一个;仅填 `iccid_20` 时脚本自动派生前 19 位,但奇成主表只按 20 位精确查询;`is_industry` 可选 - **cards.csv**:`iccid_19`/`iccid_20` 至少填一个;仅填 `iccid_20` 时脚本自动派生前 19 位,但奇成主表只按 20 位精确查询;`is_industry` 可选
- **devices.csv**:`imei` / `virtual_no` 至少填一个;`device_name` / `device_model` / `device_type` / `max_sim_slots` 可选;`sim_iccid_*` 引用 `cards.csv` 中的 ICCID(可填 19 或 20 位;若 19 位前缀对应多张 20 位卡,必须填完整 20 位) - **devices.csv** 可选。只有卡时不要放这个文件;有设备时再提供:`imei` / `virtual_no` 至少填一个;`device_name` / `device_model` / `device_type` / `max_sim_slots` 可选;`sim_iccid_*` 引用 `cards.csv` 中的 ICCID;`current_slot` / `package_source_slot` 可选,空值由 `mapping.yaml` 批量规则填充
字段说明详见 [`resources/README.md`](resources/README.md)。 字段说明详见 [`resources/README.md`](resources/README.md)。
### 3.2 我们扫描 + 填映射 ### 3.2 扫描 + 填映射
```bash ```bash
pip install -r requirements.txt pip install -r requirements.txt
@@ -61,18 +66,102 @@ python3 scan_legacy.py
vim config/mapping.yaml vim config/mapping.yaml
``` ```
`scan_legacy.py` 可以反复跑(增量合并),已填的 `target_*` 不会被覆盖。 `scan_legacy.py` 可以反复跑(增量合并),已填的 `target_*` 不会被覆盖。写回的 `mapping.yaml`
会带中文注释,说明哪些字段来自奇成、哪些字段需要人工填写。
### 3.3 生成 SQL ### 3.3 只有卡的执行顺序
只有卡时,`resources/` 下只放 `cards.csv`,不要放 `devices.csv`
```bash ```bash
python3 migrate_assets.py # → output/step1_*.sql + errors.csv python3 scan_legacy.py
python3 migrate_runtime.py # → output/step2_*.sql + warnings.csv # 填 config/mapping.yaml:
# - ownership_rules.standalone_card.mode
# - ownership_rules.default_target_shop_code (mode=default_shop 时必填)
# - agents[].target_shop_code (mode=legacy_agent 时按奇成代理分店铺)
# - carriers[].target_*
# - packages[].target_package_id
python3 migrate_assets.py
``` ```
如果 `errors.csv` 有内容,意味着 `mapping.yaml` 还没填完整、或者某些 ICCID 在奇成查不到,处理完再重跑。 先审核:
### 3.4 测试环境 / 线上执行 - `output/ownership_resolution.csv`:只会有独立卡归属决策。
- `output/errors.csv`:非空时不要执行 SQL。
- `output/summary.txt`:确认独立卡数量、分配卡数。
再按编号执行 step1。纯卡迁移时 `step1_02_devices.sql``step1_05_sim_bindings.sql`
只包含文件头或空操作,可以执行也可以跳过;为了减少漏执行,建议仍按编号执行:
```bash
psql -h $HOST -1 -f output/step1_01_iot_cards.sql
psql -h $HOST -1 -f output/step1_02_devices.sql
psql -h $HOST -1 -f output/step1_03_asset_identifiers.sql
psql -h $HOST -1 -f output/step1_04_asset_wallets.sql
psql -h $HOST -1 -f output/step1_05_sim_bindings.sql
psql -h $HOST -1 -f output/step1_06_shop_allocations.sql
```
step1 执行完成后再生成并执行 step2:
```bash
python3 migrate_runtime.py
```
审核 `package_resolution.csv` / `errors.csv` / `warnings.csv` / `summary.txt` 后,按编号执行:
```bash
psql -h $HOST -1 -f output/step2_01_migration_orders.sql
psql -h $HOST -1 -f output/step2_02_package_usages.sql
psql -h $HOST -1 -f output/step2_03_card_data_usage_update.sql
psql -h $HOST -1 -f output/step2_04_agent_wallet_init.sql
psql -h $HOST -1 -f output/step2_05_agent_wallet_transactions.sql
psql -h $HOST -1 -f output/step2_06_asset_series_update.sql
```
### 3.4 卡 + 设备的执行顺序
有设备时,同时准备 `cards.csv``devices.csv`。所有设备槽位里引用的 ICCID 必须存在于
`cards.csv`
```bash
python3 scan_legacy.py
# 填 config/mapping.yaml:
# - ownership_rules.device.mode/current_slot/package_source_slot
# - ownership_rules.standalone_card.mode
# - agents[].target_shop_code (mode=legacy_agent 时按奇成代理分店铺)
# - overrides.devices/cards (如有特殊设备或特殊卡)
# - carriers[].target_*
# - packages[].target_package_id
python3 migrate_assets.py
```
先审核:
- `ownership_resolution.csv`:核对每台设备归属、当前槽位、旧套餐读取槽位;核对独立卡归属。
- `errors.csv`:非空时不要执行 SQL。
- `summary.txt`:确认卡数、设备数、分配数量。
再执行 step1,然后生成 step2,审核 `package_resolution.csv`,最后执行 step2。命令顺序与
“只有卡”相同。
### 3.5 生成 SQL 的审核要求
```bash
python3 migrate_assets.py # → output/step1_*.sql + ownership_resolution.csv + errors.csv
python3 migrate_runtime.py # → output/step2_*.sql + package_resolution.csv + errors.csv + warnings.csv
```
执行 SQL 前必须先审核:
- `ownership_resolution.csv`:核对每个设备/独立卡的目标店铺、决策来源、当前槽位和旧套餐读取槽位
- `package_resolution.csv`:核对 active/pending 套餐、目标套餐、优先级、跳过原因和阻断原因
- `summary.txt`:核对分配数、active/pending 套餐数、跳过数、错误数
- `errors.csv`:非空时不执行 SQL,先修正配置或输入后重跑
### 3.6 目标库执行
```bash ```bash
# 按编号顺序执行 # 按编号顺序执行
@@ -112,6 +201,28 @@ go run ./cmd/migration-finalize --batch-no MIGRATION-QICHENG-YYYYMMDD --apply
migration_user_id: 0 # 后台超级管理员 ID,作为所有 creator/updater migration_user_id: 0 # 后台超级管理员 ID,作为所有 creator/updater
migration_batch_no: "MIGRATION-QICHENG-YYYYMMDD" # 本次迁移批次号,线上每批必须唯一 migration_batch_no: "MIGRATION-QICHENG-YYYYMMDD" # 本次迁移批次号,线上每批必须唯一
ownership_rules:
default_target_shop_code: KWTX # default_shop 模式的批量目标店铺;legacy_agent/none 可为 null
device:
mode: none # default_shop / legacy_agent / none;只有卡时保持 none
current_slot: null # 设备当前卡槽,写 is_current;只有卡时 null
package_source_slot: null # 设备旧套餐读取槽位;只有卡时 null
standalone_card:
mode: legacy_agent # default_shop / legacy_agent / none
package_rules:
migrate_statuses: [active, pending] # 迁移当前生效 + 未生效待生效正式套餐
overrides:
devices:
- virtual_no: "862639073940258"
target_shop_code: OTHER
current_slot: 1
package_source_slot: 1
cards:
- iccid: "89861590172420360956"
target_shop_code: OTHER
carriers: # 由 scan_legacy 自动发现 legacy_*,你只填 target_* carriers: # 由 scan_legacy 自动发现 legacy_*,你只填 target_*
- legacy_account_id: "72493DB64A2A4F01B5E59E49D58A2A34" - legacy_account_id: "72493DB64A2A4F01B5E59E49D58A2A34"
legacy_account_name: "联通10-1" legacy_account_name: "联通10-1"
@@ -131,23 +242,52 @@ series: # 仅保留奇成套餐系列快照,生成器不要求填
legacy_series_name: "国内卡 无预存30天" legacy_series_name: "国内卡 无预存30天"
target_series_id: null target_series_id: null
agents: # 决定所有卡/设备的店铺归属 agents: # legacy_agent 归属模式 + 代理钱包初始化
- legacy_agent_id: "D15EC779767B412A82B8448AC2324F04" - legacy_agent_id: "D15EC779767B412A82B8448AC2324F04"
legacy_agent_name: "叶发伟" legacy_agent_name: "叶发伟"
target_shop_code: "SHOP20260428103424CZZQ" # null = 该代理的卡进平台库存 target_shop_code: "SHOP20260428103424CZZQ" # legacy_agent 模式下 null=该代理资产进平台库存
``` ```
### 店铺归属是怎么决定的 ### 店铺归属是怎么决定的
CSV 里**没有** `target_shop_code`,所有归属都从 `mapping.agents` 推导: CSV 里**没有** `target_shop_code`。资产归属由 `mapping.yaml` 显式决定:
| 资产 | 归属规则 | | 资产 | 归属规则 |
|------|---------| |------|---------|
| 独立卡 | 奇成 `tbl_card.agent_id``mapping.agents[].target_shop_code` | | 独立卡 | `overrides.cards` 优先,否则 `ownership_rules.standalone_card` |
| 设备 | 主卡(`sim_iccid_1`)的 agent_id → 同上 | | 设备 | `overrides.devices` 优先,否则 `ownership_rules.device` |
| 设备上的卡 | 跟随设备所属店铺(`shop_id` 与设备一致,设备分配时卡状态同步为已分销) | | 设备上的卡 | 跟随设备所属店铺(`shop_id` 与设备一致,设备分配时卡状态同步为已分销) |
如果 agent 在 `mapping.agents``target_shop_code` 留空 → 该资产进平台库存。 `ownership_rules.*.mode` 有三种:
| mode | 含义 | 适用场景 |
|------|------|----------|
| `default_shop` | 全部分到 `default_target_shop_code` | 单店铺批次 |
| `legacy_agent` | 按奇成 `tbl_card.agent_id``agents[].target_shop_code` | 一批卡分散到多个店铺 |
| `none` | 不分配店铺,进入平台库存 | 先入库后分配 |
一万张卡分散到多个店铺时,不要写一万条 `overrides.cards`。做法是:
```yaml
ownership_rules:
default_target_shop_code: null
standalone_card:
mode: legacy_agent
agents:
- legacy_agent_id: "0072250EBB88422C88A573AB06870D17"
legacy_agent_name: "酷蛙通讯"
target_shop_code: "KWTX"
- legacy_agent_id: "..."
legacy_agent_name: "另一个代理"
target_shop_code: "SHOP_A"
```
这样脚本会按每张卡在奇成 `tbl_card.agent_id` 自动分流到不同店铺。设备批次如果也要按代理分店铺,设置
`ownership_rules.device.mode: legacy_agent`,脚本会用设备 `package_source_slot` 那张绑定卡的 `agent_id`
决定设备归属。
设备槽位解析优先级是 `devices.csv` 行级配置 > `overrides.devices` > `ownership_rules.device``current_slot` 对应 `tb_device_sim_binding.is_current`;`package_source_slot` 只决定从哪张绑定卡读取奇成旧套餐,新系统套餐仍写到设备资产。
## 5. 识别迁移数据(后台报表过滤用) ## 5. 识别迁移数据(后台报表过滤用)
@@ -157,9 +297,9 @@ CSV 里**没有** `target_shop_code` 列,所有归属都从 `mapping.agents` 推
| 迁移分配记录 | `WHERE allocation_no LIKE 'MIG-ALLOC-%'` | | 迁移分配记录 | `WHERE allocation_no LIKE 'MIG-ALLOC-%'` |
| 迁移钱包流水 | `WHERE metadata->>'source' = 'qicheng'` | | 迁移钱包流水 | `WHERE metadata->>'source' = 'qicheng'` |
迁移伪订单号固定为 `MIG-CARD-<ICCID>`: 迁移伪订单号固定为 `MIG-<ICCID>-<priority>`:
- 20 位卡总长 29,19 位卡总长 28,符合新库 `order_no varchar(30)` 限制。 - 20 位卡加优先级总长小于 30,符合新库 `order_no varchar(30)` 限制。
- 订单号不包含生成时间,`step2_01_migration_orders.sql``step2_02_package_usages.sql` 分开重跑也能稳定匹配。 - 订单号不包含生成时间,`step2_01_migration_orders.sql``step2_02_package_usages.sql` 分开重跑也能稳定匹配。
## 6. 流量用量迁移规则(真用量 / 虚用量) ## 6. 流量用量迁移规则(真用量 / 虚用量)
@@ -182,12 +322,12 @@ CSV 里**没有** `target_shop_code` 列,所有归属都从 `mapping.agents` 推
real_used_mb = total_bytes_cnt / (1 + flow_add_discount / 100) real_used_mb = total_bytes_cnt / (1 + flow_add_discount / 100)
``` ```
其中 `flow_add_discount` 取自该卡在 `tbl_card_life`(status=1, expire_date 最大)对应的当前生效套餐 其中 `flow_add_discount` 取自该卡在 `tbl_card_life`(status=1, type!=1, expire_date 最大)对应的当前生效正式套餐。奇成 `type=1` 是叠加包/加油包,不能作为新系统主套餐迁入
### 6.3 精度与近似 ### 6.3 精度与近似
- **跨多个套餐周期的累计**:`total_bytes_cnt` 是按各时段对应套餐虚比加权累加的,这里只用"当前生效套餐"的虚比反算,**有误差但可接受**(奇成没有按套餐分段的用量明细,无更好办法)。 - **跨多个套餐周期的累计**:`total_bytes_cnt` 是按各时段对应套餐虚比加权累加的,这里只用"当前生效正式套餐"的虚比反算,**有误差但可接受**(奇成没有按套餐分段的用量明细,无更好办法)。
- **无当前生效套餐却有 `total_bytes_cnt`**:按 `flow_add_discount=0` 处理(真=虚),并写入 `warnings.csv`(`kind=usage_without_active_package`)供人工核查。 - **无当前生效正式套餐却有 `total_bytes_cnt`**:按 `flow_add_discount=0` 处理(真=虚),并写入 `warnings.csv`(`kind=usage_without_active_package`)供人工核查。
- **整数取整规则**(`data_usage_mb` 是 BIGINT): - **整数取整规则**(`data_usage_mb` 是 BIGINT):
- `real_used_mb = 0` → 写 0 - `real_used_mb = 0` → 写 0
- `0 < real_used_mb < 1`**上取整为 1**,避免"明明用过的卡显示 0" - `0 < real_used_mb < 1`**上取整为 1**,避免"明明用过的卡显示 0"
@@ -196,7 +336,7 @@ real_used_mb = total_bytes_cnt / (1 + flow_add_discount / 100)
### 6.4 套餐使用记录的虚流量快照 ### 6.4 套餐使用记录的虚流量快照
`tb_package_usage``virtual_total_mb_snapshot` / `display_gain_ratio_snapshot` / `enable_virtual_data_snapshot` 三个字段**全部从新系统 `tb_package` 当场快照**,与奇成无关: `tb_package_usage``virtual_total_mb_snapshot` / `display_gain_ratio_snapshot` / `enable_virtual_data_snapshot` 三个字段**全部从新系统 `tb_package` 当场快照**,与奇成无关。迁移脚本只允许 `target_package_id` 指向新系统 `package_type='formal'` 的套餐;如果误指向 `addon` 加油包,生成的 step2 SQL 会直接报错并回滚:
| 快照字段 | 取值 | | 快照字段 | 取值 |
|---|---| |---|---|
@@ -206,9 +346,20 @@ real_used_mb = total_bytes_cnt / (1 + flow_add_discount / 100)
| `data_limit_mb` | 始终 `real_data_mb`(真总量) | | `data_limit_mb` | 始终 `real_data_mb`(真总量) |
| `data_usage_mb` | `LEAST(real_used_mb_floor, virtual_total_mb_snapshot)`(防卫上限) | | `data_usage_mb` | `LEAST(real_used_mb_floor, virtual_total_mb_snapshot)`(防卫上限) |
### 6.5 设备绑定卡的套餐归属 ### 6.5 当前生效 + 待生效套餐
设备上的卡在新系统运行态会按设备维度判断有效套餐和复机条件,因此迁移时只用设备当前槽位卡(默认 `sim_iccid_1`)作为奇成套餐来源,生成一条 `usage_type=device``device_id=设备ID` 的套餐使用记录。其他槽位绑定卡不重复生成主套餐,避免同一设备出现多条并列生效套餐 脚本读取奇成 `tbl_card_life` 当前/历史正式套餐和 `tbl_next_month_card_life` 次月待生效套餐:
- `type=1` 加油包不迁移,写入 `package_resolution.csv` 跳过原因
- 已过期套餐不迁移,写入跳过原因
- active 写 `tb_package_usage.status=1`,保留 `activated_at` / `expires_at` 和用量快照
- pending 写 `tb_package_usage.status=0`,保留 `expires_at`,用量为 0,按 `start_date ASC, expire_date ASC, legacy_life_id ASC` 生成稳定 `priority`
- 同一资产出现多个 active 正式套餐时写入 `errors.csv` 并阻断该资产套餐 SQL
- legacy 套餐未映射或目标套餐缺失时写入 `errors.csv` 并阻断对应套餐 SQL
### 6.6 设备绑定卡的套餐归属
设备上的卡在新系统运行态会按设备维度判断有效套餐和复机条件,因此迁移时只用设备 `package_source_slot` 对应绑定卡读取奇成旧套餐生命周期。生成的新系统套餐使用记录仍然是 `usage_type=device``device_id=设备ID`,不会挂到这张卡资产上。其他槽位绑定卡不读取旧套餐,避免同一设备重复迁入多份主套餐。
设备和卡的 `series_id` 以新系统目标套餐为准:脚本通过 `mapping.packages[].target_package_id` 对应的 `tb_package.series_id` 写入资产,并额外生成 `step2_06_asset_series_update.sql`,用于修复已经导入但因 `ON CONFLICT DO NOTHING` 没有被 step1 更新到的资产。 设备和卡的 `series_id` 以新系统目标套餐为准:脚本通过 `mapping.packages[].target_package_id` 对应的 `tb_package.series_id` 写入资产,并额外生成 `step2_06_asset_series_update.sql`,用于修复已经导入但因 `ON CONFLICT DO NOTHING` 没有被 step1 更新到的资产。
@@ -216,6 +367,8 @@ real_used_mb = total_bytes_cnt / (1 + flow_add_discount / 100)
- ❗ 所有奇成查询走 `SET TRANSACTION READ ONLY`,严禁误写。 - ❗ 所有奇成查询走 `SET TRANSACTION READ ONLY`,严禁误写。
- ❗ 所有 INSERT 用 `ON CONFLICT DO NOTHING`,严禁更新已有数据。 - ❗ 所有 INSERT 用 `ON CONFLICT DO NOTHING`,严禁更新已有数据。
- ❗ 目标店铺和目标套餐都有 SQL 前置校验,缺失时直接报错并回滚当前 SQL 文件。
-`errors.csv` 非空时必须先修复后重跑,不要执行已生成 SQL。
- ❗ 伪订单的 `payment_status=2 commission_status=2 commission_result=2`,避免被支付回调、佣金调度器扫到。 - ❗ 伪订单的 `payment_status=2 commission_status=2 commission_result=2`,避免被支付回调、佣金调度器扫到。
- ❗ 迁移卡默认 `status=1(在库)` / `activation_status=0(未激活)`,激活由后续业务流程驱动。 - ❗ 迁移卡默认 `status=1(在库)` / `activation_status=0(未激活)`,激活由后续业务流程驱动。
@@ -228,9 +381,13 @@ real_used_mb = total_bytes_cnt / (1 + flow_add_discount / 100)
| `errors.csv` 出现 `carrier_target_missing` | `mapping.yaml.carriers` 里有这一项但 `target_*` 还是 null,手动填 | | `errors.csv` 出现 `carrier_target_missing` | `mapping.yaml.carriers` 里有这一项但 `target_*` 还是 null,手动填 |
| `errors.csv` 出现 `not_in_legacy` | 该 ICCID 在奇成 `tbl_card` 查不到,业务方确认 | | `errors.csv` 出现 `not_in_legacy` | 该 ICCID 在奇成 `tbl_card` 查不到,业务方确认 |
| `errors.csv` 出现 `legacy_duplicate` | 同一 ICCID 在奇成命中多条 `tbl_card`,业务方人工确认是哪张物理卡 | | `errors.csv` 出现 `legacy_duplicate` | 同一 ICCID 在奇成命中多条 `tbl_card`,业务方人工确认是哪张物理卡 |
| `errors.csv` 出现 `current_slot_empty` | `current_slot` 指向的设备槽位无卡,修正 devices.csv 或 mapping.yaml |
| `errors.csv` 出现 `package_source_slot_empty` | `package_source_slot` 指向的设备槽位无卡,修正后重跑 |
| `errors.csv` 出现 `package_not_in_mapping` | 可迁移 legacy 套餐未配置 `target_package_id`,补齐 mapping.yaml |
| `errors.csv` 出现 `multiple_active_packages` | 同一资产多个 active 主套餐,需人工裁决后用配置或数据修正 |
| `errors.csv` 出现 `carrier_length_conflict` | 业务方填的 `iccid_20` 与奇成查到的运营商不匹配(CTCC 不该有 20 位 / 非 CTCC 必须有 20 位) | | `errors.csv` 出现 `carrier_length_conflict` | 业务方填的 `iccid_20` 与奇成查到的运营商不匹配(CTCC 不该有 20 位 / 非 CTCC 必须有 20 位) |
| `errors.csv` 出现 `iccid_20_prefix_mismatch` | `iccid_20` 前 19 位 ≠ `iccid_19`,业务方校对 | | `errors.csv` 出现 `iccid_20_prefix_mismatch` | `iccid_20` 前 19 位 ≠ `iccid_19`,业务方校对 |
| `warnings.csv` 出现 `agent_no_legacy` | `mapping.agents` 配了但奇成没有 tbl_agent 记录,业务方确认 | | `warnings.csv` 出现 `agent_no_legacy` | `mapping.agents` 配了但奇成没有 tbl_agent 记录,业务方确认 |
| `warnings.csv` 出现 `usage_without_active_package` | 该卡奇成 `total_bytes_cnt > 0``tbl_card_life` 中没有 status=1 的当前生效套餐,无法精确反算真用量(按真=虚处理),详见 §6.3 | | `warnings.csv` 出现 `usage_without_active_package` | 该卡奇成 `total_bytes_cnt > 0``tbl_card_life` 中没有 status=1 且 type!=1 的当前生效正式套餐,无法精确反算真用量(按真=虚处理),详见 §6.3 |
| 线上报 `null value in column "creator"` | `migration_user_id` 是 0,某些约束可能拒绝;改成真实管理员 ID | | 线上报 `null value in column "creator"` | `migration_user_id` 是 0,某些约束可能拒绝;改成真实管理员 ID |
| 线上报 `null shop_id` | `mapping.agents.target_shop_code` 指向的店铺在新库不存在,先建店铺 | | 线上报目标店铺不存在 | `ownership_rules` `overrides`shop_code 在新库不存在,先建店铺或修正配置 |

View File

@@ -15,6 +15,43 @@ migration_user_id: 0
# 本次迁移批次号。线上每次迁移必须使用独立批次,用于导入后收尾、验收和回滚定位。 # 本次迁移批次号。线上每次迁移必须使用独立批次,用于导入后收尾、验收和回滚定位。
migration_batch_no: "MIGRATION-QICHENG-YYYYMMDD" migration_batch_no: "MIGRATION-QICHENG-YYYYMMDD"
# ============== 资产归属与槽位规则 ==============
# mapping.yaml 是迁移决策源。奇成 agent_id 不会默认参与归属;只有显式 legacy_agent 模式才读取 agents。
ownership_rules:
# default_shop 模式的批量目标店铺。legacy_agent/none 模式可保持 null。
default_target_shop_code: KWTX
device:
# default_shop=默认店铺;legacy_agent=按旧套餐读取槽位的绑定卡 agent_id 分店铺;none=进平台库存
mode: default_shop
# 批量默认当前槽位,用于 tb_device_sim_binding.is_current
current_slot: 2
# 批量默认旧套餐读取槽位。新系统套餐挂设备,这里只决定从奇成哪张绑定卡读套餐生命周期
package_source_slot: 2
standalone_card:
# default_shop=默认店铺;legacy_agent=按卡 agent_id 分店铺;none=进平台库存
mode: legacy_agent
# ============== 套餐迁移规则 ==============
package_rules:
# active=当前生效正式套餐,pending=未生效待生效正式套餐
migrate_statuses:
- active
- pending
# 可选:也可把套餐映射写在这里;scan_legacy 仍会增量维护顶层 packages。
packages: []
# ============== 少量资产级覆盖 ==============
# 覆盖优先级:devices.csv 行级槽位 > overrides.devices 槽位 > ownership_rules.device 默认槽位。
overrides:
devices:
- virtual_no: "862639073940258"
target_shop_code: OTHER
current_slot: 1
package_source_slot: 1
cards:
- iccid: "89861590172420360956"
target_shop_code: OTHER
# ============== 运营商映射 ============== # ============== 运营商映射 ==============
# scan_legacy 会按"奇成 tbl_card.account_id/account_name"分组写入,你只需填 target_* # scan_legacy 会按"奇成 tbl_card.account_id/account_name"分组写入,你只需填 target_*
# legacy_category 只是奇成运营商大类,用于人工判断 target_carrier_type,不要把它当具体运营商。 # legacy_category 只是奇成运营商大类,用于人工判断 target_carrier_type,不要把它当具体运营商。
@@ -41,10 +78,11 @@ series:
legacy_series_name: "国内卡 无预存30天" legacy_series_name: "国内卡 无预存30天"
target_series_id: null # 可选参考字段,生成 SQL 不依赖这里 target_series_id: null # 可选参考字段,生成 SQL 不依赖这里
# ============== 代理-店铺映射 ============== # ============== 代理-店铺映射(legacy_agent 归属 + 代理钱包) ==============
# scan_legacy 会按"奇成 tbl_card.agent_id"分组写入,你只需填 target_shop_code # scan_legacy 会按"奇成 tbl_card.agent_id"分组写入
# 留空(null)的代理:其卡进平台库存(除非 cards.csv 里显式指定 target_shop_code) # standalone_card/device mode=legacy_agent 时,资产按这里的 target_shop_code 分配。
# 不使用 legacy_agent 时,这里只影响 migrate_runtime.py 的代理钱包初始化。
agents: agents:
- legacy_agent_id: "D15EC779767B412A82B8448AC2324F04" - legacy_agent_id: "D15EC779767B412A82B8448AC2324F04"
legacy_agent_name: "叶发伟" legacy_agent_name: "叶发伟"
target_shop_code: null # 新库 tb_shop.shop_code,null=不归任何店铺 target_shop_code: null # 新库 tb_shop.shop_code;legacy_agent 模式下 null=该代理资产进平台库存

View File

@@ -1,9 +1,10 @@
"""cards.csv / devices.csv 加载与跨文件一致性校验。 """cards.csv / devices.csv 加载与跨文件一致性校验。
新设计:CSV 只承载"业务方决策"—— 新设计:CSV 只承载"资产清单和行级槽位决策"——
- cards.csv 只读 iccid + 可选覆盖列(target_shop_code、is_industry) - cards.csv 只读 iccid + is_industry
- devices.csv 读设备元数据 + sim_iccid_* - devices.csv 读设备元数据 + sim_iccid_*
- 卡的运营商、套餐、msisdn、agent_id 等元数据全部从奇成查(见 legacy_query.fetch_card_meta) - 卡的运营商、套餐、msisdn、agent_id 等元数据全部从奇成查(见 legacy_query.fetch_card_meta)
- 归属、默认当前槽位、默认旧套餐读取槽位从 mapping.yaml 解析
""" """
from __future__ import annotations from __future__ import annotations
@@ -47,7 +48,7 @@ class DeviceRow:
"""devices.csv 单行的解析结果。 """devices.csv 单行的解析结果。
简化为业务方核心关心的列:imei / virtual_no / device_* / max_sim_slots / sim_iccid_*。 简化为业务方核心关心的列:imei / virtual_no / device_* / max_sim_slots / sim_iccid_*。
店铺归属"主卡 sim_iccid_1 的 agent_id → mapping.agents.target_shop_code"推导 店铺归属、批量默认 current_slot 和旧套餐读取槽位 package_source_slot 由 mapping.yaml 决定
其他字段(manufacturer 等)直接用默认值,不暴露给业务方。 其他字段(manufacturer 等)直接用默认值,不暴露给业务方。
""" """
@@ -60,6 +61,9 @@ class DeviceRow:
device_type: str = "" device_type: str = ""
max_sim_slots: int = 1 # 不填时按非空 sim_iccid_* 的最大槽位推导 max_sim_slots: int = 1 # 不填时按非空 sim_iccid_* 的最大槽位推导
current_slot: int = 1 # 默认 slot 1 为当前生效 current_slot: int = 1 # 默认 slot 1 为当前生效
package_source_slot: int = 1 # 设备旧套餐读取槽位
current_slot_source: str = "default"
package_source_slot_source: str = "default"
@dataclass @dataclass
@@ -123,7 +127,7 @@ def _parse_int(raw: str, default: int = 0) -> int:
# ---------------- 主入口 ---------------- # ---------------- 主入口 ----------------
def load_assets(resources_dir: Path) -> tuple[list[CardRow], list[DeviceRow], list[ErrorRow]]: def load_assets(resources_dir: Path, mapping=None) -> tuple[list[CardRow], list[DeviceRow], list[ErrorRow]]:
"""读 cards.csv + devices.csv,做格式校验与跨文件一致性校验。 """读 cards.csv + devices.csv,做格式校验与跨文件一致性校验。
返回:(合法卡列表, 合法设备列表, 异常行列表)。 返回:(合法卡列表, 合法设备列表, 异常行列表)。
@@ -139,6 +143,8 @@ def load_assets(resources_dir: Path) -> tuple[list[CardRow], list[DeviceRow], li
devices_path = resources_dir / "devices.csv" devices_path = resources_dir / "devices.csv"
if devices_path.exists(): if devices_path.exists():
_load_devices(devices_path, result) _load_devices(devices_path, result)
if mapping is not None:
_apply_device_slot_rules(result, mapping)
_cross_check(result) _cross_check(result)
@@ -276,6 +282,23 @@ def _load_devices(path: Path, result: LoadResult) -> None:
) )
continue continue
current_slot, current_slot_source = _parse_slot_with_source(raw.get("current_slot", ""), 1)
if current_slot < 1 or current_slot > 4:
result.errors.append(
ErrorRow("devices.csv", idx, "current_slot", raw.get("current_slot", ""), "invalid_slot",
"current_slot 必须是 1-4 的整数")
)
continue
package_source_slot, package_source_slot_source = _parse_slot_with_source(
raw.get("package_source_slot", ""), current_slot
)
if package_source_slot < 1 or package_source_slot > 4:
result.errors.append(
ErrorRow("devices.csv", idx, "package_source_slot", raw.get("package_source_slot", ""), "invalid_slot",
"package_source_slot 必须是 1-4 的整数")
)
continue
result.devices.append( result.devices.append(
DeviceRow( DeviceRow(
line_no=idx, line_no=idx,
@@ -286,10 +309,63 @@ def _load_devices(path: Path, result: LoadResult) -> None:
device_type=device_type, device_type=device_type,
sim_iccids=sim_iccids, sim_iccids=sim_iccids,
max_sim_slots=max_sim_slots, max_sim_slots=max_sim_slots,
current_slot=current_slot,
package_source_slot=package_source_slot,
current_slot_source=current_slot_source,
package_source_slot_source=package_source_slot_source,
) )
) )
def _parse_slot_with_source(raw: str, default: int) -> tuple[int, str]:
"""解析槽位并返回来源。"""
v = _normalize_cell(raw)
if not v:
return default, "default"
try:
return int(v), "row"
except ValueError:
return 0, "row"
def _apply_device_slot_rules(result: LoadResult, mapping) -> None:
"""按 行配置 > 覆盖项 > 批量默认值 解析设备槽位。"""
for device in result.devices:
override = mapping.lookup_device_override(device.virtual_no)
if device.current_slot_source != "row":
if override and override.current_slot is not None:
device.current_slot = override.current_slot
device.current_slot_source = "override"
elif mapping.ownership_rules.device.current_slot is not None:
device.current_slot = mapping.ownership_rules.device.current_slot
device.current_slot_source = "ownership_rules.device"
if device.package_source_slot_source != "row":
if override and override.package_source_slot is not None:
device.package_source_slot = override.package_source_slot
device.package_source_slot_source = "override"
elif mapping.ownership_rules.device.package_source_slot is not None:
device.package_source_slot = mapping.ownership_rules.device.package_source_slot
device.package_source_slot_source = "ownership_rules.device"
else:
device.package_source_slot = device.current_slot
device.package_source_slot_source = device.current_slot_source
for field_name, slot in (
("current_slot", device.current_slot),
("package_source_slot", device.package_source_slot),
):
if slot < 1 or slot > 4:
result.errors.append(
ErrorRow("devices.csv", device.line_no, field_name, str(slot), "invalid_slot",
f"{field_name} 必须是 1-4 的整数")
)
elif slot > device.max_sim_slots:
result.errors.append(
ErrorRow("devices.csv", device.line_no, field_name, str(slot), "slot_overflow",
f"{field_name}={slot} 超出 max_sim_slots={device.max_sim_slots}")
)
def _cross_check(result: LoadResult) -> None: def _cross_check(result: LoadResult) -> None:
"""校验设备与卡之间的一致性,并把 bound_device_virtual_no 回填到 CardRow。 """校验设备与卡之间的一致性,并把 bound_device_virtual_no 回填到 CardRow。

View File

@@ -43,23 +43,43 @@ def _import_pymysql():
@dataclass(frozen=True) @dataclass(frozen=True)
class LegacyPackage: class LegacyPackage:
"""奇成当前生效套餐快照(取自 tbl_card_life)。""" """奇成当前生效正式套餐快照(取自 tbl_card_life)。"""
iccid: str iccid: str
meal_id: str meal_id: str
meal_name: str meal_name: str
meal_type: str
start_date: Optional[str] start_date: Optional[str]
expire_date: Optional[str] expire_date: Optional[str]
flow_size_mb: int # 套餐总量,单位 MB flow_size_mb: int # 套餐总量,单位 MB
status: int status: int
@dataclass(frozen=True)
class LegacyPackageLifecycle:
"""奇成正式套餐生命周期记录。"""
iccid: str
source_table: str
life_id: str
meal_id: str
meal_name: str
meal_type: str
status: int
start_date: Optional[str]
expire_date: Optional[str]
flow_size_mb: int
stable_sort_key: str
migration_status: str
skip_reason: str = ""
@dataclass(frozen=True) @dataclass(frozen=True)
class LegacyCardUsage: class LegacyCardUsage:
"""奇成卡累计流量快照。 """奇成卡累计流量快照。
奇成 ``tbl_card.total_bytes_cnt`` 存的是经过当前套餐 ``flow_add_discount`` 奇成 ``tbl_card.total_bytes_cnt`` 存的是经过当前套餐 ``flow_add_discount``
加成后的"虚用量",我们这里同时拉取当前生效套餐(``tbl_card_life`` status=1 加成后的"虚用量",我们这里同时拉取当前生效正式套餐(``tbl_card_life`` status=1,type!=1
且 ``expire_date`` 最大)的 ``flow_add_discount``,反算回新系统期望的"真用量" 且 ``expire_date`` 最大)的 ``flow_add_discount``,反算回新系统期望的"真用量"
精度说明:跨多个套餐周期累计时,``total_bytes_cnt`` 是按各时段对应套餐 精度说明:跨多个套餐周期累计时,``total_bytes_cnt`` 是按各时段对应套餐
@@ -70,7 +90,7 @@ class LegacyCardUsage:
iccid: str # 业务方视角的完整 ICCID iccid: str # 业务方视角的完整 ICCID
virtual_used_mb: Decimal # 奇成 total_bytes_cnt 原始值(虚 MB) virtual_used_mb: Decimal # 奇成 total_bytes_cnt 原始值(虚 MB)
flow_add_discount_pct: Decimal # 当前生效套餐虚比百分比(0~100) flow_add_discount_pct: Decimal # 当前生效套餐虚比百分比(0~100)
has_active_package: bool # 是否在 tbl_card_life 中找到当前生效套餐 has_active_package: bool # 是否在 tbl_card_life 中找到当前生效正式套餐
@property @property
def real_used_mb(self) -> Decimal: def real_used_mb(self) -> Decimal:
@@ -135,8 +155,9 @@ class LegacyCardMeta:
agent_name: str agent_name: str
msisdn: str # tbl_card.phone msisdn: str # tbl_card.phone
virtual_no: str # tbl_virtual_number.vcode(如有) virtual_no: str # tbl_virtual_number.vcode(如有)
current_meal_id: str # 当前生效套餐 ID(可空,取自 tbl_card_life status=1 expire_date 最大) current_meal_id: str # 当前生效正式套餐 ID(可空,取自 tbl_card_life status=1,type!=1,expire_date 最大)
current_meal_name: str current_meal_name: str
current_meal_type: str # tbl_card_life.type,1=叠加包/加油包,2=月卡,3=季卡,4=半年卡,5=年卡
current_expire_date: Optional[str] # ISO 字符串 current_expire_date: Optional[str] # ISO 字符串
@@ -217,7 +238,7 @@ def _to_mb_decimal(value) -> Decimal:
def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPackage]: def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPackage]:
"""查每张卡的当前生效套餐:status=1 且 expire_date 最大。 """查每张卡的当前生效套餐:status=1、type!=1 且 expire_date 最大。
返回: iccid → LegacyPackage(找不到的 iccid 不在结果中) 返回: iccid → LegacyPackage(找不到的 iccid 不在结果中)
""" """
@@ -231,6 +252,7 @@ def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPacka
iccid_mark, iccid_mark,
meal_id, meal_id,
meal_name, meal_name,
COALESCE(CAST(type AS CHAR), '') AS meal_type,
start_date, start_date,
expire_date, expire_date,
COALESCE(flow_size, 0) AS flow_size, COALESCE(flow_size, 0) AS flow_size,
@@ -238,6 +260,7 @@ def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPacka
FROM tbl_card_life FROM tbl_card_life
WHERE iccid_mark IN %s WHERE iccid_mark IN %s
AND status = 1 AND status = 1
AND COALESCE(CAST(type AS CHAR), '') <> '1'
ORDER BY iccid_mark ASC, expire_date DESC ORDER BY iccid_mark ASC, expire_date DESC
""" """
seen: set[str] = set() seen: set[str] = set()
@@ -253,6 +276,7 @@ def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPacka
iccid=iccid, iccid=iccid,
meal_id=row.get("meal_id") or "", meal_id=row.get("meal_id") or "",
meal_name=row.get("meal_name") or "", meal_name=row.get("meal_name") or "",
meal_type=str(row.get("meal_type") or ""),
start_date=row["start_date"].strftime("%Y-%m-%d %H:%M:%S") if row.get("start_date") else None, start_date=row["start_date"].strftime("%Y-%m-%d %H:%M:%S") if row.get("start_date") else None,
expire_date=row["expire_date"].strftime("%Y-%m-%d %H:%M:%S") if row.get("expire_date") else None, expire_date=row["expire_date"].strftime("%Y-%m-%d %H:%M:%S") if row.get("expire_date") else None,
flow_size_mb=int(row.get("flow_size") or 0), flow_size_mb=int(row.get("flow_size") or 0),
@@ -261,11 +285,172 @@ def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPacka
return out return out
def fetch_package_lifecycles(conn, iccid_pairs: Iterable[tuple]) -> dict[str, list[LegacyPackageLifecycle]]:
"""查当前生效和未生效待生效正式套餐生命周期。
只返回可迁移范围(active/pending)和需要在审核文件中说明的跳过记录。
加油包(type=1)和已过期记录不生成套餐 SQL,但保留 skip_reason 供审核产物输出。
"""
pairs = _normalize_iccid_pairs(iccid_pairs)
out: dict[str, list[LegacyPackageLifecycle]] = {}
if not pairs:
return out
query_to_full: dict[str, str] = {}
for i19, i20, allow_i19_lookup in pairs:
card_key = i20 or i19
query_to_full[card_key] = card_key
if i20 and allow_i19_lookup:
query_to_full[i19] = card_key
if not i20:
query_to_full[i19] = card_key
iccid_list = sorted(query_to_full.keys())
sql = """
SELECT
'tbl_card_life' AS source_table,
COALESCE(CAST(id AS CHAR), '') AS life_id,
iccid_mark,
COALESCE(meal_id, '') AS meal_id,
COALESCE(meal_name, '') AS meal_name,
COALESCE(CAST(type AS CHAR), '') AS meal_type,
status,
start_date,
expire_date,
COALESCE(flow_size, 0) AS flow_size
FROM tbl_card_life
WHERE iccid_mark IN %s
ORDER BY
iccid_mark ASC,
COALESCE(start_date, '1970-01-01') ASC,
COALESCE(expire_date, '9999-12-31') ASC,
id ASC
"""
with conn.cursor() as cur:
for batch in _chunks(iccid_list):
cur.execute(sql, (tuple(batch),))
for row in cur.fetchall():
legacy_iccid = row.get("iccid_mark") or ""
if not legacy_iccid:
continue
iccid = query_to_full.get(legacy_iccid)
if not iccid:
continue
start = row.get("start_date")
expire = row.get("expire_date")
start_str = start.strftime("%Y-%m-%d %H:%M:%S") if start else None
expire_str = expire.strftime("%Y-%m-%d %H:%M:%S") if expire else None
meal_type = str(row.get("meal_type") or "")
raw_status = int(row.get("status") or 0)
migration_status, skip_reason = _classify_package_lifecycle(meal_type, raw_status, start, expire)
life_id = str(row.get("life_id") or "")
stable_sort_key = "|".join([start_str or "", expire_str or "", life_id])
item = LegacyPackageLifecycle(
iccid=iccid,
source_table=row.get("source_table") or "tbl_card_life",
life_id=life_id,
meal_id=row.get("meal_id") or "",
meal_name=row.get("meal_name") or "",
meal_type=meal_type,
status=raw_status,
start_date=start_str,
expire_date=expire_str,
flow_size_mb=int(row.get("flow_size") or 0),
stable_sort_key=stable_sort_key,
migration_status=migration_status,
skip_reason=skip_reason,
)
out.setdefault(iccid, []).append(item)
next_sql = """
SELECT
'tbl_next_month_card_life' AS source_table,
COALESCE(CAST(id AS CHAR), '') AS life_id,
iccid_mark,
COALESCE(meal_id, '') AS meal_id,
COALESCE(meal_name, '') AS meal_name,
COALESCE(CAST(order_type AS CHAR), '') AS meal_type,
status,
effect_begin_time AS start_date,
effect_complete_time AS expire_date,
COALESCE(flow_size, 0) AS flow_size
FROM tbl_next_month_card_life
WHERE iccid_mark IN %s
ORDER BY
iccid_mark ASC,
COALESCE(effect_begin_time, '1970-01-01') ASC,
COALESCE(effect_complete_time, '9999-12-31') ASC,
id ASC
"""
with conn.cursor() as cur:
for batch in _chunks(iccid_list):
cur.execute(next_sql, (tuple(batch),))
for row in cur.fetchall():
legacy_iccid = row.get("iccid_mark") or ""
if not legacy_iccid:
continue
iccid = query_to_full.get(legacy_iccid)
if not iccid:
continue
start = row.get("start_date")
expire = row.get("expire_date")
start_str = start.strftime("%Y-%m-%d %H:%M:%S") if start else None
expire_str = expire.strftime("%Y-%m-%d %H:%M:%S") if expire else None
raw_status = int(row.get("status") or 0)
migration_status, skip_reason = _classify_next_month_lifecycle(raw_status)
life_id = str(row.get("life_id") or "")
stable_sort_key = "|".join([start_str or "", expire_str or "", f"next:{life_id}"])
out.setdefault(iccid, []).append(LegacyPackageLifecycle(
iccid=iccid,
source_table=row.get("source_table") or "tbl_next_month_card_life",
life_id=life_id,
meal_id=row.get("meal_id") or "",
meal_name=row.get("meal_name") or "",
meal_type=str(row.get("meal_type") or ""),
status=raw_status,
start_date=start_str,
expire_date=expire_str,
flow_size_mb=int(row.get("flow_size") or 0),
stable_sort_key=stable_sort_key,
migration_status=migration_status,
skip_reason=skip_reason,
))
return out
def _classify_package_lifecycle(meal_type: str, status: int, start_date, expire_date) -> tuple[str, str]:
"""把奇成生命周期记录分类为 active/pending/skipped。"""
if meal_type == "1":
return "skipped", "加油包不迁移"
from datetime import datetime
now = datetime.now()
if expire_date and expire_date <= now:
return "skipped", "已过期套餐不迁移"
if status == 1:
if start_date and start_date > now:
return "pending", ""
return "active", ""
if start_date and start_date > now:
return "pending", ""
return "skipped", f"奇成状态 {status} 不在迁移范围"
def _classify_next_month_lifecycle(status: int) -> tuple[str, str]:
"""把奇成次月待生效套餐记录分类为 pending/skipped。"""
if status == 1:
return "pending", ""
if status == 2:
return "skipped", "次月待生效记录已插入 tbl_card_life,避免重复迁移"
return "skipped", f"奇成次月待生效状态 {status} 不在迁移范围"
def fetch_card_usage(conn, iccid_pairs: Iterable[tuple]) -> dict[str, LegacyCardUsage]: def fetch_card_usage(conn, iccid_pairs: Iterable[tuple]) -> dict[str, LegacyCardUsage]:
"""查每张卡的累计已用流量,并同步取当前生效套餐的 flow_add_discount。 """查每张卡的累计已用流量,并同步取当前生效正式套餐的 flow_add_discount。
奇成 ``tbl_card.total_bytes_cnt`` 是虚用量(已被套餐虚比加成)。 奇成 ``tbl_card.total_bytes_cnt`` 是虚用量(已被套餐虚比加成)。
这里 JOIN 当前生效套餐(``tbl_card_life`` status=1 且 ``expire_date`` 最大) 这里 JOIN 当前生效正式套餐(``tbl_card_life`` status=1,type!=1 且 ``expire_date`` 最大)
+ ``tbl_set_meal`` 取 ``flow_add_discount``,放进 ``LegacyCardUsage``; + ``tbl_set_meal`` 取 ``flow_add_discount``,放进 ``LegacyCardUsage``;
上层通过 ``real_used_mb_floor`` 反算真用量写入新系统。 上层通过 ``real_used_mb_floor`` 反算真用量写入新系统。
@@ -312,12 +497,15 @@ def fetch_card_usage(conn, iccid_pairs: Iterable[tuple]) -> dict[str, LegacyCard
INNER JOIN ( INNER JOIN (
SELECT iccid_mark, MAX(expire_date) AS max_expire SELECT iccid_mark, MAX(expire_date) AS max_expire
FROM tbl_card_life FROM tbl_card_life
WHERE status = 1 AND iccid_mark IN %s WHERE status = 1
AND COALESCE(CAST(type AS CHAR), '') <> '1'
AND iccid_mark IN %s
GROUP BY iccid_mark GROUP BY iccid_mark
) latest ) latest
ON latest.iccid_mark = cl1.iccid_mark ON latest.iccid_mark = cl1.iccid_mark
AND latest.max_expire = cl1.expire_date AND latest.max_expire = cl1.expire_date
WHERE cl1.status = 1 WHERE cl1.status = 1
AND COALESCE(CAST(cl1.type AS CHAR), '') <> '1'
) cl ON ( ) cl ON (
cl.iccid_mark = c.iccid_mark cl.iccid_mark = c.iccid_mark
OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(cl.iccid_mark, 19) = c.iccid_mark) OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(cl.iccid_mark, 19) = c.iccid_mark)
@@ -440,6 +628,7 @@ def fetch_card_meta(
COALESCE(vn.vcode, '') AS vcode, COALESCE(vn.vcode, '') AS vcode,
COALESCE(cl.meal_id, '') AS meal_id, COALESCE(cl.meal_id, '') AS meal_id,
COALESCE(cl.meal_name, '') AS meal_name, COALESCE(cl.meal_name, '') AS meal_name,
COALESCE(CAST(cl.meal_type AS CHAR), '') AS meal_type,
cl.expire_date AS expire_date cl.expire_date AS expire_date
FROM tbl_card c FROM tbl_card c
LEFT JOIN tbl_vendor_category vc ON vc.category = c.category LEFT JOIN tbl_vendor_category vc ON vc.category = c.category
@@ -448,17 +637,20 @@ def fetch_card_meta(
OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(vn.iccid_mark, 19) = c.iccid_mark) OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(vn.iccid_mark, 19) = c.iccid_mark)
) )
LEFT JOIN ( LEFT JOIN (
SELECT cl1.iccid_mark, cl1.meal_id, cl1.meal_name, cl1.expire_date SELECT cl1.iccid_mark, cl1.meal_id, cl1.meal_name, cl1.type AS meal_type, cl1.expire_date
FROM tbl_card_life cl1 FROM tbl_card_life cl1
INNER JOIN ( INNER JOIN (
SELECT iccid_mark, MAX(expire_date) AS max_expire SELECT iccid_mark, MAX(expire_date) AS max_expire
FROM tbl_card_life FROM tbl_card_life
WHERE status = 1 AND iccid_mark IN %s WHERE status = 1
AND COALESCE(CAST(type AS CHAR), '') <> '1'
AND iccid_mark IN %s
GROUP BY iccid_mark GROUP BY iccid_mark
) latest ) latest
ON latest.iccid_mark = cl1.iccid_mark ON latest.iccid_mark = cl1.iccid_mark
AND latest.max_expire = cl1.expire_date AND latest.max_expire = cl1.expire_date
WHERE cl1.status = 1 WHERE cl1.status = 1
AND COALESCE(CAST(cl1.type AS CHAR), '') <> '1'
) cl ON ( ) cl ON (
cl.iccid_mark = c.iccid_mark cl.iccid_mark = c.iccid_mark
OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(cl.iccid_mark, 19) = c.iccid_mark) OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(cl.iccid_mark, 19) = c.iccid_mark)
@@ -511,6 +703,7 @@ def fetch_card_meta(
virtual_no=row.get("vcode") or "", virtual_no=row.get("vcode") or "",
current_meal_id=row.get("meal_id") or "", current_meal_id=row.get("meal_id") or "",
current_meal_name=row.get("meal_name") or "", current_meal_name=row.get("meal_name") or "",
current_meal_type=str(row.get("meal_type") or ""),
current_expire_date=expire_str, current_expire_date=expire_str,
) )
return metas, duplicates return metas, duplicates

View File

@@ -7,6 +7,8 @@
""" """
from __future__ import annotations from __future__ import annotations
import json
import re
from dataclasses import dataclass, field from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
from typing import Any, Optional from typing import Any, Optional
@@ -27,6 +29,69 @@ class MissingTargetError(Exception):
) )
SHOP_CODE_RE = re.compile(r"^[A-Za-z0-9_-]{2,64}$")
ALLOWED_OWNERSHIP_MODES = {"default_shop", "legacy_agent", "none"}
ALLOWED_PACKAGE_STATUSES = {"active", "pending"}
@dataclass(frozen=True)
class OwnershipDeviceRule:
"""设备批量归属规则。"""
mode: str = "none"
current_slot: Optional[int] = None
package_source_slot: Optional[int] = None
@dataclass(frozen=True)
class OwnershipStandaloneCardRule:
"""独立卡批量归属规则。"""
mode: str = "none"
@dataclass(frozen=True)
class OwnershipRules:
"""资产归属决策规则。"""
default_target_shop_code: Optional[str] = None
device: OwnershipDeviceRule = field(default_factory=OwnershipDeviceRule)
standalone_card: OwnershipStandaloneCardRule = field(default_factory=OwnershipStandaloneCardRule)
@dataclass(frozen=True)
class PackageRules:
"""套餐迁移规则。"""
migrate_statuses: tuple[str, ...] = ("active", "pending")
@dataclass(frozen=True)
class DeviceOverride:
"""设备级覆盖项。"""
virtual_no: str
target_shop_code: Optional[str] = None
current_slot: Optional[int] = None
package_source_slot: Optional[int] = None
@dataclass(frozen=True)
class CardOverride:
"""卡级覆盖项。"""
iccid: str
target_shop_code: Optional[str] = None
@dataclass(frozen=True)
class Overrides:
"""少量资产级覆盖项。"""
devices: dict[str, DeviceOverride] = field(default_factory=dict)
cards: dict[str, CardOverride] = field(default_factory=dict)
@dataclass(frozen=True) @dataclass(frozen=True)
class CarrierMapping: class CarrierMapping:
legacy_account_id: str legacy_account_id: str
@@ -73,7 +138,7 @@ class SeriesMapping:
class AgentMapping: class AgentMapping:
legacy_agent_id: str legacy_agent_id: str
legacy_agent_name: str legacy_agent_name: str
target_shop_code: Optional[str] # 允许 null,表示该代理的卡进平台库存 target_shop_code: Optional[str] # legacy_agent 模式下允许 null,表示该代理资产进平台库存
@dataclass @dataclass
@@ -82,6 +147,9 @@ class Mapping:
migration_user_id: int = 0 migration_user_id: int = 0
migration_batch_no: str = "MIGRATION-QICHENG" migration_batch_no: str = "MIGRATION-QICHENG"
ownership_rules: OwnershipRules = field(default_factory=OwnershipRules)
package_rules: PackageRules = field(default_factory=PackageRules)
overrides: Overrides = field(default_factory=Overrides)
carriers: dict[str, CarrierMapping] = field(default_factory=dict) carriers: dict[str, CarrierMapping] = field(default_factory=dict)
packages: dict[str, PackageMapping] = field(default_factory=dict) packages: dict[str, PackageMapping] = field(default_factory=dict)
series: dict[str, SeriesMapping] = field(default_factory=dict) series: dict[str, SeriesMapping] = field(default_factory=dict)
@@ -108,6 +176,18 @@ class Mapping:
code = (agent.target_shop_code or "").strip() code = (agent.target_shop_code or "").strip()
return code or None return code or None
def lookup_device_override(self, virtual_no: str) -> Optional[DeviceOverride]:
"""按设备号查覆盖项。"""
return self.overrides.devices.get(str(virtual_no or "").strip())
def lookup_card_override(self, *iccids: str) -> Optional[CardOverride]:
"""按完整 ICCID 或 19 位 ICCID 查覆盖项。"""
for iccid in iccids:
item = self.overrides.cards.get(str(iccid or "").strip())
if item is not None:
return item
return None
# ---------------- 加载 ---------------- # ---------------- 加载 ----------------
@@ -125,10 +205,18 @@ def load_mapping(config_dir: Path) -> Mapping:
raw = yaml.safe_load(f) or {} raw = yaml.safe_load(f) or {}
if not isinstance(raw, dict): if not isinstance(raw, dict):
raise ValueError(f"{path} 顶层必须是 mapping 类型") raise ValueError(f"{path} 顶层必须是 mapping 类型")
if "ownership_rules" not in raw:
raise ValueError(
f"{path} 缺少 ownership_rules。新迁移流程必须显式声明资产归属模式,请参考 "
"config/mapping.yaml.example 添加 ownership_rules、package_rules 和 overrides 后重跑。"
)
m = Mapping( m = Mapping(
migration_user_id=int(raw.get("migration_user_id", 0)), migration_user_id=int(raw.get("migration_user_id", 0)),
migration_batch_no=_opt_str(raw.get("migration_batch_no")) or "MIGRATION-QICHENG", migration_batch_no=_opt_str(raw.get("migration_batch_no")) or "MIGRATION-QICHENG",
ownership_rules=_load_ownership_rules(raw.get("ownership_rules") or {}),
package_rules=_load_package_rules(raw.get("package_rules") or {}),
overrides=_load_overrides(raw.get("overrides") or {}),
) )
for item in raw.get("carriers") or []: for item in raw.get("carriers") or []:
@@ -147,7 +235,11 @@ def load_mapping(config_dir: Path) -> Mapping:
) )
m.carriers[c.legacy_account_id] = c m.carriers[c.legacy_account_id] = c
for item in raw.get("packages") or []: package_items = list(raw.get("packages") or [])
package_rules_raw = raw.get("package_rules") or {}
if isinstance(package_rules_raw, dict):
package_items.extend(package_rules_raw.get("packages") or [])
for item in package_items:
p = PackageMapping( p = PackageMapping(
legacy_meal_id=str(item["legacy_meal_id"]).strip(), legacy_meal_id=str(item["legacy_meal_id"]).strip(),
legacy_meal_name=str(item.get("legacy_meal_name") or ""), legacy_meal_name=str(item.get("legacy_meal_name") or ""),
@@ -167,13 +259,105 @@ def load_mapping(config_dir: Path) -> Mapping:
a = AgentMapping( a = AgentMapping(
legacy_agent_id=str(item["legacy_agent_id"]).strip(), legacy_agent_id=str(item["legacy_agent_id"]).strip(),
legacy_agent_name=str(item.get("legacy_agent_name") or ""), legacy_agent_name=str(item.get("legacy_agent_name") or ""),
target_shop_code=_opt_str(item.get("target_shop_code")), target_shop_code=_opt_shop_code(item.get("target_shop_code"), "agents[].target_shop_code"),
) )
m.agents[a.legacy_agent_id] = a m.agents[a.legacy_agent_id] = a
return m return m
def _load_ownership_rules(raw: dict) -> OwnershipRules:
"""加载归属规则配置。"""
if not isinstance(raw, dict):
raise ValueError("mapping.yaml.ownership_rules 必须是 mapping 类型")
default_shop_code = _opt_shop_code(raw.get("default_target_shop_code"), "ownership_rules.default_target_shop_code")
device_raw = raw.get("device") or {}
standalone_raw = raw.get("standalone_card") or {}
if not isinstance(device_raw, dict):
raise ValueError("mapping.yaml.ownership_rules.device 必须是 mapping 类型")
if not isinstance(standalone_raw, dict):
raise ValueError("mapping.yaml.ownership_rules.standalone_card 必须是 mapping 类型")
device_mode = _mode(device_raw.get("mode"), "ownership_rules.device.mode")
standalone_mode = _mode(standalone_raw.get("mode"), "ownership_rules.standalone_card.mode")
if (device_mode == "default_shop" or standalone_mode == "default_shop") and not default_shop_code:
raise ValueError("mapping.yaml.ownership_rules.default_target_shop_code 不能为空:default_shop 模式必须指定目标店铺码")
return OwnershipRules(
default_target_shop_code=default_shop_code,
device=OwnershipDeviceRule(
mode=device_mode,
current_slot=_slot_or_none(device_raw.get("current_slot"), "ownership_rules.device.current_slot"),
package_source_slot=_slot_or_none(
device_raw.get("package_source_slot"), "ownership_rules.device.package_source_slot"
),
),
standalone_card=OwnershipStandaloneCardRule(mode=standalone_mode),
)
def _load_package_rules(raw: dict) -> PackageRules:
"""加载套餐迁移规则。"""
if not isinstance(raw, dict):
raise ValueError("mapping.yaml.package_rules 必须是 mapping 类型")
statuses_raw = raw.get("migrate_statuses")
if statuses_raw is None:
statuses = ("active", "pending")
elif not isinstance(statuses_raw, list):
raise ValueError("mapping.yaml.package_rules.migrate_statuses 必须是列表")
else:
statuses = tuple(str(x).strip() for x in statuses_raw if str(x).strip())
invalid = [x for x in statuses if x not in ALLOWED_PACKAGE_STATUSES]
if invalid:
raise ValueError(f"mapping.yaml.package_rules.migrate_statuses 仅支持 active/pending,无效值: {invalid}")
if not statuses:
raise ValueError("mapping.yaml.package_rules.migrate_statuses 至少要包含 active 或 pending")
return PackageRules(migrate_statuses=statuses)
def _load_overrides(raw: dict) -> Overrides:
"""加载资产覆盖项。"""
if not isinstance(raw, dict):
raise ValueError("mapping.yaml.overrides 必须是 mapping 类型")
devices: dict[str, DeviceOverride] = {}
for idx, item in enumerate(raw.get("devices") or [], start=1):
if not isinstance(item, dict):
raise ValueError(f"mapping.yaml.overrides.devices[{idx}] 必须是 mapping 类型")
virtual_no = _opt_str(item.get("virtual_no"))
if not virtual_no:
raise ValueError(f"mapping.yaml.overrides.devices[{idx}].virtual_no 不能为空")
if virtual_no in devices:
raise ValueError(f"mapping.yaml.overrides.devices virtual_no 重复: {virtual_no}")
devices[virtual_no] = DeviceOverride(
virtual_no=virtual_no,
target_shop_code=_opt_shop_code(item.get("target_shop_code"), f"overrides.devices[{idx}].target_shop_code"),
current_slot=_slot_or_none(item.get("current_slot"), f"overrides.devices[{idx}].current_slot"),
package_source_slot=_slot_or_none(
item.get("package_source_slot"), f"overrides.devices[{idx}].package_source_slot"
),
)
cards: dict[str, CardOverride] = {}
for idx, item in enumerate(raw.get("cards") or [], start=1):
if not isinstance(item, dict):
raise ValueError(f"mapping.yaml.overrides.cards[{idx}] 必须是 mapping 类型")
iccid = _opt_str(item.get("iccid"))
if not iccid:
raise ValueError(f"mapping.yaml.overrides.cards[{idx}].iccid 不能为空")
if len(iccid) not in (19, 20):
raise ValueError(f"mapping.yaml.overrides.cards[{idx}].iccid 必须是 19 或 20 位")
if iccid in cards:
raise ValueError(f"mapping.yaml.overrides.cards iccid 重复: {iccid}")
cards[iccid] = CardOverride(
iccid=iccid,
target_shop_code=_opt_shop_code(item.get("target_shop_code"), f"overrides.cards[{idx}].target_shop_code"),
)
return Overrides(devices=devices, cards=cards)
def merge_and_save(config_dir: Path, mapping: Mapping) -> Path: def merge_and_save(config_dir: Path, mapping: Mapping) -> Path:
"""把 mapping 写回 config/mapping.yaml,保留人工已填的 target_* 字段。 """把 mapping 写回 config/mapping.yaml,保留人工已填的 target_* 字段。
@@ -181,49 +365,233 @@ def merge_and_save(config_dir: Path, mapping: Mapping) -> Path:
""" """
config_dir.mkdir(parents=True, exist_ok=True) config_dir.mkdir(parents=True, exist_ok=True)
path = config_dir / MAPPING_FILE path = config_dir / MAPPING_FILE
data = { path.write_text(_render_commented_mapping_yaml(mapping), encoding="utf-8")
"migration_user_id": mapping.migration_user_id, return path
"migration_batch_no": mapping.migration_batch_no,
"carriers": [
def _render_commented_mapping_yaml(mapping: Mapping) -> str:
"""渲染带中文注释的 mapping.yaml。
PyYAML 写回时会丢弃注释,这里用固定模板承载说明,动态列表仍按扫描结果增量输出。
"""
lines: list[str] = [
"# 奇成迁移映射配置(由 scan_legacy.py 生成/增量补全)",
"#",
"# 使用方式:",
"# 1. cards.csv 必填;devices.csv 可选。没有 devices.csv 时就是纯卡迁移。",
"# 2. legacy_* 字段来自奇成扫描,一般不要手工修改。",
"# 3. target_* 字段需要人工填写,否则生成 SQL 时会写入 errors.csv 并阻断对应资产。",
"# 4. 资产归属先看 overrides,再看 ownership_rules;只有显式 legacy_agent 模式才读取 agents。",
"",
"# ============== 全局配置 ==============",
"# 迁移操作者账号 ID,写入 creator/updater/operator_id。线上建议使用后台超级管理员 ID。",
f"migration_user_id: {_yaml_scalar(mapping.migration_user_id)}",
"# 本次迁移批次号。每批线上迁移保持唯一,方便收尾、验收和回滚定位。",
f"migration_batch_no: {_yaml_scalar(mapping.migration_batch_no)}",
"",
"# ============== 资产归属与槽位规则 ==============",
"# default_shop 模式下的批量目标店铺编码;mode=legacy_agent/none 时可保持 null。",
"ownership_rules:",
f" default_target_shop_code: {_yaml_scalar(mapping.ownership_rules.default_target_shop_code)}",
" # 设备规则只在存在 resources/devices.csv 时生效。",
" device:",
" # default_shop=默认店铺;legacy_agent=按旧套餐读取槽位绑定卡的奇成 agent_id 映射店铺;none=平台库存。",
f" mode: {_yaml_scalar(mapping.ownership_rules.device.mode)}",
" # 当前卡槽,写入 tb_device_sim_binding.is_current;纯卡迁移可保持 null。",
f" current_slot: {_yaml_scalar(mapping.ownership_rules.device.current_slot)}",
" # 设备旧套餐读取槽位。新系统套餐挂设备,这里只决定从奇成哪张绑定卡读套餐生命周期;纯卡迁移可保持 null。",
f" package_source_slot: {_yaml_scalar(mapping.ownership_rules.device.package_source_slot)}",
" standalone_card:",
" # default_shop=默认店铺;legacy_agent=按奇成 agent_id 映射店铺;none=平台库存。",
f" mode: {_yaml_scalar(mapping.ownership_rules.standalone_card.mode)}",
"",
"# ============== 套餐迁移规则 ==============",
"package_rules:",
" # active=当前生效正式套餐;pending=未生效待生效正式套餐。",
" migrate_statuses:",
]
for status in mapping.package_rules.migrate_statuses:
lines.append(f" - {_yaml_scalar(status)}")
lines.extend([
"",
"# ============== 少量资产级覆盖 ==============",
"# 覆盖优先级:devices.csv 行级槽位 > overrides.devices 槽位 > ownership_rules.device 默认槽位。",
"overrides:",
" # 设备级特殊处理。virtual_no 对应 devices.csv 的 virtual_no/imei 兜底值。",
])
_append_device_overrides(lines, mapping.overrides)
lines.append(" # 独立卡特殊处理。iccid 可填 19 位或 20 位。")
_append_card_overrides(lines, mapping.overrides)
lines.extend([
"",
"# ============== 运营商映射 ==============",
"# scan_legacy 按奇成 tbl_card.account_id/account_name 分组写入;只需要人工填写 target_*。",
"# target_carrier_type 只能填 CMCC / CUCC / CTCC / CBN。",
])
_append_carriers(lines, mapping)
lines.extend([
"",
"# ============== 套餐映射 ==============",
"# scan_legacy 按奇成 tbl_card_life.meal_id 分组写入;target_package_id 必须指向新库正式套餐。",
])
_append_packages(lines, mapping)
lines.extend([
"",
"# ============== 套餐系列快照(可选) ==============",
"# 仅保留奇成套餐系列供人工参考;资产 series_id 会从 target_package_id 对应的 tb_package.series_id 反查。",
])
_append_series(lines, mapping)
lines.extend([
"",
"# ============== 代理-店铺映射(legacy_agent 归属 + 代理钱包) ==============",
"# standalone_card/device mode=legacy_agent 时,资产按这里的 target_shop_code 分配。",
"# 不使用 legacy_agent 时,这里只影响 migrate_runtime.py 的代理钱包初始化。",
])
_append_agents(lines, mapping)
return "\n".join(lines) + "\n"
def _append_device_overrides(lines: list[str], overrides: Overrides) -> None:
if not overrides.devices:
lines.append(" devices: []")
return
lines.append(" devices:")
for item in sorted(overrides.devices.values(), key=lambda x: x.virtual_no):
lines.extend([
f" - virtual_no: {_yaml_scalar(item.virtual_no)}",
f" target_shop_code: {_yaml_scalar(item.target_shop_code)}",
f" current_slot: {_yaml_scalar(item.current_slot)}",
f" package_source_slot: {_yaml_scalar(item.package_source_slot)}",
])
def _append_card_overrides(lines: list[str], overrides: Overrides) -> None:
if not overrides.cards:
lines.append(" cards: []")
return
lines.append(" cards:")
for item in sorted(overrides.cards.values(), key=lambda x: x.iccid):
lines.extend([
f" - iccid: {_yaml_scalar(item.iccid)}",
f" target_shop_code: {_yaml_scalar(item.target_shop_code)}",
])
def _append_carriers(lines: list[str], mapping: Mapping) -> None:
if not mapping.carriers:
lines.append("carriers: []")
return
lines.append("carriers:")
for item in sorted(mapping.carriers.values(), key=lambda x: (x.legacy_account_name, x.legacy_account_id)):
lines.extend([
f" - legacy_account_id: {_yaml_scalar(item.legacy_account_id)}",
f" legacy_account_name: {_yaml_scalar(item.legacy_account_name)}",
f" legacy_category: {_yaml_scalar(item.legacy_category)}",
f" legacy_category_name: {_yaml_scalar(item.legacy_category_name)}",
f" target_carrier_id: {_yaml_scalar(item.target_carrier_id)}",
f" target_carrier_type: {_yaml_scalar(item.target_carrier_type)}",
f" target_carrier_name: {_yaml_scalar(item.target_carrier_name)}",
])
def _append_packages(lines: list[str], mapping: Mapping) -> None:
if not mapping.packages:
lines.append("packages: []")
return
lines.append("packages:")
for item in sorted(mapping.packages.values(), key=lambda x: x.legacy_meal_id):
lines.extend([
f" - legacy_meal_id: {_yaml_scalar(item.legacy_meal_id)}",
f" legacy_meal_name: {_yaml_scalar(item.legacy_meal_name)}",
f" target_package_id: {_yaml_scalar(item.target_package_id)}",
])
def _append_series(lines: list[str], mapping: Mapping) -> None:
if not mapping.series:
lines.append("series: []")
return
lines.append("series:")
for item in sorted(mapping.series.values(), key=lambda x: x.legacy_series_id):
lines.extend([
f" - legacy_series_id: {_yaml_scalar(item.legacy_series_id)}",
f" legacy_series_name: {_yaml_scalar(item.legacy_series_name)}",
f" target_series_id: {_yaml_scalar(item.target_series_id)}",
])
def _append_agents(lines: list[str], mapping: Mapping) -> None:
if not mapping.agents:
lines.append("agents: []")
return
lines.append("agents:")
for item in sorted(mapping.agents.values(), key=lambda x: x.legacy_agent_id):
lines.extend([
f" - legacy_agent_id: {_yaml_scalar(item.legacy_agent_id)}",
f" legacy_agent_name: {_yaml_scalar(item.legacy_agent_name)}",
f" target_shop_code: {_yaml_scalar(item.target_shop_code)}",
])
def _yaml_scalar(value: Any) -> str:
"""把 Python 标量转成稳定 YAML 标量。"""
if value is None:
return "null"
if isinstance(value, bool):
return "true" if value else "false"
if isinstance(value, int):
return str(value)
return json.dumps(str(value), ensure_ascii=False)
def _dump_ownership_rules(rules: OwnershipRules) -> dict:
"""序列化归属规则。"""
return {
"default_target_shop_code": rules.default_target_shop_code,
"device": {
"mode": rules.device.mode,
"current_slot": rules.device.current_slot,
"package_source_slot": rules.device.package_source_slot,
},
"standalone_card": {
"mode": rules.standalone_card.mode,
},
}
def _dump_package_rules(rules: PackageRules) -> dict:
"""序列化套餐规则。"""
return {
"migrate_statuses": list(rules.migrate_statuses),
}
def _dump_overrides(overrides: Overrides) -> dict:
"""序列化资产覆盖项。"""
return {
"devices": [
{ {
"legacy_account_id": c.legacy_account_id, "virtual_no": x.virtual_no,
"legacy_account_name": c.legacy_account_name, "target_shop_code": x.target_shop_code,
"legacy_category": c.legacy_category, "current_slot": x.current_slot,
"legacy_category_name": c.legacy_category_name, "package_source_slot": x.package_source_slot,
"target_carrier_id": c.target_carrier_id,
"target_carrier_type": c.target_carrier_type,
"target_carrier_name": c.target_carrier_name,
} }
for c in sorted(mapping.carriers.values(), key=lambda x: (x.legacy_account_name, x.legacy_account_id)) for x in sorted(overrides.devices.values(), key=lambda item: item.virtual_no)
], ],
"packages": [ "cards": [
{ {
"legacy_meal_id": p.legacy_meal_id, "iccid": x.iccid,
"legacy_meal_name": p.legacy_meal_name, "target_shop_code": x.target_shop_code,
"target_package_id": p.target_package_id,
} }
for p in sorted(mapping.packages.values(), key=lambda x: x.legacy_meal_id) for x in sorted(overrides.cards.values(), key=lambda item: item.iccid)
],
"series": [
{
"legacy_series_id": s.legacy_series_id,
"legacy_series_name": s.legacy_series_name,
"target_series_id": s.target_series_id,
}
for s in sorted(mapping.series.values(), key=lambda x: x.legacy_series_id)
],
"agents": [
{
"legacy_agent_id": a.legacy_agent_id,
"legacy_agent_name": a.legacy_agent_name,
"target_shop_code": a.target_shop_code,
}
for a in sorted(mapping.agents.values(), key=lambda x: x.legacy_agent_id)
], ],
} }
with path.open("w", encoding="utf-8") as f:
yaml.safe_dump(data, f, allow_unicode=True, sort_keys=False, default_flow_style=False)
return path
def load_legacy_dsn(config_dir: Path) -> dict: def load_legacy_dsn(config_dir: Path) -> dict:
@@ -253,3 +621,31 @@ def _opt_str(v: Any) -> Optional[str]:
return None return None
s = str(v).strip() s = str(v).strip()
return s or None return s or None
def _opt_shop_code(v: Any, field_name: str) -> Optional[str]:
code = _opt_str(v)
if code is None:
return None
if not SHOP_CODE_RE.match(code):
raise ValueError(f"mapping.yaml.{field_name} 店铺码格式无效: {code!r},仅允许 2-64 位字母/数字/_/-")
return code
def _mode(v: Any, field_name: str) -> str:
mode = _opt_str(v) or "none"
if mode not in ALLOWED_OWNERSHIP_MODES:
raise ValueError(f"mapping.yaml.{field_name} 仅支持 default_shop/legacy_agent/none,无效值: {mode!r}")
return mode
def _slot_or_none(v: Any, field_name: str) -> Optional[int]:
if v is None or v == "":
return None
try:
slot = int(v)
except (TypeError, ValueError) as exc:
raise ValueError(f"mapping.yaml.{field_name} 必须是 1-4 的整数") from exc
if slot < 1 or slot > 4:
raise ValueError(f"mapping.yaml.{field_name} 必须在 1-4 之间")
return slot

File diff suppressed because it is too large Load Diff

View File

@@ -40,7 +40,7 @@ def main() -> int:
output_dir.mkdir(parents=True, exist_ok=True) output_dir.mkdir(parents=True, exist_ok=True)
mapping = mapping_loader.load_mapping(config_dir) mapping = mapping_loader.load_mapping(config_dir)
cards, devices, errors = csv_loader.load_assets(resources_dir) cards, devices, errors = csv_loader.load_assets(resources_dir, mapping)
# 连奇成查卡元数据(运营商分类、agent_id、msisdn、virtual_no、当前生效套餐) # 连奇成查卡元数据(运营商分类、agent_id、msisdn、virtual_no、当前生效套餐)
iccid_pairs = [(c.iccid_19, c.iccid_20, c.allow_iccid_19_lookup) for c in cards] iccid_pairs = [(c.iccid_19, c.iccid_20, c.allow_iccid_19_lookup) for c in cards]

View File

@@ -37,7 +37,7 @@ def main() -> int:
output_dir.mkdir(parents=True, exist_ok=True) output_dir.mkdir(parents=True, exist_ok=True)
mapping = mapping_loader.load_mapping(config_dir) mapping = mapping_loader.load_mapping(config_dir)
cards, devices, errors = csv_loader.load_assets(resources_dir) cards, devices, errors = csv_loader.load_assets(resources_dir, mapping)
if errors: if errors:
sql_builder.write_runtime_input_errors(output_dir, errors) sql_builder.write_runtime_input_errors(output_dir, errors)
print(f"cards.csv/devices.csv 存在 {len(errors)} 个基础错误,已写入 {output_dir / 'errors.csv'}", file=sys.stderr) print(f"cards.csv/devices.csv 存在 {len(errors)} 个基础错误,已写入 {output_dir / 'errors.csv'}", file=sys.stderr)
@@ -50,6 +50,7 @@ def main() -> int:
with legacy_query.connect_readonly(dsn) as conn: with legacy_query.connect_readonly(dsn) as conn:
card_metas, _dup_iccids = legacy_query.fetch_card_meta(conn, iccid_pairs) card_metas, _dup_iccids = legacy_query.fetch_card_meta(conn, iccid_pairs)
usage_snapshots = legacy_query.fetch_card_usage(conn, iccid_pairs) usage_snapshots = legacy_query.fetch_card_usage(conn, iccid_pairs)
package_lifecycles = legacy_query.fetch_package_lifecycles(conn, iccid_pairs)
commission_snapshots = legacy_query.fetch_commission_accounts(conn, agent_ids) if agent_ids else {} commission_snapshots = legacy_query.fetch_commission_accounts(conn, agent_ids) if agent_ids else {}
agent_balances = legacy_query.fetch_agent_balances(conn, agent_ids) if agent_ids else {} agent_balances = legacy_query.fetch_agent_balances(conn, agent_ids) if agent_ids else {}
@@ -60,6 +61,7 @@ def main() -> int:
mapping=mapping, mapping=mapping,
card_metas=card_metas, card_metas=card_metas,
usage_snapshots=usage_snapshots, usage_snapshots=usage_snapshots,
package_lifecycles=package_lifecycles,
commission_snapshots=commission_snapshots, commission_snapshots=commission_snapshots,
agent_balances=agent_balances, agent_balances=agent_balances,
) )

View File

@@ -2,8 +2,13 @@
业务方按以下规则把数据放到本目录,**重命名去掉 `.example` 后缀**,即可被脚本读取。 业务方按以下规则把数据放到本目录,**重命名去掉 `.example` 后缀**,即可被脚本读取。
> 设计原则:**业务方只填"是哪张卡""在哪台设备上",归属、运营商、套餐等元数据全部由 > 设计原则:**业务方只填"是哪张卡""在哪台设备上"和少量行级槽位覆盖**。
> 脚本从奇成自动获取,通过 `config/mapping.yaml` 决定如何落到新系统。** > 归属、批量默认槽位、套餐迁移范围由 `config/mapping.yaml` 决定;运营商、套餐、号码等历史事实从奇成只读查询。
`cards.csv` 必填。`devices.csv` 可选:
- 只有卡迁移:只放 `cards.csv`,不要放 `devices.csv`
- 卡 + 设备迁移:同时放 `cards.csv``devices.csv`,设备槽位引用的卡必须出现在 `cards.csv`
## cards.csv 卡资产清单 ## cards.csv 卡资产清单
@@ -24,13 +29,14 @@
- 非 CTCC 卡 `iccid_20` 留空 → 同上阻断 - 非 CTCC 卡 `iccid_20` 留空 → 同上阻断
- 同一完整 ICCID 在奇成 `tbl_card` 命中多条记录 → `errors.csv: legacy_duplicate` 阻断,请业务方人工确认 - 同一完整 ICCID 在奇成 `tbl_card` 命中多条记录 → `errors.csv: legacy_duplicate` 阻断,请业务方人工确认
**店铺归属不在 CSV 里填**,`mapping.yaml.agents[].target_shop_code` 通过奇成 **店铺归属不在 CSV 里填**。独立卡`mapping.yaml.ownership_rules.standalone_card`
`tbl_card.agent_id` 自动推导。如果该卡在奇成没有 agent_id 或代理映射未填 shop_code, `overrides.cards` 决定。多店铺批次使用 `standalone_card.mode: legacy_agent`,
卡进平台库存。 再按奇成 `agent_id``mapping.yaml.agents[].target_shop_code`。设备上的卡跟随设备归属,
不单独生成分配记录。
## devices.csv 设备资产清单 ## devices.csv 设备资产清单(可选)
每台要迁移的设备一行。 每台要迁移的设备一行。没有设备时不要创建这个文件,脚本会把所有卡按独立卡处理。
| 列名 | 必填 | 说明 | | 列名 | 必填 | 说明 |
|------|------|------| |------|------|------|
@@ -41,12 +47,22 @@
| `device_type` | 否 | 设备类型,写入 `tb_device.device_type` | | `device_type` | 否 | 设备类型,写入 `tb_device.device_type` |
| `max_sim_slots` | 否 | 最大卡槽数,范围 1-4。不填时按非空 `sim_iccid_*` 的最大槽位推导 | | `max_sim_slots` | 否 | 最大卡槽数,范围 1-4。不填时按非空 `sim_iccid_*` 的最大槽位推导 |
| `sim_iccid_1` ~ `sim_iccid_4` | 否 | 各插槽 ICCID,引用 `cards.csv` 的卡。可填 19 或 20 位;20 位精确匹配完整 ICCID,19 位仅在该前缀唯一对应一张卡时允许(至少要有 1 个) | | `sim_iccid_1` ~ `sim_iccid_4` | 否 | 各插槽 ICCID,引用 `cards.csv` 的卡。可填 19 或 20 位;20 位精确匹配完整 ICCID,19 位仅在该前缀唯一对应一张卡时允许(至少要有 1 个) |
| `current_slot` | 否 | 当前槽位,范围 1-4。写入 `tb_device_sim_binding.is_current`;空值用批量规则或覆盖项 |
| `package_source_slot` | 否 | 设备旧套餐读取槽位,范围 1-4。新系统套餐仍归属设备;该字段只决定从奇成哪张绑定卡读取正式套餐生命周期 |
**店铺归属规则**:用 `sim_iccid_1`(主卡)的 agent_id 走 `mapping.agents` **槽位优先级**:
推导,与卡归属同口径。 1. `devices.csv` 行级 `current_slot` / `package_source_slot`
2. `mapping.yaml.overrides.devices` 中同设备号覆盖项
3. `mapping.yaml.ownership_rules.device` 批量默认槽位
`current_slot``package_source_slot` 对应的 `sim_iccid_N` 为空,脚本会写入
`errors.csv` 并阻断该设备相关 SQL。非 `package_source_slot` 的设备绑定卡只生成
基础卡和绑定 SQL,不会再读取它的奇成套餐,避免同一设备重复迁入多份主套餐。
**其他字段自动取默认值**:`manufacturer` 留空,`max_sim_slots` 按已填写卡槽推导, **其他字段自动取默认值**:`manufacturer` 留空,`max_sim_slots` 按已填写卡槽推导,
`current_slot=1`(以 `sim_iccid_1` 为当前生效卡)。 设备归属由 `mapping.yaml.ownership_rules.device``overrides.devices` 决定。设备使用
`device.mode: legacy_agent` 时,脚本会按 `package_source_slot` 对应绑定卡的奇成 `agent_id`
`mapping.yaml.agents[].target_shop_code`
## 一致性约束 ## 一致性约束

View File

@@ -17,6 +17,7 @@ from __future__ import annotations
import argparse import argparse
import csv import csv
import sys import sys
from dataclasses import dataclass
from pathlib import Path from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent)) sys.path.insert(0, str(Path(__file__).resolve().parent))
@@ -31,15 +32,30 @@ from lib.mapping_loader import ( # noqa: E402
) )
def _read_iccid_pairs(cards_csv: Path) -> list[tuple[str, str, bool]]: @dataclass(frozen=True)
class ICCIDInput:
"""业务方提供的单行 ICCID 输入。"""
line_no: int
iccid_19: str
iccid_20: str
allow_iccid_19_lookup: bool
@property
def card_key(self) -> str:
"""奇成扫描结果使用的业务方完整键。"""
return self.iccid_20 or self.iccid_19
def _read_iccid_pairs(cards_csv: Path) -> list[ICCIDInput]:
"""从 cards.csv 读 iccid_19 + iccid_20 列(只关心这两列,其他列忽略)。 """从 cards.csv 读 iccid_19 + iccid_20 列(只关心这两列,其他列忽略)。
返回 (iccid_19, iccid_20, allow_iccid_19_lookup) 列表。 返回带行号的 ICCID 输入列表。
只填 iccid_20 时,iccid_19 从前 19 位派生,但不用于奇成主表兜底查询。 只填 iccid_20 时,iccid_19 从前 19 位派生,但不用于奇成主表兜底查询。
""" """
if not cards_csv.exists(): if not cards_csv.exists():
raise FileNotFoundError(f"找不到 {cards_csv},请按 resources/README.md 准备") raise FileNotFoundError(f"找不到 {cards_csv},请按 resources/README.md 准备")
pairs: dict[str, tuple[str, str, bool]] = {} pairs: dict[str, ICCIDInput] = {}
first_seen_line: dict[str, int] = {} first_seen_line: dict[str, int] = {}
errors: list[str] = [] errors: list[str] = []
with cards_csv.open("r", encoding="utf-8-sig", newline="") as f: with cards_csv.open("r", encoding="utf-8-sig", newline="") as f:
@@ -75,7 +91,12 @@ def _read_iccid_pairs(cards_csv: Path) -> list[tuple[str, str, bool]]:
errors.append(f"{line_no}{'iccid_20' if i20 else 'iccid_19'} 与第 {first_seen_line[dedup_key]} 行重复: {dedup_key!r}") errors.append(f"{line_no}{'iccid_20' if i20 else 'iccid_19'} 与第 {first_seen_line[dedup_key]} 行重复: {dedup_key!r}")
continue continue
first_seen_line[dedup_key] = line_no first_seen_line[dedup_key] = line_no
pairs[dedup_key] = (i19, i20, allow_i19_lookup) pairs[dedup_key] = ICCIDInput(
line_no=line_no,
iccid_19=i19,
iccid_20=i20,
allow_iccid_19_lookup=allow_i19_lookup,
)
if errors: if errors:
preview = "\n".join(f" - {msg}" for msg in errors[:20]) preview = "\n".join(f" - {msg}" for msg in errors[:20])
if len(errors) > 20: if len(errors) > 20:
@@ -106,7 +127,10 @@ def main() -> int:
dsn = mapping_loader.load_legacy_dsn(config_dir) dsn = mapping_loader.load_legacy_dsn(config_dir)
with legacy_query.connect_readonly(dsn) as conn: with legacy_query.connect_readonly(dsn) as conn:
card_metas, dup_iccids = legacy_query.fetch_card_meta(conn, iccid_pairs) card_metas, dup_iccids = legacy_query.fetch_card_meta(
conn,
[(x.iccid_19, x.iccid_20, x.allow_iccid_19_lookup) for x in iccid_pairs],
)
meal_ids = {m.current_meal_id for m in card_metas.values() if m.current_meal_id} meal_ids = {m.current_meal_id for m in card_metas.values() if m.current_meal_id}
meal_metas = legacy_query.fetch_meal_meta(conn, meal_ids) meal_metas = legacy_query.fetch_meal_meta(conn, meal_ids)
@@ -120,8 +144,11 @@ def main() -> int:
merged = _merge(existing, card_metas, meal_metas) merged = _merge(existing, card_metas, meal_metas)
out_path = mapping_loader.merge_and_save(config_dir, merged) out_path = mapping_loader.merge_and_save(config_dir, merged)
todo = _collect_todo(merged) todo = _collect_todo(merged)
unmatched = _collect_unmatched_iccids(iccid_pairs, card_metas, dup_iccids)
_write_todo(config_dir / "mapping_todo.txt", todo) _write_todo(config_dir / "mapping_todo.txt", todo)
_print_summary(out_path, card_metas, meal_metas, todo) unmatched_path = config_dir / "unmatched_iccids.csv"
_write_unmatched_iccids(unmatched_path, unmatched)
_print_summary(out_path, card_metas, meal_metas, todo, unmatched_path, len(unmatched))
return 0 return 0
@@ -134,6 +161,9 @@ def _merge(
merged = Mapping( merged = Mapping(
migration_user_id=existing.migration_user_id, migration_user_id=existing.migration_user_id,
migration_batch_no=existing.migration_batch_no, migration_batch_no=existing.migration_batch_no,
ownership_rules=existing.ownership_rules,
package_rules=existing.package_rules,
overrides=existing.overrides,
carriers=dict(existing.carriers), carriers=dict(existing.carriers),
packages=dict(existing.packages), packages=dict(existing.packages),
series=dict(existing.series), series=dict(existing.series),
@@ -210,6 +240,19 @@ def _collect_todo(mapping: Mapping) -> dict[str, list[str]]:
return todo return todo
def _collect_unmatched_iccids(
iccid_pairs: list[ICCIDInput],
card_metas: dict[str, "legacy_query.LegacyCardMeta"],
duplicate_iccids: set[str],
) -> list[ICCIDInput]:
"""收集未在奇成 tbl_card 命中的 ICCID。
重复命中的卡属于脏数据异常,不是未命中,因此从未命中清单中排除。
"""
resolved_keys = set(card_metas) | set(duplicate_iccids)
return [item for item in iccid_pairs if item.card_key not in resolved_keys]
def _write_todo(path: Path, todo: dict[str, list[str]]) -> None: def _write_todo(path: Path, todo: dict[str, list[str]]) -> None:
lines = ["奇成迁移映射待填写清单", "=" * 40, ""] lines = ["奇成迁移映射待填写清单", "=" * 40, ""]
for kind in ("carriers", "packages", "agents"): for kind in ("carriers", "packages", "agents"):
@@ -220,13 +263,47 @@ def _write_todo(path: Path, todo: dict[str, list[str]]) -> None:
path.write_text("\n".join(lines), encoding="utf-8") path.write_text("\n".join(lines), encoding="utf-8")
def _print_summary(out_path: Path, card_metas, meal_metas, todo: dict[str, list[str]]) -> None: def _write_unmatched_iccids(path: Path, unmatched: list[ICCIDInput]) -> None:
"""把未命中的 ICCID 写成 CSV,便于转给业务方核对。"""
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8-sig", newline="") as f:
writer = csv.DictWriter(
f,
fieldnames=["line_no", "iccid_19", "iccid_20", "lookup_keys", "note"],
)
writer.writeheader()
for item in unmatched:
lookup_keys = [item.iccid_20 or item.iccid_19]
if item.iccid_20 and item.allow_iccid_19_lookup:
lookup_keys.append(item.iccid_19)
writer.writerow(
{
"line_no": item.line_no,
"iccid_19": item.iccid_19,
"iccid_20": item.iccid_20,
"lookup_keys": ";".join(lookup_keys),
"note": "奇成 tbl_card 未命中",
}
)
def _print_summary(
out_path: Path,
card_metas,
meal_metas,
todo: dict[str, list[str]],
unmatched_path: Path,
unmatched_count: int,
) -> None:
print(f"扫描完成,mapping 写入: {out_path}") print(f"扫描完成,mapping 写入: {out_path}")
print(f" 奇成卡命中 : {len(card_metas)}") print(f" 奇成卡命中 : {len(card_metas)}")
print(f" 奇成卡未命中 : {unmatched_count}")
print(f" 奇成套餐命中 : {len(meal_metas)}") print(f" 奇成套餐命中 : {len(meal_metas)}")
print(f" 待填 carriers : {len(todo['carriers'])}") print(f" 待填 carriers : {len(todo['carriers'])}")
print(f" 待填 packages : {len(todo['packages'])}") print(f" 待填 packages : {len(todo['packages'])}")
print(f" 待填 agents : {len(todo['agents'])}") print(f" 待填 agents : {len(todo['agents'])}")
if unmatched_count:
print(f"未命中清单: {unmatched_path}")
if any(todo.values()): if any(todo.values()):
print(f"详见: {out_path.parent / 'mapping_todo.txt'}") print(f"详见: {out_path.parent / 'mapping_todo.txt'}")