数据迁移脚本
All checks were successful
构建并部署到测试环境(无 SSH) / build-and-deploy (push) Successful in 54s

This commit is contained in:
Break
2026-05-27 11:10:56 +08:00
parent 0d96a94e5f
commit 58863e8ec5
20 changed files with 3343 additions and 0 deletions

View File

@@ -0,0 +1 @@
3.13.7

227
scripts/migration/README.md Normal file
View File

@@ -0,0 +1,227 @@
# 奇成数据迁移脚本
> 详细方案见 [`docs/脚本/奇成数据迁移方案.md`](../../docs/脚本/奇成数据迁移方案.md)。
## 1. 四步走
```
业务方 我们 运维
│ │ │
│ cards.csv │ scan_legacy.py → mapping.yaml │
│ devices.csv │ ↓ 人工填 target_* │
├─────────────▶│ migrate_assets.py → step1_*.sql │
│ │ migrate_runtime.py → step2_*.sql │
│ ├────────────────────────────────────▶│ psql -1 -f step*.sql
│ │ │ migration-finalize
│ │ │ 重建轮询队列/缓存
```
## 2. 文件清单
```
scripts/migration/
├── README.md
├── requirements.txt
├── scan_legacy.py ← 第1步:扫描奇成,生成 mapping.yaml 模板
├── migrate_assets.py ← 第2步:生成资产 SQL
├── migrate_runtime.py ← 第3步:生成关联数据 SQL
├── lib/ ← 公共库
├── resources/
│ ├── README.md
│ ├── cards.csv.example ← 业务方提供:iccid + 可选覆盖列
│ └── devices.csv.example ← 业务方提供:设备 + 卡槽
├── config/
│ ├── mapping.yaml.example ← 唯一映射配置(全局 + carrier + package + series + agent)
│ └── legacy_dsn.yaml.example← 奇成 MySQL 只读 DSN
└── output/ ← 生成的 SQL 与日志
```
## 3. 详细流程
### 3.1 业务方提供数据
把两个 CSV 放到 `resources/` 下(命名为 `cards.csv` / `devices.csv`):
- **cards.csv**:`iccid_19` 必填(所有卡前 19 位),`iccid_20` 视运营商必填或留空,`is_industry` 可选
- **devices.csv**:`virtual_no` 必填,`sim_iccid_*` 引用 `cards.csv` 中的 ICCID(可填 19 或 20 位)
字段说明详见 [`resources/README.md`](resources/README.md)。
### 3.2 我们扫描 + 填映射
```bash
pip install -r requirements.txt
cp config/legacy_dsn.yaml.example config/legacy_dsn.yaml # 填真实 DSN
cp config/mapping.yaml.example config/mapping.yaml # 第一次跑要有这个文件
# 扫描奇成,把发现的 carrier/package/agent 写入 mapping.yaml
python3 scan_legacy.py
# 看 config/mapping_todo.txt,把 target_* 全部填好
vim config/mapping.yaml
```
`scan_legacy.py` 可以反复跑(增量合并),已填的 `target_*` 不会被覆盖。
### 3.3 生成 SQL
```bash
python3 migrate_assets.py # → output/step1_*.sql + errors.csv
python3 migrate_runtime.py # → output/step2_*.sql + warnings.csv
```
如果 `errors.csv` 有内容,意味着 `mapping.yaml` 还没填完整、或者某些 ICCID 在奇成查不到,处理完再重跑。
### 3.4 测试环境 / 线上执行
```bash
# 按编号顺序执行
psql -h $HOST -1 -f output/step1_01_iot_cards.sql
psql -h $HOST -1 -f output/step1_02_devices.sql
psql -h $HOST -1 -f output/step1_03_asset_identifiers.sql
psql -h $HOST -1 -f output/step1_04_asset_wallets.sql
psql -h $HOST -1 -f output/step1_05_sim_bindings.sql
psql -h $HOST -1 -f output/step1_06_shop_allocations.sql
psql -h $HOST -1 -f output/step2_01_migration_orders.sql
psql -h $HOST -1 -f output/step2_02_package_usages.sql
psql -h $HOST -1 -f output/step2_03_card_data_usage_update.sql
psql -h $HOST -1 -f output/step2_04_agent_wallet_init.sql
psql -h $HOST -1 -f output/step2_05_agent_wallet_transactions.sql
```
`-1` = 单事务,失败回滚整文件;`ON CONFLICT DO NOTHING` 保证重跑安全。
SQL 导入完成后必须按批次执行迁移收尾。SQL 只负责落库,不会触发 Service 生命周期,
也不会自动写入 Redis 轮询队列或卡缓存:
```bash
# 先看本批次会处理多少卡、会写哪些轮询任务,不会写 Redis
go run ./cmd/migration-finalize --batch-no MIGRATION-QICHENG-YYYYMMDD
# dry-run 无误后再执行,会按 tb_polling_config 重建本批卡的轮询队列和卡缓存
go run ./cmd/migration-finalize --batch-no MIGRATION-QICHENG-YYYYMMDD --apply
```
## 4. 配置说明 (`mapping.yaml`)
唯一需要人工填写的文件:
```yaml
migration_user_id: 0 # 后台超级管理员 ID,作为所有 creator/updater
migration_batch_no: "MIGRATION-QICHENG-YYYYMMDD" # 本次迁移批次号,线上每批必须唯一
carriers: # 由 scan_legacy 自动发现 legacy_*,你只填 target_*
- legacy_category: 9
legacy_category_name: "中国联通"
target_carrier_id: 1585 ← 必填:新库 tb_carrier.id
target_carrier_type: CUCC ← 必填:CMCC/CUCC/CTCC/CBN
target_carrier_name: "联通22-..."
packages: # 同理
- legacy_meal_id: "D6554244711498752"
legacy_meal_name: "三网充电宝全国畅玩年卡套餐..."
target_package_id: 2 ← 必填:新库 tb_package.id
series: # 可选(留 null 表示不写入 tb_iot_card.series_id)
- legacy_series_id: "..."
legacy_series_name: "国内卡 无预存30天"
target_series_id: null
agents: # 决定所有卡/设备的店铺归属
- legacy_agent_id: "D15EC779767B412A82B8448AC2324F04"
legacy_agent_name: "叶发伟"
target_shop_code: "SHOP20260428103424CZZQ" # 留 null = 该代理的卡进平台库存
```
### 店铺归属是怎么决定的
CSV 里**没有** `target_shop_code` 列,所有归属都从 `mapping.agents` 推导:
| 资产 | 归属规则 |
|------|---------|
| 独立卡 | 奇成 `tbl_card.agent_id``mapping.agents[].target_shop_code` |
| 设备 | 主卡(`sim_iccid_1`)的 agent_id → 同上 |
| 设备上的卡 | 跟随设备(`shop_id=NULL`,由触发器+绑定关系维护) |
如果 agent 在 `mapping.agents``target_shop_code` 留空 → 该资产进平台库存。
## 5. 识别迁移数据(后台报表过滤用)
| 数据 | 识别条件 |
|------|---------|
| 迁移伪订单 | `WHERE order_no LIKE 'MIG-%'` |
| 迁移分配记录 | `WHERE allocation_no LIKE 'MIG-ALLOC-%'` |
| 迁移钱包流水 | `WHERE metadata->>'source' = 'qicheng'` |
迁移伪订单号固定为 `MIG-CARD-<ICCID>`:
- 20 位卡总长 29,19 位卡总长 28,符合新库 `order_no varchar(30)` 限制。
- 订单号不包含生成时间,`step2_01_migration_orders.sql``step2_02_package_usages.sql` 分开重跑也能稳定匹配。
## 6. 流量用量迁移规则(真用量 / 虚用量)
奇成与新系统的"虚流量"模型口径不同,必须做一次反算才能落对地方。
### 6.1 两边的口径差异
| | 奇成 | 新系统 |
|---|---|---|
| 用量存储 | `tbl_card.total_bytes_cnt`(varchar 小数 MB)= **虚用量**(真 × `(1 + flow_add_discount/100)`) | `tb_iot_card.data_usage_mb`(BIGINT MB)= **真用量**(网关返回的物理读数) |
| 加成配置 | `tbl_set_meal.flow_add_discount`(0~100 百分比,作用于"已用量") | `tb_package.enable_virtual_data` + `virtual_data_mb` + `virtual_ratio`(`= real_data_mb / virtual_data_mb`) |
| 用户看到的"已用" | = 数据库 `total_bytes_cnt`(已加成) | 前端用 `data_usage_mb × display_gain_ratio_snapshot` 现算 |
### 6.2 反算公式
迁移脚本对每张卡按下式得到真用量,再写入 `data_usage_mb` / `current_month_usage_mb` / `last_gateway_reading_mb`:
```
real_used_mb = total_bytes_cnt / (1 + flow_add_discount / 100)
```
其中 `flow_add_discount` 取自该卡在 `tbl_card_life`(status=1, expire_date 最大)对应的当前生效套餐。
### 6.3 精度与近似
- **跨多个套餐周期的累计**:`total_bytes_cnt` 是按各时段对应套餐虚比加权累加的,这里只用"当前生效套餐"的虚比反算,**有误差但可接受**(奇成没有按套餐分段的用量明细,无更好办法)。
- **无当前生效套餐却有 `total_bytes_cnt`**:按 `flow_add_discount=0` 处理(真=虚),并写入 `warnings.csv`(`kind=usage_without_active_package`)供人工核查。
- **整数取整规则**(`data_usage_mb` 是 BIGINT):
- `real_used_mb = 0` → 写 0
- `0 < real_used_mb < 1`**上取整为 1**,避免"明明用过的卡显示 0"
- `real_used_mb ≥ 1` → 向下取整
- **小数字段保留精度**:`current_month_usage_mb`(numeric)与 `last_gateway_reading_mb`(double)直接写 Decimal 真值,作为下次轮询增量计算的基线,**避免轮询第一次回写时产生跳变**。
### 6.4 套餐使用记录的虚流量快照
`tb_package_usage``virtual_total_mb_snapshot` / `display_gain_ratio_snapshot` / `enable_virtual_data_snapshot` 三个字段**全部从新系统 `tb_package` 当场快照**,与奇成无关:
| 快照字段 | 取值 |
|---|---|
| `virtual_total_mb_snapshot` | 启用虚:`virtual_data_mb`;否则:`real_data_mb` |
| `display_gain_ratio_snapshot` | 启用虚:`virtual_ratio`(即 `real_data_mb/virtual_data_mb`);否则:`1.0` |
| `enable_virtual_data_snapshot` | 启用虚:`TRUE`;否则:`FALSE` |
| `data_limit_mb` | 始终 `real_data_mb`(真总量) |
| `data_usage_mb` | `LEAST(real_used_mb_floor, virtual_total_mb_snapshot)`(防卫上限) |
## 7. 重要约束
- ❗ 所有奇成查询走 `SET TRANSACTION READ ONLY`,严禁误写。
- ❗ 所有 INSERT 用 `ON CONFLICT DO NOTHING`,严禁更新已有数据。
- ❗ 伪订单的 `payment_status=2 commission_status=2 commission_result=2`,避免被支付回调、佣金调度器扫到。
- ❗ 迁移卡默认 `status=1(在库)` / `activation_status=0(未激活)`,激活由后续业务流程驱动。
## 8. 常见问题
| 现象 | 排查 |
|------|------|
| `scan_legacy.py``pymysql 缺失` | `pip install -r requirements.txt` |
| `errors.csv` 出现 `carrier_not_in_mapping` | `mapping.yaml.carriers` 没覆盖到这个 category,重跑 `scan_legacy.py` 自动补 |
| `errors.csv` 出现 `carrier_target_missing` | `mapping.yaml.carriers` 里有这一项但 `target_*` 还是 null,手动填 |
| `errors.csv` 出现 `not_in_legacy` | 该 ICCID 在奇成 `tbl_card` 查不到,业务方确认 |
| `errors.csv` 出现 `legacy_duplicate` | 同一 `iccid_19` 在奇成命中多条 `tbl_card`,业务方人工确认是哪张物理卡 |
| `errors.csv` 出现 `carrier_length_conflict` | 业务方填的 `iccid_20` 与奇成查到的运营商不匹配(CTCC 不该有 20 位 / 非 CTCC 必须有 20 位) |
| `errors.csv` 出现 `iccid_20_prefix_mismatch` | `iccid_20` 前 19 位 ≠ `iccid_19`,业务方校对 |
| `warnings.csv` 出现 `agent_no_legacy` | `mapping.agents` 配了但奇成没有 tbl_agent 记录,业务方确认 |
| `warnings.csv` 出现 `usage_without_active_package` | 该卡奇成 `total_bytes_cnt > 0``tbl_card_life` 中没有 status=1 的当前生效套餐,无法精确反算真用量(按真=虚处理),详见 §6.3 |
| 线上报 `null value in column "creator"` | `migration_user_id` 是 0,某些约束可能拒绝;改成真实管理员 ID |
| 线上报 `null shop_id` | `mapping.agents.target_shop_code` 指向的店铺在新库不存在,先建店铺 |

View File

@@ -0,0 +1,11 @@
# 新系统 PostgreSQL 测试库连接配置(只读)
# 脚本仅用它做"测试环境 id 探测",生成的线上 SQL 不会硬编码这些 id
# 实际使用时复制为 main_test_dsn.yaml(已 .gitignore)
host: "cxd.whcxd.cn"
port: 16159
user: "erp_pgsql"
password: "erp_2025"
database: "junhong_cmp_test"
sslmode: "disable"
connect_timeout: 10

View File

@@ -0,0 +1,47 @@
# 奇成迁移映射配置(单文件)
# 复制为 mapping.yaml 后使用,会被 scan_legacy.py 自动增量补全
#
# 工作流:
# 1. 业务方提供 resources/cards.csv(只需 iccid 列)
# 2. python3 scan_legacy.py → 自动从奇成扫描元数据,把发现的 carrier/package/agent 补到本文件
# 3. 人工填写所有 target_* 字段
# 4. python3 migrate_assets.py / migrate_runtime.py 正常出 SQL
# ============== 全局配置 ==============
# 迁移操作者(creator/updater/operator_id),通常是后台超级管理员账号 ID
# 上线前确认: SELECT id FROM tb_account WHERE login_account = 'admin' LIMIT 1;
migration_user_id: 0
# 本次迁移批次号。线上每次迁移必须使用独立批次,用于导入后收尾、验收和回滚定位。
migration_batch_no: "MIGRATION-QICHENG-YYYYMMDD"
# ============== 运营商映射 ==============
# scan_legacy 会按"奇成 tbl_card.category"分组写入,你只需填 target_*
carriers:
- legacy_category: 9
legacy_category_name: "中国联通"
target_carrier_id: null # 必填:新库 tb_carrier.id
target_carrier_type: null # 必填:CMCC / CUCC / CTCC / CBN
target_carrier_name: null # 必填:新库 carrier_name 快照
# ============== 套餐映射 ==============
# scan_legacy 会按"奇成 tbl_card_life.meal_id"分组写入
packages:
- legacy_meal_id: "D6554244711498752"
legacy_meal_name: "三网充电宝全国畅玩年卡套餐1000G/月12个月"
target_package_id: null # 必填:新库 tb_package.id
# ============== 套餐系列映射(可选) ==============
# 仅当你需要给迁移卡设置 tb_iot_card.series_id 时才填
series:
- legacy_series_id: "95A038B9D3014F9DA1539EC092D9DBAA"
legacy_series_name: "国内卡 无预存30天"
target_series_id: null # 可选:新库 tb_package_series.id,null=不写入
# ============== 代理-店铺映射 ==============
# scan_legacy 会按"奇成 tbl_card.agent_id"分组写入,你只需填 target_shop_code
# 留空(null)的代理:其卡进平台库存(除非 cards.csv 里显式指定 target_shop_code)
agents:
- legacy_agent_id: "D15EC779767B412A82B8448AC2324F04"
legacy_agent_name: "叶发伟"
target_shop_code: null # 新库 tb_shop.shop_code,null=不归任何店铺

View File

@@ -0,0 +1,9 @@
"""奇成数据迁移脚本公共库。
模块组成:
iccid_utils - ICCID 长度校验 / 19-20 位拆分
csv_loader - cards.csv / devices.csv 读取与清洗
mapping_loader - yaml 映射文件加载
legacy_query - 奇成 MySQL 只读查询封装
sql_builder - 输出 SQL 构造(CTE + ON CONFLICT)
"""

View File

@@ -0,0 +1,272 @@
"""cards.csv / devices.csv 加载与跨文件一致性校验。
新设计:CSV 只承载"业务方决策"——
- cards.csv 只读 iccid + 可选覆盖列(target_shop_code、is_industry)
- devices.csv 读设备元数据 + sim_iccid_*
- 卡的运营商、套餐、msisdn、agent_id 等元数据全部从奇成查(见 legacy_query.fetch_card_meta)
"""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
CARD_HEADERS_REQUIRED = ("iccid_19",)
# 设备列至少包含 imei 或 virtual_no 之一(运行时校验,允许只填一列)
DEVICE_HEADERS_AT_LEAST_ONE = ("imei", "virtual_no")
@dataclass
class CardRow:
"""cards.csv 单行的解析结果(只含业务方决策字段)。
业务方必须同时提供 iccid_19 和 iccid_20(20 位运营商):
- iccid_19:前 19 位,所有卡必填,作为跨文件稳定 ID
- iccid_20:20 位完整 ICCID,CMCC/CUCC/CBN 必填、CTCC 必须留空
- 运营商类型与 iccid_20 的一致性在查完奇成后由 sql_builder 校验
新系统 iccid 唯一键 = iccid_20 优先,否则 iccid_19。
"""
line_no: int # 原始 CSV 行号(从 1 开始,含表头)
iccid_19: str
iccid_20: str = "" # 留空表示 CTCC(查完奇成会再校验)
is_industry: bool = False # 默认 False(普通卡)
# 跨文件校验后填充
bound_device_virtual_no: str = ""
@property
def iccid_full(self) -> str:
"""完整 ICCID:有 iccid_20 用 iccid_20,否则用 iccid_19。"""
return self.iccid_20 or self.iccid_19
@dataclass
class DeviceRow:
"""devices.csv 单行的解析结果。
简化为业务方核心关心的列:imei / virtual_no / sim_iccid_*。
店铺归属由"主卡 sim_iccid_1 的 agent_id → mapping.agents.target_shop_code"推导。
其他字段(device_name/device_model/manufacturer 等)直接用默认值,不暴露给业务方。
"""
line_no: int
imei: str
virtual_no: str # 留空时用 imei 兜底,见 _load_devices
sim_iccids: dict[int, str] # slot → iccid(只保留非空)
max_sim_slots: int = 4 # 默认 4 槽位
current_slot: int = 1 # 默认 slot 1 为当前生效
@dataclass
class ErrorRow:
"""异常行汇总(写入 output/errors.csv)。"""
source_file: str
line_no: int
field: str
value: str
error_code: str
message: str
def to_csv_row(self) -> list[str]:
return [self.source_file, str(self.line_no), self.field, self.value, self.error_code, self.message]
@dataclass
class LoadResult:
cards: list[CardRow] = field(default_factory=list)
devices: list[DeviceRow] = field(default_factory=list)
errors: list[ErrorRow] = field(default_factory=list)
# ---------------- 字段清洗 ----------------
def _normalize_cell(raw: str) -> str:
if raw is None:
return ""
s = raw.strip()
if s.startswith(""):
s = s[1:]
if s.startswith("'"):
s = s[1:]
s = s.replace(" ", "")
return s.strip()
def _parse_bool(raw: str) -> Optional[bool]:
v = _normalize_cell(raw).lower()
if v == "":
return False
if v in ("true", "1", "yes", "y", "t", "", "行业卡"):
return True
if v in ("false", "0", "no", "n", "f", "", "普通卡"):
return False
return None
def _parse_int(raw: str, default: int = 0) -> int:
v = _normalize_cell(raw)
if not v:
return default
try:
return int(v)
except ValueError:
return default
# ---------------- 主入口 ----------------
def load_assets(resources_dir: Path) -> tuple[list[CardRow], list[DeviceRow], list[ErrorRow]]:
"""读 cards.csv + devices.csv,做格式校验与跨文件一致性校验。
返回:(合法卡列表, 合法设备列表, 异常行列表)。
元数据(carrier/agent_id/msisdn 等)由调用方再用 legacy_query 补充。
"""
result = LoadResult()
cards_path = resources_dir / "cards.csv"
if not cards_path.exists():
raise FileNotFoundError(f"缺少 {cards_path},请按 resources/README.md 准备数据")
_load_cards(cards_path, result)
devices_path = resources_dir / "devices.csv"
if devices_path.exists():
_load_devices(devices_path, result)
_cross_check(result)
return result.cards, result.devices, result.errors
def _load_cards(path: Path, result: LoadResult) -> None:
import csv
from . import iccid_utils
with path.open("r", encoding="utf-8-sig", newline="") as f:
reader = csv.DictReader(f)
header = [_normalize_cell(h) for h in (reader.fieldnames or [])]
missing = [k for k in CARD_HEADERS_REQUIRED if k not in header]
if missing:
raise ValueError(f"{path} 缺少必备列: {missing}")
if "iccid_20" not in header:
raise ValueError(f"{path} 必须包含 iccid_20 列(20 位运营商卡填写,CTCC 留空)")
seen_iccid_19: dict[str, int] = {}
for idx, raw in enumerate(reader, start=2):
iccid_19 = _normalize_cell(raw.get("iccid_19", ""))
iccid_20 = _normalize_cell(raw.get("iccid_20", ""))
err = iccid_utils.validate_iccid_19(iccid_19)
if err is not None:
result.errors.append(ErrorRow("cards.csv", idx, "iccid_19", iccid_19, err.code, err.message))
continue
err = iccid_utils.validate_iccid_20(iccid_20, iccid_19)
if err is not None:
result.errors.append(ErrorRow("cards.csv", idx, "iccid_20", iccid_20, err.code, err.message))
continue
if iccid_19 in seen_iccid_19:
result.errors.append(
ErrorRow("cards.csv", idx, "iccid_19", iccid_19, "duplicate",
f"与第 {seen_iccid_19[iccid_19]} 行 iccid_19 重复")
)
continue
seen_iccid_19[iccid_19] = idx
is_industry = _parse_bool(raw.get("is_industry", ""))
if is_industry is None:
result.errors.append(
ErrorRow("cards.csv", idx, "is_industry", raw.get("is_industry", ""), "invalid_bool", "is_industry 必须是 true/false")
)
continue
result.cards.append(
CardRow(line_no=idx, iccid_19=iccid_19, iccid_20=iccid_20, is_industry=is_industry)
)
def _load_devices(path: Path, result: LoadResult) -> None:
import csv
with path.open("r", encoding="utf-8-sig", newline="") as f:
reader = csv.DictReader(f)
header = [_normalize_cell(h) for h in (reader.fieldnames or [])]
if not any(k in header for k in DEVICE_HEADERS_AT_LEAST_ONE):
raise ValueError(f"{path} 必须至少包含 imei 或 virtual_no 列")
seen_virtual_no: dict[str, int] = {}
for idx, raw in enumerate(reader, start=2):
imei = _normalize_cell(raw.get("imei", ""))
virtual_no = _normalize_cell(raw.get("virtual_no", ""))
# virtual_no 留空时用 imei 兜底(业务方说"虚拟号一般会提供,没有就用 imei")
effective_vno = virtual_no or imei
if not effective_vno:
result.errors.append(
ErrorRow("devices.csv", idx, "virtual_no", "", "missing_identity",
"imei 和 virtual_no 不能同时为空")
)
continue
if effective_vno in seen_virtual_no:
result.errors.append(
ErrorRow("devices.csv", idx, "virtual_no", effective_vno, "duplicate",
f"与第 {seen_virtual_no[effective_vno]} 行的设备号重复")
)
continue
seen_virtual_no[effective_vno] = idx
sim_iccids: dict[int, str] = {}
for slot in (1, 2, 3, 4):
v = _normalize_cell(raw.get(f"sim_iccid_{slot}", ""))
if v:
sim_iccids[slot] = v
result.devices.append(
DeviceRow(
line_no=idx,
imei=imei,
virtual_no=effective_vno,
sim_iccids=sim_iccids,
)
)
def _cross_check(result: LoadResult) -> None:
"""校验设备与卡之间的一致性,并把 bound_device_virtual_no 回填到 CardRow。
devices.csv 里 sim_iccid_X 业务方可能填 19 位也可能填 20 位,
统一截取前 19 位回查 cards.iccid_19。
"""
from . import iccid_utils
cards_by_iccid_19: dict[str, CardRow] = {c.iccid_19: c for c in result.cards}
device_card_refs: dict[str, tuple[str, int]] = {}
for device in result.devices:
for slot, raw_iccid in device.sim_iccids.items():
key19 = iccid_utils.prefix_19(raw_iccid)
if key19 not in cards_by_iccid_19:
result.errors.append(
ErrorRow(
"devices.csv", device.line_no, f"sim_iccid_{slot}", raw_iccid,
"card_not_found", "引用的 ICCID(前 19 位)在 cards.csv 中不存在",
)
)
continue
if key19 in device_card_refs:
prev_vno, prev_line = device_card_refs[key19]
result.errors.append(
ErrorRow(
"devices.csv", device.line_no, f"sim_iccid_{slot}", raw_iccid,
"card_used_twice", f"该 ICCID 已被设备 {prev_vno}(第 {prev_line} 行)使用",
)
)
continue
device_card_refs[key19] = (device.virtual_no, device.line_no)
cards_by_iccid_19[key19].bound_device_virtual_no = device.virtual_no
# 标准化:slot 里的值统一改写为 iccid_19,sql_builder 直接用它查 cards / card_metas
device.sim_iccids[slot] = key19

View File

@@ -0,0 +1,158 @@
"""ICCID 19/20 位拆分与校验工具。
移植自 pkg/utils/iccid.go,并按运营商类型增加长度合规校验。
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Optional
# 运营商标准 ICCID 长度
CARRIER_ICCID_LENGTH = {
"CTCC": 19, # 电信
"CMCC": 20, # 移动
"CUCC": 20, # 联通
"CBN": 20, # 广电
}
VALID_CARRIER_TYPES = set(CARRIER_ICCID_LENGTH.keys())
@dataclass(frozen=True)
class SplitResult:
"""ICCID 拆分结果。"""
iccid: str # 原始 ICCID(去空格)
iccid_19: str # 前 19 位(所有卡必填)
iccid_20: Optional[str] # 完整 20 位(仅 20 位运营商卡有值,19 位卡为 None → SQL NULL)
@dataclass(frozen=True)
class ValidationError:
"""ICCID 校验失败信息。"""
code: str # 错误码:length_invalid / length_mismatch
message: str # 中文描述
def split_iccid(iccid: str) -> Optional[SplitResult]:
"""按 ICCID 长度拆分双列存储值。
与 Go 版本一致:
19 位 → (iccid, iccid, None)
20 位 → (iccid, iccid[:19], iccid)
其他 → None(调用方按异常处理)
"""
iccid = iccid.strip()
if len(iccid) == 19:
return SplitResult(iccid=iccid, iccid_19=iccid, iccid_20=None)
if len(iccid) == 20:
return SplitResult(iccid=iccid, iccid_19=iccid[:19], iccid_20=iccid)
return None
def validate_and_split(iccid: str, carrier_type: str) -> tuple[Optional[SplitResult], Optional[ValidationError]]:
"""按运营商类型校验 ICCID 长度并拆分。
返回 (拆分结果, 错误信息) 二元组:
- 长度不是 19/20:返回 (None, length_invalid 错误)
- 长度与 carrier_type 不匹配:返回 (拆分结果, length_mismatch 警告) → 调用方决定要不要放行
- 合法:返回 (拆分结果, None)
设计说明:不直接拒绝长度不匹配的情况(电信卡居然是 20 位 / 非电信卡是 19 位),
保留拆分结果让脚本写入 errors.csv 让业务方确认。
"""
iccid = iccid.strip()
if carrier_type not in VALID_CARRIER_TYPES:
return None, ValidationError(
code="carrier_type_invalid",
message=f"无效的运营商类型 {carrier_type!r},应为 {sorted(VALID_CARRIER_TYPES)} 之一",
)
result = split_iccid(iccid)
if result is None:
return None, ValidationError(
code="length_invalid",
message=f"ICCID 长度 {len(iccid)} 不合法,只支持 19/20 位",
)
expected_len = CARRIER_ICCID_LENGTH[carrier_type]
if len(iccid) != expected_len:
return result, ValidationError(
code="length_mismatch",
message=f"{carrier_type} 应为 {expected_len} 位但 ICCID 是 {len(iccid)} 位,请人工确认",
)
return result, None
def prefix_19(iccid: str) -> str:
"""ICCID 前 19 位(稳定 ID,无校验位)。
用于 devices.csv 引用 cards 时的兼容查找:业务方在 devices 里填 19 还是 20 位都接受,
内部统一截取前 19 位回查 cards.iccid_19。
"""
s = (iccid or "").strip()
if len(s) >= 19:
return s[:19]
return s
def validate_iccid_19(iccid_19: str) -> Optional[ValidationError]:
"""iccid_19 必须 19 位纯数字。"""
if not iccid_19:
return ValidationError(code="iccid_19_required", message="iccid_19 不能为空")
if len(iccid_19) != 19:
return ValidationError(
code="iccid_19_length",
message=f"iccid_19 必须是 19 位,当前 {len(iccid_19)}",
)
if not iccid_19.isdigit():
return ValidationError(code="iccid_19_format", message="iccid_19 必须是纯数字")
return None
def validate_iccid_20(iccid_20: str, iccid_19: str) -> Optional[ValidationError]:
"""iccid_20 可空(CTCC 卡);非空时必须 20 位纯数字且前 19 位 == iccid_19。"""
if not iccid_20:
return None
if len(iccid_20) != 20:
return ValidationError(
code="iccid_20_length",
message=f"iccid_20 必须是 20 位,当前 {len(iccid_20)}",
)
if not iccid_20.isdigit():
return ValidationError(code="iccid_20_format", message="iccid_20 必须是纯数字")
if iccid_20[:19] != iccid_19:
return ValidationError(
code="iccid_20_prefix_mismatch",
message=f"iccid_20 前 19 位 ({iccid_20[:19]!r}) 与 iccid_19 ({iccid_19!r}) 不一致",
)
return None
def validate_carrier_iccid_match(
iccid_20: str, carrier_type: str
) -> Optional[ValidationError]:
"""业务方填写的 iccid_20 与奇成查到的 carrier_type 是否匹配。
- CTCC(19 位运营商):iccid_20 必须为空
- CMCC/CUCC/CBN(20 位运营商):iccid_20 必填
"""
if carrier_type not in VALID_CARRIER_TYPES:
return ValidationError(
code="carrier_type_invalid",
message=f"未知运营商类型 {carrier_type!r}",
)
expected_len = CARRIER_ICCID_LENGTH[carrier_type]
if expected_len == 19 and iccid_20:
return ValidationError(
code="carrier_length_conflict",
message=f"{carrier_type} 是 19 位运营商,iccid_20 应留空,实际填了 {iccid_20!r}",
)
if expected_len == 20 and not iccid_20:
return ValidationError(
code="carrier_length_conflict",
message=f"{carrier_type}{expected_len} 位运营商,iccid_20 必填",
)
return None

View File

@@ -0,0 +1,510 @@
"""奇成 MySQL 只读查询封装。
接入 kyhl 库,仅 SELECT,强制 READ ONLY 事务防止误写。
所有查询按 ICCID/agent_id 批量分块,避免单条 SQL 过大。
"""
from __future__ import annotations
import contextlib
from dataclasses import dataclass
from decimal import Decimal
from typing import Any, Iterable, Iterator, Optional
BATCH_SIZE = 500
def _import_pymysql():
"""延迟导入 pymysql,这样脚本1(纯离线)不需要安装 pymysql 也能运行。"""
try:
import pymysql
from pymysql.cursors import DictCursor
return pymysql, DictCursor
except ImportError as exc:
raise ImportError("缺少 pymysql,请先 pip install -r requirements.txt") from exc
@dataclass(frozen=True)
class LegacyPackage:
"""奇成当前生效套餐快照(取自 tbl_card_life)。"""
iccid: str
meal_id: str
meal_name: str
start_date: Optional[str]
expire_date: Optional[str]
flow_size_mb: int # 套餐总量,单位 MB
status: int
@dataclass(frozen=True)
class LegacyCardUsage:
"""奇成卡累计流量快照。
奇成 ``tbl_card.total_bytes_cnt`` 存的是经过当前套餐 ``flow_add_discount``
加成后的"虚用量",我们这里同时拉取当前生效套餐(``tbl_card_life`` status=1
且 ``expire_date`` 最大)的 ``flow_add_discount``,反算回新系统期望的"真用量"
精度说明:跨多个套餐周期累计时,``total_bytes_cnt`` 是按各时段对应套餐
虚比加权累加的,这里只用"当前生效套餐"的虚比反算,**有误差但可接受**——
奇成没有按套餐分段的用量明细,无更好做法。
"""
iccid: str # 业务方视角的 iccid_19
virtual_used_mb: Decimal # 奇成 total_bytes_cnt 原始值(虚 MB)
flow_add_discount_pct: Decimal # 当前生效套餐虚比百分比(0~100)
has_active_package: bool # 是否在 tbl_card_life 中找到当前生效套餐
@property
def real_used_mb(self) -> Decimal:
"""反算真用量(Decimal MB,保留小数)。
虚 = 真 × (1 + pct/100) → 真 = 虚 / (1 + pct/100)
"""
if self.virtual_used_mb <= 0:
return Decimal("0")
ratio = Decimal(1) + self.flow_add_discount_pct / Decimal(100)
return self.virtual_used_mb / ratio
@property
def real_used_mb_floor(self) -> int:
"""适配新系统 ``data_usage_mb``(BIGINT)的整数 MB。
取整规则:0 → 0,(0, 1) → 1,≥1 → 向下取整。
sub-MB 上取整为 1,避免"明明用过的卡显示 0"
"""
real = self.real_used_mb
if real <= 0:
return 0
if real < 1:
return 1
return int(real)
@dataclass(frozen=True)
class LegacyCommissionAccount:
"""奇成代理佣金账户快照。"""
agent_id: str
agent_name: str
can_draw_amount_fen: int # 当前可提现佣金(分)
applying_draw_amount_fen: int # 申请提现中佣金(分)
total_commission_amount_fen: int # 累计总佣金(分,做参考)
total_draw_amount_fen: int # 累计已提现(分,做参考)
@dataclass(frozen=True)
class LegacyAgentBalance:
"""奇成代理主钱包余额快照(取自 tbl_agent.balance_money)。"""
agent_id: str
balance_fen: int
@dataclass(frozen=True)
class LegacyCardMeta:
"""奇成卡基础元数据(由 scan_legacy / migrate_assets 共享)。
业务方提供的 iccid_19 + iccid_20 才是新系统的权威值,奇成的 iccid_mark 长度不可信。
legacy_raw_iccid 仅作调试参考。
"""
legacy_raw_iccid: str # 奇成实际存的 iccid_mark(可能 19 或 20 位)
category: int # tbl_card.category(运营商分类编号)
category_name: str # tbl_vendor_category.category_name
agent_id: str
agent_name: str
msisdn: str # tbl_card.phone
virtual_no: str # tbl_virtual_number.vcode(如有)
current_meal_id: str # 当前生效套餐 ID(可空,取自 tbl_card_life status=1 expire_date 最大)
current_meal_name: str
current_expire_date: Optional[str] # ISO 字符串
@dataclass(frozen=True)
class LegacyMealMeta:
"""奇成套餐 + 系列元数据(由 scan_legacy 用)。"""
meal_id: str
meal_name: str
series_id: str
series_name: str
# ---------------- 连接管理 ----------------
@contextlib.contextmanager
def connect_readonly(dsn: dict) -> Iterator[Any]:
"""打开只读连接。
用法:
with connect_readonly(dsn) as conn:
rows = fetch_current_packages(conn, iccids)
"""
pymysql, DictCursor = _import_pymysql()
conn = pymysql.connect(
host=dsn["host"],
port=int(dsn.get("port", 3306)),
user=dsn["user"],
password=dsn["password"],
database=dsn["database"],
charset=dsn.get("charset", "utf8mb4"),
connect_timeout=int(dsn.get("connect_timeout", 10)),
cursorclass=DictCursor,
autocommit=False,
)
try:
with conn.cursor() as cur:
cur.execute("SET SESSION TRANSACTION READ ONLY")
cur.execute("SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ")
conn.begin()
yield conn
conn.rollback() # 只读事务,回滚即结束
finally:
conn.close()
def _chunks(items: list[str], size: int = BATCH_SIZE) -> Iterator[list[str]]:
for start in range(0, len(items), size):
yield items[start : start + size]
def _to_fen(value) -> int:
"""元 → 分。奇成的 decimal 是元单位。"""
if value is None:
return 0
try:
d = Decimal(str(value))
except Exception:
return 0
return int((d * Decimal("100")).quantize(Decimal("1")))
def _to_mb_decimal(value) -> Decimal:
"""字符串/数字 → Decimal MB。奇成 total_bytes_cnt 是小数 MB 读数。"""
if value is None or value == "":
return Decimal("0")
try:
d = Decimal(str(value))
if d < 0:
return Decimal("0")
return d
except Exception:
return Decimal("0")
# ---------------- 查询函数 ----------------
def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPackage]:
"""查每张卡的当前生效套餐:status=1 且 expire_date 最大。
返回: iccid → LegacyPackage(找不到的 iccid 不在结果中)
"""
iccid_list = sorted({i for i in iccids if i})
out: dict[str, LegacyPackage] = {}
if not iccid_list:
return out
sql = """
SELECT
iccid_mark,
meal_id,
meal_name,
start_date,
expire_date,
COALESCE(flow_size, 0) AS flow_size,
status
FROM tbl_card_life
WHERE iccid_mark IN %s
AND status = 1
ORDER BY iccid_mark ASC, expire_date DESC
"""
seen: set[str] = set()
with conn.cursor() as cur:
for batch in _chunks(iccid_list):
cur.execute(sql, (tuple(batch),))
for row in cur.fetchall():
iccid = row["iccid_mark"]
if not iccid or iccid in seen:
continue
seen.add(iccid)
out[iccid] = LegacyPackage(
iccid=iccid,
meal_id=row.get("meal_id") or "",
meal_name=row.get("meal_name") or "",
start_date=row["start_date"].strftime("%Y-%m-%d %H:%M:%S") if row.get("start_date") else None,
expire_date=row["expire_date"].strftime("%Y-%m-%d %H:%M:%S") if row.get("expire_date") else None,
flow_size_mb=int(row.get("flow_size") or 0),
status=int(row.get("status") or 0),
)
return out
def fetch_card_usage(conn, iccid_pairs: Iterable[tuple[str, str]]) -> dict[str, LegacyCardUsage]:
"""查每张卡的累计已用流量,并同步取当前生效套餐的 flow_add_discount。
奇成 ``tbl_card.total_bytes_cnt`` 是虚用量(已被套餐虚比加成)。
这里 JOIN 当前生效套餐(``tbl_card_life`` status=1 且 ``expire_date`` 最大)
+ ``tbl_set_meal`` 取 ``flow_add_discount``,放进 ``LegacyCardUsage``;
上层通过 ``real_used_mb_floor`` 反算真用量写入新系统。
Args:
iccid_pairs: (iccid_19, iccid_20) 列表;iccid_20 可为空字符串。
Returns:
{iccid_19: LegacyCardUsage}。
"""
pairs = [(p[0], p[1]) for p in iccid_pairs if p[0]]
out: dict[str, LegacyCardUsage] = {}
if not pairs:
return out
full_to_19: dict[str, str] = {}
for i19, i20 in pairs:
full_to_19[i19] = i19
if i20:
full_to_19[i20] = i19
full_set = sorted(full_to_19.keys())
# tbl_card 主表用 iccid_mark IN 全长精确匹配;tbl_card_life 从表用
# LEFT(iccid_mark, 19) 兜底,兼容奇成两张表 iccid 长度不一致的脏数据。
sql = """
SELECT
c.iccid_mark,
c.total_bytes_cnt,
COALESCE(sm.flow_add_discount, 0) AS flow_add_discount,
CASE WHEN cl.iccid_mark IS NOT NULL THEN 1 ELSE 0 END AS has_active_package
FROM tbl_card c
LEFT JOIN (
SELECT cl1.iccid_mark, cl1.meal_id
FROM tbl_card_life cl1
INNER JOIN (
SELECT iccid_mark, MAX(expire_date) AS max_expire
FROM tbl_card_life
WHERE status = 1 AND iccid_mark IN %s
GROUP BY iccid_mark
) latest
ON latest.iccid_mark = cl1.iccid_mark
AND latest.max_expire = cl1.expire_date
WHERE cl1.status = 1
) cl ON LEFT(cl.iccid_mark, 19) = LEFT(c.iccid_mark, 19)
LEFT JOIN tbl_set_meal sm ON sm.id = cl.meal_id
WHERE c.iccid_mark IN %s
"""
with conn.cursor() as cur:
for batch in _chunks(full_set):
cur.execute(sql, (tuple(batch), tuple(batch)))
for row in cur.fetchall():
legacy_iccid = row["iccid_mark"]
if not legacy_iccid:
continue
i19 = full_to_19.get(legacy_iccid)
if i19 is None:
continue
# 取第一个匹配,后续 fetch_card_meta 已经会把重复命中的卡标记为 errors
if i19 in out:
continue
out[i19] = LegacyCardUsage(
iccid=i19,
virtual_used_mb=_to_mb_decimal(row.get("total_bytes_cnt")),
flow_add_discount_pct=_to_mb_decimal(row.get("flow_add_discount")),
has_active_package=bool(row.get("has_active_package")),
)
return out
def fetch_commission_accounts(conn, agent_ids: Iterable[str]) -> dict[str, LegacyCommissionAccount]:
"""查代理佣金账户。"""
agent_list = sorted({a for a in agent_ids if a})
out: dict[str, LegacyCommissionAccount] = {}
if not agent_list:
return out
sql = """
SELECT
agent_id, agent_name,
can_draw_amount, applying_draw_amount,
total_commision_amount, total_draw_amount
FROM tbl_agent_commission_account
WHERE agent_id IN %s
"""
with conn.cursor() as cur:
for batch in _chunks(agent_list):
cur.execute(sql, (tuple(batch),))
for row in cur.fetchall():
aid = row["agent_id"]
if not aid:
continue
out[aid] = LegacyCommissionAccount(
agent_id=aid,
agent_name=row.get("agent_name") or "",
can_draw_amount_fen=_to_fen(row.get("can_draw_amount")),
applying_draw_amount_fen=_to_fen(row.get("applying_draw_amount")),
total_commission_amount_fen=_to_fen(row.get("total_commision_amount")),
total_draw_amount_fen=_to_fen(row.get("total_draw_amount")),
)
return out
def fetch_card_meta(
conn,
iccid_pairs: Iterable[tuple[str, str]],
) -> tuple[dict[str, LegacyCardMeta], set[str]]:
"""查每张卡的元数据(卡本体 + 当前生效套餐 + 虚拟号)。
匹配策略:
- tbl_card 主表用 `iccid_mark IN (iccid_19 iccid_20)` 全长精确;
这样不会因前 19 位相同(校验位不同的两张 20 位卡)而误匹配多张
- tbl_card_life / tbl_virtual_number 从表用 LEFT(iccid_mark, 19) 兜底,
应对奇成两张表 iccid 长度不一致的脏数据
Args:
iccid_pairs: (iccid_19, iccid_20) 列表;iccid_20 可为空字符串(CTCC 卡)。
Returns:
(metas, duplicate_iccid_19):
- metas: {iccid_19: LegacyCardMeta} 单匹结果
- duplicate_iccid_19: 一份业务方 iccid_19 命中奇成 tbl_card 多条记录的 set,
这些卡不进 metas,sql_builder 会写 errors.csv 阻断
"""
pairs = [(p[0], p[1]) for p in iccid_pairs if p[0]]
if not pairs:
return {}, set()
# 全长 ICCID → 业务方 iccid_19。同时维护 iccid_20 优先级:
# 奇成 iccid_mark 命中时,优先取等于 iccid_20 的映射(20 位精确),
# 找不到再退到 iccid_19(奇成存了 19 位的兜底)。
by_full: dict[str, str] = {} # 任意长度 ICCID → iccid_19
for i19, i20 in pairs:
by_full[i19] = i19
if i20:
by_full[i20] = i19
full_set = sorted(by_full.keys())
sql = """
SELECT
c.iccid_mark AS iccid,
COALESCE(c.category, 0) AS category,
COALESCE(vc.category_name, '') AS category_name,
COALESCE(c.agent_id, '') AS agent_id,
COALESCE(c.agent_name, '') AS agent_name,
COALESCE(c.phone, '') AS phone,
COALESCE(vn.vcode, '') AS vcode,
COALESCE(cl.meal_id, '') AS meal_id,
COALESCE(cl.meal_name, '') AS meal_name,
cl.expire_date AS expire_date
FROM tbl_card c
LEFT JOIN tbl_vendor_category vc ON vc.category = c.category
LEFT JOIN tbl_virtual_number vn ON LEFT(vn.iccid_mark, 19) = LEFT(c.iccid_mark, 19)
LEFT JOIN (
SELECT cl1.iccid_mark, cl1.meal_id, cl1.meal_name, cl1.expire_date
FROM tbl_card_life cl1
INNER JOIN (
SELECT iccid_mark, MAX(expire_date) AS max_expire
FROM tbl_card_life
WHERE status = 1 AND iccid_mark IN %s
GROUP BY iccid_mark
) latest
ON latest.iccid_mark = cl1.iccid_mark
AND latest.max_expire = cl1.expire_date
WHERE cl1.status = 1
) cl ON LEFT(cl.iccid_mark, 19) = LEFT(c.iccid_mark, 19)
WHERE c.iccid_mark IN %s
"""
# 收集每个 iccid_19 命中的奇成行(可能多条 = 脏数据)
hits: dict[str, list[dict]] = {}
with conn.cursor() as cur:
for batch in _chunks(full_set):
cur.execute(sql, (tuple(batch), tuple(batch)))
for row in cur.fetchall():
legacy_iccid = row["iccid"]
if not legacy_iccid:
continue
i19 = by_full.get(legacy_iccid)
if i19 is None:
# WHERE 已经限定 IN,理论不会出现;保险起见再按前 19 位反查
i19 = by_full.get(legacy_iccid[:19])
if i19 is None:
continue
hits.setdefault(i19, []).append(row)
metas: dict[str, LegacyCardMeta] = {}
duplicates: set[str] = set()
for i19, rows in hits.items():
if len(rows) > 1:
duplicates.add(i19)
continue
row = rows[0]
expire = row.get("expire_date")
expire_str = expire.strftime("%Y-%m-%d %H:%M:%S") if expire else None
metas[i19] = LegacyCardMeta(
legacy_raw_iccid=row["iccid"],
category=int(row.get("category") or 0),
category_name=row.get("category_name") or "",
agent_id=row.get("agent_id") or "",
agent_name=row.get("agent_name") or "",
msisdn=row.get("phone") or "",
virtual_no=row.get("vcode") or "",
current_meal_id=row.get("meal_id") or "",
current_meal_name=row.get("meal_name") or "",
current_expire_date=expire_str,
)
return metas, duplicates
def fetch_meal_meta(conn, meal_ids: Iterable[str]) -> dict[str, LegacyMealMeta]:
"""查套餐 + 系列元数据(供 scan_legacy 收集映射 legacy_* 字段)。"""
id_list = sorted({m for m in meal_ids if m})
out: dict[str, LegacyMealMeta] = {}
if not id_list:
return out
sql = """
SELECT
sm.id AS meal_id,
COALESCE(sm.name, '') AS meal_name,
COALESCE(sm.meal_series_id, '') AS series_id,
COALESCE(sms.name, '') AS series_name
FROM tbl_set_meal sm
LEFT JOIN tbl_set_meal_series sms ON sms.id = sm.meal_series_id
WHERE sm.id IN %s
"""
with conn.cursor() as cur:
for batch in _chunks(id_list):
cur.execute(sql, (tuple(batch),))
for row in cur.fetchall():
mid = row["meal_id"]
if not mid:
continue
out[mid] = LegacyMealMeta(
meal_id=mid,
meal_name=row.get("meal_name") or "",
series_id=row.get("series_id") or "",
series_name=row.get("series_name") or "",
)
return out
def fetch_agent_balances(conn, agent_ids: Iterable[str]) -> dict[str, LegacyAgentBalance]:
"""查代理主钱包余额(tbl_agent.balance_money,元单位)。"""
agent_list = sorted({a for a in agent_ids if a})
out: dict[str, LegacyAgentBalance] = {}
if not agent_list:
return out
sql = """
SELECT id, balance_money
FROM tbl_agent
WHERE id IN %s
"""
with conn.cursor() as cur:
for batch in _chunks(agent_list):
cur.execute(sql, (tuple(batch),))
for row in cur.fetchall():
aid = row["id"]
if not aid:
continue
out[aid] = LegacyAgentBalance(agent_id=aid, balance_fen=_to_fen(row.get("balance_money")))
return out

View File

@@ -0,0 +1,245 @@
"""mapping.yaml 单文件加载与增量合并。
设计要点:
- 一个文件搞定所有映射(migration_user_id / carriers / packages / series / agents)
- 由 scan_legacy.py 增量补全 legacy_* 字段;人工只需填 target_*
- lookup 时检查 target_* 是否已填,未填则抛 MissingTargetError 让调用方写入 errors.csv
"""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Optional
import yaml
class MissingTargetError(Exception):
"""target_* 字段未填的具体错误,带 legacy 标识信息。"""
def __init__(self, kind: str, legacy_key: str, legacy_name: str, missing_fields: list[str]):
self.kind = kind
self.legacy_key = legacy_key
self.legacy_name = legacy_name
self.missing_fields = missing_fields
super().__init__(
f"{kind} 映射未填 target_*: legacy_key={legacy_key!r} ({legacy_name!r}), 缺失={missing_fields}"
)
@dataclass(frozen=True)
class CarrierMapping:
legacy_category: int
legacy_category_name: str
target_carrier_id: Optional[int]
target_carrier_type: Optional[str]
target_carrier_name: Optional[str]
def require_target(self) -> "CarrierMapping":
missing = [
n for n, v in (
("target_carrier_id", self.target_carrier_id),
("target_carrier_type", self.target_carrier_type),
("target_carrier_name", self.target_carrier_name),
) if v in (None, "", 0)
]
if missing:
raise MissingTargetError("carrier", str(self.legacy_category), self.legacy_category_name, missing)
return self
@dataclass(frozen=True)
class PackageMapping:
legacy_meal_id: str
legacy_meal_name: str
target_package_id: Optional[int]
def require_target(self) -> "PackageMapping":
if self.target_package_id in (None, 0):
raise MissingTargetError("package", self.legacy_meal_id, self.legacy_meal_name, ["target_package_id"])
return self
@dataclass(frozen=True)
class SeriesMapping:
legacy_series_id: str
legacy_series_name: str
target_series_id: Optional[int] # 允许 null,表示不写入新库 series_id
@dataclass(frozen=True)
class AgentMapping:
legacy_agent_id: str
legacy_agent_name: str
target_shop_code: Optional[str] # 允许 null,表示该代理的卡进平台库存
@dataclass
class Mapping:
"""整体映射(对应单个 mapping.yaml)。"""
migration_user_id: int = 0
migration_batch_no: str = "MIGRATION-QICHENG"
carriers: dict[int, CarrierMapping] = field(default_factory=dict)
packages: dict[str, PackageMapping] = field(default_factory=dict)
series: dict[str, SeriesMapping] = field(default_factory=dict)
agents: dict[str, AgentMapping] = field(default_factory=dict)
def lookup_carrier(self, legacy_category: int) -> Optional[CarrierMapping]:
return self.carriers.get(int(legacy_category))
def lookup_package(self, legacy_meal_id: str) -> Optional[PackageMapping]:
return self.packages.get(str(legacy_meal_id).strip())
def lookup_series(self, legacy_series_id: str) -> Optional[SeriesMapping]:
if not legacy_series_id:
return None
return self.series.get(str(legacy_series_id).strip())
def lookup_agent_shop_code(self, legacy_agent_id: str) -> Optional[str]:
"""按 agent_id 查 target_shop_code,空字符串视为 None(进平台库存)。"""
if not legacy_agent_id:
return None
agent = self.agents.get(str(legacy_agent_id).strip())
if agent is None:
return None
code = (agent.target_shop_code or "").strip()
return code or None
# ---------------- 加载 ----------------
MAPPING_FILE = "mapping.yaml"
DSN_FILE = "legacy_dsn.yaml"
def load_mapping(config_dir: Path) -> Mapping:
"""加载 config/mapping.yaml。文件不存在时返回空 Mapping(供 scan_legacy 首次创建)。"""
path = config_dir / MAPPING_FILE
if not path.exists():
return Mapping()
with path.open("r", encoding="utf-8") as f:
raw = yaml.safe_load(f) or {}
if not isinstance(raw, dict):
raise ValueError(f"{path} 顶层必须是 mapping 类型")
m = Mapping(
migration_user_id=int(raw.get("migration_user_id", 0)),
migration_batch_no=_opt_str(raw.get("migration_batch_no")) or "MIGRATION-QICHENG",
)
for item in raw.get("carriers") or []:
c = CarrierMapping(
legacy_category=int(item["legacy_category"]),
legacy_category_name=str(item.get("legacy_category_name") or ""),
target_carrier_id=_opt_int(item.get("target_carrier_id")),
target_carrier_type=_opt_str(item.get("target_carrier_type")),
target_carrier_name=_opt_str(item.get("target_carrier_name")),
)
m.carriers[c.legacy_category] = c
for item in raw.get("packages") or []:
p = PackageMapping(
legacy_meal_id=str(item["legacy_meal_id"]).strip(),
legacy_meal_name=str(item.get("legacy_meal_name") or ""),
target_package_id=_opt_int(item.get("target_package_id")),
)
m.packages[p.legacy_meal_id] = p
for item in raw.get("series") or []:
s = SeriesMapping(
legacy_series_id=str(item["legacy_series_id"]).strip(),
legacy_series_name=str(item.get("legacy_series_name") or ""),
target_series_id=_opt_int(item.get("target_series_id")),
)
m.series[s.legacy_series_id] = s
for item in raw.get("agents") or []:
a = AgentMapping(
legacy_agent_id=str(item["legacy_agent_id"]).strip(),
legacy_agent_name=str(item.get("legacy_agent_name") or ""),
target_shop_code=_opt_str(item.get("target_shop_code")),
)
m.agents[a.legacy_agent_id] = a
return m
def merge_and_save(config_dir: Path, mapping: Mapping) -> Path:
"""把 mapping 写回 config/mapping.yaml,保留人工已填的 target_* 字段。
由 scan_legacy 调用:先 load,再用扫描结果 _补充_ legacy_* 项(不存在的追加),最后调用本函数写回。
"""
config_dir.mkdir(parents=True, exist_ok=True)
path = config_dir / MAPPING_FILE
data = {
"migration_user_id": mapping.migration_user_id,
"migration_batch_no": mapping.migration_batch_no,
"carriers": [
{
"legacy_category": c.legacy_category,
"legacy_category_name": c.legacy_category_name,
"target_carrier_id": c.target_carrier_id,
"target_carrier_type": c.target_carrier_type,
"target_carrier_name": c.target_carrier_name,
}
for c in sorted(mapping.carriers.values(), key=lambda x: x.legacy_category)
],
"packages": [
{
"legacy_meal_id": p.legacy_meal_id,
"legacy_meal_name": p.legacy_meal_name,
"target_package_id": p.target_package_id,
}
for p in sorted(mapping.packages.values(), key=lambda x: x.legacy_meal_id)
],
"series": [
{
"legacy_series_id": s.legacy_series_id,
"legacy_series_name": s.legacy_series_name,
"target_series_id": s.target_series_id,
}
for s in sorted(mapping.series.values(), key=lambda x: x.legacy_series_id)
],
"agents": [
{
"legacy_agent_id": a.legacy_agent_id,
"legacy_agent_name": a.legacy_agent_name,
"target_shop_code": a.target_shop_code,
}
for a in sorted(mapping.agents.values(), key=lambda x: x.legacy_agent_id)
],
}
with path.open("w", encoding="utf-8") as f:
yaml.safe_dump(data, f, allow_unicode=True, sort_keys=False, default_flow_style=False)
return path
def load_legacy_dsn(config_dir: Path) -> dict:
"""加载奇成 MySQL 只读连接配置。"""
path = config_dir / DSN_FILE
if not path.exists():
raise FileNotFoundError(f"找不到 {path},请基于 legacy_dsn.yaml.example 创建并填写真实 DSN")
with path.open("r", encoding="utf-8") as f:
data = yaml.safe_load(f) or {}
for required in ("host", "port", "user", "password", "database"):
if required not in data:
raise ValueError(f"{path} 缺少必填项 {required}")
return data
# ---------------- 小工具 ----------------
def _opt_int(v: Any) -> Optional[int]:
if v is None or v == "":
return None
return int(v)
def _opt_str(v: Any) -> Optional[str]:
if v is None:
return None
s = str(v).strip()
return s or None

View File

@@ -0,0 +1,845 @@
"""SQL 输出构造。
输出文件:
step1_01_iot_cards.sql - 卡资产
step1_02_devices.sql - 设备资产
step1_03_asset_identifiers.sql- 全局标识符(iccid / virtual_no)
step1_04_asset_wallets.sql - 资产钱包
step1_05_sim_bindings.sql - 设备-卡绑定
step1_06_shop_allocations.sql - 店铺归属的分配记录
step2_01_migration_orders.sql - 迁移伪订单
step2_02_package_usages.sql - 套餐使用记录
step2_03_card_data_usage_update.sql - 卡累计流量 UPDATE
step2_04_agent_wallet_init.sql - 代理钱包初始化(main + commission)
step2_05_agent_wallet_transactions.sql - 代理钱包初始化流水
设计原则:
- 所有 INSERT 通过 SELECT 子查询动态取 id,不硬编码新库 id
- 使用 ON CONFLICT (...) WHERE ... DO NOTHING 保证幂等(PG 14+ 支持 partial index 推断)
- 字符串 ' 转义为 ''
- 卡的元数据(carrier/agent_id/msisdn/virtual_no/当前套餐)从奇成 fetch_card_meta 取,
cards.csv 只承载 is_industry 覆盖,店铺归属完全由 mapping.agents 推导
"""
from __future__ import annotations
import csv
from datetime import datetime
from decimal import Decimal
from pathlib import Path
from typing import Optional
from . import iccid_utils
from .csv_loader import CardRow, DeviceRow, ErrorRow
from .legacy_query import (
LegacyAgentBalance,
LegacyCardMeta,
LegacyCardUsage,
LegacyCommissionAccount,
)
from .mapping_loader import Mapping, MissingTargetError
# 写死的默认值(对应原 migration.yaml,实际不需要业务侧关心)
ORDER_NO_PREFIX_CARD = "MIG-CARD-"
ALLOCATION_NO_PREFIX = "MIG-ALLOC-"
METADATA_SOURCE = "qicheng"
# ---------------- 通用工具 ----------------
def _sql_str(value: Optional[str]) -> str:
if value is None:
return "NULL"
s = str(value)
return "'" + s.replace("'", "''") + "'"
def _sql_jsonb(d: dict) -> str:
import json
s = json.dumps(d, ensure_ascii=False, separators=(",", ":"))
return "'" + s.replace("'", "''") + "'::jsonb"
def _sql_decimal(value: Decimal) -> str:
"""生成 SQL decimal 字面量,保留奇成小数 MB 精度。"""
return format(value, "f")
def _shop_id_sub(shop_code: Optional[str]) -> str:
"""生成 shop_id 子查询表达式;空 shop_code 返回 'NULL'(平台库存)。"""
if not shop_code:
return "NULL"
return f"(SELECT id FROM tb_shop WHERE shop_code = {_sql_str(shop_code)} AND deleted_at IS NULL LIMIT 1)"
def _now_for_no() -> str:
return datetime.now().strftime("%Y%m%d%H%M%S")
def _file_header(title: str) -> str:
return (
"-- ============================================================\n"
f"-- {title}\n"
f"-- 生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n"
"-- 重跑安全: ON CONFLICT DO NOTHING\n"
"-- 建议执行: psql ... -1 -f <此文件>\n"
"-- ============================================================\n\n"
)
def _resolve_card_shop_code(card: CardRow, meta: LegacyCardMeta, mapping: Mapping) -> str:
"""决定卡最终 shop_code:设备上的卡→空(跟设备走);否则按 agent 映射;查不到→平台库存。"""
if card.bound_device_virtual_no:
return ""
if meta and meta.agent_id:
shop = mapping.lookup_agent_shop_code(meta.agent_id)
if shop:
return shop
return ""
def _resolve_device_shop_code(
device: DeviceRow,
card_metas: dict[str, LegacyCardMeta],
mapping: Mapping,
) -> str:
"""决定设备最终 shop_code:按主卡(slot 1)的 agent 映射;无主卡或映射不到→平台库存。"""
primary_iccid = device.sim_iccids.get(1) or next(iter(device.sim_iccids.values()), "")
if not primary_iccid:
return ""
meta = card_metas.get(primary_iccid)
if not meta or not meta.agent_id:
return ""
return mapping.lookup_agent_shop_code(meta.agent_id) or ""
# ============================================================
# 脚本1:资产导入
# ============================================================
def write_step1(
output_dir: Path,
*,
cards: list[CardRow],
devices: list[DeviceRow],
mapping: Mapping,
card_metas: dict[str, LegacyCardMeta],
duplicate_iccid_19s: set[str],
errors: list[ErrorRow],
) -> None:
"""生成 step1_* SQL + errors.csv + summary.txt。
会就地修改 errors 列表,把映射缺失等问题追加进去。
Args:
card_metas: {iccid_19: LegacyCardMeta}
duplicate_iccid_19s: 同一 iccid_19 在奇成命中多条 tbl_card 记录的集合,直接报错阻断
"""
user_id = mapping.migration_user_id
# 预检 carrier 映射 + 运营商一致性 + 重复命中,把异常卡剔除
valid_cards = _filter_cards_by_carrier(cards, card_metas, duplicate_iccid_19s, mapping, errors)
cards_by_iccid_19 = {c.iccid_19: c for c in valid_cards}
_write_iot_cards(output_dir / "step1_01_iot_cards.sql", valid_cards, card_metas, mapping, user_id)
_write_devices(output_dir / "step1_02_devices.sql", devices, card_metas, mapping, user_id)
_write_asset_identifiers(output_dir / "step1_03_asset_identifiers.sql", valid_cards, devices, card_metas)
_write_asset_wallets(output_dir / "step1_04_asset_wallets.sql", valid_cards, devices)
_write_sim_bindings(output_dir / "step1_05_sim_bindings.sql", devices, cards_by_iccid_19)
_write_shop_allocations(
output_dir / "step1_06_shop_allocations.sql", valid_cards, devices, card_metas, mapping, user_id
)
_write_errors(output_dir / "errors.csv", errors)
_write_summary_step1(output_dir / "summary.txt", valid_cards, devices, errors)
def _filter_cards_by_carrier(
cards: list[CardRow],
card_metas: dict[str, LegacyCardMeta],
duplicate_iccid_19s: set[str],
mapping: Mapping,
errors: list[ErrorRow],
) -> list[CardRow]:
"""筛掉:奇成查不到 / 命中多条 / carrier 映射缺失 / target_* 未填 / 运营商与 iccid 长度不一致 的卡。"""
valid: list[CardRow] = []
for c in cards:
if c.iccid_19 in duplicate_iccid_19s:
errors.append(ErrorRow(
"cards.csv", c.line_no, "iccid_19", c.iccid_19,
"legacy_duplicate", "奇成 tbl_card 中同一 iccid_19 命中多条记录,请业务方人工确认",
))
continue
meta = card_metas.get(c.iccid_19)
if meta is None:
errors.append(ErrorRow(
"cards.csv", c.line_no, "iccid_19", c.iccid_19,
"not_in_legacy", "奇成 tbl_card 中找不到该 ICCID(iccid_19/iccid_20 都没命中)",
))
continue
if not meta.category:
errors.append(ErrorRow(
"cards.csv", c.line_no, "iccid_19", c.iccid_19,
"no_category", "奇成 tbl_card.category 为空,无法定位运营商映射",
))
continue
carrier = mapping.lookup_carrier(meta.category)
if carrier is None:
errors.append(ErrorRow(
"cards.csv", c.line_no, "category", str(meta.category),
"carrier_not_in_mapping", f"mapping.yaml.carriers 未配置 category={meta.category} ({meta.category_name})",
))
continue
try:
carrier.require_target()
except MissingTargetError as exc:
errors.append(ErrorRow(
"cards.csv", c.line_no, "category", str(meta.category),
"carrier_target_missing", str(exc),
))
continue
# 运营商类型与业务方 iccid_20 一致性校验
err = iccid_utils.validate_carrier_iccid_match(c.iccid_20, carrier.target_carrier_type or "")
if err is not None:
errors.append(ErrorRow(
"cards.csv", c.line_no, "iccid_20", c.iccid_20,
err.code, err.message,
))
continue
valid.append(c)
return valid
def _write_iot_cards(
path: Path,
cards: list[CardRow],
card_metas: dict[str, LegacyCardMeta],
mapping: Mapping,
user_id: int,
) -> None:
"""写 tb_iot_card。元数据从奇成 card_metas + mapping 推导,iccid_19/iccid_20 用业务方两列。"""
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step1_01 卡资产导入 (tb_iot_card)"))
for c in cards:
meta = card_metas[c.iccid_19]
carrier = mapping.lookup_carrier(meta.category)
assert carrier is not None # 已被 _filter_cards_by_carrier 保证
# 业务方直接给的两列;iccid 唯一键 = 有 iccid_20 用 iccid_20,否则 iccid_19
iccid_20_expr = _sql_str(c.iccid_20) if c.iccid_20 else "NULL"
card_category = "industry" if c.is_industry else "normal"
shop_code = _resolve_card_shop_code(c, meta, mapping)
series = mapping.lookup_series(_meal_to_series_id(meta, mapping))
series_id_expr = str(series.target_series_id) if series and series.target_series_id else "NULL"
f.write(
"INSERT INTO tb_iot_card (\n"
" iccid, iccid_19, iccid_20,\n"
" carrier_id, carrier_type, carrier_name,\n"
" card_category, batch_no,\n"
" status, activation_status, real_name_status, network_status,\n"
" asset_status, generation, is_standalone,\n"
" realname_policy, stop_reason,\n"
" msisdn, virtual_no, shop_id, series_id,\n"
" device_virtual_no, gateway_extend,\n"
" creator, updater, created_at, updated_at\n"
") VALUES (\n"
f" {_sql_str(c.iccid_full)}, {_sql_str(c.iccid_19)}, {iccid_20_expr},\n"
f" {carrier.target_carrier_id}, {_sql_str(carrier.target_carrier_type)}, {_sql_str(carrier.target_carrier_name)},\n"
f" {_sql_str(card_category)}, {_sql_str(mapping.migration_batch_no)},\n"
" 1, 0, 0, 0,\n"
" 1, 1, TRUE,\n"
" 'after_order', '',\n"
f" {_sql_str(meta.msisdn) if meta.msisdn else 'NULL'}, "
f"{_sql_str(meta.virtual_no) if meta.virtual_no else 'NULL'}, "
f"{_shop_id_sub(shop_code)}, {series_id_expr},\n"
" '', '',\n"
f" {user_id}, {user_id}, NOW(), NOW()\n"
")\n"
"ON CONFLICT (iccid) WHERE deleted_at IS NULL DO NOTHING;\n\n"
)
def _meal_to_series_id(meta: LegacyCardMeta, mapping: Mapping) -> str:
"""由 meta 的 meal_id 经 mapping.packages 间接拿不到 series,所以这里直接靠
mapping.series 用 legacy_series_id 反查就行。简化:返回空字符串(让上层走 series 映射)。
实际上 series_id 仅在套餐有 meal_series_id 时才有意义。考虑到 LegacyCardMeta 没存
series_id,这里返回 '' — 让 series_id_expr 走 NULL 分支。后续若需要,scan_legacy
可把 series_id 也下沉到 card_meta。
"""
_ = meta
_ = mapping
return ""
def _write_devices(
path: Path,
devices: list[DeviceRow],
card_metas: dict[str, LegacyCardMeta],
mapping: Mapping,
user_id: int,
) -> None:
"""设备表 INSERT。
业务方不关心的字段都写默认值:
device_name/device_model/manufacturer = NULL(可后台编辑)
max_sim_slots = 4
status / asset_status / online_status = 1 / 1 / 0
enable_polling = TRUE,realname_policy = 'after_order'
software_version = '',switch_mode = '0'(自动切卡)
"""
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step1_02 设备资产导入 (tb_device)"))
for d in devices:
shop_code = _resolve_device_shop_code(d, card_metas, mapping)
f.write(
"INSERT INTO tb_device (\n"
" virtual_no, max_sim_slots, imei,\n"
" status, asset_status, generation, online_status,\n"
" enable_polling, realname_policy,\n"
" software_version, switch_mode,\n"
" batch_no, shop_id,\n"
" creator, updater, created_at, updated_at\n"
") VALUES (\n"
f" {_sql_str(d.virtual_no)}, {d.max_sim_slots}, "
f"{_sql_str(d.imei) if d.imei else 'NULL'},\n"
" 1, 1, 1, 0,\n"
" TRUE, 'after_order',\n"
" '', '0',\n"
f" {_sql_str(mapping.migration_batch_no)}, {_shop_id_sub(shop_code)},\n"
f" {user_id}, {user_id}, NOW(), NOW()\n"
")\n"
"ON CONFLICT (virtual_no) WHERE deleted_at IS NULL DO NOTHING;\n\n"
)
def _write_asset_identifiers(
path: Path,
cards: list[CardRow],
devices: list[DeviceRow],
card_metas: dict[str, LegacyCardMeta],
) -> None:
"""卡的 iccid + 虚拟号(来自奇成)和设备的 virtual_no 都注册。"""
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step1_03 资产标识符注册 (tb_asset_identifier)"))
for c in cards:
meta = card_metas[c.iccid_19]
f.write(
"INSERT INTO tb_asset_identifier (identifier, asset_type, asset_id)\n"
f"SELECT {_sql_str(c.iccid_full)}, 'iot_card', id FROM tb_iot_card\n"
f"WHERE iccid = {_sql_str(c.iccid_full)} AND deleted_at IS NULL\n"
"ON CONFLICT (identifier) DO NOTHING;\n\n"
)
if meta.virtual_no:
f.write(
"INSERT INTO tb_asset_identifier (identifier, asset_type, asset_id)\n"
f"SELECT {_sql_str(meta.virtual_no)}, 'iot_card', id FROM tb_iot_card\n"
f"WHERE iccid = {_sql_str(c.iccid_full)} AND deleted_at IS NULL\n"
"ON CONFLICT (identifier) DO NOTHING;\n\n"
)
for d in devices:
f.write(
"INSERT INTO tb_asset_identifier (identifier, asset_type, asset_id)\n"
f"SELECT {_sql_str(d.virtual_no)}, 'device', id FROM tb_device\n"
f"WHERE virtual_no = {_sql_str(d.virtual_no)} AND deleted_at IS NULL\n"
"ON CONFLICT (identifier) DO NOTHING;\n\n"
)
def _write_asset_wallets(path: Path, cards: list[CardRow], devices: list[DeviceRow]) -> None:
"""卡和设备各一条钱包,初始余额 0。"""
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step1_04 资产钱包初始化 (tb_asset_wallet)"))
for c in cards:
f.write(
"INSERT INTO tb_asset_wallet (\n"
" resource_type, resource_id, balance, frozen_balance,\n"
" currency, status, version, shop_id_tag,\n"
" created_at, updated_at\n"
")\n"
"SELECT 'iot_card', id, 0, 0,\n"
" 'CNY', 1, 0, COALESCE(shop_id, 0),\n"
" NOW(), NOW()\n"
"FROM tb_iot_card\n"
f"WHERE iccid = {_sql_str(c.iccid_full)} AND deleted_at IS NULL\n"
"ON CONFLICT (resource_type, resource_id) WHERE deleted_at IS NULL DO NOTHING;\n\n"
)
for d in devices:
f.write(
"INSERT INTO tb_asset_wallet (\n"
" resource_type, resource_id, balance, frozen_balance,\n"
" currency, status, version, shop_id_tag,\n"
" created_at, updated_at\n"
")\n"
"SELECT 'device', id, 0, 0,\n"
" 'CNY', 1, 0, COALESCE(shop_id, 0),\n"
" NOW(), NOW()\n"
"FROM tb_device\n"
f"WHERE virtual_no = {_sql_str(d.virtual_no)} AND deleted_at IS NULL\n"
"ON CONFLICT (resource_type, resource_id) WHERE deleted_at IS NULL DO NOTHING;\n\n"
)
def _write_sim_bindings(
path: Path,
devices: list[DeviceRow],
cards_by_iccid_19: dict[str, CardRow],
) -> None:
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step1_05 设备-卡绑定 (tb_device_sim_binding)"))
for d in devices:
for slot, iccid_19 in sorted(d.sim_iccids.items()):
card = cards_by_iccid_19.get(iccid_19)
if card is None:
continue # 该卡可能被 carrier 校验剔除,跳过此 slot 绑定
is_current = "TRUE" if slot == d.current_slot else "FALSE"
f.write(
"INSERT INTO tb_device_sim_binding (\n"
" device_id, iot_card_id, slot_position, bind_status, is_current,\n"
" bind_time, creator, updater, created_at, updated_at\n"
")\n"
f"SELECT d.id, c.id, {slot}, 1, {is_current},\n"
" NOW(), 0, 0, NOW(), NOW()\n"
"FROM tb_device d\n"
"CROSS JOIN tb_iot_card c\n"
f"WHERE d.virtual_no = {_sql_str(d.virtual_no)} AND d.deleted_at IS NULL\n"
f" AND c.iccid = {_sql_str(card.iccid_full)} AND c.deleted_at IS NULL\n"
"ON CONFLICT (device_id, slot_position) WHERE bind_status = 1 AND deleted_at IS NULL DO NOTHING;\n\n"
)
def _write_shop_allocations(
path: Path,
cards: list[CardRow],
devices: list[DeviceRow],
card_metas: dict[str, LegacyCardMeta],
mapping: Mapping,
user_id: int,
) -> None:
"""只为带店铺归属的资产生成分配记录;设备上的卡跟随设备分配。"""
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step1_06 店铺分配记录 (tb_asset_allocation_record)"))
no_ts = _now_for_no()
for c in cards:
if c.bound_device_virtual_no:
continue
meta = card_metas[c.iccid_19]
shop_code = _resolve_card_shop_code(c, meta, mapping)
if not shop_code:
continue
allocation_no = f"{ALLOCATION_NO_PREFIX}CARD-{no_ts}-{c.line_no}"
f.write(
"INSERT INTO tb_asset_allocation_record (\n"
" allocation_no, allocation_type, asset_type, asset_id, asset_identifier,\n"
" from_owner_type, to_owner_type, to_owner_id,\n"
" operator_id, remark,\n"
" creator, updater, created_at, updated_at\n"
")\n"
f"SELECT {_sql_str(allocation_no)}, 'allocate', 'iot_card', c.id, {_sql_str(c.iccid_full)},\n"
" 'platform', 'shop', s.id,\n"
f" {user_id}, '奇成数据迁移初始化分配',\n"
f" {user_id}, {user_id}, NOW(), NOW()\n"
"FROM tb_iot_card c\n"
"CROSS JOIN tb_shop s\n"
f"WHERE c.iccid = {_sql_str(c.iccid_full)} AND c.deleted_at IS NULL\n"
f" AND s.shop_code = {_sql_str(shop_code)} AND s.deleted_at IS NULL;\n\n"
)
for d in devices:
shop_code = _resolve_device_shop_code(d, card_metas, mapping)
if not shop_code:
continue
allocation_no = f"{ALLOCATION_NO_PREFIX}DEV-{no_ts}-{d.line_no}"
f.write(
"INSERT INTO tb_asset_allocation_record (\n"
" allocation_no, allocation_type, asset_type, asset_id, asset_identifier,\n"
" from_owner_type, to_owner_type, to_owner_id,\n"
" operator_id, remark,\n"
" creator, updater, created_at, updated_at\n"
")\n"
f"SELECT {_sql_str(allocation_no)}, 'allocate', 'device', dev.id, {_sql_str(d.virtual_no)},\n"
" 'platform', 'shop', s.id,\n"
f" {user_id}, '奇成数据迁移初始化分配',\n"
f" {user_id}, {user_id}, NOW(), NOW()\n"
"FROM tb_device dev\n"
"CROSS JOIN tb_shop s\n"
f"WHERE dev.virtual_no = {_sql_str(d.virtual_no)} AND dev.deleted_at IS NULL\n"
f" AND s.shop_code = {_sql_str(shop_code)} AND s.deleted_at IS NULL;\n\n"
)
def _write_errors(path: Path, errors: list[ErrorRow]) -> None:
with path.open("w", encoding="utf-8", newline="") as f:
writer = csv.writer(f)
writer.writerow(["source_file", "line_no", "field", "value", "error_code", "message"])
for e in errors:
writer.writerow(e.to_csv_row())
def _write_summary_step1(path: Path, cards: list[CardRow], devices: list[DeviceRow], errors: list[ErrorRow]) -> None:
bound = sum(1 for c in cards if c.bound_device_virtual_no)
lines = [
"奇成迁移 - 脚本1 资产导入摘要",
"=" * 48,
f"生成时间 : {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
f"卡有效总数 : {len(cards)}",
f" 绑设备 : {bound}",
f" 独立卡 : {len(cards) - bound}",
f"设备总数 : {len(devices)}",
f"异常行 : {len(errors)} (详见 errors.csv)",
"",
"店铺归属:由 mapping.agents 自动推导(详见 step1_01/step1_02/step1_06 内的 shop_id 子查询)",
]
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
# ============================================================
# 脚本2:关联数据导入
# ============================================================
def write_step2(
output_dir: Path,
*,
cards: list[CardRow],
mapping: Mapping,
card_metas: dict[str, LegacyCardMeta],
usage_snapshots: dict[str, LegacyCardUsage],
commission_snapshots: dict[str, LegacyCommissionAccount],
agent_balances: dict[str, LegacyAgentBalance],
) -> list[str]:
"""生成 step2_* SQL + warnings.csv + summary.txt。"""
user_id = mapping.migration_user_id
warnings: list[tuple[str, str, str]] = []
# 卡 → package(从 card_meta.current_meal_id 经 mapping.packages 查 target)
resolved_packages: dict[str, tuple[int, str]] = {} # iccid_19 → (target_package_id, expire_date)
for c in cards:
meta = card_metas.get(c.iccid_19)
if meta is None or not meta.current_meal_id:
continue
pkg_map = mapping.lookup_package(meta.current_meal_id)
if pkg_map is None:
warnings.append(("package_not_in_mapping", c.iccid_19,
f"奇成 meal_id={meta.current_meal_id} ({meta.current_meal_name}) 未在 mapping.yaml.packages"))
continue
try:
pkg_map.require_target()
except MissingTargetError as exc:
warnings.append(("package_target_missing", c.iccid_19, str(exc)))
continue
resolved_packages[c.iccid_19] = (pkg_map.target_package_id, meta.current_expire_date or "")
# 用量预检:虚用量 > 0 但没找到当前生效套餐的卡,无法精确反算真用量
# (按 flow_add_discount=0 处理,即真=虚,可能高估真用量;不阻断)
for c in cards:
snap = usage_snapshots.get(c.iccid_19)
if snap is None or snap.virtual_used_mb <= 0:
continue
if not snap.has_active_package:
warnings.append((
"usage_without_active_package", c.iccid_19,
f"奇成虚用量 {snap.virtual_used_mb}MB,但 tbl_card_life 中没有当前生效套餐"
"(status=1),无法精确反算真用量,按 flow_add_discount=0 处理(真=虚)",
))
_write_migration_orders(output_dir / "step2_01_migration_orders.sql", cards, resolved_packages, user_id)
_write_package_usages(
output_dir / "step2_02_package_usages.sql", cards, resolved_packages, usage_snapshots, user_id,
)
_write_card_data_usage(output_dir / "step2_03_card_data_usage_update.sql", cards, usage_snapshots)
# 代理钱包:只为 mapping.agents 中 target_shop_code 已填的项写入
eligible_agents = [a for a in mapping.agents.values() if (a.target_shop_code or "").strip()]
_write_agent_wallet_init(
output_dir / "step2_04_agent_wallet_init.sql",
eligible_agents, agent_balances, commission_snapshots, warnings,
)
_write_agent_wallet_transactions(
output_dir / "step2_05_agent_wallet_transactions.sql",
eligible_agents, agent_balances, commission_snapshots, user_id,
)
_write_warnings(output_dir / "warnings.csv", warnings)
_write_summary_step2(
output_dir / "summary.txt",
cards=cards,
resolved=resolved_packages,
agents=eligible_agents,
warnings=warnings,
)
return [w[2] for w in warnings]
def _make_card_order_no(iccid_full: str) -> str:
"""生成稳定迁移订单号。
新库 order_no 限制为 varchar(30);MIG-CARD- + 20 位 ICCID 总长 29。
固定使用 ICCID 还能保证 step2_01/step2_02 分开重跑时仍能匹配同一订单。
"""
return f"{ORDER_NO_PREFIX_CARD}{iccid_full}"
def _write_migration_orders(
path: Path,
cards: list[CardRow],
resolved_packages: dict[str, tuple[int, str]],
user_id: int,
) -> None:
"""伪订单关键字段(详见方案文档 10.2):source=admin / buyer_type=personal /
buyer_id=0 / payment_method=offline / payment_status=2 / commission_status=2 /
commission_result=2 / amount=0。"""
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step2_01 迁移伪订单 (tb_order)"))
for c in cards:
if c.iccid_19 not in resolved_packages:
continue
order_no = _make_card_order_no(c.iccid_full)
f.write(
"INSERT INTO tb_order (\n"
" order_no, order_type, buyer_type, buyer_id,\n"
" iot_card_id, asset_identifier,\n"
" total_amount, actual_paid_amount,\n"
" payment_method, payment_status, paid_at,\n"
" commission_status, commission_result, commission_config_version,\n"
" source, generation,\n"
" operator_account_type, operator_account_name,\n"
" creator, updater, created_at, updated_at\n"
")\n"
f"SELECT {_sql_str(order_no)}, 'single_card', 'personal', 0,\n"
f" c.id, {_sql_str(c.iccid_full)},\n"
" 0, 0,\n"
" 'offline', 2, NOW(),\n"
" 2, 2, 0,\n"
" 'admin', 1,\n"
" 'platform', '奇成数据迁移',\n"
f" {user_id}, {user_id}, NOW(), NOW()\n"
"FROM tb_iot_card c\n"
f"WHERE c.iccid = {_sql_str(c.iccid_full)} AND c.deleted_at IS NULL\n"
"ON CONFLICT (order_no) WHERE deleted_at IS NULL DO NOTHING;\n\n"
)
def _write_package_usages(
path: Path,
cards: list[CardRow],
resolved_packages: dict[str, tuple[int, str]],
usage_snapshots: dict[str, LegacyCardUsage],
user_id: int,
) -> None:
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step2_02 套餐使用记录 (tb_package_usage)"))
for c in cards:
if c.iccid_19 not in resolved_packages:
continue
target_pkg_id, expire = resolved_packages[c.iccid_19]
order_no = _make_card_order_no(c.iccid_full)
expires_expr = f"{_sql_str(expire)}::timestamp" if expire else "NULL"
snap = usage_snapshots.get(c.iccid_19)
used_mb = snap.real_used_mb_floor if snap is not None else 0
effective_limit_expr = (
"CASE WHEN p.enable_virtual_data AND p.virtual_data_mb > 0 "
"THEN p.virtual_data_mb ELSE p.real_data_mb END"
)
usage_expr = f"LEAST({used_mb}, {effective_limit_expr})"
f.write(
"INSERT INTO tb_package_usage (\n"
" order_id, order_no, package_id, usage_type,\n"
" iot_card_id,\n"
" data_limit_mb, data_usage_mb,\n"
" virtual_total_mb_snapshot, display_gain_ratio_snapshot, enable_virtual_data_snapshot,\n"
" status, priority, generation,\n"
" activated_at, expires_at,\n"
" package_name,\n"
" package_price_config_status, package_is_gift,\n"
" data_reset_cycle,\n"
" creator, updater, created_at, updated_at\n"
")\n"
"SELECT o.id, o.order_no, p.id, 'single_card',\n"
" o.iot_card_id,\n"
f" p.real_data_mb, {usage_expr},\n"
f" {effective_limit_expr},\n"
" CASE WHEN p.enable_virtual_data AND p.virtual_data_mb > 0 THEN p.virtual_ratio ELSE 1.0 END,\n"
" CASE WHEN p.enable_virtual_data AND p.virtual_data_mb > 0 THEN TRUE ELSE FALSE END,\n"
f" CASE WHEN {effective_limit_expr} > 0 AND {usage_expr} >= {effective_limit_expr} THEN 2 ELSE 1 END, 1, 1,\n"
f" NULL, {expires_expr},\n"
" p.package_name,\n"
" p.price_config_status, p.is_gift,\n"
" p.data_reset_cycle,\n"
f" {user_id}, {user_id}, NOW(), NOW()\n"
"FROM tb_order o\n"
f"JOIN tb_package p ON p.id = {target_pkg_id} AND p.deleted_at IS NULL\n"
f"WHERE o.order_no = {_sql_str(order_no)} AND o.deleted_at IS NULL\n"
"ON CONFLICT (order_id, package_id) WHERE deleted_at IS NULL DO NOTHING;\n\n"
)
def _write_card_data_usage(path: Path, cards: list[CardRow], usage_snapshots: dict[str, LegacyCardUsage]) -> None:
"""写卡累计流量基线 UPDATE。
三个字段都按"真用量"语义落库:
- ``data_usage_mb``(BIGINT)= ``real_used_mb_floor``,sub-MB 上取整为 1
- ``current_month_usage_mb``(numeric)= ``real_used_mb`` 原始 Decimal,保留小数
- ``last_gateway_reading_mb``(double)= 同上,用作下次轮询增量计算的基线
"""
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step2_03 卡累计流量更新 (tb_iot_card 流量基线)"))
for c in cards:
snap = usage_snapshots.get(c.iccid_19)
if snap is None or snap.real_used_mb <= 0:
continue
real_decimal = _sql_decimal(snap.real_used_mb)
real_int = snap.real_used_mb_floor
f.write(
"UPDATE tb_iot_card\n"
"SET\n"
f" data_usage_mb = {real_int},\n"
f" current_month_usage_mb = {real_decimal},\n"
f" last_gateway_reading_mb = {real_decimal},\n"
" current_month_start_date = COALESCE(current_month_start_date, DATE_TRUNC('month', NOW())::date),\n"
" last_data_check_at = NOW(),\n"
" last_sync_time = NOW(),\n"
" updated_at = NOW()\n"
f"WHERE iccid = {_sql_str(c.iccid_full)} AND deleted_at IS NULL;\n\n"
)
def _write_agent_wallet_init(
path: Path,
eligible_agents,
agent_balances: dict[str, LegacyAgentBalance],
commission_snapshots: dict[str, LegacyCommissionAccount],
warnings: list[tuple[str, str, str]],
) -> None:
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step2_04 代理钱包初始化 (tb_agent_wallet, main + commission)"))
for a in eligible_agents:
balance = agent_balances.get(a.legacy_agent_id)
commission = commission_snapshots.get(a.legacy_agent_id)
if balance is None and commission is None:
warnings.append((
"agent_no_legacy", a.legacy_agent_id,
f"奇成代理 {a.legacy_agent_id} ({a.legacy_agent_name}) 没有 tbl_agent / 佣金账户记录",
))
continue
main_balance = balance.balance_fen if balance else 0
comm_balance = commission.can_draw_amount_fen if commission else 0
comm_frozen = commission.applying_draw_amount_fen if commission else 0
shop_sub = _shop_id_sub(a.target_shop_code)
f.write(
"INSERT INTO tb_agent_wallet (\n"
" shop_id, wallet_type, balance, frozen_balance,\n"
" currency, status, version, shop_id_tag,\n"
" created_at, updated_at\n"
") VALUES (\n"
f" {shop_sub}, 'main', {main_balance}, 0,\n"
f" 'CNY', 1, 0, {shop_sub},\n"
" NOW(), NOW()\n"
")\n"
"ON CONFLICT (shop_id, wallet_type) WHERE deleted_at IS NULL DO NOTHING;\n\n"
)
f.write(
"INSERT INTO tb_agent_wallet (\n"
" shop_id, wallet_type, balance, frozen_balance,\n"
" currency, status, version, shop_id_tag,\n"
" created_at, updated_at\n"
") VALUES (\n"
f" {shop_sub}, 'commission', {comm_balance}, {comm_frozen},\n"
f" 'CNY', 1, 0, {shop_sub},\n"
" NOW(), NOW()\n"
")\n"
"ON CONFLICT (shop_id, wallet_type) WHERE deleted_at IS NULL DO NOTHING;\n\n"
)
def _write_agent_wallet_transactions(
path: Path,
eligible_agents,
agent_balances: dict[str, LegacyAgentBalance],
commission_snapshots: dict[str, LegacyCommissionAccount],
user_id: int,
) -> None:
with path.open("w", encoding="utf-8") as f:
f.write(_file_header("step2_05 代理钱包初始化流水 (tb_agent_wallet_transaction)"))
for a in eligible_agents:
balance = agent_balances.get(a.legacy_agent_id)
commission = commission_snapshots.get(a.legacy_agent_id)
if balance is None and commission is None:
continue
main_amount = balance.balance_fen if balance else 0
comm_amount = commission.can_draw_amount_fen if commission else 0
comm_frozen = commission.applying_draw_amount_fen if commission else 0
shop_sub = _shop_id_sub(a.target_shop_code)
for wallet_type, amount in (("main", main_amount), ("commission", comm_amount)):
if amount <= 0:
continue
meta = {
"source": METADATA_SOURCE,
"legacy_agent_id": a.legacy_agent_id,
"legacy_agent_name": a.legacy_agent_name,
"legacy_field": "balance_money" if wallet_type == "main" else "can_draw_amount",
}
if wallet_type == "commission":
meta["legacy_applying_draw_amount_fen"] = comm_frozen
if commission:
meta["legacy_total_commission_fen"] = commission.total_commission_amount_fen
meta["legacy_total_draw_fen"] = commission.total_draw_amount_fen
remark = (
f"奇成数据迁移初始化({'主钱包' if wallet_type == 'main' else '分佣钱包'})"
f"源 agent_id={a.legacy_agent_id}"
)
f.write(
"INSERT INTO tb_agent_wallet_transaction (\n"
" agent_wallet_id, shop_id, user_id,\n"
" transaction_type, amount, balance_before, balance_after,\n"
" status, reference_type, reference_id,\n"
" remark, metadata,\n"
" creator, shop_id_tag, created_at, updated_at\n"
")\n"
f"SELECT w.id, w.shop_id, {user_id},\n"
f" 'recharge', {amount}, 0, {amount},\n"
" 1, 'topup', NULL,\n"
f" {_sql_str(remark)}, {_sql_jsonb(meta)},\n"
f" {user_id}, w.shop_id, NOW(), NOW()\n"
"FROM tb_agent_wallet w\n"
f"WHERE w.shop_id = {shop_sub}\n"
f" AND w.wallet_type = {_sql_str(wallet_type)}\n"
" AND w.deleted_at IS NULL\n"
" AND NOT EXISTS (\n"
" SELECT 1 FROM tb_agent_wallet_transaction t\n"
" WHERE t.agent_wallet_id = w.id\n"
f" AND t.metadata->>'source' = {_sql_str(METADATA_SOURCE)}\n"
f" AND t.metadata->>'legacy_agent_id' = {_sql_str(a.legacy_agent_id)}\n"
f" AND t.metadata->>'legacy_field' = {_sql_str(meta['legacy_field'])}\n"
" AND t.deleted_at IS NULL\n"
" );\n\n"
)
def _write_warnings(path: Path, warnings: list[tuple[str, str, str]]) -> None:
with path.open("w", encoding="utf-8", newline="") as f:
writer = csv.writer(f)
writer.writerow(["kind", "key", "message"])
for kind, key, message in warnings:
writer.writerow([kind, key, message])
def _write_summary_step2(path: Path, *, cards, resolved, agents, warnings) -> None:
lines = [
"奇成迁移 - 脚本2 关联数据导入摘要",
"=" * 48,
f"生成时间 : {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
f"卡总数 : {len(cards)}",
f" 生成伪订单+套餐 : {len(resolved)}",
f"代理钱包初始化 : {len(agents)}",
f"警告条目 : {len(warnings)} (详见 warnings.csv)",
]
path.write_text("\n".join(lines) + "\n", encoding="utf-8")

View File

@@ -0,0 +1,67 @@
#!/usr/bin/env python3
"""奇成数据迁移脚本1:资产导入。
工作流:
1. 读 mapping.yaml(必须先跑过 scan_legacy.py)
2. 读 cards.csv / devices.csv
3. 连奇成查每张卡的元数据(category/agent_id/msisdn/virtual_no)
4. 应用 mapping 把 legacy_* 翻译成 target_*,自动决定归属
5. 生成 output/step1_*.sql + errors.csv + summary.txt
约束:
- 对奇成只读;所有 INSERT 用 ON CONFLICT 保证幂等
- mapping 中任意 target_* 缺失会让对应卡进 errors.csv 跳过
用法:
python3 migrate_assets.py [--config-dir config] [--resources-dir resources] [--output-dir output]
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from lib import csv_loader, legacy_query, mapping_loader, sql_builder # noqa: E402
def main() -> int:
parser = argparse.ArgumentParser(description="奇成数据迁移脚本1:资产导入")
parser.add_argument("--config-dir", default="config", help="yaml 配置目录")
parser.add_argument("--resources-dir", default="resources", help="CSV 资源目录")
parser.add_argument("--output-dir", default="output", help="SQL/CSV 输出目录")
args = parser.parse_args()
base = Path(__file__).resolve().parent
config_dir = (base / args.config_dir).resolve()
resources_dir = (base / args.resources_dir).resolve()
output_dir = (base / args.output_dir).resolve()
output_dir.mkdir(parents=True, exist_ok=True)
mapping = mapping_loader.load_mapping(config_dir)
cards, devices, errors = csv_loader.load_assets(resources_dir)
# 连奇成查卡元数据(运营商分类、agent_id、msisdn、virtual_no、当前生效套餐)
iccid_pairs = [(c.iccid_19, c.iccid_20) for c in cards]
dsn = mapping_loader.load_legacy_dsn(config_dir)
with legacy_query.connect_readonly(dsn) as conn:
card_metas, dup_iccid_19 = legacy_query.fetch_card_meta(conn, iccid_pairs)
sql_builder.write_step1(
output_dir,
cards=cards,
devices=devices,
mapping=mapping,
card_metas=card_metas,
duplicate_iccid_19s=dup_iccid_19,
errors=errors,
)
print(f"完成:卡 {len(cards)} 张,设备 {len(devices)} 台,异常 {len(errors)}")
print(f"SQL 输出: {output_dir}")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,68 @@
#!/usr/bin/env python3
"""奇成数据迁移脚本2:关联数据导入。
前提:脚本1的 SQL 已在新系统执行(线上库已包含本次迁移所有 ICCID)。
工作流:
1. 读 mapping.yaml + cards.csv
2. 连奇成查:卡元数据(card_meta) + 累计流量(card_usage) + 代理佣金/主钱包余额
3. 卡 → 当前生效套餐 → mapping.packages → target_package_id
4. 生成 step2_*.sql
用法:
python3 migrate_runtime.py [--config-dir config] [--resources-dir resources] [--output-dir output]
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from lib import csv_loader, legacy_query, mapping_loader, sql_builder # noqa: E402
def main() -> int:
parser = argparse.ArgumentParser(description="奇成数据迁移脚本2:关联数据导入")
parser.add_argument("--config-dir", default="config", help="yaml 配置目录")
parser.add_argument("--resources-dir", default="resources", help="CSV 资源目录")
parser.add_argument("--output-dir", default="output", help="SQL/CSV 输出目录")
args = parser.parse_args()
base = Path(__file__).resolve().parent
config_dir = (base / args.config_dir).resolve()
resources_dir = (base / args.resources_dir).resolve()
output_dir = (base / args.output_dir).resolve()
output_dir.mkdir(parents=True, exist_ok=True)
mapping = mapping_loader.load_mapping(config_dir)
cards, _devices, _errors = csv_loader.load_assets(resources_dir)
iccid_pairs = [(c.iccid_19, c.iccid_20) for c in cards]
agent_ids = [a.legacy_agent_id for a in mapping.agents.values() if (a.target_shop_code or "").strip()]
dsn = mapping_loader.load_legacy_dsn(config_dir)
with legacy_query.connect_readonly(dsn) as conn:
card_metas, _dup_iccid_19 = legacy_query.fetch_card_meta(conn, iccid_pairs)
usage_snapshots = legacy_query.fetch_card_usage(conn, iccid_pairs)
commission_snapshots = legacy_query.fetch_commission_accounts(conn, agent_ids) if agent_ids else {}
agent_balances = legacy_query.fetch_agent_balances(conn, agent_ids) if agent_ids else {}
warnings = sql_builder.write_step2(
output_dir,
cards=cards,
mapping=mapping,
card_metas=card_metas,
usage_snapshots=usage_snapshots,
commission_snapshots=commission_snapshots,
agent_balances=agent_balances,
)
print(f"完成:卡 {len(cards)} 张,代理 {len(agent_ids)} 个,警告 {len(warnings)}")
print(f"SQL 输出: {output_dir}")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

View File

@@ -0,0 +1,4 @@
# 奇成数据迁移脚本依赖
# Python >= 3.10 推荐
PyMySQL>=1.1.0 # 连接奇成 MySQL(只读)
PyYAML>=6.0 # 加载映射配置

View File

@@ -0,0 +1,49 @@
# 业务方资料投放目录
业务方按以下规则把数据放到本目录,**重命名去掉 `.example` 后缀**,即可被脚本读取。
> 设计原则:**业务方只填"是哪张卡"和"在哪台设备上",归属、运营商、套餐等元数据全部由
> 脚本从奇成自动获取,通过 `config/mapping.yaml` 决定如何落到新系统。**
## cards.csv 卡资产清单
每张要迁移的卡一行。**必须同时提供 iccid_19 和 iccid_20**(因为奇成历史数据
长度不可信,业务方提供两列作为权威值)。
| 列名 | 必填 | 说明 |
|------|------|------|
| `iccid_19` | 是 | ICCID 前 19 位(所有卡都有,作为跨文件稳定 ID;必须 19 位纯数字) |
| `iccid_20` | 视运营商 | 完整 20 位 ICCID;CMCC/CUCC/CBN(20 位运营商) **必填**,CTCC(19 位)**必须留空**;非空时前 19 位必须等于 `iccid_19` |
| `is_industry` | 否 | 是否行业卡,留空 = `false`(普通卡) |
运营商类型从奇成 `tbl_card.category` 自动推断,在查完奇成后会做一致性校验:
- CTCC 卡填了 `iccid_20``errors.csv: carrier_length_conflict` 阻断
- 非 CTCC 卡 `iccid_20` 留空 → 同上阻断
- `iccid_19` 在奇成 `tbl_card` 命中多条记录 → `errors.csv: legacy_duplicate` 阻断,请业务方人工确认
**店铺归属不在 CSV 里填**,由 `mapping.yaml.agents[].target_shop_code` 通过奇成
`tbl_card.agent_id` 自动推导。如果该卡在奇成没有 agent_id 或代理映射未填 shop_code,
卡进平台库存。
## devices.csv 设备资产清单
每台要迁移的设备一行。
| 列名 | 必填 | 说明 |
|------|------|------|
| `imei` | 否 | 设备 IMEI(非蜂窝设备可留空) |
| `virtual_no` | 否 | 设备虚拟号/设备号。留空时用 `imei` 兜底;两者都为空则该行报错 |
| `sim_iccid_1` ~ `sim_iccid_4` | 否 | 各插槽 ICCID,引用 `cards.csv` 的卡。可填 19 或 20 位,内部统一按前 19 位回查 `cards.iccid_19`(至少要有 1 个) |
**店铺归属规则**:用 `sim_iccid_1`(主卡)的 agent_id 走 `mapping.agents`
推导,与卡归属同口径。
**其他字段自动取默认值**:`device_name` / `device_model` / `manufacturer` 留空,
`max_sim_slots=4`,`current_slot=1`(以 `sim_iccid_1` 为当前生效卡)。
## 一致性约束
- `devices.csv` 任何 `sim_iccid_*` 引用的 ICCID **必须**在 `cards.csv` 中存在
- 同一 ICCID 不允许在 `cards.csv` 中重复出现
- `imei``virtual_no` 至少要有一个(用作设备唯一标识)
- 同一 ICCID 不能同时被多个设备/插槽引用

View File

@@ -0,0 +1,4 @@
iccid_19,iccid_20,is_industry
8985200026333877243,89852000263338772439,
8986062300000000001,,true
8986006654000000000,89860066540000000002,

View File

@@ -0,0 +1,4 @@
imei,virtual_no,sim_iccid_1,sim_iccid_2,sim_iccid_3,sim_iccid_4
860000000000001,DEV-202604-0001,89852000263338772439,8986062300000000001,,
,DEV-202604-0002,89860066540000000002,,,
860000000000003,,89860000000000004,,,

View File

@@ -0,0 +1,191 @@
#!/usr/bin/env python3
"""扫描奇成,自动生成/增量补全 config/mapping.yaml。
工作流:
1. 读 resources/cards.csv 拿到 iccid 列表
2. 连奇成只读查询每张卡的 category / agent_id / 当前生效套餐 / 套餐系列
3. 加载已有的 config/mapping.yaml(若没有则创建空的)
4. 把扫描到的 legacy_* 增量补到 mapping 里(已有项的 target_* 字段保留,新项的 target_* 为 null)
5. 写回 config/mapping.yaml
6. 输出"待人工填写"清单到 console + config/mapping_todo.txt
用法:
python3 scan_legacy.py [--config-dir config] [--resources-dir resources]
"""
from __future__ import annotations
import argparse
import csv
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from lib import legacy_query, mapping_loader # noqa: E402
from lib.mapping_loader import ( # noqa: E402
AgentMapping,
CarrierMapping,
Mapping,
PackageMapping,
SeriesMapping,
)
def _read_iccid_pairs(cards_csv: Path) -> list[tuple[str, str]]:
"""从 cards.csv 读 iccid_19 + iccid_20 列(只关心这两列,其他列忽略)。
返回 (iccid_19, iccid_20) 列表;iccid_20 可为空字符串(CTCC 卡留空)。
"""
if not cards_csv.exists():
raise FileNotFoundError(f"找不到 {cards_csv},请按 resources/README.md 准备")
pairs: dict[str, str] = {}
with cards_csv.open("r", encoding="utf-8-sig", newline="") as f:
reader = csv.DictReader(f)
headers = reader.fieldnames or []
if "iccid_19" not in headers:
raise ValueError(f"{cards_csv} 必须包含 iccid_19 列")
if "iccid_20" not in headers:
raise ValueError(f"{cards_csv} 必须包含 iccid_20 列(CTCC 卡留空)")
for row in reader:
i19 = (row.get("iccid_19") or "").strip()
i20 = (row.get("iccid_20") or "").strip()
if i19:
pairs[i19] = i20
return sorted(pairs.items())
def main() -> int:
parser = argparse.ArgumentParser(description="奇成元数据预扫描,自动生成 mapping.yaml")
parser.add_argument("--config-dir", default="config", help="yaml 配置目录")
parser.add_argument("--resources-dir", default="resources", help="CSV 资源目录")
args = parser.parse_args()
base = Path(__file__).resolve().parent
config_dir = (base / args.config_dir).resolve()
resources_dir = (base / args.resources_dir).resolve()
iccid_pairs = _read_iccid_pairs(resources_dir / "cards.csv")
if not iccid_pairs:
print("cards.csv 中没有 iccid_19,无需扫描", file=sys.stderr)
return 1
print(f"读到 {len(iccid_pairs)} 个 ICCID,连接奇成扫描元数据...")
dsn = mapping_loader.load_legacy_dsn(config_dir)
with legacy_query.connect_readonly(dsn) as conn:
card_metas, dup_iccid_19 = legacy_query.fetch_card_meta(conn, iccid_pairs)
meal_ids = {m.current_meal_id for m in card_metas.values() if m.current_meal_id}
meal_metas = legacy_query.fetch_meal_meta(conn, meal_ids)
if dup_iccid_19:
print(f"警告:{len(dup_iccid_19)} 张卡的 iccid_19 在奇成 tbl_card 命中多条记录:")
for x in sorted(dup_iccid_19):
print(f" - {x}")
print("这些卡 migrate_assets 阶段会写 errors.csv 阻断,请业务方人工确认。")
existing = mapping_loader.load_mapping(config_dir)
merged = _merge(existing, card_metas, meal_metas)
out_path = mapping_loader.merge_and_save(config_dir, merged)
todo = _collect_todo(merged)
_write_todo(config_dir / "mapping_todo.txt", todo)
_print_summary(out_path, card_metas, meal_metas, todo)
return 0
def _merge(
existing: Mapping,
card_metas: dict[str, "legacy_query.LegacyCardMeta"],
meal_metas: dict[str, "legacy_query.LegacyMealMeta"],
) -> Mapping:
"""把扫描到的 legacy_* 增量合并到现有 mapping(保留已填的 target_*)。"""
merged = Mapping(
migration_user_id=existing.migration_user_id,
migration_batch_no=existing.migration_batch_no,
carriers=dict(existing.carriers),
packages=dict(existing.packages),
series=dict(existing.series),
agents=dict(existing.agents),
)
for meta in card_metas.values():
# carrier
if meta.category and meta.category not in merged.carriers:
merged.carriers[meta.category] = CarrierMapping(
legacy_category=meta.category,
legacy_category_name=meta.category_name,
target_carrier_id=None,
target_carrier_type=None,
target_carrier_name=None,
)
elif meta.category in merged.carriers and not merged.carriers[meta.category].legacy_category_name:
# 补全 legacy_category_name(如果之前是空的)
old = merged.carriers[meta.category]
merged.carriers[meta.category] = CarrierMapping(
legacy_category=old.legacy_category,
legacy_category_name=meta.category_name,
target_carrier_id=old.target_carrier_id,
target_carrier_type=old.target_carrier_type,
target_carrier_name=old.target_carrier_name,
)
# agent
if meta.agent_id and meta.agent_id not in merged.agents:
merged.agents[meta.agent_id] = AgentMapping(
legacy_agent_id=meta.agent_id,
legacy_agent_name=meta.agent_name,
target_shop_code=None,
)
for mm in meal_metas.values():
if mm.meal_id not in merged.packages:
merged.packages[mm.meal_id] = PackageMapping(
legacy_meal_id=mm.meal_id,
legacy_meal_name=mm.meal_name,
target_package_id=None,
)
if mm.series_id and mm.series_id not in merged.series:
merged.series[mm.series_id] = SeriesMapping(
legacy_series_id=mm.series_id,
legacy_series_name=mm.series_name,
target_series_id=None,
)
return merged
def _collect_todo(mapping: Mapping) -> dict[str, list[str]]:
"""收集所有待填写的项,按类别分组。"""
todo: dict[str, list[str]] = {"carriers": [], "packages": [], "agents": []}
for c in sorted(mapping.carriers.values(), key=lambda x: x.legacy_category):
if not c.target_carrier_id or not c.target_carrier_type or not c.target_carrier_name:
todo["carriers"].append(f"category={c.legacy_category} ({c.legacy_category_name})")
for p in sorted(mapping.packages.values(), key=lambda x: x.legacy_meal_id):
if not p.target_package_id:
todo["packages"].append(f"meal_id={p.legacy_meal_id} ({p.legacy_meal_name})")
for a in sorted(mapping.agents.values(), key=lambda x: x.legacy_agent_id):
if not (a.target_shop_code or "").strip():
todo["agents"].append(f"agent_id={a.legacy_agent_id} ({a.legacy_agent_name})")
return todo
def _write_todo(path: Path, todo: dict[str, list[str]]) -> None:
lines = ["奇成迁移映射待填写清单", "=" * 40, ""]
for kind in ("carriers", "packages", "agents"):
lines.append(f"## {kind} (待填: {len(todo[kind])})")
for item in todo[kind]:
lines.append(f" - {item}")
lines.append("")
path.write_text("\n".join(lines), encoding="utf-8")
def _print_summary(out_path: Path, card_metas, meal_metas, todo: dict[str, list[str]]) -> None:
print(f"扫描完成,mapping 写入: {out_path}")
print(f" 奇成卡命中 : {len(card_metas)}")
print(f" 奇成套餐命中 : {len(meal_metas)}")
print(f" 待填 carriers : {len(todo['carriers'])}")
print(f" 待填 packages : {len(todo['packages'])}")
print(f" 待填 agents : {len(todo['agents'])}")
if any(todo.values()):
print(f"详见: {out_path.parent / 'mapping_todo.txt'}")
if __name__ == "__main__":
sys.exit(main())