init

2026-06-16 15:15:50 +08:00
parent 84ab3bad99
commit 1f634eb465
21 changed files with 2776 additions and 41 deletions
--- a/.codex/skills/export-datasource/SKILL.md
+++ b/.codex/skills/export-datasource/SKILL.md
@@ -0,0 +1,198 @@
+---
+name: export-datasource
+description: Project-specific guide for implementing, modifying, or reviewing export data sources in junhong_cmp_fiber. Use when Codex needs to add a new export scene, extend filters or dynamic columns, register an export scene, change export task query behavior, or explain/debug the DataSource-based export system.
+---
+
+# Export Datasource
+
+Use this skill to work on this project's DataSource-based export system. It covers developer-facing export scene implementation, not one-off manual export operations.
+
+## First Reads
+
+Before editing export code, read the relevant current files:
+
+- `internal/exporter/datasource.go`
+- `internal/exporter/query_params.go`
+- `internal/exporter/filter_helpers.go`
+- `internal/exporter/registry.go`
+- Existing scene closest to the new scene:
+  - `internal/exporter/device_scene.go`
+  - `internal/exporter/iot_card_scene.go`
+- For request/scene validation:
+  - `internal/model/dto/export_task_dto.go`
+  - `pkg/constants/constants.go`
+- For behavior across worker stages:
+  - `internal/task/export_dispatch.go`
+  - `internal/task/export_shard.go`
+  - `internal/task/export_finalize.go`
+
+Read `references/export-scene-template.md` when adding a new scene or when a concrete code skeleton is useful.
+
+## Mental Model
+
+The framework owns async execution, sharding, file generation, OSS upload, and download URLs. A scene implementation owns only data semantics:
+
+```go
+type DataSource interface {
+    Scene() string
+    Count(ctx context.Context, params ExportParams) (int, error)
+    Headers(ctx context.Context, params ExportParams) ([]string, error)
+    Fetch(ctx context.Context, params ExportParams, offset, limit int) ([][]string, error)
+}
+```
+
+Execution flow:
+
+1. Admin API creates `tb_export_task` and enqueues `export:dispatch`.
+2. Dispatch parses `query_json.filters` plus permission snapshot into `ExportParams`.
+3. Dispatch calls `Headers` once and stores `query_json.resolved_headers`.
+4. Dispatch calls `Count` and creates `tb_export_shard_task` rows with `shard_offset` and `shard_limit`.
+5. Shard calls `Fetch`, writes headerless CSV shard files, and uploads them.
+6. Finalize downloads shard CSV files in shard order, writes one header row, uploads final CSV or converts CSV to XLSX.
+
+Do not reintroduce keyset cursor logic. New scenes must use offset/limit through `Fetch`.
+
+## Implementation Workflow
+
+1. Add a scene constant in `pkg/constants/constants.go`.
+2. Update `internal/model/dto/export_task_dto.go` validation and descriptions for `scene`.
+3. Implement `internal/exporter/<scene>_scene.go` with `DataSource`.
+4. Register the source in `NewDefaultRegistry`.
+5. Update `IsSupportedScene` if it is used by the current code path.
+6. Add or update migrations only if the exported domain needs schema/index changes. Do not change export task tables unless the framework contract changes.
+7. Build and manually verify. This repository forbids automated tests unless the user explicitly requests them.
+
+## DataSource Rules
+
+- `Scene` must return the constant, not a string literal.
+- `Count` and `Fetch` must apply the same filters and permission scope.
+- `Headers` defines the exact column contract for the whole task. Dispatch stores it once; shards and finalize reuse it.
+- `Fetch` must return rows aligned to `Headers`; the framework pads/truncates as a fallback, but the source should be correct.
+- `Fetch` must use stable ordering, normally `ORDER BY id ASC`.
+- Return string values only. Format time as `2006-01-02 15:04:05` unless the surrounding scene establishes another convention.
+- Use GORM only. Do not use `database/sql`.
+- Keep SQL parameters bound through GORM placeholders. Do not concatenate user-controlled values into SQL.
+- Keep comments, logs, errors, and documentation in Chinese per project rules.
+
+## Filters And Permissions
+
+Incoming task query shape:
+
+```json
+{
+  "filters": {
+    "status": 1,
+    "shop_id": 1
+  }
+}
+```
+
+`ParseExportParams` exposes:
+
+- `Filters`: `query_json.filters`
+- `ScopeShopIDs`: shop permission snapshot captured at task creation
+- `UserType`: creator user type snapshot
+
+Apply permission scope inside each DataSource:
+
+```go
+query = applyExportShopScope(query, params, "shop_id")
+```
+
+For aliased queries:
+
+```go
+query = applyExportShopScope(query, params, "o.shop_id")
+```
+
+Use helpers from `filter_helpers.go`:
+
+- `filterInt`
+- `filterUint`
+- `filterString`
+- `filterBool`
+- `filterTime`
+- `formatOptionalUint`
+- `formatOptionalTime`
+
+Do not read live permissions from context inside a DataSource. Export tasks must use the permission snapshot stored at creation time.
+
+## Dynamic Columns
+
+Use dynamic headers only when the task parameters or data require it. If `Headers` changes based on filters, `Fetch` must produce the same shape for every shard under the same `ExportParams`.
+
+Example pattern:
+
+```go
+headers := []string{"ID", "ICCID", "状态"}
+if filterBool(params.Filters, "with_package") {
+    headers = append(headers, "套餐名称", "套餐状态")
+}
+return headers, nil
+```
+
+## Join Queries
+
+JOIN-based exports are allowed. Keep these constraints:
+
+- Preserve one output row per intended exported entity unless the scene explicitly exports detail rows.
+- If a JOIN can multiply rows, make `Count` match the exported row semantics exactly.
+- Use table aliases consistently in filters and scope columns.
+- Prefer explicit `Select` into a local row struct for multi-table exports.
+- Keep `Order`, `Limit`, and `Offset` on the final query used by `Fetch`.
+
+## User-Facing API Notes
+
+Current API group:
+
+- `POST /api/admin/export-tasks`
+- `GET /api/admin/export-tasks`
+- `GET /api/admin/export-tasks/:id`
+- `POST /api/admin/export-tasks/:id/cancel`
+
+Creation request:
+
+```json
+{
+  "scene": "iot_card",
+  "format": "csv",
+  "query": {
+    "filters": {
+      "shop_id": 1,
+      "with_package": true
+    }
+  }
+}
+```
+
+Formats are `csv` and `xlsx`. Final download URLs are returned from task detail after completion.
+
+## Verification
+
+This project forbids automated tests and `_test.go` files unless the user explicitly asks for tests. Use manual verification:
+
+```bash
+go build ./internal/exporter/...
+go build ./internal/task/...
+go build ./...
+```
+
+Then create an export task through the API and inspect:
+
+- `tb_export_task.query_json` contains original `filters` and generated `resolved_headers`.
+- `tb_export_task.total_rows` matches the filtered query.
+- `tb_export_shard_task.shard_offset` and `shard_limit` are filled.
+- Shards reach success and final task reaches completed status.
+- Downloaded file has one header row, expected row count, correct filtering, and valid CSV/XLSX format.
+
+Use PostgreSQL MCP/manual SQL for data validation when needed.
+
+## Common Mistakes
+
+- Updating only `Fetch` and forgetting `Count`, causing wrong shard planning.
+- Returning dynamic rows whose column count does not match `Headers`.
+- Filtering by requested `shop_id` without also applying `ScopeShopIDs`.
+- Using context/user middleware inside worker DataSource logic.
+- Registering the source but forgetting DTO `oneof`, so API rejects the new scene.
+- Writing XLSX shard files. Shards should be CSV; finalize handles final format.
+- Adding automated tests despite the repository ban.
--- a/.codex/skills/export-datasource/agents/openai.yaml
+++ b/.codex/skills/export-datasource/agents/openai.yaml
@@ -0,0 +1,4 @@
+interface:
+  display_name: "导出数据源开发"
+  short_description: "指导新增和维护项目导出数据源场景"
+  default_prompt: "Use $export-datasource to add a new export scene for this project."
--- a/.codex/skills/export-datasource/references/export-scene-template.md
+++ b/.codex/skills/export-datasource/references/export-scene-template.md
@@ -0,0 +1,222 @@
+# Export Scene Template
+
+Use this reference when adding a new export scene.
+
+## Minimal Checklist
+
+- Add `ExportTaskSceneXxx` in `pkg/constants/constants.go`.
+- Add the scene to `CreateExportTaskRequest.Scene` and `ListExportTaskRequest.Scene` validation/description.
+- Create `internal/exporter/<scene>_scene.go`.
+- Register `NewXxxDataSource(db)` in `internal/exporter/registry.go`.
+- Update `IsSupportedScene`.
+- Run `gofmt` on changed Go files.
+- Run `go build ./internal/exporter/...`, `go build ./internal/task/...`, and `go build ./...`.
+- Manually create a task and validate database rows plus downloaded file.
+
+## Skeleton
+
+```go
+package exporter
+
+import (
+    "context"
+    "strconv"
+
+    "gorm.io/gorm"
+
+    "github.com/break/junhong_cmp_fiber/internal/model"
+    "github.com/break/junhong_cmp_fiber/pkg/constants"
+)
+
+// XxxDataSource Xxx 导出数据源。
+type XxxDataSource struct {
+    db *gorm.DB
+}
+
+// NewXxxDataSource 创建 Xxx 导出数据源。
+func NewXxxDataSource(db *gorm.DB) *XxxDataSource {
+    return &XxxDataSource{db: db}
+}
+
+// Scene 返回导出场景编码。
+func (s *XxxDataSource) Scene() string {
+    return constants.ExportTaskSceneXxx
+}
+
+// Count 统计 Xxx 导出行数。
+func (s *XxxDataSource) Count(ctx context.Context, params ExportParams) (int, error) {
+    var total int64
+    query := s.applyFilters(s.db.WithContext(ctx).Model(&model.Xxx{}), params)
+    if err := query.Count(&total).Error; err != nil {
+        return 0, err
+    }
+    return int(total), nil
+}
+
+// Headers 返回 Xxx 导出表头。
+func (s *XxxDataSource) Headers(ctx context.Context, params ExportParams) ([]string, error) {
+    return []string{"ID", "名称", "状态", "店铺ID", "创建时间"}, nil
+}
+
+// Fetch 按 offset/limit 查询 Xxx 导出数据。
+func (s *XxxDataSource) Fetch(ctx context.Context, params ExportParams, offset, limit int) ([][]string, error) {
+    if limit <= 0 {
+        return [][]string{}, nil
+    }
+
+    var items []model.Xxx
+    query := s.applyFilters(s.db.WithContext(ctx).Model(&model.Xxx{}), params).
+        Order("id ASC").
+        Limit(limit).
+        Offset(offset)
+    if err := query.Find(&items).Error; err != nil {
+        return nil, err
+    }
+
+    rows := make([][]string, 0, len(items))
+    for _, item := range items {
+        rows = append(rows, []string{
+            strconv.FormatUint(uint64(item.ID), 10),
+            item.Name,
+            strconv.Itoa(item.Status),
+            formatOptionalUint(item.ShopID),
+            item.CreatedAt.Format(exportTimeLayout),
+        })
+    }
+    return rows, nil
+}
+
+func (s *XxxDataSource) applyFilters(query *gorm.DB, params ExportParams) *gorm.DB {
+    query = applyExportShopScope(query, params, "shop_id")
+
+    if status, ok := filterInt(params.Filters, "status"); ok {
+        query = query.Where("status = ?", status)
+    }
+    if shopID, ok := filterUint(params.Filters, "shop_id"); ok {
+        query = query.Where("shop_id = ?", shopID)
+    }
+    if start, ok := filterTime(params.Filters, "created_at_start"); ok {
+        query = query.Where("created_at >= ?", start)
+    }
+    if end, ok := filterTime(params.Filters, "created_at_end"); ok {
+        query = query.Where("created_at <= ?", end)
+    }
+    return query
+}
+```
+
+## JOIN Skeleton
+
+Use this shape for multi-table exports:
+
+```go
+type xxxExportRow struct {
+    ID        uint
+    Name      string
+    Status    int
+    ShopID    *uint
+    ExtraName string
+    CreatedAt time.Time
+}
+
+func (s *XxxDataSource) Fetch(ctx context.Context, params ExportParams, offset, limit int) ([][]string, error) {
+    if limit <= 0 {
+        return [][]string{}, nil
+    }
+
+    var items []xxxExportRow
+    query := s.applyFilters(s.db.WithContext(ctx).Table("tb_xxx AS x"), params, "x.").
+        Select(`
+            x.id,
+            x.name,
+            x.status,
+            x.shop_id,
+            COALESCE(e.name, '') AS extra_name,
+            x.created_at
+        `).
+        Joins("LEFT JOIN tb_extra AS e ON e.xxx_id = x.id AND e.deleted_at IS NULL").
+        Where("x.deleted_at IS NULL").
+        Order("x.id ASC").
+        Limit(limit).
+        Offset(offset)
+    if err := query.Scan(&items).Error; err != nil {
+        return nil, err
+    }
+
+    rows := make([][]string, 0, len(items))
+    for _, item := range items {
+        rows = append(rows, []string{
+            strconv.FormatUint(uint64(item.ID), 10),
+            item.Name,
+            strconv.Itoa(item.Status),
+            formatOptionalUint(item.ShopID),
+            item.ExtraName,
+            item.CreatedAt.Format(exportTimeLayout),
+        })
+    }
+    return rows, nil
+}
+
+func (s *XxxDataSource) applyFilters(query *gorm.DB, params ExportParams, prefix string) *gorm.DB {
+    query = applyExportShopScope(query, params, prefix+"shop_id")
+    if status, ok := filterInt(params.Filters, "status"); ok {
+        query = query.Where(prefix+"status = ?", status)
+    }
+    return query
+}
+```
+
+If the JOIN multiplies rows, update `Count` to count the same exported row set. Do not count only the base table unless `Fetch` also returns one row per base record.
+
+## Registry Patch
+
+```go
+func NewDefaultRegistry(db *gorm.DB) *Registry {
+    return NewRegistry(
+        NewDeviceDataSource(db),
+        NewIotCardDataSource(db),
+        NewXxxDataSource(db),
+    )
+}
+
+func IsSupportedScene(scene string) bool {
+    switch scene {
+    case constants.ExportTaskSceneDevice,
+        constants.ExportTaskSceneIotCard,
+        constants.ExportTaskSceneXxx:
+        return true
+    default:
+        return false
+    }
+}
+```
+
+## Manual API Smoke
+
+```bash
+curl -X POST 'http://localhost:端口/api/admin/export-tasks' \
+  -H 'Authorization: Bearer <token>' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "scene": "xxx",
+    "format": "csv",
+    "query": {
+      "filters": {
+        "status": 1
+      }
+    }
+  }'
+```
+
+Check database:
+
+```sql
+SELECT id, scene, format, status, total_rows, total_shards, query_json
+FROM tb_export_task
+WHERE id = <task_id>;
+
+SELECT shard_no, status, shard_offset, shard_limit, row_count, file_key
+FROM tb_export_shard_task
+WHERE task_id = <task_id>
+ORDER BY shard_no;
+```