junhong_cmp_fiber/.codex/skills/export-datasource/SKILL.md

name, description

name	description
export-datasource	Project-specific guide for implementing, modifying, or reviewing export data sources in junhong_cmp_fiber. Use when Codex needs to add a new export scene, extend filters or dynamic columns, register an export scene, change export task query behavior, or explain/debug the DataSource-based export system.

Export Datasource

Use this skill to work on this project's DataSource-based export system. It covers developer-facing export scene implementation, not one-off manual export operations.

First Reads

Before editing export code, read the relevant current files:

• internal/exporter/datasource.go
• internal/exporter/query_params.go
• internal/exporter/filter_helpers.go
• internal/exporter/registry.go
• Existing scene closest to the new scene:
  • internal/exporter/device_scene.go
  • internal/exporter/iot_card_scene.go
• For request/scene validation:
  • internal/model/dto/export_task_dto.go
  • pkg/constants/constants.go
• For behavior across worker stages:
  • internal/task/export_dispatch.go
  • internal/task/export_shard.go
  • internal/task/export_finalize.go

Read references/export-scene-template.md when adding a new scene or when a concrete code skeleton is useful.

Mental Model

The framework owns async execution, sharding, file generation, OSS upload, and download URLs. A scene implementation owns only data semantics:

type DataSource interface {
    Scene() string
    Count(ctx context.Context, params ExportParams) (int, error)
    Headers(ctx context.Context, params ExportParams) ([]string, error)
    Fetch(ctx context.Context, params ExportParams, offset, limit int) ([][]string, error)
}

Execution flow:

1. Admin API creates tb_export_task and enqueues export:dispatch.
2. Dispatch parses query_json.filters plus permission snapshot into ExportParams.
3. Dispatch calls Headers once and stores query_json.resolved_headers.
4. Dispatch calls Count and creates tb_export_shard_task rows with shard_offset and shard_limit.
5. Shard calls Fetch, writes headerless CSV shard files, and uploads them.
6. Finalize downloads shard CSV files in shard order, writes one header row, uploads final CSV or converts CSV to XLSX.

Do not reintroduce keyset cursor logic. New scenes must use offset/limit through Fetch.

Implementation Workflow

1. Add a scene constant in pkg/constants/constants.go.
2. Update internal/model/dto/export_task_dto.go validation and descriptions for scene.
3. Implement internal/exporter/<scene>_scene.go with DataSource.
4. Register the source in NewDefaultRegistry.
5. Update IsSupportedScene if it is used by the current code path.
6. Add or update migrations only if the exported domain needs schema/index changes. Do not change export task tables unless the framework contract changes.
7. Build and manually verify. This repository forbids automated tests unless the user explicitly requests them.

DataSource Rules

• Scene must return the constant, not a string literal.
• Count and Fetch must apply the same filters and permission scope.
• Headers defines the exact column contract for the whole task. Dispatch stores it once; shards and finalize reuse it.
• Fetch must return rows aligned to Headers; the framework pads/truncates as a fallback, but the source should be correct.
• Fetch must use stable ordering, normally ORDER BY id ASC.
• Return string values only. Format time as 2006-01-02 15:04:05 unless the surrounding scene establishes another convention.
• Use GORM only. Do not use database/sql.
• Keep SQL parameters bound through GORM placeholders. Do not concatenate user-controlled values into SQL.
• Keep comments, logs, errors, and documentation in Chinese per project rules.

Filters And Permissions

Incoming task query shape:

{
  "filters": {
    "status": 1,
    "shop_id": 1
  }
}

ParseExportParams exposes:

• Filters: query_json.filters
• ScopeShopIDs: shop permission snapshot captured at task creation
• UserType: creator user type snapshot

Apply permission scope inside each DataSource:

query = applyExportShopScope(query, params, "shop_id")

For aliased queries:

query = applyExportShopScope(query, params, "o.shop_id")

Use helpers from filter_helpers.go:

• filterInt
• filterUint
• filterString
• filterBool
• filterTime
• formatOptionalUint
• formatOptionalTime

Do not read live permissions from context inside a DataSource. Export tasks must use the permission snapshot stored at creation time.

Dynamic Columns

Use dynamic headers only when the task parameters or data require it. If Headers changes based on filters, Fetch must produce the same shape for every shard under the same ExportParams.

Example pattern:

headers := []string{"ID", "ICCID", "状态"}
if filterBool(params.Filters, "with_package") {
    headers = append(headers, "套餐名称", "套餐状态")
}
return headers, nil

Join Queries

JOIN-based exports are allowed. Keep these constraints:

• Preserve one output row per intended exported entity unless the scene explicitly exports detail rows.
• If a JOIN can multiply rows, make Count match the exported row semantics exactly.
• Use table aliases consistently in filters and scope columns.
• Prefer explicit Select into a local row struct for multi-table exports.
• Keep Order, Limit, and Offset on the final query used by Fetch.

User-Facing API Notes

Current API group:

• POST /api/admin/export-tasks
• GET /api/admin/export-tasks
• GET /api/admin/export-tasks/:id
• POST /api/admin/export-tasks/:id/cancel

Creation request:

{
  "scene": "iot_card",
  "format": "csv",
  "query": {
    "filters": {
      "shop_id": 1,
      "with_package": true
    }
  }
}

Formats are csv and xlsx. Final download URLs are returned from task detail after completion.

Verification

This project forbids automated tests and _test.go files unless the user explicitly asks for tests. Use manual verification:

go build ./internal/exporter/...
go build ./internal/task/...
go build ./...

Then create an export task through the API and inspect:

• tb_export_task.query_json contains original filters and generated resolved_headers.
• tb_export_task.total_rows matches the filtered query.
• tb_export_shard_task.shard_offset and shard_limit are filled.
• Shards reach success and final task reaches completed status.
• Downloaded file has one header row, expected row count, correct filtering, and valid CSV/XLSX format.

Use PostgreSQL MCP/manual SQL for data validation when needed.

Common Mistakes

• Updating only Fetch and forgetting Count, causing wrong shard planning.
• Returning dynamic rows whose column count does not match Headers.
• Filtering by requested shop_id without also applying ScopeShopIDs.
• Using context/user middleware inside worker DataSource logic.
• Registering the source but forgetting DTO oneof, so API rejects the new scene.
• Writing XLSX shard files. Shards should be CSV; finalize handles final format.
• Adding automated tests despite the repository ban.