junhong_cmp_fiber/.codex/skills/export-datasource/SKILL.md

---
name: export-datasource
description: Project-specific guide for implementing, modifying, or reviewing export data sources in junhong_cmp_fiber. Use when Codex needs to add a new export scene, extend filters or dynamic columns, register an export scene, change export task query behavior, or explain/debug the DataSource-based export system.
---

# Export Datasource

Use this skill to work on this project's DataSource-based export system. It covers developer-facing export scene implementation, not one-off manual export operations.

## First Reads

Before editing export code, read the relevant current files:

- `internal/exporter/datasource.go`
- `internal/exporter/query_params.go`
- `internal/exporter/filter_helpers.go`
- `internal/exporter/registry.go`
- Existing scene closest to the new scene:
  - `internal/exporter/device_scene.go`
  - `internal/exporter/iot_card_scene.go`
- For request/scene validation:
  - `internal/model/dto/export_task_dto.go`
  - `pkg/constants/constants.go`
- For behavior across worker stages:
  - `internal/task/export_dispatch.go`
  - `internal/task/export_shard.go`
  - `internal/task/export_finalize.go`

Read `references/export-scene-template.md` when adding a new scene or when a concrete code skeleton is useful.

## Mental Model

The framework owns async execution, sharding, file generation, OSS upload, and download URLs. A scene implementation owns only data semantics:

```go
type DataSource interface {
    Scene() string
    Count(ctx context.Context, params ExportParams) (int, error)
    Headers(ctx context.Context, params ExportParams) ([]string, error)
    Fetch(ctx context.Context, params ExportParams, offset, limit int) ([][]string, error)
}
```

Execution flow:

1. Admin API creates `tb_export_task` and enqueues `export:dispatch`.
2. Dispatch parses `query_json.filters` plus permission snapshot into `ExportParams`.
3. Dispatch calls `Headers` once and stores `query_json.resolved_headers`.
4. Dispatch calls `Count` and creates `tb_export_shard_task` rows with `shard_offset` and `shard_limit`.
5. Shard calls `Fetch`, writes headerless CSV shard files, and uploads them.
6. Finalize downloads shard CSV files in shard order, writes one header row, uploads final CSV or converts CSV to XLSX.

Do not reintroduce keyset cursor logic. New scenes must use offset/limit through `Fetch`.

## Implementation Workflow

1. Add a scene constant in `pkg/constants/constants.go`.
2. Update `internal/model/dto/export_task_dto.go` validation and descriptions for `scene`.
3. Implement `internal/exporter/<scene>_scene.go` with `DataSource`.
4. Register the source in `NewDefaultRegistry`.
5. Update `IsSupportedScene` if it is used by the current code path.
6. Add or update migrations only if the exported domain needs schema/index changes. Do not change export task tables unless the framework contract changes.
7. Build and manually verify. This repository forbids automated tests unless the user explicitly requests them.

## DataSource Rules

- `Scene` must return the constant, not a string literal.
- `Count` and `Fetch` must apply the same filters and permission scope.
- `Headers` defines the exact column contract for the whole task. Dispatch stores it once; shards and finalize reuse it.
- `Fetch` must return rows aligned to `Headers`; the framework pads/truncates as a fallback, but the source should be correct.
- `Fetch` must use stable ordering, normally `ORDER BY id ASC`.
- Return string values only. Format time as `2006-01-02 15:04:05` unless the surrounding scene establishes another convention.
- Use GORM only. Do not use `database/sql`.
- Keep SQL parameters bound through GORM placeholders. Do not concatenate user-controlled values into SQL.
- Keep comments, logs, errors, and documentation in Chinese per project rules.

## Filters And Permissions

Incoming task query shape:

```json
{
  "filters": {
    "status": 1,
    "shop_id": 1
  }
}
```

`ParseExportParams` exposes:

- `Filters`: `query_json.filters`
- `ScopeShopIDs`: shop permission snapshot captured at task creation
- `UserType`: creator user type snapshot

Apply permission scope inside each DataSource:

```go
query = applyExportShopScope(query, params, "shop_id")
```

For aliased queries:

```go
query = applyExportShopScope(query, params, "o.shop_id")
```

Use helpers from `filter_helpers.go`:

- `filterInt`
- `filterUint`
- `filterString`
- `filterBool`
- `filterTime`
- `formatOptionalUint`
- `formatOptionalTime`

Do not read live permissions from context inside a DataSource. Export tasks must use the permission snapshot stored at creation time.

## Dynamic Columns

Use dynamic headers only when the task parameters or data require it. If `Headers` changes based on filters, `Fetch` must produce the same shape for every shard under the same `ExportParams`.

Example pattern:

```go
headers := []string{"ID", "ICCID", "状态"}
if filterBool(params.Filters, "with_package") {
    headers = append(headers, "套餐名称", "套餐状态")
}
return headers, nil
```

## Join Queries

JOIN-based exports are allowed. Keep these constraints:

- Preserve one output row per intended exported entity unless the scene explicitly exports detail rows.
- If a JOIN can multiply rows, make `Count` match the exported row semantics exactly.
- Use table aliases consistently in filters and scope columns.
- Prefer explicit `Select` into a local row struct for multi-table exports.
- Keep `Order`, `Limit`, and `Offset` on the final query used by `Fetch`.

## User-Facing API Notes

Current API group:

- `POST /api/admin/export-tasks`
- `GET /api/admin/export-tasks`
- `GET /api/admin/export-tasks/:id`
- `POST /api/admin/export-tasks/:id/cancel`

Creation request:

```json
{
  "scene": "iot_card",
  "format": "csv",
  "query": {
    "filters": {
      "shop_id": 1,
      "with_package": true
    }
  }
}
```

Formats are `csv` and `xlsx`. Final download URLs are returned from task detail after completion.

## Verification

This project forbids automated tests and `_test.go` files unless the user explicitly asks for tests. Use manual verification:

```bash
go build ./internal/exporter/...
go build ./internal/task/...
go build ./...
```

Then create an export task through the API and inspect:

- `tb_export_task.query_json` contains original `filters` and generated `resolved_headers`.
- `tb_export_task.total_rows` matches the filtered query.
- `tb_export_shard_task.shard_offset` and `shard_limit` are filled.
- Shards reach success and final task reaches completed status.
- Downloaded file has one header row, expected row count, correct filtering, and valid CSV/XLSX format.

Use PostgreSQL MCP/manual SQL for data validation when needed.

## Common Mistakes

- Updating only `Fetch` and forgetting `Count`, causing wrong shard planning.
- Returning dynamic rows whose column count does not match `Headers`.
- Filtering by requested `shop_id` without also applying `ScopeShopIDs`.
- Using context/user middleware inside worker DataSource logic.
- Registering the source but forgetting DTO `oneof`, so API rejects the new scene.
- Writing XLSX shard files. Shards should be CSV; finalize handles final format.
- Adding automated tests despite the repository ban.