This commit is contained in:
Break
198
.codex/skills/export-datasource/SKILL.md
Normal file
198
.codex/skills/export-datasource/SKILL.md
Normal file
@@ -0,0 +1,198 @@
|
||||
---
|
||||
name: export-datasource
|
||||
description: Project-specific guide for implementing, modifying, or reviewing export data sources in junhong_cmp_fiber. Use when Codex needs to add a new export scene, extend filters or dynamic columns, register an export scene, change export task query behavior, or explain/debug the DataSource-based export system.
|
||||
---
|
||||
|
||||
# Export Datasource
|
||||
|
||||
Use this skill to work on this project's DataSource-based export system. It covers developer-facing export scene implementation, not one-off manual export operations.
|
||||
|
||||
## First Reads
|
||||
|
||||
Before editing export code, read the relevant current files:
|
||||
|
||||
- `internal/exporter/datasource.go`
|
||||
- `internal/exporter/query_params.go`
|
||||
- `internal/exporter/filter_helpers.go`
|
||||
- `internal/exporter/registry.go`
|
||||
- Existing scene closest to the new scene:
|
||||
- `internal/exporter/device_scene.go`
|
||||
- `internal/exporter/iot_card_scene.go`
|
||||
- For request/scene validation:
|
||||
- `internal/model/dto/export_task_dto.go`
|
||||
- `pkg/constants/constants.go`
|
||||
- For behavior across worker stages:
|
||||
- `internal/task/export_dispatch.go`
|
||||
- `internal/task/export_shard.go`
|
||||
- `internal/task/export_finalize.go`
|
||||
|
||||
Read `references/export-scene-template.md` when adding a new scene or when a concrete code skeleton is useful.
|
||||
|
||||
## Mental Model
|
||||
|
||||
The framework owns async execution, sharding, file generation, OSS upload, and download URLs. A scene implementation owns only data semantics:
|
||||
|
||||
```go
|
||||
type DataSource interface {
|
||||
Scene() string
|
||||
Count(ctx context.Context, params ExportParams) (int, error)
|
||||
Headers(ctx context.Context, params ExportParams) ([]string, error)
|
||||
Fetch(ctx context.Context, params ExportParams, offset, limit int) ([][]string, error)
|
||||
}
|
||||
```
|
||||
|
||||
Execution flow:
|
||||
|
||||
1. Admin API creates `tb_export_task` and enqueues `export:dispatch`.
|
||||
2. Dispatch parses `query_json.filters` plus permission snapshot into `ExportParams`.
|
||||
3. Dispatch calls `Headers` once and stores `query_json.resolved_headers`.
|
||||
4. Dispatch calls `Count` and creates `tb_export_shard_task` rows with `shard_offset` and `shard_limit`.
|
||||
5. Shard calls `Fetch`, writes headerless CSV shard files, and uploads them.
|
||||
6. Finalize downloads shard CSV files in shard order, writes one header row, uploads final CSV or converts CSV to XLSX.
|
||||
|
||||
Do not reintroduce keyset cursor logic. New scenes must use offset/limit through `Fetch`.
|
||||
|
||||
## Implementation Workflow
|
||||
|
||||
1. Add a scene constant in `pkg/constants/constants.go`.
|
||||
2. Update `internal/model/dto/export_task_dto.go` validation and descriptions for `scene`.
|
||||
3. Implement `internal/exporter/<scene>_scene.go` with `DataSource`.
|
||||
4. Register the source in `NewDefaultRegistry`.
|
||||
5. Update `IsSupportedScene` if it is used by the current code path.
|
||||
6. Add or update migrations only if the exported domain needs schema/index changes. Do not change export task tables unless the framework contract changes.
|
||||
7. Build and manually verify. This repository forbids automated tests unless the user explicitly requests them.
|
||||
|
||||
## DataSource Rules
|
||||
|
||||
- `Scene` must return the constant, not a string literal.
|
||||
- `Count` and `Fetch` must apply the same filters and permission scope.
|
||||
- `Headers` defines the exact column contract for the whole task. Dispatch stores it once; shards and finalize reuse it.
|
||||
- `Fetch` must return rows aligned to `Headers`; the framework pads/truncates as a fallback, but the source should be correct.
|
||||
- `Fetch` must use stable ordering, normally `ORDER BY id ASC`.
|
||||
- Return string values only. Format time as `2006-01-02 15:04:05` unless the surrounding scene establishes another convention.
|
||||
- Use GORM only. Do not use `database/sql`.
|
||||
- Keep SQL parameters bound through GORM placeholders. Do not concatenate user-controlled values into SQL.
|
||||
- Keep comments, logs, errors, and documentation in Chinese per project rules.
|
||||
|
||||
## Filters And Permissions
|
||||
|
||||
Incoming task query shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"filters": {
|
||||
"status": 1,
|
||||
"shop_id": 1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`ParseExportParams` exposes:
|
||||
|
||||
- `Filters`: `query_json.filters`
|
||||
- `ScopeShopIDs`: shop permission snapshot captured at task creation
|
||||
- `UserType`: creator user type snapshot
|
||||
|
||||
Apply permission scope inside each DataSource:
|
||||
|
||||
```go
|
||||
query = applyExportShopScope(query, params, "shop_id")
|
||||
```
|
||||
|
||||
For aliased queries:
|
||||
|
||||
```go
|
||||
query = applyExportShopScope(query, params, "o.shop_id")
|
||||
```
|
||||
|
||||
Use helpers from `filter_helpers.go`:
|
||||
|
||||
- `filterInt`
|
||||
- `filterUint`
|
||||
- `filterString`
|
||||
- `filterBool`
|
||||
- `filterTime`
|
||||
- `formatOptionalUint`
|
||||
- `formatOptionalTime`
|
||||
|
||||
Do not read live permissions from context inside a DataSource. Export tasks must use the permission snapshot stored at creation time.
|
||||
|
||||
## Dynamic Columns
|
||||
|
||||
Use dynamic headers only when the task parameters or data require it. If `Headers` changes based on filters, `Fetch` must produce the same shape for every shard under the same `ExportParams`.
|
||||
|
||||
Example pattern:
|
||||
|
||||
```go
|
||||
headers := []string{"ID", "ICCID", "状态"}
|
||||
if filterBool(params.Filters, "with_package") {
|
||||
headers = append(headers, "套餐名称", "套餐状态")
|
||||
}
|
||||
return headers, nil
|
||||
```
|
||||
|
||||
## Join Queries
|
||||
|
||||
JOIN-based exports are allowed. Keep these constraints:
|
||||
|
||||
- Preserve one output row per intended exported entity unless the scene explicitly exports detail rows.
|
||||
- If a JOIN can multiply rows, make `Count` match the exported row semantics exactly.
|
||||
- Use table aliases consistently in filters and scope columns.
|
||||
- Prefer explicit `Select` into a local row struct for multi-table exports.
|
||||
- Keep `Order`, `Limit`, and `Offset` on the final query used by `Fetch`.
|
||||
|
||||
## User-Facing API Notes
|
||||
|
||||
Current API group:
|
||||
|
||||
- `POST /api/admin/export-tasks`
|
||||
- `GET /api/admin/export-tasks`
|
||||
- `GET /api/admin/export-tasks/:id`
|
||||
- `POST /api/admin/export-tasks/:id/cancel`
|
||||
|
||||
Creation request:
|
||||
|
||||
```json
|
||||
{
|
||||
"scene": "iot_card",
|
||||
"format": "csv",
|
||||
"query": {
|
||||
"filters": {
|
||||
"shop_id": 1,
|
||||
"with_package": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Formats are `csv` and `xlsx`. Final download URLs are returned from task detail after completion.
|
||||
|
||||
## Verification
|
||||
|
||||
This project forbids automated tests and `_test.go` files unless the user explicitly asks for tests. Use manual verification:
|
||||
|
||||
```bash
|
||||
go build ./internal/exporter/...
|
||||
go build ./internal/task/...
|
||||
go build ./...
|
||||
```
|
||||
|
||||
Then create an export task through the API and inspect:
|
||||
|
||||
- `tb_export_task.query_json` contains original `filters` and generated `resolved_headers`.
|
||||
- `tb_export_task.total_rows` matches the filtered query.
|
||||
- `tb_export_shard_task.shard_offset` and `shard_limit` are filled.
|
||||
- Shards reach success and final task reaches completed status.
|
||||
- Downloaded file has one header row, expected row count, correct filtering, and valid CSV/XLSX format.
|
||||
|
||||
Use PostgreSQL MCP/manual SQL for data validation when needed.
|
||||
|
||||
## Common Mistakes
|
||||
|
||||
- Updating only `Fetch` and forgetting `Count`, causing wrong shard planning.
|
||||
- Returning dynamic rows whose column count does not match `Headers`.
|
||||
- Filtering by requested `shop_id` without also applying `ScopeShopIDs`.
|
||||
- Using context/user middleware inside worker DataSource logic.
|
||||
- Registering the source but forgetting DTO `oneof`, so API rejects the new scene.
|
||||
- Writing XLSX shard files. Shards should be CSV; finalize handles final format.
|
||||
- Adding automated tests despite the repository ban.
|
||||
Reference in New Issue
Block a user