junhong_cmp_fiber/specs/002-gorm-postgres-asynq/tasks.md

Tasks: 数据持久化与异步任务处理集成

Feature: 002-gorm-postgres-asynq
Input: Design documents from /specs/002-gorm-postgres-asynq/
Prerequisites: plan.md, spec.md, data-model.md, contracts/api.yaml, research.md, quickstart.md

Organization: Tasks are grouped by user story (US1: 数据存储与检索, US2: 异步任务处理, US3: 连接管理与监控) to enable independent implementation and testing.

Format: [ID] [P?] [Story] Description

• [P]: Can run in parallel (different files, no dependencies)
• [Story]: Which user story this task belongs to (US1, US2, US3)
• Include exact file paths in descriptions

---

Phase 1: Setup (Shared Infrastructure)

Purpose: Project initialization and basic structure (project already exists, validate/enhance)

• T001 Validate project structure matches plan.md (internal/, pkg/, cmd/, configs/, migrations/, tests/)
• T002 Validate Go dependencies for Fiber + GORM + Asynq + Viper + Zap + golang-migrate
• T003 [P] Validate unified error codes in pkg/errors/codes.go and pkg/errors/errors.go
• T004 [P] Validate unified API response in pkg/response/response.go
• T005 [P] Add database configuration constants in pkg/constants/constants.go (DefaultMaxOpenConns=25, DefaultMaxIdleConns=10, etc.)
• T006 [P] Add task queue constants in pkg/constants/constants.go (TaskTypeEmailSend, TaskTypeDataSync, QueueCritical, QueueDefault, etc.)
• T007 [P] Add user/order status constants in pkg/constants/constants.go (UserStatusActive, OrderStatusPending, etc.)
• T008 [P] Add Redis key generation functions in pkg/constants/redis.go (RedisTaskLockKey, RedisTaskStatusKey)

---

Phase 2: Foundational (Blocking Prerequisites)

Purpose: Core infrastructure that MUST be complete before ANY user story can be implemented

⚠️ CRITICAL: No user story work can begin until this phase is complete

• T009 Implement PostgreSQL connection initialization in pkg/database/postgres.go (GORM + connection pool)
• T010 Validate Redis connection initialization in pkg/database/redis.go (connection pool: PoolSize=10, MinIdleConns=5)
• T011 [P] Add DatabaseConfig to pkg/config/config.go (Host, Port, User, Password, MaxOpenConns, MaxIdleConns, ConnMaxLifetime)
• T012 [P] Add QueueConfig to pkg/config/config.go (Concurrency, Queues, RetryMax, Timeout)
• T013 [P] Update config.yaml files with database and queue configurations (config.dev.yaml, config.staging.yaml, config.prod.yaml)
• T014 Implement Asynq client initialization in pkg/queue/client.go (EnqueueTask with logging)
• T015 Implement Asynq server initialization in pkg/queue/server.go (with queue priorities and error handler)
• T016 Create base Store structure in internal/store/store.go with transaction support
• T017 Initialize postgres store in internal/store/postgres/store.go (embed UserStore, OrderStore)
• T018 Validate migrations directory structure (migrations/000001_init_schema.up.sql and .down.sql exist)

Checkpoint: Foundation ready - user story implementation can now begin in parallel

---

Phase 3: User Story 1 - 可靠的数据存储与检索 (Priority: P1) 🎯 MVP

Goal: 实现可靠的数据持久化存储和高效的 CRUD 操作，确保数据一致性和完整性

Independent Test: 通过创建、读取、更新、删除用户和订单数据验证。包括基本 CRUD、事务提交、数据一致性验证等场景。

Tests for User Story 1 (REQUIRED per Constitution)

NOTE: Write these tests FIRST, ensure they FAIL before implementation

• T019 [P] [US1] Unit tests for User Store layer in tests/unit/store_test.go (Create, GetByID, Update, Delete, List)
• T020 [P] [US1] Unit tests for Order Store layer in tests/unit/store_test.go (Create, GetByID, Update, Delete, ListByUserID)
• T021 [P] [US1] Unit tests for User Service layer in tests/unit/service_test.go (business logic validation)
• T022 [P] [US1] Integration tests for User API endpoints in tests/integration/database_test.go (POST/GET/PUT/DELETE /users)
• T023 [P] [US1] Transaction rollback tests in tests/unit/store_test.go (verify atomic operations)

Implementation for User Story 1

Models & DTOs:

• T024 [P] [US1] Validate BaseModel in internal/model/base.go (ID, CreatedAt, UpdatedAt, DeletedAt)
• T025 [P] [US1] Validate User model in internal/model/user.go with GORM tags (Username, Email, Password, Status)
• T026 [P] [US1] Validate Order model in internal/model/order.go with GORM tags (OrderID, UserID, Amount, Status)
• T027 [P] [US1] Validate User DTOs in internal/model/user_dto.go (CreateUserRequest, UpdateUserRequest, UserResponse, ListUsersResponse)
• T028 [P] [US1] Create Order DTOs in internal/model/order_dto.go (CreateOrderRequest, UpdateOrderRequest, OrderResponse, ListOrdersResponse)

Store Layer (Data Access):

• T029 [US1] Implement UserStore in internal/store/postgres/user_store.go (Create, GetByID, Update, Delete, List with pagination)
• T030 [US1] Implement OrderStore in internal/store/postgres/order_store.go (Create, GetByID, Update, Delete, ListByUserID)
• T031 [US1] Add context timeout handling (3s default) and slow query logging (>100ms) in Store methods

Service Layer (Business Logic):

• T032 [US1] Implement UserService in internal/service/user/service.go (CreateUser, GetUserByID, UpdateUser, DeleteUser, ListUsers)
• T033 [US1] Implement OrderService in internal/service/order/service.go (CreateOrder, GetOrderByID, UpdateOrder, DeleteOrder, ListOrdersByUserID)
• T034 [US1] Add password hashing (bcrypt) in UserService.CreateUser
• T035 [US1] Add validation logic in Service layer using Validator
• T036 [US1] Implement transaction example in OrderService (CreateOrderWithUser)

Handler Layer (HTTP Endpoints):

• T037 [US1] Validate/enhance User Handler in internal/handler/user.go (Create, GetByID, Update, Delete, List endpoints)
• T038 [US1] Create Order Handler in internal/handler/order.go (Create, GetByID, Update, Delete, List endpoints)
• T039 [US1] Add request validation using Validator in handlers
• T040 [US1] Add unified error handling using pkg/errors/ and pkg/response/ in handlers
• T041 [US1] Add structured logging with Zap in handlers (log user_id, order_id, operation, duration)
• T042 [US1] Register User routes in cmd/api/main.go (POST/GET/PUT/DELETE /api/v1/users, /api/v1/users/:id)
• T043 [US1] Register Order routes in cmd/api/main.go (POST/GET/PUT/DELETE /api/v1/orders, /api/v1/orders/:id)

Database Migrations:

• T044 [US1] Validate migration 000001_init_schema.up.sql (tb_user and tb_order tables with indexes)
• T045 [US1] Validate migration 000001_init_schema.down.sql (DROP tables)
• T046 [US1] Test migration up/down with scripts/migrate.sh

Checkpoint: At this point, User Story 1 should be fully functional and testable independently

---

Phase 4: User Story 2 - 异步任务处理能力 (Priority: P2)

Goal: 实现耗时操作的后台异步执行，避免阻塞用户请求，提升系统响应速度

Independent Test: 提交耗时任务（如发送邮件），验证任务被成功加入队列，用户请求立即返回，后台 Worker 完成任务执行。

Tests for User Story 2 (REQUIRED per Constitution)

• T047 [P] [US2] Unit tests for Email task handler in tests/unit/task_handler_test.go (HandleEmailSend idempotency)
• T048 [P] [US2] Unit tests for Sync task handler in tests/unit/task_handler_test.go (HandleDataSync idempotency)
• T049 [P] [US2] Integration tests for task submission in tests/integration/task_test.go (EnqueueEmailTask, EnqueueSyncTask)
• T050 [P] [US2] Integration tests for task queue in tests/integration/task_test.go (verify Worker processes tasks)

Implementation for User Story 2

Task Payloads:

• T051 [P] [US2] Validate EmailPayload in internal/task/email.go (RequestID, To, Subject, Body, CC, Attachments)
• T052 [P] [US2] Validate DataSyncPayload in internal/task/sync.go (RequestID, SyncType, StartDate, EndDate, BatchSize)
• T053 [P] [US2] Create SIMStatusSyncPayload in internal/task/sim.go (RequestID, ICCIDs, ForceSync)

Task Handlers (Worker):

• T054 [US2] Implement HandleEmailSend in internal/task/email.go (with Redis idempotency lock and retry)
• T055 [US2] Implement HandleDataSync in internal/task/sync.go (with idempotency and batch processing)
• T056 [US2] Implement HandleSIMStatusSync in internal/task/sim.go (with idempotency)
• T057 [US2] Add structured logging in task handlers (task_id, task_type, request_id, duration)
• T058 [US2] Add error handling and retry logic in task handlers (max 5 retries, exponential backoff)

Service Integration:

• T059 [US2] Implement EmailService in internal/service/email/service.go (SendWelcomeEmail, EnqueueEmailTask)
• T060 [US2] Implement SyncService in internal/service/sync/service.go (EnqueueDataSyncTask, EnqueueSIMStatusSyncTask)
• T061 [US2] Add Queue Client dependency injection in Service constructors

Handler Layer (Task Submission):

• T062 [US2] Validate/enhance Task Handler in internal/handler/task.go (SubmitEmailTask, SubmitSyncTask endpoints)
• T063 [US2] Add request validation for task payloads in handler
• T064 [US2] Add priority queue selection logic (critical/default/low) in handler
• T065 [US2] Register task routes in cmd/api/main.go (POST /api/v1/tasks/email, POST /api/v1/tasks/sync)

Worker Process:

• T066 [US2] Validate Worker main in cmd/worker/main.go (initialize Server, register handlers, graceful shutdown)
• T067 [US2] Register task handlers in Worker (HandleEmailSend, HandleDataSync, HandleSIMStatusSync)
• T068 [US2] Add signal handling for graceful shutdown in Worker (SIGINT, SIGTERM)

Checkpoint: At this point, User Stories 1 AND 2 should both work independently

---

Phase 5: User Story 3 - 数据库连接管理与监控 (Priority: P3)

Goal: 监控数据库连接状态、查询性能和任务队列健康度，确保系统稳定运行

Independent Test: 通过健康检查接口验证数据库和 Redis 连接状态，模拟连接失败场景验证容错能力。

Tests for User Story 3 (REQUIRED per Constitution)

• T069 [P] [US3] Integration tests for health check in tests/integration/health_test.go (GET /health returns 200 when healthy)
• T070 [P] [US3] Integration tests for degraded state in tests/integration/health_test.go (503 when database down)
• T071 [P] [US3] Unit tests for graceful shutdown in tests/unit/shutdown_test.go (verify connections closed)

Implementation for User Story 3

Health Check:

• T072 [US3] Validate/enhance Health Handler in internal/handler/health.go (check PostgreSQL and Redis status)
• T073 [US3] Add database Ping check with timeout in Health Handler
• T074 [US3] Add Redis Ping check with timeout in Health Handler
• T075 [US3] Return appropriate status codes (200 ok, 503 degraded/unavailable)
• T076 [US3] Register health route in cmd/api/main.go (GET /health)

Connection Management:

• T077 [US3] Add connection pool monitoring in pkg/database/postgres.go (log Stats: OpenConnections, InUse, Idle)
• T078 [US3] Add connection retry logic in pkg/database/postgres.go (max 5 retries, exponential backoff)
• T079 [US3] Add slow query logging middleware in pkg/logger/middleware.go (log queries >100ms)

Graceful Shutdown:

• T080 [US3] Implement graceful shutdown in cmd/api/main.go (close DB, Redis, wait for requests, max 30s timeout)
• T081 [US3] Validate graceful shutdown in cmd/worker/main.go (stop accepting tasks, wait for completion, max 30s)
• T082 [US3] Add signal handling (SIGINT, SIGTERM) in both API and Worker processes

Checkpoint: All user stories should now be independently functional

---

Phase 6: Polish & Quality Gates

Purpose: Improvements that affect multiple user stories and final quality checks

Documentation (Constitution Principle VII - REQUIRED)

• T083 [P] Create feature summary doc in docs/002-gorm-postgres-asynq/功能总结.md (Chinese filename and content)
• T084 [P] Create usage guide in docs/002-gorm-postgres-asynq/使用指南.md (Chinese filename and content)
• T085 [P] Create architecture doc in docs/002-gorm-postgres-asynq/架构说明.md (Chinese filename and content)
• T086 Update README.md with brief feature description (2-3 sentences in Chinese)

Code Quality

• T087 Code cleanup: Remove unused imports, variables, and functions
• T088 Code refactoring: Extract duplicate logic into helper functions
• T089 Performance optimization: Add database indexes for common queries (username, email, order_id, user_id, status)
• T090 Performance testing: Verify API response time P95 < 200ms, P99 < 500ms
• T091 [P] Additional unit tests to reach 70%+ overall coverage, 90%+ for Service layer
• T092 Security audit: Verify no SQL injection (GORM uses prepared statements)
• T093 Security audit: Verify password storage uses bcrypt hashing
• T094 Security audit: Verify sensitive data not logged (passwords, tokens)
• T095 Run quickstart.md validation (test all curl examples work)

Quality Gates (Constitution Compliance)

• T096 Quality Gate: Run go test ./... (all tests pass)
• T097 Quality Gate: Run gofmt -l . (no formatting issues)
• T098 Quality Gate: Run go vet ./... (no issues)
• T099 Quality Gate: Run golangci-lint run (no critical issues)
• T100 Quality Gate: Verify test coverage with go test -cover ./... (70%+ overall, 90%+ Service)
• T101 Quality Gate: Check no TODO/FIXME remains (or documented in GitHub issues)
• T102 Quality Gate: Verify database migrations work (up and down)
• T103 Quality Gate: Verify API documentation in contracts/api.yaml matches implementation
• T104 Quality Gate: Verify no hardcoded constants (all use pkg/constants/)
• T105 Quality Gate: Verify no duplicate hardcoded values (3+ identical literals must be constants)
• T106 Quality Gate: Verify defined constants are used (no duplicate hardcoding)
• T107 Quality Gate: Verify code comments use Chinese (implementation comments in Chinese)
• T108 Quality Gate: Verify log messages use Chinese (logger.Info/Warn/Error/Debug in Chinese)
• T109 Quality Gate: Verify error messages support Chinese (user-facing errors have Chinese text)
• T110 Quality Gate: Verify no Java-style patterns (no getter/setter, no I-prefix, no Impl-suffix)
• T111 Quality Gate: Verify Go naming conventions (UserID not userId, HTTPServer not HttpServer)
• T112 Quality Gate: Verify error handling is explicit (no panic/recover in business logic)
• T113 Quality Gate: Verify uses goroutines/channels for concurrency (not thread pools)
• T114 Quality Gate: Verify no ORM associations (foreignKey, belongsTo tags - use manual joins)
• T115 Quality Gate: Verify feature docs created in docs/002-gorm-postgres-asynq/ with Chinese filenames
• T116 Quality Gate: Verify summary doc content uses Chinese
• T117 Quality Gate: Verify README.md updated with brief description
• T118 Quality Gate: Verify ALL HTTP requests logged to access.log (via pkg/logger/Middleware())
• T119 Quality Gate: Verify access log includes request/response bodies (limited to 50KB)
• T120 Quality Gate: Verify no middleware bypasses logging (test auth failures, rate limits)
• T121 Quality Gate: Verify access log has all required fields (method, path, status, duration_ms, request_id, ip, user_agent, request_body, response_body)

---

Dependencies & Execution Order

Phase Dependencies

• Setup (Phase 1): No dependencies - can start immediately
• Foundational (Phase 2): Depends on Setup completion - BLOCKS all user stories
• User Stories (Phase 3-5): All depend on Foundational phase completion
  • User Story 1 (P1): Can start after Foundational - No dependencies on other stories
  • User Story 2 (P2): Can start after Foundational - Independent (may integrate with US1 for examples)
  • User Story 3 (P3): Can start after Foundational - Independent
• Polish (Phase 6): Depends on all user stories being complete

User Story Independence

• US1 (P1): Fully independent - can be tested and deployed alone (MVP)
• US2 (P2): Fully independent - can be tested and deployed alone (may reference US1 models as examples)
• US3 (P3): Fully independent - can be tested and deployed alone

Within Each User Story

• Tests MUST be written and FAIL before implementation
• Models → Store → Service → Handler → Routes
• Core implementation before integration
• Story complete before moving to next priority

Parallel Opportunities

Phase 1 (Setup):

• T003, T004, T005, T006, T007, T008 can all run in parallel

Phase 2 (Foundational):

• T011, T012, T013 (config) can run in parallel
• T014, T015 (queue client/server) can run in parallel after config

Phase 3 (User Story 1):

• T019-T023 (all tests) can run in parallel
• T024-T028 (all models/DTOs) can run in parallel
• T029, T030 (Store implementations) can run in parallel after models
• T032, T033 (Service implementations) can run in parallel after Store

Phase 4 (User Story 2):

• T047-T050 (all tests) can run in parallel
• T051-T053 (all payloads) can run in parallel
• T054-T056 (all task handlers) can run in parallel after payloads
• T059, T060 (Service implementations) can run in parallel after handlers

Phase 5 (User Story 3):

• T069-T071 (all tests) can run in parallel
• T073, T074 (Ping checks) can run in parallel
• T077, T078, T079 (connection management) can run in parallel

Phase 6 (Polish):

• T083-T085 (all docs) can run in parallel
• T096-T121 (quality gates) run sequentially but can be automated in CI

---

Parallel Example: User Story 1

# Launch all tests together:
go test -v tests/unit/store_test.go &       # T019, T020
go test -v tests/unit/service_test.go &     # T021
go test -v tests/integration/database_test.go &  # T022
wait

# Launch all models together:
Task: "Validate User model in internal/model/user.go"   # T025
Task: "Validate Order model in internal/model/order.go" # T026
Task: "Validate User DTOs"                               # T027
Task: "Create Order DTOs"                                # T028

# Launch both Store implementations together:
Task: "Implement UserStore"  # T029
Task: "Implement OrderStore" # T030

---

Implementation Strategy

MVP First (User Story 1 Only)

1. Complete Phase 1: Setup (T001-T008)
2. Complete Phase 2: Foundational (T009-T018) - CRITICAL
3. Complete Phase 3: User Story 1 (T019-T046)
4. STOP and VALIDATE: Test CRUD operations independently
5. Deploy/demo if ready

Incremental Delivery

1. Setup + Foundational → Foundation ready
2. Add User Story 1 → Test independently → Deploy/Demo (MVP! 🎯)
3. Add User Story 2 → Test independently → Deploy/Demo
4. Add User Story 3 → Test independently → Deploy/Demo
5. Polish → Final quality checks → Production ready

Parallel Team Strategy

With multiple developers:

1. Team completes Setup (Phase 1) + Foundational (Phase 2) together
2. Once Foundational is done:
   • Developer A: User Story 1 (T019-T046)
   • Developer B: User Story 2 (T047-T068)
   • Developer C: User Story 3 (T069-T082)
3. Stories complete and integrate independently
4. Team reconvenes for Polish (Phase 6)

---

Notes

• [P] tasks = different files, no dependencies, can run in parallel
• [Story] label (US1, US2, US3) maps task to specific user story for traceability
• Each user story is independently completable and testable
• Tests are written FIRST and should FAIL before implementation (TDD approach)
• Commit after each task or logical group
• Stop at any checkpoint to validate story independently
• Project structure already exists - tasks validate/enhance existing code where noted
• Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence

---

Task Count Summary

• Total Tasks: 121
• Phase 1 (Setup): 8 tasks
• Phase 2 (Foundational): 10 tasks
• Phase 3 (User Story 1): 28 tasks (5 tests + 23 implementation)
• Phase 4 (User Story 2): 22 tasks (4 tests + 18 implementation)
• Phase 5 (User Story 3): 14 tasks (3 tests + 11 implementation)
• Phase 6 (Polish): 39 tasks (4 docs + 35 quality gates)

Parallel Opportunities: ~40 tasks marked [P] can run in parallel within their phases

Suggested MVP Scope: Phase 1 + Phase 2 + Phase 3 (User Story 1) = 46 tasks