Compare commits
16 Commits
ab9da7bb33
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| d2e08dbbec | |||
| 026d4908d8 | |||
| 31232ea899 | |||
| b38df737e1 | |||
| 346156ee9b | |||
| 0d79130e07 | |||
| b3fb8c7a82 | |||
| 44fb21eb6a | |||
| 8f738ffbe8 | |||
| 6db152bb2f | |||
| fc6af43baa | |||
| 52bdbbae25 | |||
| 76f657c986 | |||
| a2793f7bec | |||
| b5b7f9a41e | |||
| b930662817 |
30
.agents/rules/ponytail.md
Normal file
30
.agents/rules/ponytail.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# Ponytail, lazy senior dev mode
|
||||
|
||||
You are a lazy senior developer. Lazy means efficient, not careless. The best code is the code never written.
|
||||
|
||||
Before writing any code, stop at the first rung that holds:
|
||||
|
||||
1. Does this need to be built at all? (YAGNI)
|
||||
2. Does it already exist in this codebase? Reuse the helper, util, or pattern that's already here, don't re-write it.
|
||||
3. Does the standard library already do this? Use it.
|
||||
4. Does a native platform feature cover it? Use it.
|
||||
5. Does an already-installed dependency solve it? Use it.
|
||||
6. Can this be one line? Make it one line.
|
||||
7. Only then: write the minimum code that works.
|
||||
|
||||
The ladder runs after you understand the problem, not instead of it: read the task and the code it touches, trace the real flow end to end, then climb.
|
||||
|
||||
Bug fix = root cause, not symptom: a report names a symptom. Grep every caller of the function you touch and fix the shared function once — one guard there is a smaller diff than one per caller, and patching only the path the ticket names leaves a sibling caller still broken.
|
||||
|
||||
Rules:
|
||||
|
||||
- No abstractions that weren't explicitly requested.
|
||||
- No new dependency if it can be avoided.
|
||||
- No boilerplate nobody asked for.
|
||||
- Deletion over addition. Boring over clever. Fewest files possible.
|
||||
- Shortest working diff wins, but only once you understand the problem. The smallest change in the wrong place isn't lazy, it's a second bug.
|
||||
- Question complex requests: "Do you actually need X, or does Y cover it?"
|
||||
- Pick the edge-case-correct option when two stdlib approaches are the same size, lazy means less code, not the flimsier algorithm.
|
||||
- Mark intentional simplifications with a `ponytail:` comment. If the shortcut has a known ceiling (global lock, O(n²) scan, naive heuristic), the comment names the ceiling and the upgrade path.
|
||||
|
||||
Not lazy about: understanding the problem (read it fully and trace the real flow before picking a rung, a small diff you don't understand is just laziness dressed up as efficiency), input validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs (the platform is never the spec ideal, a clock drifts, a sensor reads off), anything explicitly requested. Lazy code without its check is unfinished: non-trivial logic leaves ONE runnable check behind, the smallest thing that fails if the logic breaks (an assert-based demo/self-check or one small test file; no frameworks, no fixtures). Trivial one-liners need no test.
|
||||
@@ -1,10 +1,7 @@
|
||||
---
|
||||
name: grill-me
|
||||
description: Interview the user relentlessly about a plan or design until reaching shared understanding, resolving each branch of the decision tree. Use when user wants to stress-test a plan, get grilled on their design, or mentions "grill me".
|
||||
description: A relentless interview to sharpen a plan or design.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
Interview me relentlessly about every aspect of this plan until we reach a shared understanding. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer.
|
||||
|
||||
Ask the questions one at a time.
|
||||
|
||||
If a question can be answered by exploring the codebase, explore the codebase instead.
|
||||
Run a `/grilling` session.
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
# ADR Format
|
||||
|
||||
ADRs live in `docs/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc.
|
||||
|
||||
Create the `docs/adr/` directory lazily — only when the first ADR is needed.
|
||||
|
||||
## Template
|
||||
|
||||
```md
|
||||
# {Short title of the decision}
|
||||
|
||||
{1-3 sentences: what's the context, what did we decide, and why.}
|
||||
```
|
||||
|
||||
That's it. An ADR can be a single paragraph. The value is in recording *that* a decision was made and *why* — not in filling out sections.
|
||||
|
||||
## Optional sections
|
||||
|
||||
Only include these when they add genuine value. Most ADRs won't need them.
|
||||
|
||||
- **Status** frontmatter (`proposed | accepted | deprecated | superseded by ADR-NNNN`) — useful when decisions are revisited
|
||||
- **Considered Options** — only when the rejected alternatives are worth remembering
|
||||
- **Consequences** — only when non-obvious downstream effects need to be called out
|
||||
|
||||
## Numbering
|
||||
|
||||
Scan `docs/adr/` for the highest existing number and increment by one.
|
||||
|
||||
## When to offer an ADR
|
||||
|
||||
All three of these must be true:
|
||||
|
||||
1. **Hard to reverse** — the cost of changing your mind later is meaningful
|
||||
2. **Surprising without context** — a future reader will look at the code and wonder "why on earth did they do it this way?"
|
||||
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons
|
||||
|
||||
If a decision is easy to reverse, skip it — you'll just reverse it. If it's not surprising, nobody will wonder why. If there was no real alternative, there's nothing to record beyond "we did the obvious thing."
|
||||
|
||||
### What qualifies
|
||||
|
||||
- **Architectural shape.** "We're using a monorepo." "The write model is event-sourced, the read model is projected into Postgres."
|
||||
- **Integration patterns between contexts.** "Ordering and Billing communicate via domain events, not synchronous HTTP."
|
||||
- **Technology choices that carry lock-in.** Database, message bus, auth provider, deployment target. Not every library — just the ones that would take a quarter to swap out.
|
||||
- **Boundary and scope decisions.** "Customer data is owned by the Customer context; other contexts reference it by ID only." The explicit no-s are as valuable as the yes-s.
|
||||
- **Deliberate deviations from the obvious path.** "We're using manual SQL instead of an ORM because X." Anything where a reasonable reader would assume the opposite. These stop the next engineer from "fixing" something that was deliberate.
|
||||
- **Constraints not visible in the code.** "We can't use AWS because of compliance requirements." "Response times must be under 200ms because of the partner API contract."
|
||||
- **Rejected alternatives when the rejection is non-obvious.** If you considered GraphQL and picked REST for subtle reasons, record it — otherwise someone will suggest GraphQL again in six months.
|
||||
@@ -1,60 +0,0 @@
|
||||
# CONTEXT.md Format
|
||||
|
||||
## Structure
|
||||
|
||||
```md
|
||||
# {Context Name}
|
||||
|
||||
{One or two sentence description of what this context is and why it exists.}
|
||||
|
||||
## Language
|
||||
|
||||
**Order**:
|
||||
{A one or two sentence description of the term}
|
||||
_Avoid_: Purchase, transaction
|
||||
|
||||
**Invoice**:
|
||||
A request for payment sent to a customer after delivery.
|
||||
_Avoid_: Bill, payment request
|
||||
|
||||
**Customer**:
|
||||
A person or organization that places orders.
|
||||
_Avoid_: Client, buyer, account
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
- **Be opinionated.** When multiple words exist for the same concept, pick the best one and list the others under `_Avoid_`.
|
||||
- **Keep definitions tight.** One or two sentences max. Define what it IS, not what it does.
|
||||
- **Only include terms specific to this project's context.** General programming concepts (timeouts, error types, utility patterns) don't belong even if the project uses them extensively. Before adding a term, ask: is this a concept unique to this context, or a general programming concept? Only the former belongs.
|
||||
- **Group terms under subheadings** when natural clusters emerge. If all terms belong to a single cohesive area, a flat list is fine.
|
||||
|
||||
## Single vs multi-context repos
|
||||
|
||||
**Single context (most repos):** One `CONTEXT.md` at the repo root.
|
||||
|
||||
**Multiple contexts:** A `CONTEXT-MAP.md` at the repo root lists the contexts, where they live, and how they relate to each other:
|
||||
|
||||
```md
|
||||
# Context Map
|
||||
|
||||
## Contexts
|
||||
|
||||
- [Ordering](./src/ordering/CONTEXT.md) — receives and tracks customer orders
|
||||
- [Billing](./src/billing/CONTEXT.md) — generates invoices and processes payments
|
||||
- [Fulfillment](./src/fulfillment/CONTEXT.md) — manages warehouse picking and shipping
|
||||
|
||||
## Relationships
|
||||
|
||||
- **Ordering → Fulfillment**: Ordering emits `OrderPlaced` events; Fulfillment consumes them to start picking
|
||||
- **Fulfillment → Billing**: Fulfillment emits `ShipmentDispatched` events; Billing consumes them to generate invoices
|
||||
- **Ordering ↔ Billing**: Shared types for `CustomerId` and `Money`
|
||||
```
|
||||
|
||||
The skill infers which structure applies:
|
||||
|
||||
- If `CONTEXT-MAP.md` exists, read it to find contexts
|
||||
- If only a root `CONTEXT.md` exists, single context
|
||||
- If neither exists, create a root `CONTEXT.md` lazily when the first term is resolved
|
||||
|
||||
When multiple contexts exist, infer which one the current topic relates to. If unclear, ask.
|
||||
@@ -1,88 +1,7 @@
|
||||
---
|
||||
name: grill-with-docs
|
||||
description: Grilling session that challenges your plan against the existing domain model, sharpens terminology, and updates documentation (CONTEXT.md, ADRs) inline as decisions crystallise. Use when user wants to stress-test a plan against their project's language and documented decisions.
|
||||
description: A relentless interview to sharpen a plan or design, which also creates docs (ADR's and glossary) as we go.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
<what-to-do>
|
||||
|
||||
Interview me relentlessly about every aspect of this plan until we reach a shared understanding. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer.
|
||||
|
||||
Ask the questions one at a time, waiting for feedback on each question before continuing.
|
||||
|
||||
If a question can be answered by exploring the codebase, explore the codebase instead.
|
||||
|
||||
</what-to-do>
|
||||
|
||||
<supporting-info>
|
||||
|
||||
## Domain awareness
|
||||
|
||||
During codebase exploration, also look for existing documentation:
|
||||
|
||||
### File structure
|
||||
|
||||
Most repos have a single context:
|
||||
|
||||
```
|
||||
/
|
||||
├── CONTEXT.md
|
||||
├── docs/
|
||||
│ └── adr/
|
||||
│ ├── 0001-event-sourced-orders.md
|
||||
│ └── 0002-postgres-for-write-model.md
|
||||
└── src/
|
||||
```
|
||||
|
||||
If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The map points to where each one lives:
|
||||
|
||||
```
|
||||
/
|
||||
├── CONTEXT-MAP.md
|
||||
├── docs/
|
||||
│ └── adr/ ← system-wide decisions
|
||||
├── src/
|
||||
│ ├── ordering/
|
||||
│ │ ├── CONTEXT.md
|
||||
│ │ └── docs/adr/ ← context-specific decisions
|
||||
│ └── billing/
|
||||
│ ├── CONTEXT.md
|
||||
│ └── docs/adr/
|
||||
```
|
||||
|
||||
Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `docs/adr/` exists, create it when the first ADR is needed.
|
||||
|
||||
## During the session
|
||||
|
||||
### Challenge against the glossary
|
||||
|
||||
When the user uses a term that conflicts with the existing language in `CONTEXT.md`, call it out immediately. "Your glossary defines 'cancellation' as X, but you seem to mean Y — which is it?"
|
||||
|
||||
### Sharpen fuzzy language
|
||||
|
||||
When the user uses vague or overloaded terms, propose a precise canonical term. "You're saying 'account' — do you mean the Customer or the User? Those are different things."
|
||||
|
||||
### Discuss concrete scenarios
|
||||
|
||||
When domain relationships are being discussed, stress-test them with specific scenarios. Invent scenarios that probe edge cases and force the user to be precise about the boundaries between concepts.
|
||||
|
||||
### Cross-reference with code
|
||||
|
||||
When the user states how something works, check whether the code agrees. If you find a contradiction, surface it: "Your code cancels entire Orders, but you just said partial cancellation is possible — which is right?"
|
||||
|
||||
### Update CONTEXT.md inline
|
||||
|
||||
When a term is resolved, update `CONTEXT.md` right there. Don't batch these up — capture them as they happen. Use the format in [CONTEXT-FORMAT.md](./CONTEXT-FORMAT.md).
|
||||
|
||||
`CONTEXT.md` should be totally devoid of implementation details. Do not treat `CONTEXT.md` as a spec, a scratch pad, or a repository for implementation decisions. It is a glossary and nothing else.
|
||||
|
||||
### Offer ADRs sparingly
|
||||
|
||||
Only offer to create an ADR when all three are true:
|
||||
|
||||
1. **Hard to reverse** — the cost of changing your mind later is meaningful
|
||||
2. **Surprising without context** — a future reader will wonder "why did they do it this way?"
|
||||
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons
|
||||
|
||||
If any of the three is missing, skip the ADR. Use the format in [ADR-FORMAT.md](./ADR-FORMAT.md).
|
||||
|
||||
</supporting-info>
|
||||
Run a `/grilling` session, using the `/domain-modeling` skill.
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
name: handoff
|
||||
description: Compact the current conversation into a handoff document for another agent to pick up.
|
||||
argument-hint: "What will the next session be used for?"
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
Write a handoff document summarising the current conversation so a fresh agent can continue the work. Save to the temporary directory of the user's OS - not the current workspace.
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
# Deepening
|
||||
|
||||
How to deepen a cluster of shallow modules safely, given its dependencies. Assumes the vocabulary in [LANGUAGE.md](LANGUAGE.md) — **module**, **interface**, **seam**, **adapter**.
|
||||
|
||||
## Dependency categories
|
||||
|
||||
When assessing a candidate for deepening, classify its dependencies. The category determines how the deepened module is tested across its seam.
|
||||
|
||||
### 1. In-process
|
||||
|
||||
Pure computation, in-memory state, no I/O. Always deepenable — merge the modules and test through the new interface directly. No adapter needed.
|
||||
|
||||
### 2. Local-substitutable
|
||||
|
||||
Dependencies that have local test stand-ins (PGLite for Postgres, in-memory filesystem). Deepenable if the stand-in exists. The deepened module is tested with the stand-in running in the test suite. The seam is internal; no port at the module's external interface.
|
||||
|
||||
### 3. Remote but owned (Ports & Adapters)
|
||||
|
||||
Your own services across a network boundary (microservices, internal APIs). Define a **port** (interface) at the seam. The deep module owns the logic; the transport is injected as an **adapter**. Tests use an in-memory adapter. Production uses an HTTP/gRPC/queue adapter.
|
||||
|
||||
Recommendation shape: *"Define a port at the seam, implement an HTTP adapter for production and an in-memory adapter for testing, so the logic sits in one deep module even though it's deployed across a network."*
|
||||
|
||||
### 4. True external (Mock)
|
||||
|
||||
Third-party services (Stripe, Twilio, etc.) you don't control. The deepened module takes the external dependency as an injected port; tests provide a mock adapter.
|
||||
|
||||
## Seam discipline
|
||||
|
||||
- **One adapter means a hypothetical seam. Two adapters means a real one.** Don't introduce a port unless at least two adapters are justified (typically production + test). A single-adapter seam is just indirection.
|
||||
- **Internal seams vs external seams.** A deep module can have internal seams (private to its implementation, used by its own tests) as well as the external seam at its interface. Don't expose internal seams through the interface just because tests use them.
|
||||
|
||||
## Testing strategy: replace, don't layer
|
||||
|
||||
- Old unit tests on shallow modules become waste once tests at the deepened module's interface exist — delete them.
|
||||
- Write new tests at the deepened module's interface. The **interface is the test surface**.
|
||||
- Tests assert on observable outcomes through the interface, not internal state.
|
||||
- Tests should survive internal refactors — they describe behaviour, not implementation. If a test has to change when the implementation changes, it's testing past the interface.
|
||||
@@ -39,7 +39,7 @@ Repo name, date, and a compact legend: solid box = module, dashed line = seam, r
|
||||
|
||||
## Candidate card
|
||||
|
||||
The diagrams carry the weight. Prose is sparse, plain, and uses the glossary terms ([LANGUAGE.md](LANGUAGE.md)) without ceremony.
|
||||
The diagrams carry the weight. Prose is sparse, plain, and uses the glossary terms (from the `/codebase-design` skill) without ceremony.
|
||||
|
||||
Each candidate is one `<article>`:
|
||||
|
||||
@@ -105,7 +105,7 @@ One larger card. Candidate name, one sentence on why, anchor link to its card. T
|
||||
|
||||
## Tone
|
||||
|
||||
Plain English, concise — but the architectural nouns and verbs come straight from [LANGUAGE.md](LANGUAGE.md). Concision is not an excuse to drift.
|
||||
Plain English, concise — but the architectural nouns and verbs come straight from the `/codebase-design` skill. Concision is not an excuse to drift.
|
||||
|
||||
**Use exactly:** module, interface, implementation, depth, deep, shallow, seam, adapter, leverage, locality.
|
||||
|
||||
@@ -120,4 +120,4 @@ Plain English, concise — but the architectural nouns and verbs come straight f
|
||||
|
||||
**Wins bullets** name the gain in glossary terms: *"locality: bugs concentrate in one module"*, *"leverage: one interface, N call sites"*, *"interface shrinks; implementation absorbs the wrappers"*. Don't write *"easier to maintain"* or *"cleaner code"* — those terms aren't in the glossary and don't earn their place.
|
||||
|
||||
No hedging, no throat-clearing, no "it's worth noting that…". If a sentence could be a bullet, make it a bullet. If a bullet could be cut, cut it. If a term isn't in [LANGUAGE.md](LANGUAGE.md), reach for one that is before inventing a new one.
|
||||
No hedging, no throat-clearing, no "it's worth noting that…". If a sentence could be a bullet, make it a bullet. If a bullet could be cut, cut it. If a term isn't in the `/codebase-design` glossary, reach for one that is before inventing a new one.
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
# Interface Design
|
||||
|
||||
When the user wants to explore alternative interfaces for a chosen deepening candidate, use this parallel sub-agent pattern. Based on "Design It Twice" (Ousterhout) — your first idea is unlikely to be the best.
|
||||
|
||||
Uses the vocabulary in [LANGUAGE.md](LANGUAGE.md) — **module**, **interface**, **seam**, **adapter**, **leverage**.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Frame the problem space
|
||||
|
||||
Before spawning sub-agents, write a user-facing explanation of the problem space for the chosen candidate:
|
||||
|
||||
- The constraints any new interface would need to satisfy
|
||||
- The dependencies it would rely on, and which category they fall into (see [DEEPENING.md](DEEPENING.md))
|
||||
- A rough illustrative code sketch to ground the constraints — not a proposal, just a way to make the constraints concrete
|
||||
|
||||
Show this to the user, then immediately proceed to Step 2. The user reads and thinks while the sub-agents work in parallel.
|
||||
|
||||
### 2. Spawn sub-agents
|
||||
|
||||
Spawn 3+ sub-agents in parallel using the Agent tool. Each must produce a **radically different** interface for the deepened module.
|
||||
|
||||
Prompt each sub-agent with a separate technical brief (file paths, coupling details, dependency category from [DEEPENING.md](DEEPENING.md), what sits behind the seam). The brief is independent of the user-facing problem-space explanation in Step 1. Give each agent a different design constraint:
|
||||
|
||||
- Agent 1: "Minimize the interface — aim for 1–3 entry points max. Maximise leverage per entry point."
|
||||
- Agent 2: "Maximise flexibility — support many use cases and extension."
|
||||
- Agent 3: "Optimise for the most common caller — make the default case trivial."
|
||||
- Agent 4 (if applicable): "Design around ports & adapters for cross-seam dependencies."
|
||||
|
||||
Include both [LANGUAGE.md](LANGUAGE.md) vocabulary and CONTEXT.md vocabulary in the brief so each sub-agent names things consistently with the architecture language and the project's domain language.
|
||||
|
||||
Each sub-agent outputs:
|
||||
|
||||
1. Interface (types, methods, params — plus invariants, ordering, error modes)
|
||||
2. Usage example showing how callers use it
|
||||
3. What the implementation hides behind the seam
|
||||
4. Dependency strategy and adapters (see [DEEPENING.md](DEEPENING.md))
|
||||
5. Trade-offs — where leverage is high, where it's thin
|
||||
|
||||
### 3. Present and compare
|
||||
|
||||
Present designs sequentially so the user can absorb each one, then compare them in prose. Contrast by **depth** (leverage at the interface), **locality** (where change concentrates), and **seam placement**.
|
||||
|
||||
After comparing, give your own recommendation: which design you think is strongest and why. If elements from different designs would combine well, propose a hybrid. Be opinionated — the user wants a strong read, not a menu.
|
||||
@@ -1,53 +0,0 @@
|
||||
# Language
|
||||
|
||||
Shared vocabulary for every suggestion this skill makes. Use these terms exactly — don't substitute "component," "service," "API," or "boundary." Consistent language is the whole point.
|
||||
|
||||
## Terms
|
||||
|
||||
**Module**
|
||||
Anything with an interface and an implementation. Deliberately scale-agnostic — applies equally to a function, class, package, or tier-spanning slice.
|
||||
_Avoid_: unit, component, service.
|
||||
|
||||
**Interface**
|
||||
Everything a caller must know to use the module correctly. Includes the type signature, but also invariants, ordering constraints, error modes, required configuration, and performance characteristics.
|
||||
_Avoid_: API, signature (too narrow — those refer only to the type-level surface).
|
||||
|
||||
**Implementation**
|
||||
What's inside a module — its body of code. Distinct from **Adapter**: a thing can be a small adapter with a large implementation (a Postgres repo) or a large adapter with a small implementation (an in-memory fake). Reach for "adapter" when the seam is the topic; "implementation" otherwise.
|
||||
|
||||
**Depth**
|
||||
Leverage at the interface — the amount of behaviour a caller (or test) can exercise per unit of interface they have to learn. A module is **deep** when a large amount of behaviour sits behind a small interface. A module is **shallow** when the interface is nearly as complex as the implementation.
|
||||
|
||||
**Seam** _(from Michael Feathers)_
|
||||
A place where you can alter behaviour without editing in that place. The *location* at which a module's interface lives. Choosing where to put the seam is its own design decision, distinct from what goes behind it.
|
||||
_Avoid_: boundary (overloaded with DDD's bounded context).
|
||||
|
||||
**Adapter**
|
||||
A concrete thing that satisfies an interface at a seam. Describes *role* (what slot it fills), not substance (what's inside).
|
||||
|
||||
**Leverage**
|
||||
What callers get from depth. More capability per unit of interface they have to learn. One implementation pays back across N call sites and M tests.
|
||||
|
||||
**Locality**
|
||||
What maintainers get from depth. Change, bugs, knowledge, and verification concentrate at one place rather than spreading across callers. Fix once, fixed everywhere.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Depth is a property of the interface, not the implementation.** A deep module can be internally composed of small, mockable, swappable parts — they just aren't part of the interface. A module can have **internal seams** (private to its implementation, used by its own tests) as well as the **external seam** at its interface.
|
||||
- **The deletion test.** Imagine deleting the module. If complexity vanishes, the module wasn't hiding anything (it was a pass-through). If complexity reappears across N callers, the module was earning its keep.
|
||||
- **The interface is the test surface.** Callers and tests cross the same seam. If you want to test *past* the interface, the module is probably the wrong shape.
|
||||
- **One adapter means a hypothetical seam. Two adapters means a real one.** Don't introduce a seam unless something actually varies across it.
|
||||
|
||||
## Relationships
|
||||
|
||||
- A **Module** has exactly one **Interface** (the surface it presents to callers and tests).
|
||||
- **Depth** is a property of a **Module**, measured against its **Interface**.
|
||||
- A **Seam** is where a **Module**'s **Interface** lives.
|
||||
- An **Adapter** sits at a **Seam** and satisfies the **Interface**.
|
||||
- **Depth** produces **Leverage** for callers and **Locality** for maintainers.
|
||||
|
||||
## Rejected framings
|
||||
|
||||
- **Depth as ratio of implementation-lines to interface-lines** (Ousterhout): rewards padding the implementation. We use depth-as-leverage instead.
|
||||
- **"Interface" as the TypeScript `interface` keyword or a class's public methods**: too narrow — interface here includes every fact a caller must know.
|
||||
- **"Boundary"**: overloaded with DDD's bounded context. Say **seam** or **interface**.
|
||||
@@ -1,38 +1,23 @@
|
||||
---
|
||||
name: improve-codebase-architecture
|
||||
description: Find deepening opportunities in a codebase, informed by the domain language in CONTEXT.md and the decisions in docs/adr/. Use when the user wants to improve architecture, find refactoring opportunities, consolidate tightly-coupled modules, or make a codebase more testable and AI-navigable.
|
||||
description: Scan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# Improve Codebase Architecture
|
||||
|
||||
Surface architectural friction and propose **deepening opportunities** — refactors that turn shallow modules into deep ones. The aim is testability and AI-navigability.
|
||||
|
||||
## Glossary
|
||||
This command is _informed_ by the project's domain model and built on a shared design vocabulary:
|
||||
|
||||
Use these terms exactly in every suggestion. Consistent language is the point — don't drift into "component," "service," "API," or "boundary." Full definitions in [LANGUAGE.md](LANGUAGE.md).
|
||||
|
||||
- **Module** — anything with an interface and an implementation (function, class, package, slice).
|
||||
- **Interface** — everything a caller must know to use the module: types, invariants, error modes, ordering, config. Not just the type signature.
|
||||
- **Implementation** — the code inside.
|
||||
- **Depth** — leverage at the interface: a lot of behaviour behind a small interface. **Deep** = high leverage. **Shallow** = interface nearly as complex as the implementation.
|
||||
- **Seam** — where an interface lives; a place behaviour can be altered without editing in place. (Use this, not "boundary.")
|
||||
- **Adapter** — a concrete thing satisfying an interface at a seam.
|
||||
- **Leverage** — what callers get from depth.
|
||||
- **Locality** — what maintainers get from depth: change, bugs, knowledge concentrated in one place.
|
||||
|
||||
Key principles (see [LANGUAGE.md](LANGUAGE.md) for the full list):
|
||||
|
||||
- **Deletion test**: imagine deleting the module. If complexity vanishes, it was a pass-through. If complexity reappears across N callers, it was earning its keep.
|
||||
- **The interface is the test surface.**
|
||||
- **One adapter = hypothetical seam. Two adapters = real seam.**
|
||||
|
||||
This skill is _informed_ by the project's domain model. The domain language gives names to good seams; ADRs record decisions the skill should not re-litigate.
|
||||
- Run the `/codebase-design` skill for the architecture vocabulary (**module**, **interface**, **depth**, **seam**, **adapter**, **leverage**, **locality**) and its principles (the deletion test, "the interface is the test surface", "one adapter = hypothetical seam, two = real"). Use these terms exactly in every suggestion — don't drift into "component," "service," "API," or "boundary."
|
||||
- The domain language in `CONTEXT.md` gives names to good seams; ADRs in `docs/adr/` record decisions this command should not re-litigate.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Explore
|
||||
|
||||
Read the project's domain glossary and any ADRs in the area you're touching first.
|
||||
Read the project's domain glossary (`CONTEXT.md`) and any ADRs in the area you're touching first.
|
||||
|
||||
Then use the Agent tool with `subagent_type=Explore` to walk the codebase. Don't follow rigid heuristics — explore organically and note where you experience friction:
|
||||
|
||||
@@ -50,7 +35,7 @@ Write a self-contained HTML file to the OS temp directory so nothing lands in th
|
||||
|
||||
The report uses **Tailwind via CDN** for layout and styling, and **Mermaid via CDN** for diagrams where a graph/flow/sequence reliably communicates the structure. Mix Mermaid with hand-crafted CSS/SVG visuals — use Mermaid when relationships are graph-shaped (call graphs, dependencies, sequences), and hand-built divs/SVG when you want something more editorial (mass diagrams, cross-sections, collapse animations). Each candidate gets a **before/after visualisation**. Be visual.
|
||||
|
||||
For each candidate, the same template as before, but rendered as a card:
|
||||
For each candidate, render a card with:
|
||||
|
||||
- **Files** — which files/modules are involved
|
||||
- **Problem** — why the current architecture is causing friction
|
||||
@@ -61,7 +46,7 @@ For each candidate, the same template as before, but rendered as a card:
|
||||
|
||||
End the report with a **Top recommendation** section: which candidate you'd tackle first and why.
|
||||
|
||||
**Use CONTEXT.md vocabulary for the domain, and [LANGUAGE.md](LANGUAGE.md) vocabulary for the architecture.** If `CONTEXT.md` defines "Order," talk about "the Order intake module" — not "the FooBarHandler," and not "the Order service."
|
||||
**Use CONTEXT.md vocabulary for the domain, and the `/codebase-design` vocabulary for the architecture.** If `CONTEXT.md` defines "Order," talk about "the Order intake module" — not "the FooBarHandler," and not "the Order service."
|
||||
|
||||
**ADR conflicts**: if a candidate contradicts an existing ADR, only surface it when the friction is real enough to warrant revisiting the ADR. Mark it clearly in the card (e.g. a warning callout: _"contradicts ADR-0007 — but worth reopening because…"_). Don't list every theoretical refactor an ADR forbids.
|
||||
|
||||
@@ -71,11 +56,11 @@ Do NOT propose interfaces yet. After the file is written, ask the user: "Which o
|
||||
|
||||
### 3. Grilling loop
|
||||
|
||||
Once the user picks a candidate, drop into a grilling conversation. Walk the design tree with them — constraints, dependencies, the shape of the deepened module, what sits behind the seam, what tests survive.
|
||||
Once the user picks a candidate, run the `/grilling` skill to walk the design tree with them — constraints, dependencies, the shape of the deepened module, what sits behind the seam, what tests survive.
|
||||
|
||||
Side effects happen inline as decisions crystallize:
|
||||
Side effects happen inline as decisions crystallize — run the `/domain-modeling` skill to keep the domain model current as you go:
|
||||
|
||||
- **Naming a deepened module after a concept not in `CONTEXT.md`?** Add the term to `CONTEXT.md` — same discipline as `/grill-with-docs` (see [CONTEXT-FORMAT.md](../grill-with-docs/CONTEXT-FORMAT.md)). Create the file lazily if it doesn't exist.
|
||||
- **Naming a deepened module after a concept not in `CONTEXT.md`?** Add the term to `CONTEXT.md`. Create the file lazily if it doesn't exist.
|
||||
- **Sharpening a fuzzy term during the conversation?** Update `CONTEXT.md` right there.
|
||||
- **User rejects the candidate with a load-bearing reason?** Offer an ADR, framed as: _"Want me to record this as an ADR so future architecture reviews don't re-suggest it?"_ Only offer when the reason would actually be needed by a future explorer to avoid re-suggesting the same thing — skip ephemeral reasons ("not worth it right now") and self-evident ones. See [ADR-FORMAT.md](../grill-with-docs/ADR-FORMAT.md).
|
||||
- **Want to explore alternative interfaces for the deepened module?** See [INTERFACE-DESIGN.md](INTERFACE-DESIGN.md).
|
||||
- **User rejects the candidate with a load-bearing reason?** Offer an ADR, framed as: _"Want me to record this as an ADR so future architecture reviews don't re-suggest it?"_ Only offer when the reason would actually be needed by a future explorer to avoid re-suggesting the same thing — skip ephemeral reasons ("not worth it right now") and self-evident ones.
|
||||
- **Want to explore alternative interfaces for the deepened module?** Run the `/codebase-design` skill and use its design-it-twice parallel sub-agent pattern.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: prototype
|
||||
description: Build a throwaway prototype to flesh out a design before committing to it. Routes between two branches — a runnable terminal app for state/business-logic questions, or several radically different UI variations toggleable from one route. Use when the user wants to prototype, sanity-check a data model or state machine, mock up a UI, explore design options, or says "prototype this", "let me play with it", "try a few designs".
|
||||
description: Build a throwaway prototype to answer a design question. Use when the user wants to sanity-check whether a state model or logic feels right, or explore what a UI should look like.
|
||||
---
|
||||
|
||||
# Prototype
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: setup-matt-pocock-skills
|
||||
description: Sets up an `## Agent skills` block in AGENTS.md/CLAUDE.md and `docs/agents/` so the engineering skills know this repo's issue tracker (GitHub or local markdown), triage label vocabulary, and domain doc layout. Run before first use of `to-issues`, `to-prd`, `triage`, `diagnose`, `tdd`, `improve-codebase-architecture`, or `zoom-out` — or if those skills appear to be missing context about the issue tracker, triage labels, or domain docs.
|
||||
description: Configure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
@@ -44,6 +44,12 @@ Default posture: these skills were designed for GitHub. If a `git remote` points
|
||||
- **Local markdown** — issues live as files under `.scratch/<feature>/` in this repo (good for solo projects or repos without a remote)
|
||||
- **Other** (Jira, Linear, etc.) — ask the user to describe the workflow in one paragraph; the skill will record it as freeform prose
|
||||
|
||||
If — and only if — the user picked **GitHub** or **GitLab**, ask one follow-up:
|
||||
|
||||
> Explainer: Open-source repos often receive feature requests as pull requests, not just issues — a PR is an issue with attached code. If you turn this on, `/triage` pulls *external* PRs into the same queue and runs them through the same labels and states as issues (collaborators' in-flight PRs are left alone). Leave it off if PRs aren't a request surface for you.
|
||||
|
||||
- **PRs as a request surface** — yes / no (default: no). Record the answer in `docs/agents/issue-tracker.md`. For local-markdown and other trackers, skip this question — there are no PRs.
|
||||
|
||||
**Section B — Triage label vocabulary.**
|
||||
|
||||
> Explainer: When the `triage` skill processes an incoming issue, it moves it through a state machine — needs evaluation, waiting on reporter, ready for an AFK agent to pick up, ready for a human, or won't fix. To do that, it needs to apply labels (or the equivalent in your issue tracker) that match strings *you've actually configured*. If your repo already uses different label names (e.g. `bug:triage` instead of `needs-triage`), map them here so the skill applies the right ones instead of creating duplicates.
|
||||
@@ -60,7 +66,7 @@ Default: each role's string equals its name. Ask the user if they want to overri
|
||||
|
||||
**Section C — Domain docs.**
|
||||
|
||||
> Explainer: Some skills (`improve-codebase-architecture`, `diagnose`, `tdd`) read a `CONTEXT.md` file to learn the project's domain language, and `docs/adr/` for past architectural decisions. They need to know whether the repo has one global context or multiple (e.g. a monorepo with separate frontend/backend contexts) so they look in the right place.
|
||||
> Explainer: Some skills (`improve-codebase-architecture`, `diagnosing-bugs`, `tdd`) read a `CONTEXT.md` file to learn the project's domain language, and `docs/adr/` for past architectural decisions. They need to know whether the repo has one global context or multiple (e.g. a monorepo with separate frontend/backend contexts) so they look in the right place.
|
||||
|
||||
Confirm the layout:
|
||||
|
||||
@@ -95,7 +101,7 @@ The block:
|
||||
|
||||
### Issue tracker
|
||||
|
||||
[one-line summary of where issues are tracked]. See `docs/agents/issue-tracker.md`.
|
||||
[one-line summary of where issues are tracked, plus whether external PRs are a triage surface]. See `docs/agents/issue-tracker.md`.
|
||||
|
||||
### Triage labels
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ How the engineering skills should consume this repo's domain documentation when
|
||||
- **`CONTEXT-MAP.md`** at the repo root if it exists — it points at one `CONTEXT.md` per context. Read each one relevant to the topic.
|
||||
- **`docs/adr/`** — read ADRs that touch the area you're about to work in. In multi-context repos, also check `src/<context>/docs/adr/` for context-scoped decisions.
|
||||
|
||||
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The producer skill (`/grill-with-docs`) creates them lazily when terms or decisions actually get resolved.
|
||||
If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
|
||||
|
||||
## File structure
|
||||
|
||||
@@ -42,7 +42,7 @@ Multi-context repo (presence of `CONTEXT-MAP.md` at the root):
|
||||
|
||||
When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids.
|
||||
|
||||
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/grill-with-docs`).
|
||||
If the concept you need isn't in the glossary yet, that's a signal — either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
|
||||
|
||||
## Flag ADR conflicts
|
||||
|
||||
|
||||
@@ -13,6 +13,18 @@ Issues and PRDs for this repo live as GitHub issues. Use the `gh` CLI for all op
|
||||
|
||||
Infer the repo from `git remote -v` — `gh` does this automatically when run inside a clone.
|
||||
|
||||
## Pull requests as a triage surface
|
||||
|
||||
**PRs as a request surface: no.** _(Set to `yes` if this repo treats external PRs as feature requests; `/triage` reads this flag.)_
|
||||
|
||||
When set to `yes`, PRs run through the same labels and states as issues, using the `gh pr` equivalents:
|
||||
|
||||
- **Read a PR**: `gh pr view <number> --comments` and `gh pr diff <number>` for the diff.
|
||||
- **List external PRs for triage**: `gh pr list --state open --json number,title,body,labels,author,authorAssociation,comments` then keep only `authorAssociation` of `CONTRIBUTOR`, `FIRST_TIME_CONTRIBUTOR`, or `NONE` (drop `OWNER`/`MEMBER`/`COLLABORATOR`).
|
||||
- **Comment / label / close**: `gh pr comment`, `gh pr edit --add-label`/`--remove-label`, `gh pr close`.
|
||||
|
||||
GitHub shares one number space across issues and PRs, so a bare `#42` may be either — resolve with `gh pr view 42` and fall back to `gh issue view 42`.
|
||||
|
||||
## When a skill says "publish to the issue tracker"
|
||||
|
||||
Create a GitHub issue.
|
||||
|
||||
@@ -14,6 +14,18 @@ Issues and PRDs for this repo live as GitLab issues. Use the [`glab`](https://gi
|
||||
|
||||
Infer the repo from `git remote -v` — `glab` does this automatically when run inside a clone.
|
||||
|
||||
## Merge requests as a triage surface
|
||||
|
||||
**MRs as a request surface: no.** _(Set to `yes` if this repo treats external merge requests as feature requests; `/triage` reads this flag.)_
|
||||
|
||||
When set to `yes`, MRs run through the same labels and states as issues, using the `glab mr` equivalents:
|
||||
|
||||
- **Read an MR**: `glab mr view <number> --comments` and `glab mr diff <number>` for the diff.
|
||||
- **List external MRs for triage**: `glab mr list -F json`, then keep only MRs whose author is not a project member/owner (a contributor's MR, not a maintainer's in-flight work).
|
||||
- **Comment / label / close**: `glab mr note`, `glab mr update --label`/`--unlabel`, `glab mr close`.
|
||||
|
||||
Unlike GitHub, GitLab numbers issues and MRs separately, so `#42` is unambiguous once you know which surface the maintainer means.
|
||||
|
||||
## When a skill says "publish to the issue tracker"
|
||||
|
||||
Create a GitLab issue.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: tdd
|
||||
description: Test-driven development with red-green-refactor loop. Use when user wants to build features or fix bugs using TDD, mentions "red-green-refactor", wants integration tests, or asks for test-first development.
|
||||
description: Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.
|
||||
---
|
||||
|
||||
# Test-Driven Development
|
||||
@@ -13,6 +13,8 @@ description: Test-driven development with red-green-refactor loop. Use when user
|
||||
|
||||
**Bad tests** are coupled to implementation. They mock internal collaborators, test private methods, or verify through external means (like querying a database directly instead of using the interface). The warning sign: your test breaks when you refactor, but behavior hasn't changed. If you rename an internal function and tests fail, those tests were testing implementation, not behavior.
|
||||
|
||||
**Tautological tests** restate the implementation inside the assertion, so they pass by construction and give zero confidence. When the expected value is computed the way the code computes it — `expect(add(a, b)).toBe(a + b)`, snapshotting a figure you derived by hand the same way the code does, asserting a constant equals itself — the test can never disagree with the code: break the code wrong and the assertion breaks wrong with it. The expected value must come from an independent source of truth — a known-good literal, a worked example, the spec.
|
||||
|
||||
See [tests.md](tests.md) for examples and [mocking.md](mocking.md) for mocking guidelines.
|
||||
|
||||
## Anti-Pattern: Horizontal Slices
|
||||
@@ -44,14 +46,13 @@ RIGHT (vertical):
|
||||
|
||||
### 1. Planning
|
||||
|
||||
When exploring the codebase, use the project's domain glossary so that test names and interface vocabulary match the project's language, and respect ADRs in the area you're touching.
|
||||
When exploring the codebase, read `CONTEXT.md` (if it exists) so that test names and interface vocabulary match the project's domain language, and respect ADRs in the area you're touching.
|
||||
|
||||
Before writing any code:
|
||||
|
||||
- [ ] Confirm with user what interface changes are needed
|
||||
- [ ] Confirm with user which behaviors to test (prioritize)
|
||||
- [ ] Identify opportunities for [deep modules](deep-modules.md) (small interface, deep implementation)
|
||||
- [ ] Design interfaces for [testability](interface-design.md)
|
||||
- [ ] Identify opportunities for deep modules (small interface, deep implementation) — run the `/codebase-design` skill for the vocabulary and the testability checks
|
||||
- [ ] List the behaviors to test (not implementation steps)
|
||||
- [ ] Get user approval on the plan
|
||||
|
||||
@@ -104,6 +105,7 @@ After all tests pass, look for [refactor candidates](refactoring.md):
|
||||
[ ] Test describes behavior, not implementation
|
||||
[ ] Test uses public interface only
|
||||
[ ] Test would survive internal refactor
|
||||
[ ] Expected values are independent literals, not recomputed from the code
|
||||
[ ] Code is minimal for this test
|
||||
[ ] No speculative features added
|
||||
```
|
||||
|
||||
@@ -1,33 +0,0 @@
|
||||
# Deep Modules
|
||||
|
||||
From "A Philosophy of Software Design":
|
||||
|
||||
**Deep module** = small interface + lots of implementation
|
||||
|
||||
```
|
||||
┌─────────────────────┐
|
||||
│ Small Interface │ ← Few methods, simple params
|
||||
├─────────────────────┤
|
||||
│ │
|
||||
│ │
|
||||
│ Deep Implementation│ ← Complex logic hidden
|
||||
│ │
|
||||
│ │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
**Shallow module** = large interface + little implementation (avoid)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────┐
|
||||
│ Large Interface │ ← Many methods, complex params
|
||||
├─────────────────────────────────┤
|
||||
│ Thin Implementation │ ← Just passes through
|
||||
└─────────────────────────────────┘
|
||||
```
|
||||
|
||||
When designing interfaces, ask:
|
||||
|
||||
- Can I reduce the number of methods?
|
||||
- Can I simplify the parameters?
|
||||
- Can I hide more complexity inside?
|
||||
@@ -1,31 +0,0 @@
|
||||
# Interface Design for Testability
|
||||
|
||||
Good interfaces make testing natural:
|
||||
|
||||
1. **Accept dependencies, don't create them**
|
||||
|
||||
```typescript
|
||||
// Testable
|
||||
function processOrder(order, paymentGateway) {}
|
||||
|
||||
// Hard to test
|
||||
function processOrder(order) {
|
||||
const gateway = new StripeGateway();
|
||||
}
|
||||
```
|
||||
|
||||
2. **Return results, don't produce side effects**
|
||||
|
||||
```typescript
|
||||
// Testable
|
||||
function calculateDiscount(cart): Discount {}
|
||||
|
||||
// Hard to test
|
||||
function applyDiscount(cart): void {
|
||||
cart.total -= discount;
|
||||
}
|
||||
```
|
||||
|
||||
3. **Small surface area**
|
||||
- Fewer methods = fewer tests needed
|
||||
- Fewer params = simpler test setup
|
||||
@@ -59,3 +59,19 @@ test("createUser makes user retrievable", async () => {
|
||||
expect(retrieved.name).toBe("Alice");
|
||||
});
|
||||
```
|
||||
|
||||
**Tautological tests**: Expected value restates the implementation, so the test passes by construction.
|
||||
|
||||
```typescript
|
||||
// BAD: Expected value is recomputed the way the code computes it
|
||||
test("calculateTotal sums line items", () => {
|
||||
const items = [{ price: 10 }, { price: 5 }];
|
||||
const expected = items.reduce((sum, i) => sum + i.price, 0);
|
||||
expect(calculateTotal(items)).toBe(expected);
|
||||
});
|
||||
|
||||
// GOOD: Expected value is an independent, known literal
|
||||
test("calculateTotal sums line items", () => {
|
||||
expect(calculateTotal([{ price: 10 }, { price: 5 }])).toBe(15);
|
||||
});
|
||||
```
|
||||
|
||||
@@ -16,6 +16,7 @@ Treat the current directory as a teaching workspace. The state of their learning
|
||||
- `RESOURCES.md`: A list of resources which can be explored to ground your teaching in contextual knowledge, or to acquire knowledge and wisdom. Use the format in [RESOURCES-FORMAT.md](./RESOURCES-FORMAT.md).
|
||||
- `./learning-records/*.md`: A directory of learning records, which capture what the user has learned. These are loosely equivalent to architectural decision records in software development - they capture non-obvious lessons and key insights that may need to be revised later, or drive future sessions. These should be used to calculate the zone of proximal development. They are titled `0001-<dash-case-name>.md`, where the number increments each time. Use the format in [LEARNING-RECORD-FORMAT.md](./LEARNING-RECORD-FORMAT.md).
|
||||
- `./lessons/*.html`: A directory of lessons. A **lesson** is a single, self-contained HTML output that teaches one tightly-scoped thing tied to the mission. This is the primary unit of teaching in this workspace.
|
||||
- `./assets/*`: Reusable **components** shared across lessons. See [Assets](#assets).
|
||||
- `NOTES.md`: A scratchpad for you to jot down user preferences, or working notes.
|
||||
|
||||
## Philosophy
|
||||
@@ -59,6 +60,14 @@ Each lesson should recommend a primary source for the user to read or watch. Thi
|
||||
|
||||
Each lesson should contain a reminder to ask followup questions to the agent. The agent is their teacher, and can assist with anything that's unclear.
|
||||
|
||||
## Assets
|
||||
|
||||
Lessons are built from reusable **components**, stored in `./assets/`: stylesheets, quiz widgets, simulators, diagram helpers — anything a second lesson could reuse.
|
||||
|
||||
Reuse is the default, not the exception. Before authoring a lesson, read `./assets/` and build from the components already there. When a lesson needs something new and reusable, write it as a component in `./assets/` and link to it — never inline code a future lesson would duplicate.
|
||||
|
||||
A shared stylesheet is the first component every workspace earns: every lesson links it, so the lessons look like one consistent course rather than a pile of one-offs. As the workspace grows, so should the component library.
|
||||
|
||||
## The Mission
|
||||
|
||||
Every lesson should be tied into the mission - the reason that the user is interested in learning about the topic.
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
---
|
||||
name: to-issues
|
||||
description: Break a plan, spec, or PRD into independently-grabbable issues on the project issue tracker using tracer-bullet vertical slices. Use when user wants to convert a plan into issues, create implementation tickets, or break down work into issues.
|
||||
description: Break a plan, spec, or PRD into independently-grabbable issues on the project issue tracker using tracer-bullet vertical slices.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# To Issues
|
||||
@@ -19,16 +20,18 @@ Work from whatever is already in the conversation context. If the user passes an
|
||||
|
||||
If you have not already explored the codebase, do so to understand the current state of the code. Issue titles and descriptions should use the project's domain glossary vocabulary, and respect ADRs in the area you're touching.
|
||||
|
||||
Look for opportunities to prefactor the code to make the implementation easier. "Make the change easy, then make the easy change."
|
||||
|
||||
### 3. Draft vertical slices
|
||||
|
||||
Break the plan into **tracer bullet** issues. Each issue is a thin vertical slice that cuts through ALL integration layers end-to-end, NOT a horizontal slice of one layer.
|
||||
|
||||
Slices may be 'HITL' or 'AFK'. HITL slices require human interaction, such as an architectural decision or a design review. AFK slices can be implemented and merged without human interaction. Prefer AFK over HITL where possible.
|
||||
|
||||
<vertical-slice-rules>
|
||||
|
||||
- Each slice delivers a narrow but COMPLETE path through every layer (schema, API, UI, tests)
|
||||
- A completed slice is demoable or verifiable on its own
|
||||
- Prefer many thin slices over few thick ones
|
||||
- Any prefactoring should be done first
|
||||
|
||||
</vertical-slice-rules>
|
||||
|
||||
### 4. Quiz the user
|
||||
@@ -36,7 +39,6 @@ Slices may be 'HITL' or 'AFK'. HITL slices require human interaction, such as an
|
||||
Present the proposed breakdown as a numbered list. For each slice, show:
|
||||
|
||||
- **Title**: short descriptive name
|
||||
- **Type**: HITL / AFK
|
||||
- **Blocked by**: which other slices (if any) must complete first
|
||||
- **User stories covered**: which user stories this addresses (if the source material has them)
|
||||
|
||||
@@ -45,7 +47,6 @@ Ask the user:
|
||||
- Does the granularity feel right? (too coarse / too fine)
|
||||
- Are the dependency relationships correct?
|
||||
- Should any slices be merged or split further?
|
||||
- Are the correct slices marked as HITL and AFK?
|
||||
|
||||
Iterate until the user approves the breakdown.
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
---
|
||||
name: to-prd
|
||||
description: Turn the current conversation context into a PRD and publish it to the project issue tracker. Use when user wants to create a PRD from the current context.
|
||||
description: Turn the current conversation into a PRD and publish it to the project issue tracker — no interview, just synthesis of what you've already discussed.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
This skill takes the current conversation context and codebase understanding and produces a PRD. Do NOT interview the user — just synthesize what you already know.
|
||||
@@ -11,7 +12,7 @@ The issue tracker and triage label vocabulary should have been provided to you
|
||||
|
||||
1. Explore the repo to understand the current state of the codebase, if you haven't already. Use the project's domain glossary vocabulary throughout the PRD, and respect any ADRs in the area you're touching.
|
||||
|
||||
2. Sketch out the seams at which you're going to test the feature. Existing seams should be preferred to new ones. Use the highest seam possible. If new seams are needed, propose them at the highest point you can.
|
||||
2. Sketch out the seams at which you're going to test the feature. Existing seams should be preferred to new ones. Use the highest seam possible. If new seams are needed, propose them at the highest point you can. The fewer seams across the codebase, the better - the ideal number is one.
|
||||
|
||||
Check with the user that these seams match their expectations.
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
# Writing Agent Briefs
|
||||
|
||||
An agent brief is a structured comment posted on a GitHub issue when it moves to `ready-for-agent`. It is the authoritative specification that an AFK agent will work from. The original issue body and discussion are context — the agent brief is the contract.
|
||||
An agent brief is a structured comment posted on a GitHub issue or PR when it moves to `ready-for-agent`. It is the authoritative specification that an AFK agent will work from. The original body and discussion are context — the agent brief is the contract.
|
||||
|
||||
The brief states **what the agent should do**, which stretches to both surfaces: for an issue, that's building the change from nothing; for a PR, it's what's left to do *to the existing diff* — finish it, close gaps, address review points. Same principles either way; the PR example below shows the difference.
|
||||
|
||||
## Principles
|
||||
|
||||
@@ -143,6 +145,43 @@ checked for matches.
|
||||
- Bug reports (only enhancement rejections go to `.out-of-scope/`)
|
||||
```
|
||||
|
||||
### Good agent brief (PR)
|
||||
|
||||
For a PR, "Current behavior" describes the state of the diff, and the brief asks the agent to finish or fix it rather than build from scratch.
|
||||
|
||||
```markdown
|
||||
## Agent Brief
|
||||
|
||||
**Category:** enhancement
|
||||
**Summary:** Finish the contributor's `--json` output flag for `triage list`
|
||||
|
||||
**Current behavior:**
|
||||
The PR adds a `--json` flag that serializes the issue list to JSON. The happy
|
||||
path works and the diff matches the project's command structure. Two gaps
|
||||
remain: errors are still printed as human text (not JSON), and the new flag has
|
||||
no test coverage.
|
||||
|
||||
**Desired behavior:**
|
||||
With `--json`, all output — including errors — is well-formed JSON on stdout,
|
||||
and the command's exit codes are unchanged. The existing human-readable output
|
||||
is untouched when the flag is absent.
|
||||
|
||||
**Key interfaces:**
|
||||
- The command's error path should emit `{ "error": string }` under `--json`
|
||||
instead of the plain-text error
|
||||
- Reuse the existing serializer the PR already added; don't introduce a second
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] `triage list --json` emits valid JSON for both success and error cases
|
||||
- [ ] Exit codes match the non-JSON command
|
||||
- [ ] A test covers the `--json` success output and one error case
|
||||
- [ ] Default (non-JSON) output is byte-for-byte unchanged
|
||||
|
||||
**Out of scope:**
|
||||
- Adding `--json` to any other command
|
||||
- Changing the JSON shape of the success payload the PR already defined
|
||||
```
|
||||
|
||||
### Bad agent brief
|
||||
|
||||
```markdown
|
||||
|
||||
@@ -83,7 +83,11 @@ The maintainer may:
|
||||
|
||||
## When to write to `.out-of-scope/`
|
||||
|
||||
Only when an **enhancement** (not a bug) is rejected as `wontfix`. The flow:
|
||||
Only when an **enhancement** (not a bug) is *rejected* as `wontfix`. This applies to enhancement PRs exactly as it does to issues — a rejected PR is recorded here so the same request doesn't return as fresh code.
|
||||
|
||||
Do **not** write here when something is closed as `wontfix` because it's **already implemented**. That's a built feature, not a rejected one; recording it would poison the dedup checks with false rejections. Instead, the closing comment points to where the feature already lives.
|
||||
|
||||
The flow:
|
||||
|
||||
1. Maintainer decides a feature request is out of scope
|
||||
2. Check if a matching `.out-of-scope/` file already exists
|
||||
|
||||
@@ -1,12 +1,15 @@
|
||||
---
|
||||
name: triage
|
||||
description: Triage issues through a state machine driven by triage roles. Use when user wants to create an issue, triage issues, review incoming bugs or feature requests, prepare issues for an AFK agent, or manage issue workflow.
|
||||
description: Move issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.
|
||||
disable-model-invocation: true
|
||||
---
|
||||
|
||||
# Triage
|
||||
|
||||
Move issues on the project issue tracker through a small state machine of triage roles.
|
||||
|
||||
If this repo treats external pull requests as a request surface (see the issue-tracker config), triage covers them too: **a PR is an issue with attached code** — same roles, same states, same machine, with a few deltas marked "for a PR" below. Resolve a bare `#42` to an issue or PR per the tracker config.
|
||||
|
||||
Every comment or issue posted to the issue tracker during triage **must** start with this disclaimer:
|
||||
|
||||
```
|
||||
@@ -33,6 +36,8 @@ Five **state** roles:
|
||||
- `ready-for-human` — needs human implementation
|
||||
- `wontfix` — will not be actioned
|
||||
|
||||
For a PR, the same states read against the attached code: `ready-for-agent` means a brief is attached and an agent should take the next step on the diff; `ready-for-human` means it's ready for a human to merge.
|
||||
|
||||
Every triaged issue should carry exactly one category role and one state role. If state roles conflict, flag it and ask the maintainer before doing anything else.
|
||||
|
||||
These are canonical role names — the actual label strings used in the issue tracker may differ. The mapping should have been provided to you - run `/setup-matt-pocock-skills` if not.
|
||||
@@ -44,7 +49,7 @@ State transitions: an unlabeled issue normally goes to `needs-triage` first; fro
|
||||
The maintainer invokes `/triage` and describes what they want in natural language. Interpret the request and act. Examples:
|
||||
|
||||
- "Show me anything that needs my attention"
|
||||
- "Let's look at #42"
|
||||
- "Let's look at #42" (issue or PR)
|
||||
- "Move #42 to ready-for-agent"
|
||||
- "What's ready for agents to pick up?"
|
||||
|
||||
@@ -56,24 +61,28 @@ Query the issue tracker and present three buckets, oldest first:
|
||||
2. **`needs-triage`** — evaluation in progress.
|
||||
3. **`needs-info` with reporter activity since the last triage notes** — needs re-evaluation.
|
||||
|
||||
Show counts and a one-line summary per issue. Let the maintainer pick.
|
||||
When PRs are in scope, include external PRs in these buckets and tag each line `[PR]` or `[issue]`. Discovery surfaces only *external* PRs (the tracker config defines who counts as external) — a collaborator's in-flight PR is not triage work. This filter is discovery-only; an explicitly named PR is always triaged regardless of author.
|
||||
|
||||
## Triage a specific issue
|
||||
Show counts and a one-line summary per item. Let the maintainer pick.
|
||||
|
||||
1. **Gather context.** Read the full issue (body, comments, labels, reporter, dates). Parse any prior triage notes so you don't re-ask resolved questions. Explore the codebase using the project's domain glossary, respecting ADRs in the area. Read `.out-of-scope/*.md` and surface any prior rejection that resembles this issue.
|
||||
## Triage a specific issue or PR
|
||||
|
||||
2. **Recommend.** Tell the maintainer your category and state recommendation with reasoning, plus a brief codebase summary relevant to the issue. Wait for direction.
|
||||
1. **Gather context.** Read the full issue or PR (body, comments, labels, author, dates; for a PR, the diff too). Parse any prior triage notes so you don't re-ask resolved questions. Explore the codebase using the project's domain glossary, respecting ADRs in the area. Run two checks against the codebase: (a) **redundancy** — search for an existing implementation of the requested behavior by domain concept (not just the request's wording), and report where you looked. If found, it's an already-implemented `wontfix` (step 5). (b) **prior rejection** — read `.out-of-scope/*.md` and surface any that resembles this request.
|
||||
|
||||
3. **Reproduce (bugs only).** Before any grilling, attempt reproduction: read the reporter's steps, trace the relevant code, run tests or commands. Report what happened — successful repro with code path, failed repro, or insufficient detail (a strong `needs-info` signal). A confirmed repro makes a much stronger agent brief.
|
||||
2. **Recommend.** Tell the maintainer your category and state recommendation with reasoning, plus a brief codebase summary relevant to the request — including whether it's already implemented. Wait for direction.
|
||||
|
||||
4. **Grill (if needed).** If the issue needs fleshing out, run a `/grill-with-docs` session.
|
||||
3. **Verify the claim.** Before any grilling, check that the claim holds up. For a bug, reproduce it from the reporter's steps. For a PR, confirm the diff does what it claims — check it out, run the relevant tests or commands. Report what happened: confirmed (with code path), failed, or insufficient detail (a strong `needs-info` signal). A confirmed verification makes a much stronger agent brief.
|
||||
|
||||
4. **Grill (if needed).** If the request needs fleshing out, run the `/grilling` and `/domain-modeling` skills together — grill it into shape one question at a time, sharpening domain terms and updating `CONTEXT.md`/ADRs inline as decisions land.
|
||||
|
||||
5. **Apply the outcome:**
|
||||
- `ready-for-agent` — post an agent brief comment ([AGENT-BRIEF.md](AGENT-BRIEF.md)).
|
||||
- `ready-for-human` — same structure as an agent brief, but note why it can't be delegated (judgment calls, external access, design decisions, manual testing).
|
||||
- `needs-info` — post triage notes (template below).
|
||||
- `wontfix` (bug) — polite explanation, then close.
|
||||
- `wontfix` (enhancement) — write to `.out-of-scope/`, link to it from a comment, then close ([OUT-OF-SCOPE.md](OUT-OF-SCOPE.md)).
|
||||
- `wontfix` — close, with the comment depending on *why*:
|
||||
- **Already implemented** — the change already exists in the codebase. Point to where it lives; do **not** write to `.out-of-scope/` (that KB is for *rejected* requests, not built ones).
|
||||
- **Rejected (bug)** — polite explanation, then close.
|
||||
- **Rejected (enhancement)** — write to `.out-of-scope/`, link to it from a comment, then close ([OUT-OF-SCOPE.md](OUT-OF-SCOPE.md)).
|
||||
- `needs-triage` — apply the role. Optional comment if there's partial progress.
|
||||
|
||||
## Quick state override
|
||||
@@ -100,4 +109,4 @@ Capture everything resolved during grilling under "established so far" so the wo
|
||||
|
||||
## Resuming a previous session
|
||||
|
||||
If prior triage notes exist on the issue, read them, check whether the reporter has answered any outstanding questions, and present an updated picture before continuing. Don't re-ask resolved questions.
|
||||
If prior triage notes exist on the issue or PR, read them, check whether the reporter has answered any outstanding questions, and present an updated picture before continuing. Don't re-ask resolved questions.
|
||||
|
||||
@@ -346,6 +346,19 @@ func startPollingScheduler(
|
||||
appLogger,
|
||||
)
|
||||
|
||||
// 注入套餐失效前流量同步器:最小化 iot_card.Service(只需 gateway + store + redis + 流量扣减)
|
||||
trafficSyncer := iot_card_svc.New(
|
||||
runtime.db,
|
||||
runtime.pollingIotCardStore,
|
||||
nil, nil, nil, nil, nil,
|
||||
runtime.gatewayClient,
|
||||
appLogger,
|
||||
nil,
|
||||
)
|
||||
trafficSyncer.SetRedisClient(runtime.redisClient)
|
||||
trafficSyncer.SetDataDeductor(runtime.workerResult.Services.UsageService)
|
||||
activationHandler.SetTrafficSyncer(trafficSyncer)
|
||||
|
||||
pollingScheduler := polling.NewScheduler(
|
||||
runtime.redisClient,
|
||||
runtime.asynqClient,
|
||||
|
||||
343
docs/改造方向.md
Normal file
343
docs/改造方向.md
Normal file
@@ -0,0 +1,343 @@
|
||||
# 系统改造方向
|
||||
|
||||
> 本文档记录当前系统的核心问题、改造原因、改造方向和预期收益。
|
||||
> 用于在开发过程中明确"干什么、为什么干、干的好处"。
|
||||
|
||||
---
|
||||
|
||||
## 一、现状
|
||||
|
||||
### 1.1 项目规模
|
||||
|
||||
- 当前在线卡数:约 3 万张(生产)
|
||||
- 待迁移卡数:约 67 万张(来自旧系统)
|
||||
- 核心业务链路:代理购卡 → 销售分配 → 套餐购买 → 流量同步 → 停复机
|
||||
|
||||
### 1.2 轮询系统现状
|
||||
|
||||
轮询系统的作用是**定期向运营商查询卡的状态**(实名状态、流量用量、卡网络状态等),保持系统数据与运营商同步。
|
||||
|
||||
当前架构:
|
||||
|
||||
```
|
||||
全量卡 ID → Redis Sorted Set(分片,score = 下次检查时间)
|
||||
→ Scheduler 每秒扫描到期卡
|
||||
→ Asynq 队列
|
||||
→ Worker 执行检查(实名/流量/卡状态/保护期/套餐,共 5 种类型)
|
||||
```
|
||||
|
||||
**问题清单:**
|
||||
|
||||
| # | 问题 | 具体表现 |
|
||||
|---|------|---------|
|
||||
| 1 | **看不见,无法排查** | 业务反馈"某张卡一小时了还没同步实名",没有任何结构化记录可以查,只能翻日志文件靠关键词过滤 |
|
||||
| 2 | **配置改了不知道有没有用** | 轮询配置对哪些卡生效完全黑盒,改完后没有验证手段,全靠等和猜 |
|
||||
| 3 | **大量无意义轮询** | 全量卡按固定频率轮询,大多数卡状态根本没有变化,这些查询全是浪费;67 万张卡迁移后压力更大 |
|
||||
| 4 | **业务事件不触发检查** | 代理通过开放接口买了套餐,系统不知道,要等下次定时轮询碰到这张卡(可能要等几分钟到几十分钟)|
|
||||
| 5 | **两种不同职责混在一个调度器** | 卡状态同步(每 1 秒触发)和业务流程调度(套餐过期处理、流量重置,每 10 秒触发)混在同一个 Scheduler,出问题难以定位是哪一层 |
|
||||
| 6 | **调度状态全在 Redis,没有持久化** | Redis 重启或清空后,全量卡的调度状态丢失,需要重跑全量初始化才能恢复;67 万张卡的初始化需要相当长时间,期间所有卡停止轮询 |
|
||||
| 7 | **单卡状态散在四处** | 上次检查时间在 IotCard 表;下次检查时间在 Redis Sorted Set;卡状态快照在 Redis Hash;执行结果在日志文件。没有一个地方能给完整答案 |
|
||||
|
||||
### 1.3 流量数据现状
|
||||
|
||||
运营商接口返回的是**当前账期内的累计已用流量**(不是增量,是总量)。
|
||||
运营商账期重置日因运营商而异,在创建运营商时配置。
|
||||
套餐周期可以是自然月或固定日期,在创建套餐时决定。
|
||||
|
||||
当前 IotCard 表上与流量相关的字段:
|
||||
|
||||
```
|
||||
DataUsageMB 卡生命周期总用量(永不归零)
|
||||
CurrentMonthUsageMB 系统自然月累计用量(展示用)
|
||||
LastMonthTotalMB 上月结束时的流量总量(用于跨月计算)
|
||||
CurrentMonthStartDate 系统自然月起始日期
|
||||
LastGatewayReadingMB 上次轮询时运营商返回的累计读数
|
||||
```
|
||||
|
||||
PackageUsage 表上:
|
||||
|
||||
```
|
||||
DataUsageMB 套餐已用量(套餐周期到期时归零,用于判断是否耗尽)
|
||||
```
|
||||
|
||||
**问题清单:**
|
||||
|
||||
| # | 问题 | 具体表现 |
|
||||
|---|------|---------|
|
||||
| 1 | **运营商重置靠猜,有漏检风险** | 只存上一次读数(`LastGatewayReadingMB`),靠"本次读数比上次小"来猜测运营商是否重置了账期。若轮询有空窗(例如系统故障几小时),重置后的新用量超过了上次读数,则增量为正,系统判断为"没有重置",漏掉了那段用量 |
|
||||
| 2 | **无原始读数,无法审计** | 运营商告诉了什么、什么时候告诉的、算出了多少增量——计算完成后全部丢弃。流量统计不对时,没有任何数据可以追溯原因 |
|
||||
| 3 | **重置边界的流量归属不准** | 套餐在午夜重置,但上次轮询在 23:59,下次轮询在 00:01。这两分钟的增量(包含昨天剩余用量)全部算进今天,日流量套餐受影响明显 |
|
||||
| 4 | **三套"月"的概念同时存在** | 运营商账期(各自定义重置日)+ 套餐周期(自然月或固定日)+ 系统自然月(展示用)。三者不对齐,边界处理散落在代码各处,逻辑复杂且脆弱 |
|
||||
| 5 | **`CurrentMonthUsageMB` 不按时重置** | 这个字段不由重置任务触发,而是在下次轮询时内联检测跨月并重置。1 号凌晨用户看到的数据是上月的,要等到下次轮询才更新 |
|
||||
| 6 | **两套日流量记录并存** | `CardDailyUsage`(按卡)和 `PackageUsageDailyRecord`(按套餐)同时存在,来源不同,权威性不明确,出现差异时不知道信谁 |
|
||||
|
||||
### 1.4 整体架构现状
|
||||
|
||||
当前分层:`Handler → Service → Store → Model`,分层存在但执行不彻底。
|
||||
|
||||
核心问题:**贫血模型 + 上帝 Service**
|
||||
|
||||
- Model 只是数据库映射,没有任何业务行为
|
||||
- 所有业务逻辑全部堆在 Service,导致 Service 极度臃肿
|
||||
|
||||
| Service | 行数 | 混杂的业务域 |
|
||||
|---------|------|------------|
|
||||
| order/service.go | 3031 行 | 订单创建、支付、佣金计算、代购、库存扣减 |
|
||||
| iot_card/service.go | 2310 行 | 卡管理、网关调用、停复机、生命周期、流量同步 |
|
||||
| device/service.go | 2078 行 | 设备管理、绑定、批量导入 |
|
||||
|
||||
没有领域事件:业务事件(套餐支付完成、卡实名成功)不产生任何可订阅的事件,所有下游动作全部通过直接方法调用耦合,新增联动逻辑必须修改原有代码。
|
||||
|
||||
---
|
||||
|
||||
## 二、改造方向(优先级一):轮询系统 + 流量数据
|
||||
|
||||
### 2.1 轮询系统改造
|
||||
|
||||
#### 改造 A:建立单卡可查询的轮询状态
|
||||
|
||||
**要干什么:** 新增一张表 `card_polling_state`,每张卡一行,只更新不追加,记录每种检查类型的上次执行时间、结果、命中配置、下次预计时间。
|
||||
|
||||
**为什么干:** 现在"这张卡的轮询状态"散在四个地方,没有一处能完整回答问题。这张表就是单一的事实来源。
|
||||
|
||||
**好处:**
|
||||
- 业务反馈某张卡没同步 → 查这张表,30 秒给出答案(上次检查时间、结果是什么、下次什么时候)
|
||||
- 不影响任何现有轮询逻辑,只需在执行检查后多写一次 UPDATE
|
||||
|
||||
**同时新增两个接口:**
|
||||
|
||||
```
|
||||
GET /admin/polling/cards/:id/state
|
||||
→ 返回这张卡的完整轮询状态
|
||||
|
||||
GET /admin/polling/preview?card_id=12345
|
||||
→ 返回这张卡当前命中哪条配置、各检查类型的间隔是多少、下次检查时间
|
||||
```
|
||||
|
||||
第二个接口解决"改了配置不知道有没有用"的问题——改完配置后,查任意一张卡,立刻知道新配置有没有生效。
|
||||
|
||||
---
|
||||
|
||||
#### 改造 B:引入事件触发层
|
||||
|
||||
**要干什么:** 在业务动作完成时,主动安排针对性的延迟检查,而不是等定时轮询碰到这张卡。
|
||||
|
||||
**为什么干:** 现在买了套餐,系统要等下次轮询才知道套餐有没有激活。运营商激活需要时间,但我们不知道什么时候激活完成,只能等碰到。
|
||||
|
||||
**好处:**
|
||||
- 套餐购买 → 5 分钟后主动检查 → 激活状态及时同步,不用等下次定时轮询
|
||||
- 轮询有了"意义"——有事件发生才检查,而不是盲目扫全量
|
||||
|
||||
**触发规则:**
|
||||
|
||||
| 业务事件 | 触发检查 | 延迟 |
|
||||
|---------|---------|------|
|
||||
| 套餐订单支付完成 | 检查套餐激活状态 | 5 分钟 |
|
||||
| 代理开放接口购套餐 | 检查套餐激活状态 | 5 分钟 |
|
||||
| 卡首次分配给店铺 | 检查实名状态 | 30 分钟 |
|
||||
| 手动触发刷新 | 立即检查 | 0 |
|
||||
|
||||
实现方式:利用 Asynq 的定时任务功能(`asynq.ProcessAt`),在现有 Worker 框架内完成,不需要新增基础设施。
|
||||
|
||||
---
|
||||
|
||||
#### 改造 C:按卡状态分级调度
|
||||
|
||||
**要干什么:** 不同状态的卡,轮询频率不同,而不是所有卡一视同仁。
|
||||
|
||||
**为什么干:** 稳定运行了半年、没有套餐变动的卡,每 2 分钟轮询一次是纯粹的浪费。等实名的卡、刚购套餐的卡才需要频繁检查。
|
||||
|
||||
**好处:**
|
||||
- 67 万张卡迁移后,无意义轮询量大幅下降(估计减少 60%-80%)
|
||||
- 轮询资源集中在真正需要关注的卡
|
||||
|
||||
**分级规则(通过现有 PollingConfig 机制配置):**
|
||||
|
||||
| 卡状态 | 实名检查间隔 | 流量检查间隔 |
|
||||
|-------|------------|------------|
|
||||
| 等待实名 | 10 分钟 | 30 分钟 |
|
||||
| 活跃中(近期有套餐变动)| 15 分钟 | 30 分钟 |
|
||||
| 稳定运行 | 2 小时 | 2 小时 |
|
||||
| 停机 / 停用 | 每日一次 | 每日一次 |
|
||||
|
||||
---
|
||||
|
||||
#### 改造 D:分离业务调度
|
||||
|
||||
**要干什么:** 把套餐过期处理、流量重置从 `Scheduler` 里拆出去,用独立的定时任务管理。
|
||||
|
||||
**为什么干:** 现在卡状态同步(每 1 秒)和业务逻辑调度(每 10 秒)混在同一个 Scheduler 里。出了问题不知道是哪一层,改动也容易互相影响。
|
||||
|
||||
**好处:**
|
||||
- 各司其职,互不干扰
|
||||
- 业务调度(套餐过期)出问题不影响卡状态同步
|
||||
- 代码更清晰,新增调度任务不需要修改 Scheduler
|
||||
|
||||
---
|
||||
|
||||
### 2.2 流量数据改造
|
||||
|
||||
#### 改造 A:保留原始读数
|
||||
|
||||
**要干什么:** 新增 `carrier_readings` 表,每次从运营商拿到读数都记录下来,永不修改。
|
||||
|
||||
```
|
||||
carrier_readings:
|
||||
card_id 卡ID
|
||||
reading_mb 运营商返回的当期累计值
|
||||
read_at 读取时间
|
||||
cycle_start_date 本条读数所属的运营商账期开始日期
|
||||
source 来源(polling / manual / triggered)
|
||||
```
|
||||
|
||||
**为什么干:** 现在计算完增量后,原始读数就丢弃了。流量统计不对时,没有任何数据可以追溯。
|
||||
|
||||
**好处:**
|
||||
- 流量统计有问题 → 查这张表,还原每次拿到了什么数据、算出了多少增量
|
||||
- 不再是"只有最终结果,不知道过程"
|
||||
- 可以离线重算历史数据,修复过去的错误
|
||||
|
||||
**存储量控制:** 保留 30 天原始记录,超过 30 天只保留"状态变化"事件(数量极少)。分区表实现,不影响查询性能。
|
||||
|
||||
---
|
||||
|
||||
#### 改造 B:基于账期的重置检测
|
||||
|
||||
**要干什么:** 用 `cycle_start_date` 字段判断运营商是否重置了账期,替代现在靠"增量为负"来猜测的方式。
|
||||
|
||||
**为什么干:** 现在的猜测方式有漏检风险——如果轮询空窗期间运营商重置且新用量超过了旧读数,增量为正,系统无法检测到重置,漏计那段用量。
|
||||
|
||||
**好处:**
|
||||
- 重置检测从"猜"变成"判断",准确率 100%
|
||||
- 彻底消除因轮询空窗导致的流量漏计
|
||||
- 代码逻辑更简单,不再需要"重置窗口"时间判断
|
||||
|
||||
```go
|
||||
// 改前:靠负增量猜测,有漏检风险
|
||||
if increment < 0 && isRefreshResetWindow(now, resetDay) { ... }
|
||||
|
||||
// 改后:直接判断账期,准确无误
|
||||
if prev.CycleStartDate != curr.CycleStartDate {
|
||||
increment = curr.ReadingMB // 不同账期 = 运营商重置,当前读数即为增量
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### 改造 C:清理 IotCard 上的流量计算字段
|
||||
|
||||
**要干什么:** 将 `LastGatewayReadingMB`、`CurrentMonthStartDate`、`LastMonthTotalMB` 从 IotCard 表移除,这些信息改由 `carrier_readings` 表承载。
|
||||
|
||||
IotCard 只保留两个对外展示的汇总值:
|
||||
- `CurrentMonthUsageMB`:展示用,由定时任务按自然月计算并主动更新(不再等轮询时内联检测)
|
||||
- `DataUsageMB`:生命周期总量,由增量事件累加
|
||||
|
||||
**为什么干:** 现在 IotCard 同时承担了三个角色:卡的身份信息、流量计数器、流量计算中间状态。职责太多,一张表改动牵一发动全身。
|
||||
|
||||
**好处:**
|
||||
- IotCard 回归本职:描述这张卡是谁、状态是什么
|
||||
- 流量计算有了独立的数据层,不污染主体数据
|
||||
- `CurrentMonthUsageMB` 在月初会被主动更新,不再有"1号凌晨数据是上月的"问题
|
||||
|
||||
---
|
||||
|
||||
## 三、改造方向(优先级二):DDD 整体方向
|
||||
|
||||
> 这部分是更长期的方向,在优先级一完成后推进。
|
||||
|
||||
### 3.1 核心思想
|
||||
|
||||
**积木本身是安全的,拼积木组成业务流程。**
|
||||
|
||||
- **积木** = 领域对象(`IotCard`、`Order`、`Wallet`):自己知道自己的规则,外部只能通过方法操作,不能随意改字段
|
||||
- **拼积木** = 应用服务(用例):只负责"拿数据 → 调对象方法 → 保存 → 发事件",不含业务判断逻辑
|
||||
|
||||
**AI 辅助开发的约束价值:** 有了这个结构,AI 生成代码时只能往"一个用例一个文件"的模式里填,不会生成新的上帝 Service;业务规则在领域对象里,AI 调用对象方法时规则自动生效。
|
||||
|
||||
### 3.2 目标目录结构
|
||||
|
||||
```
|
||||
internal/
|
||||
├── domain/ 领域层(纯业务逻辑,不依赖任何框架)
|
||||
│ ├── order/
|
||||
│ │ ├── order.go 聚合根(Order 实体,有方法:Pay, Cancel, Reject)
|
||||
│ │ ├── events.go 领域事件(OrderPaid, OrderCancelled)
|
||||
│ │ ├── repository.go 仓储接口(interface,不含实现)
|
||||
│ │ └── values.go 值对象(OrderStatus, Amount)
|
||||
│ ├── asset/
|
||||
│ │ ├── iot_card.go IotCard 聚合根
|
||||
│ │ └── repository.go
|
||||
│ └── wallet/
|
||||
│ └── wallet.go Wallet 聚合根(Debit, Credit)
|
||||
│
|
||||
├── application/ 应用层(用例,只做编排)
|
||||
│ ├── order/
|
||||
│ │ ├── create_order.go 一个文件一个用例
|
||||
│ │ ├── pay_order.go
|
||||
│ │ └── cancel_order.go
|
||||
│ └── asset/
|
||||
│
|
||||
├── infrastructure/ 基础设施层(实现接口)
|
||||
│ ├── persistence/ 原来的 store/postgres/
|
||||
│ └── event/ 事件总线
|
||||
│
|
||||
└── interface/ 接口层
|
||||
├── http/ 原来的 handler/ + routes/
|
||||
└── task/ Asynq task handlers
|
||||
```
|
||||
|
||||
### 3.3 迁移策略(绞杀者模式)
|
||||
|
||||
不全量重写,新旧并行,逐步替换:
|
||||
|
||||
1. 建好 `domain/`、`application/` 目录和规范文件
|
||||
2. **所有新功能按新结构写**,不往旧 Service 里加代码
|
||||
3. 最痛的模块优先迁移:`order`(3031 行)和 `iot_card`(2310 行)
|
||||
4. 每次迁移一个用例,验证跑通后再迁移下一个
|
||||
|
||||
---
|
||||
|
||||
## 四、实施顺序
|
||||
|
||||
```
|
||||
第一阶段(当前)
|
||||
├── 新增 card_polling_state 表(轮询状态持久化)
|
||||
├── 新增轮询状态查询接口 + 配置预览接口(可观测性)
|
||||
├── 新增 carrier_readings 表(保留原始读数)
|
||||
├── 改造增量计算逻辑(基于账期判断重置)
|
||||
└── 事件触发层(套餐购买 → 延迟检查)
|
||||
|
||||
第二阶段
|
||||
├── 分级调度(按卡状态动态调整轮询频率)
|
||||
├── 分离业务调度(套餐过期/流量重置从 Scheduler 独立)
|
||||
└── 清理 IotCard 流量计算字段
|
||||
|
||||
第三阶段(DDD 改造)
|
||||
├── 建立目录骨架和规范示例
|
||||
├── 新功能按新结构写
|
||||
└── 逐步迁移最痛的模块
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、预期收益
|
||||
|
||||
### 轮询 + 流量改造后
|
||||
|
||||
| 场景 | 改造前 | 改造后 |
|
||||
|------|--------|--------|
|
||||
| 业务反馈"某张卡没同步" | 翻日志,靠运气,可能花几十分钟 | 查接口,30 秒内给出答案 |
|
||||
| 验证配置是否生效 | 完全黑盒,改完只能等和猜 | 配置预览接口,改完立即验证 |
|
||||
| 流量统计不对 | 无法追溯,无从下手 | 查原始读数,还原每次增量计算 |
|
||||
| 运营商重置漏计 | 有风险,轮询空窗时必现 | 账期判断,100% 准确 |
|
||||
| 套餐购买后同步延迟 | 等下次定时轮询,最多几十分钟 | 5 分钟内主动检查 |
|
||||
| 67 万卡迁移后轮询压力 | 全量扫描,压力线性增长 | 分级 + 事件触发,压力可控 |
|
||||
|
||||
### DDD 改造后
|
||||
|
||||
| 场景 | 改造前 | 改造后 |
|
||||
|------|--------|--------|
|
||||
| AI 辅助开发新功能 | 容易生成新的上帝 Service | 遵循用例模式,每次只加一个文件 |
|
||||
| 新增业务联动逻辑 | 必须修改原有 Service 代码 | 新增事件监听器,原有代码不动 |
|
||||
| 定位业务规则在哪里 | 散在几千行 Service 里找 | 在对应的领域对象方法里 |
|
||||
| 排查某个流程的问题 | 跨多个 Service 追调用链 | 用例文件即流程,一目了然 |
|
||||
@@ -56,12 +56,13 @@ func (h *AssetHandler) SetLifecycleService(svc AssetLifecycleService) {
|
||||
}
|
||||
|
||||
// resolveByIdentifier 通过标识符解析资产,用于各处理器的公共逻辑
|
||||
// 不传递 include_usage_summary,始终为 false,不影响其他 Handler 性能
|
||||
func (h *AssetHandler) resolveByIdentifier(c *fiber.Ctx) (*dto.AssetResolveResponse, error) {
|
||||
identifier := c.Params("identifier")
|
||||
if identifier == "" {
|
||||
return nil, errors.New(errors.CodeInvalidParam, "标识符不能为空")
|
||||
}
|
||||
return h.assetService.Resolve(c.UserContext(), identifier)
|
||||
return h.assetService.Resolve(c.UserContext(), identifier, false)
|
||||
}
|
||||
|
||||
// Resolve 通过任意标识符解析资产(设备或卡)
|
||||
@@ -72,7 +73,8 @@ func (h *AssetHandler) Resolve(c *fiber.Ctx) error {
|
||||
return errors.New(errors.CodeInvalidParam, "标识符不能为空")
|
||||
}
|
||||
|
||||
result, err := h.assetService.Resolve(c.UserContext(), identifier)
|
||||
includeUsageSummary := c.QueryBool("include_usage_summary", false)
|
||||
result, err := h.assetService.Resolve(c.UserContext(), identifier, includeUsageSummary)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -378,7 +380,7 @@ func (h *AssetHandler) UpdatePollingStatus(c *fiber.Ctx) error {
|
||||
return errors.New(errors.CodeInvalidParam)
|
||||
}
|
||||
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier)
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -424,7 +426,7 @@ func (h *AssetHandler) UpdateRealnamePolicy(c *fiber.Ctx) error {
|
||||
return errors.New(errors.CodeInvalidParam, "无效的实名认证策略")
|
||||
}
|
||||
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier)
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -467,7 +469,7 @@ func (h *AssetHandler) UpdateRealnameStatus(c *fiber.Ctx) error {
|
||||
return errors.New(errors.CodeInvalidParam, "无效的实名状态")
|
||||
}
|
||||
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier)
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -46,7 +46,7 @@ func (h *AssetWalletHandler) GetWallet(c *fiber.Ctx) error {
|
||||
return errors.New(errors.CodeInternalError, "资产服务未配置")
|
||||
}
|
||||
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier)
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -76,7 +76,7 @@ func (h *AssetWalletHandler) ListTransactions(c *fiber.Ctx) error {
|
||||
return errors.New(errors.CodeInternalError, "资产服务未配置")
|
||||
}
|
||||
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier)
|
||||
asset, err := h.assetService.Resolve(c.UserContext(), identifier, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -98,7 +98,7 @@ func (h *ClientAssetHandler) resolveAssetFromIdentifier(c *fiber.Ctx, identifier
|
||||
}
|
||||
|
||||
skipPermissionCtx := context.WithValue(c.UserContext(), constants.ContextKeySubordinateShopIDs, []uint{})
|
||||
assetInfo, err := h.assetService.Resolve(skipPermissionCtx, identifier)
|
||||
assetInfo, err := h.assetService.Resolve(skipPermissionCtx, identifier, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -69,7 +69,7 @@ func (h *ClientDeviceHandler) validateDeviceAsset(c *fiber.Ctx, identifier strin
|
||||
ctx := c.UserContext()
|
||||
|
||||
// 通过标识符解析资产
|
||||
asset, err := h.assetService.Resolve(ctx, identifier)
|
||||
asset, err := h.assetService.Resolve(ctx, identifier, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -84,7 +84,7 @@ func (h *ClientRealnameHandler) GetRealnameLink(c *fiber.Ctx) error {
|
||||
ctx := c.UserContext()
|
||||
|
||||
// 3. 通过标识符解析资产
|
||||
asset, err := h.assetService.Resolve(ctx, req.Identifier)
|
||||
asset, err := h.assetService.Resolve(ctx, req.Identifier, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -279,6 +279,18 @@ func (h *ClientWalletHandler) CreateRecharge(c *fiber.Ctx) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// 第三方支付方式与资产类型必须匹配:单卡只允许支付宝,设备只允许微信(已暂时注释)
|
||||
// switch resolved.Asset.AssetType {
|
||||
// case "card":
|
||||
// if req.PaymentMethod != constants.RechargeMethodAlipay {
|
||||
// return errors.New(errors.CodeInvalidParam, "单卡资产只支持支付宝充值")
|
||||
// }
|
||||
// case "device":
|
||||
// if req.PaymentMethod != constants.RechargeMethodWechat {
|
||||
// return errors.New(errors.CodeInvalidParam, "设备资产只支持微信充值")
|
||||
// }
|
||||
// }
|
||||
|
||||
config, err := h.wechatConfigService.GetActiveConfig(resolved.SkipPermissionCtx)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -540,7 +552,7 @@ func (h *ClientWalletHandler) resolveAssetFromIdentifier(c *fiber.Ctx, identifie
|
||||
}
|
||||
|
||||
skipPermissionCtx := context.WithValue(c.UserContext(), constants.ContextKeySubordinateShopIDs, []uint{})
|
||||
assetInfo, err := h.assetService.Resolve(skipPermissionCtx, identifier)
|
||||
assetInfo, err := h.assetService.Resolve(skipPermissionCtx, identifier, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -68,6 +68,9 @@ type AssetResolveResponse struct {
|
||||
NetworkStatus int `json:"network_status" description:"网络状态:0停机 1开机(asset_type=card时有效)"`
|
||||
GatewayExtend string `json:"gateway_extend" description:"Gateway 卡状态扩展字段,原样返回上游 extend,用于展示运营商侧实际停机原因"`
|
||||
GatewayCardIMEI string `json:"gateway_card_imei" description:"插拔卡业务 IMEI(由 Gateway 卡状态接口同步,非设备自身 IMEI,无业务含义,仅供查看)"`
|
||||
// 流量汇总字段(仅在 ?include_usage_summary=true 时返回非 null 值)
|
||||
TotalVirtualUsedMB *float64 `json:"total_virtual_used_mb" description:"当前世代所有套餐虚已用之和(MB),未请求时为 null"`
|
||||
TotalVirtualRemainingMB *float64 `json:"total_virtual_remaining_mb" description:"当前世代所有套餐虚剩余之和(MB),未请求时为 null"`
|
||||
}
|
||||
|
||||
// BoundCardInfo 设备绑定的卡信息
|
||||
@@ -148,9 +151,10 @@ type AssetPackagesResult struct {
|
||||
Items []*AssetPackageResponse `json:"items" description:"套餐列表"`
|
||||
}
|
||||
|
||||
// AssetResolveRequest 资产解析请求(路径参数)
|
||||
// AssetResolveRequest 资产解析请求
|
||||
type AssetResolveRequest struct {
|
||||
Identifier string `path:"identifier" description:"资产标识符(虚拟号/ICCID/IMEI/SN/MSISDN)" required:"true"`
|
||||
Identifier string `path:"identifier" description:"资产标识符(虚拟号/ICCID/IMEI/SN/MSISDN)" required:"true"`
|
||||
IncludeUsageSummary bool `query:"include_usage_summary" description:"是否返回当前世代流量汇总字段(total_virtual_used_mb / total_virtual_remaining_mb),默认 false"`
|
||||
}
|
||||
|
||||
// AssetIdentifierRequest 资产标识符路径参数请求
|
||||
|
||||
@@ -17,6 +17,12 @@ import (
|
||||
"github.com/break/junhong_cmp_fiber/pkg/errors"
|
||||
)
|
||||
|
||||
// TrafficSyncer 套餐失效前流量同步接口
|
||||
// 在主套餐标记为已过期之前,从 Gateway 拉取最新流量写入 DB
|
||||
type TrafficSyncer interface {
|
||||
RefreshCardDataByID(ctx context.Context, cardID uint) error
|
||||
}
|
||||
|
||||
// PackageActivationHandler 套餐激活检查处理器
|
||||
// 任务 19: 处理主套餐过期、加油包级联失效、待生效主套餐激活
|
||||
type PackageActivationHandler struct {
|
||||
@@ -27,6 +33,7 @@ type PackageActivationHandler struct {
|
||||
deviceSimBinding *postgres.DeviceSimBindingStore
|
||||
activationService *packagepkg.ActivationService
|
||||
stopResumeCallback packagepkg.StopResumeCallback // 停复机回调,用于套餐过期后主动停机
|
||||
trafficSyncer TrafficSyncer // 套餐失效前流量同步,可选
|
||||
logger *zap.Logger
|
||||
}
|
||||
|
||||
@@ -60,6 +67,48 @@ func NewPackageActivationHandler(
|
||||
}
|
||||
}
|
||||
|
||||
// SetTrafficSyncer 注入套餐失效前流量同步器(在 Start 前调用)
|
||||
func (h *PackageActivationHandler) SetTrafficSyncer(syncer TrafficSyncer) {
|
||||
h.trafficSyncer = syncer
|
||||
}
|
||||
|
||||
// syncTrafficBeforeExpiry 套餐失效前从 Gateway 同步一次最新流量
|
||||
// iot_card 直接同步;device 遍历所有绑定卡同步
|
||||
// 失败只记录警告,不阻断失效流程
|
||||
func (h *PackageActivationHandler) syncTrafficBeforeExpiry(ctx context.Context, carrierType string, carrierID uint) {
|
||||
if h.trafficSyncer == nil {
|
||||
return
|
||||
}
|
||||
|
||||
if carrierType == constants.AssetTypeIotCard {
|
||||
if err := h.trafficSyncer.RefreshCardDataByID(ctx, carrierID); err != nil {
|
||||
h.logger.Warn("套餐失效前流量同步失败",
|
||||
zap.String("carrier_type", carrierType),
|
||||
zap.Uint("carrier_id", carrierID),
|
||||
zap.Error(err))
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if carrierType == constants.AssetTypeDevice {
|
||||
bindings, err := h.deviceSimBinding.ListByDeviceID(ctx, carrierID)
|
||||
if err != nil {
|
||||
h.logger.Warn("套餐失效前查询设备绑定卡失败",
|
||||
zap.Uint("device_id", carrierID),
|
||||
zap.Error(err))
|
||||
return
|
||||
}
|
||||
for _, b := range bindings {
|
||||
if err := h.trafficSyncer.RefreshCardDataByID(ctx, b.IotCardID); err != nil {
|
||||
h.logger.Warn("套餐失效前流量同步失败(设备绑定卡)",
|
||||
zap.Uint("device_id", carrierID),
|
||||
zap.Uint("card_id", b.IotCardID),
|
||||
zap.Error(err))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// HandlePackageActivationCheck 任务 19.2-19.5: 处理套餐激活检查
|
||||
// 每 10 秒调度一次,检查过期主套餐并激活下一个待生效主套餐
|
||||
func (h *PackageActivationHandler) HandlePackageActivationCheck(ctx context.Context) error {
|
||||
@@ -145,20 +194,21 @@ func (h *PackageActivationHandler) findAndActivateOrphanPackages(ctx context.Con
|
||||
continue
|
||||
}
|
||||
|
||||
// 检查该载体是否已有生效主套餐
|
||||
// 检查该载体是否已有占位主套餐(生效中或已用完均视为占位,不允许激活待生效套餐)
|
||||
var activeCount int64
|
||||
var countErr error
|
||||
occupiedStatuses := []int{constants.PackageUsageStatusActive, constants.PackageUsageStatusDepleted}
|
||||
if key.carrierType == "iot_card" {
|
||||
countErr = h.db.WithContext(ctx).
|
||||
Model(&model.PackageUsage{}).
|
||||
Where("status = ?", constants.PackageUsageStatusActive).
|
||||
Where("status IN ?", occupiedStatuses).
|
||||
Where("master_usage_id IS NULL").
|
||||
Where("iot_card_id = ?", key.carrierID).
|
||||
Count(&activeCount).Error
|
||||
} else {
|
||||
countErr = h.db.WithContext(ctx).
|
||||
Model(&model.PackageUsage{}).
|
||||
Where("status = ?", constants.PackageUsageStatusActive).
|
||||
Where("status IN ?", occupiedStatuses).
|
||||
Where("master_usage_id IS NULL").
|
||||
Where("device_id = ?", key.carrierID).
|
||||
Count(&activeCount).Error
|
||||
@@ -171,7 +221,7 @@ func (h *PackageActivationHandler) findAndActivateOrphanPackages(ctx context.Con
|
||||
continue
|
||||
}
|
||||
|
||||
// 已有生效套餐,跳过
|
||||
// 已有生效或已用完(占位)套餐,跳过
|
||||
if activeCount > 0 {
|
||||
continue
|
||||
}
|
||||
@@ -219,10 +269,15 @@ func (h *PackageActivationHandler) findExpiredMainPackages(ctx context.Context)
|
||||
}
|
||||
|
||||
// processExpiredPackage 处理单个过期套餐
|
||||
// triggerStopAfterExpiry 在事务提交后才调用,避免 goroutine 在 TX 提交前读到脏数据
|
||||
// 流程:先同步最新流量 → 事务内标记过期/失效/激活下一个 → 事务提交后触发停机
|
||||
func (h *PackageActivationHandler) processExpiredPackage(ctx context.Context, pkg *model.PackageUsage) error {
|
||||
carrierType, carrierID := h.getCarrierInfo(pkg)
|
||||
|
||||
// 失效前先从 Gateway 同步一次最新流量,确保 data_usage_mb 精确
|
||||
if carrierType != "" && carrierID > 0 {
|
||||
h.syncTrafficBeforeExpiry(ctx, carrierType, carrierID)
|
||||
}
|
||||
|
||||
err := h.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
// 任务 19.3: 更新过期主套餐状态为 Expired (status=3)
|
||||
if err := tx.Model(pkg).Update("status", constants.PackageUsageStatusExpired).Error; err != nil {
|
||||
|
||||
@@ -484,7 +484,7 @@ func (s *Service) resolveOpenAPICard(ctx context.Context, cardNo string) (*model
|
||||
return nil, errors.New(errors.CodeInternalError, "开放接口资产解析依赖未配置")
|
||||
}
|
||||
|
||||
resolved, err := s.assetService.Resolve(ctx, identifier)
|
||||
resolved, err := s.assetService.Resolve(ctx, identifier, false)
|
||||
if err != nil {
|
||||
if appErr, ok := err.(*errors.AppError); ok && appErr.Code == errors.CodeNotFound {
|
||||
return s.resolveOpenAPICardByIdentifier(ctx, identifier)
|
||||
|
||||
@@ -91,7 +91,8 @@ func New(
|
||||
// Resolve 通过任意标识符解析资产
|
||||
// 主路径:查注册表(精确匹配 ICCID 或 VirtualNo)
|
||||
// Fallback:原有跨表 OR 查询(处理 IMEI/SN/MSISDN 等非注册标识符)
|
||||
func (s *Service) Resolve(ctx context.Context, identifier string) (*dto.AssetResolveResponse, error) {
|
||||
// includeUsageSummary 为 true 时额外填充当前世代流量汇总字段
|
||||
func (s *Service) Resolve(ctx context.Context, identifier string, includeUsageSummary bool) (*dto.AssetResolveResponse, error) {
|
||||
if s.assetIdentifierStore != nil {
|
||||
regRecord, regErr := s.assetIdentifierStore.FindByIdentifier(ctx, identifier)
|
||||
if regErr == nil && regRecord != nil {
|
||||
@@ -102,6 +103,9 @@ func (s *Service) Resolve(ctx context.Context, identifier string) (*dto.AssetRes
|
||||
resp, buildErr := s.buildDeviceResolveResponse(ctx, device)
|
||||
if buildErr == nil {
|
||||
resp.Identifier = identifier
|
||||
if includeUsageSummary {
|
||||
s.fillUsageSummary(ctx, resp, "device", device.ID)
|
||||
}
|
||||
}
|
||||
return resp, buildErr
|
||||
}
|
||||
@@ -111,6 +115,9 @@ func (s *Service) Resolve(ctx context.Context, identifier string) (*dto.AssetRes
|
||||
resp, buildErr := s.buildCardResolveResponse(ctx, card)
|
||||
if buildErr == nil {
|
||||
resp.Identifier = identifier
|
||||
if includeUsageSummary {
|
||||
s.fillUsageSummary(ctx, resp, "iot_card", card.ID)
|
||||
}
|
||||
}
|
||||
return resp, buildErr
|
||||
}
|
||||
@@ -123,6 +130,9 @@ func (s *Service) Resolve(ctx context.Context, identifier string) (*dto.AssetRes
|
||||
resp, buildErr := s.buildDeviceResolveResponse(ctx, device)
|
||||
if buildErr == nil {
|
||||
resp.Identifier = identifier
|
||||
if includeUsageSummary {
|
||||
s.fillUsageSummary(ctx, resp, "device", device.ID)
|
||||
}
|
||||
}
|
||||
return resp, buildErr
|
||||
}
|
||||
@@ -135,6 +145,9 @@ func (s *Service) Resolve(ctx context.Context, identifier string) (*dto.AssetRes
|
||||
resp, buildErr := s.buildCardResolveResponse(ctx, &card)
|
||||
if buildErr == nil {
|
||||
resp.Identifier = identifier
|
||||
if includeUsageSummary {
|
||||
s.fillUsageSummary(ctx, resp, "iot_card", card.ID)
|
||||
}
|
||||
}
|
||||
return resp, buildErr
|
||||
}
|
||||
@@ -336,6 +349,19 @@ func (s *Service) fillPackageInfo(ctx context.Context, resp *dto.AssetResolveRes
|
||||
resp.EnableVirtualData = usage.EnableVirtualDataSnapshot
|
||||
}
|
||||
|
||||
// fillUsageSummary 填充当前世代流量汇总字段。
|
||||
// 仅在 includeUsageSummary=true 时被调用,无套餐时返回 0.0(非 null)。
|
||||
func (s *Service) fillUsageSummary(ctx context.Context, resp *dto.AssetResolveResponse, carrierType string, carrierID uint) {
|
||||
usages, err := s.packageUsageStore.GetCurrentGenerationUsagesForSummary(ctx, carrierType, carrierID)
|
||||
if err != nil {
|
||||
logger.GetAppLogger().Error("查询流量汇总失败", zap.String("carrierType", carrierType), zap.Uint("carrierID", carrierID), zap.Error(err))
|
||||
return
|
||||
}
|
||||
totalUsed, totalRemaining := computeUsageSummary(usages)
|
||||
resp.TotalVirtualUsedMB = &totalUsed
|
||||
resp.TotalVirtualRemainingMB = &totalRemaining
|
||||
}
|
||||
|
||||
// fillShopName 填充店铺名称
|
||||
func (s *Service) fillShopName(ctx context.Context, resp *dto.AssetResolveResponse) {
|
||||
if resp.ShopID == nil || *resp.ShopID == 0 {
|
||||
@@ -971,7 +997,7 @@ func (s *Service) GetOrders(ctx context.Context, identifier string, page, pageSi
|
||||
pageSize = 100
|
||||
}
|
||||
|
||||
asset, err := s.Resolve(ctx, identifier)
|
||||
asset, err := s.Resolve(ctx, identifier, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
16
internal/service/asset/usage_summary.go
Normal file
16
internal/service/asset/usage_summary.go
Normal file
@@ -0,0 +1,16 @@
|
||||
package asset
|
||||
|
||||
import "github.com/break/junhong_cmp_fiber/internal/model"
|
||||
|
||||
// computeUsageSummary 汇总当前世代套餐的虚已用量和虚剩余量。
|
||||
// 对每条记录调用 BuildTrafficMetrics() 获取虚流量字段,累加后计算剩余。
|
||||
func computeUsageSummary(usages []*model.PackageUsage) (totalUsedMB float64, totalRemainingMB float64) {
|
||||
var totalVirtualTotal float64
|
||||
for _, u := range usages {
|
||||
m := u.BuildTrafficMetrics()
|
||||
totalUsedMB += m.VirtualUsedMB
|
||||
totalVirtualTotal += float64(m.VirtualTotalMB)
|
||||
}
|
||||
totalRemainingMB = totalVirtualTotal - totalUsedMB
|
||||
return
|
||||
}
|
||||
81
internal/service/asset/usage_summary_test.go
Normal file
81
internal/service/asset/usage_summary_test.go
Normal file
@@ -0,0 +1,81 @@
|
||||
package asset
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/break/junhong_cmp_fiber/internal/model"
|
||||
)
|
||||
|
||||
func TestComputeUsageSummary_EmptyList(t *testing.T) {
|
||||
totalUsed, totalRemaining := computeUsageSummary(nil)
|
||||
if totalUsed != 0.0 {
|
||||
t.Errorf("期望 totalUsed=0.0,实际=%v", totalUsed)
|
||||
}
|
||||
if totalRemaining != 0.0 {
|
||||
t.Errorf("期望 totalRemaining=0.0,实际=%v", totalRemaining)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComputeUsageSummary_SingleVirtualPackage(t *testing.T) {
|
||||
// 启用虚流量:真实总量=100MB,虚拟总量=200MB,倍率=0.5
|
||||
// 真实已用=40MB → 虚拟已用=min(40*0.5, 100)=20MB
|
||||
// 虚拟剩余=200-20=180MB
|
||||
usage := &model.PackageUsage{
|
||||
DataLimitMB: 100,
|
||||
DataUsageMB: 40,
|
||||
VirtualTotalMBSnapshot: 200,
|
||||
DisplayGainRatioSnapshot: 0.5,
|
||||
EnableVirtualDataSnapshot: true,
|
||||
}
|
||||
totalUsed, totalRemaining := computeUsageSummary([]*model.PackageUsage{usage})
|
||||
if totalUsed != 20.0 {
|
||||
t.Errorf("期望 totalUsed=20.0,实际=%v", totalUsed)
|
||||
}
|
||||
if totalRemaining != 180.0 {
|
||||
t.Errorf("期望 totalRemaining=180.0,实际=%v", totalRemaining)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComputeUsageSummary_SingleNonVirtualPackage(t *testing.T) {
|
||||
// 未启用虚流量:退化为真实值
|
||||
// 真实总量=100MB,真实已用=40MB → 虚拟已用=40MB,虚拟剩余=60MB
|
||||
usage := &model.PackageUsage{
|
||||
DataLimitMB: 100,
|
||||
DataUsageMB: 40,
|
||||
EnableVirtualDataSnapshot: false,
|
||||
}
|
||||
totalUsed, totalRemaining := computeUsageSummary([]*model.PackageUsage{usage})
|
||||
if totalUsed != 40.0 {
|
||||
t.Errorf("期望 totalUsed=40.0,实际=%v", totalUsed)
|
||||
}
|
||||
if totalRemaining != 60.0 {
|
||||
t.Errorf("期望 totalRemaining=60.0,实际=%v", totalRemaining)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComputeUsageSummary_MultiplePackages(t *testing.T) {
|
||||
// 套餐1(启用虚流量):VirtualUsed=20, VirtualTotal=200
|
||||
// 套餐2(未启用虚流量):VirtualUsed=10, VirtualTotal=50
|
||||
// 汇总:totalUsed=30, totalRemaining=(200+50)-30=220
|
||||
usages := []*model.PackageUsage{
|
||||
{
|
||||
DataLimitMB: 100,
|
||||
DataUsageMB: 40,
|
||||
VirtualTotalMBSnapshot: 200,
|
||||
DisplayGainRatioSnapshot: 0.5,
|
||||
EnableVirtualDataSnapshot: true,
|
||||
},
|
||||
{
|
||||
DataLimitMB: 50,
|
||||
DataUsageMB: 10,
|
||||
EnableVirtualDataSnapshot: false,
|
||||
},
|
||||
}
|
||||
totalUsed, totalRemaining := computeUsageSummary(usages)
|
||||
if totalUsed != 30.0 {
|
||||
t.Errorf("期望 totalUsed=30.0,实际=%v", totalUsed)
|
||||
}
|
||||
if totalRemaining != 220.0 {
|
||||
t.Errorf("期望 totalRemaining=220.0,实际=%v", totalRemaining)
|
||||
}
|
||||
}
|
||||
@@ -128,7 +128,7 @@ func (s *Service) CreateOrder(ctx context.Context, customerID uint, req *dto.Cli
|
||||
}
|
||||
|
||||
skipCtx := context.WithValue(ctx, constants.ContextKeySubordinateShopIDs, []uint{})
|
||||
assetInfo, err := s.assetService.Resolve(skipCtx, strings.TrimSpace(req.Identifier))
|
||||
assetInfo, err := s.assetService.Resolve(skipCtx, strings.TrimSpace(req.Identifier), false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -229,6 +229,18 @@ func (s *Service) CreateOrder(ctx context.Context, customerID uint, req *dto.Cli
|
||||
}()
|
||||
|
||||
if forceRecharge.NeedForceRecharge {
|
||||
// 强充第三方支付方式与资产类型必须匹配:单卡只允许支付宝,设备只允许微信(已暂时注释)
|
||||
// switch assetInfo.AssetType {
|
||||
// case "card":
|
||||
// if req.PaymentMethod != model.PaymentMethodAlipay {
|
||||
// return nil, errors.New(errors.CodeInvalidParam, "单卡资产强充只支持支付宝支付")
|
||||
// }
|
||||
// case constants.ResourceTypeDevice:
|
||||
// if req.PaymentMethod == model.PaymentMethodAlipay {
|
||||
// return nil, errors.New(errors.CodeInvalidParam, "设备资产强充只支持微信支付")
|
||||
// }
|
||||
// }
|
||||
|
||||
if req.PaymentMethod == model.PaymentMethodAlipay {
|
||||
// 支付宝强充:不需要 app_type / OpenID
|
||||
activeConfig, err := s.wechatConfigService.GetActiveConfig(skipCtx)
|
||||
@@ -1060,7 +1072,7 @@ func (s *Service) createAlipayForceRechargeOrder(
|
||||
func (s *Service) ListOrders(ctx context.Context, customerID uint, req *dto.ClientOrderListRequest) ([]dto.ClientOrderListItem, int64, error) {
|
||||
skipCtx := context.WithValue(ctx, constants.ContextKeySubordinateShopIDs, []uint{})
|
||||
|
||||
assetInfo, err := s.assetService.Resolve(skipCtx, strings.TrimSpace(req.Identifier))
|
||||
assetInfo, err := s.assetService.Resolve(skipCtx, strings.TrimSpace(req.Identifier), false)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
@@ -1231,7 +1243,7 @@ func (s *Service) PayOrder(ctx context.Context, customerID uint, orderID uint, r
|
||||
if err == nil && device != nil {
|
||||
assetRealnamePolicy = device.RealnamePolicy
|
||||
// 设备的 RealNameStatus 需要通过 Resolve 获取(从当前卡派生)
|
||||
resolved, resolveErr := s.assetService.Resolve(skipCtx, device.VirtualNo)
|
||||
resolved, resolveErr := s.assetService.Resolve(skipCtx, device.VirtualNo, false)
|
||||
if resolveErr == nil && resolved != nil {
|
||||
assetRealnameStatus = resolved.RealNameStatus
|
||||
}
|
||||
@@ -1241,6 +1253,20 @@ func (s *Service) PayOrder(ctx context.Context, customerID uint, orderID uint, r
|
||||
return nil, errors.New(errors.CodeNeedRealname)
|
||||
}
|
||||
|
||||
// 第三方支付方式与资产类型必须匹配:单卡只允许支付宝,设备只允许微信(已暂时注释)
|
||||
// if req.PaymentMethod == model.PaymentMethodWechat || req.PaymentMethod == model.PaymentMethodAlipay {
|
||||
// switch order.OrderType {
|
||||
// case model.OrderTypeSingleCard:
|
||||
// if req.PaymentMethod != model.PaymentMethodAlipay {
|
||||
// return nil, errors.New(errors.CodeInvalidParam, "单卡资产只支持支付宝支付")
|
||||
// }
|
||||
// case model.OrderTypeDevice:
|
||||
// if req.PaymentMethod != model.PaymentMethodWechat {
|
||||
// return nil, errors.New(errors.CodeInvalidParam, "设备资产只支持微信支付")
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
|
||||
switch req.PaymentMethod {
|
||||
case model.PaymentMethodWallet:
|
||||
if err := s.orderPaymentService.WalletPay(skipCtx, orderID, model.BuyerTypePersonal, customerID); err != nil {
|
||||
|
||||
@@ -1494,6 +1494,19 @@ func (s *Service) buildCardNotFoundFailedItems(iccids []string) []dto.CardSeries
|
||||
return items
|
||||
}
|
||||
|
||||
// RefreshCardDataByID 通过卡 ID 从 Gateway 同步卡数据
|
||||
// 内部查询 ICCID 后委托给 RefreshCardDataFromGateway,供套餐失效前流量同步使用
|
||||
func (s *Service) RefreshCardDataByID(ctx context.Context, cardID uint) error {
|
||||
card, err := s.iotCardStore.GetByID(ctx, cardID)
|
||||
if err != nil {
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
return s.RefreshCardDataFromGateway(ctx, card.ICCID)
|
||||
}
|
||||
|
||||
// RefreshCardDataFromGateway 从 Gateway 完整同步卡数据
|
||||
// 调用网关查询网络状态、实名状态、本月流量,并写回数据库
|
||||
// 流量采用增量计算(与轮询逻辑一致):increment = 当前网关读数 - 上次网关读数
|
||||
|
||||
@@ -1120,26 +1120,31 @@ func (s *Service) createOrderWithWalletPayment(ctx context.Context, order *model
|
||||
}
|
||||
actualAmount := *order.ActualPaidAmount
|
||||
|
||||
// 1. 事务外:检查钱包余额(快速失败)
|
||||
wallet, err := s.agentWalletStore.GetMainWallet(ctx, operatorShopID)
|
||||
if err != nil {
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
return errors.New(errors.CodeWalletNotFound, "钱包不存在")
|
||||
// 事务内:加锁读取钱包 + 余额检查 + 创建订单 + 扣款 + 创建流水 + 激活套餐
|
||||
// 使用 FOR UPDATE 悲观锁替代乐观锁,避免并发请求因 version 过期导致误报余额不足
|
||||
err := s.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
// 1. 加锁读取钱包,并发请求排队等待,不会冲突
|
||||
var wallet model.AgentWallet
|
||||
if err := tx.Set("gorm:query_option", "FOR UPDATE").
|
||||
Where("shop_id = ? AND wallet_type = ?", operatorShopID, constants.AgentWalletTypeMain).
|
||||
First(&wallet).Error; err != nil {
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
return errors.New(errors.CodeWalletNotFound, "钱包不存在")
|
||||
}
|
||||
return errors.Wrap(errors.CodeDatabaseError, err, "查询钱包失败")
|
||||
}
|
||||
return errors.Wrap(errors.CodeDatabaseError, err, "查询钱包失败")
|
||||
}
|
||||
if wallet.Balance < actualAmount {
|
||||
return errors.New(errors.CodeInsufficientBalance, "余额不足")
|
||||
}
|
||||
|
||||
// 2. 事务内:创建订单 + 扣款 + 创建流水 + 激活套餐
|
||||
err = s.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
// 2.1 创建订单
|
||||
// 2. 余额检查(加锁后读到的是最新余额,结果准确)
|
||||
if wallet.Balance < actualAmount {
|
||||
return errors.New(errors.CodeInsufficientBalance, "余额不足")
|
||||
}
|
||||
|
||||
// 3. 创建订单
|
||||
if err := tx.Create(order).Error; err != nil {
|
||||
return errors.Wrap(errors.CodeDatabaseError, err, "创建订单失败")
|
||||
}
|
||||
|
||||
// 2.2 创建订单明细
|
||||
// 4. 创建订单明细
|
||||
for _, item := range items {
|
||||
item.OrderID = order.ID
|
||||
}
|
||||
@@ -1147,30 +1152,24 @@ func (s *Service) createOrderWithWalletPayment(ctx context.Context, order *model
|
||||
return errors.Wrap(errors.CodeDatabaseError, err, "创建订单明细失败")
|
||||
}
|
||||
|
||||
// 2.3 扣减钱包余额(乐观锁)
|
||||
result := tx.Model(&model.AgentWallet{}).
|
||||
Where("id = ? AND balance >= ? AND version = ?", wallet.ID, actualAmount, wallet.Version).
|
||||
Updates(map[string]any{
|
||||
"balance": gorm.Expr("balance - ?", actualAmount),
|
||||
"version": gorm.Expr("version + 1"),
|
||||
})
|
||||
if result.Error != nil {
|
||||
return errors.Wrap(errors.CodeDatabaseError, result.Error, "扣减钱包余额失败")
|
||||
}
|
||||
if result.RowsAffected == 0 {
|
||||
return errors.New(errors.CodeInsufficientBalance, "余额不足或并发冲突")
|
||||
// 5. 扣减钱包余额(FOR UPDATE 已保证串行,无需 version 条件)
|
||||
if err := tx.Model(&wallet).Updates(map[string]any{
|
||||
"balance": gorm.Expr("balance - ?", actualAmount),
|
||||
"version": gorm.Expr("version + 1"),
|
||||
}).Error; err != nil {
|
||||
return errors.Wrap(errors.CodeDatabaseError, err, "扣减钱包余额失败")
|
||||
}
|
||||
|
||||
// 2.4 创建钱包流水
|
||||
// 6. 创建钱包流水
|
||||
var relatedShopID *uint
|
||||
if order.PurchaseRole == model.PurchaseRolePurchaseForSubordinate {
|
||||
relatedShopID = &buyerShopID
|
||||
}
|
||||
if err := s.createWalletTransaction(ctx, tx, wallet, order, actualAmount, order.PurchaseRole, relatedShopID); err != nil {
|
||||
if err := s.createWalletTransaction(ctx, tx, &wallet, order, actualAmount, order.PurchaseRole, relatedShopID); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 2.5 激活套餐
|
||||
// 7. 激活套餐
|
||||
if err := s.activatePackage(ctx, tx, order); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -92,19 +92,20 @@ func (s *ActivationService) ActivateByRealname(ctx context.Context, carrierType
|
||||
|
||||
// 检查是否是主套餐
|
||||
if usage.MasterUsageID == nil {
|
||||
// 主套餐:需要检查是否有已激活的主套餐
|
||||
// 主套餐:需要检查是否有已激活或已用完(仍在有效期)的主套餐
|
||||
var activeMain model.PackageUsage
|
||||
err := tx.Where("status = ?", constants.PackageUsageStatusActive).
|
||||
err := tx.Where("status IN ?", []int{constants.PackageUsageStatusActive, constants.PackageUsageStatusDepleted}).
|
||||
Where("master_usage_id IS NULL").
|
||||
Where(carrierType+"_id = ?", carrierID).
|
||||
Order("priority ASC").
|
||||
First(&activeMain).Error
|
||||
|
||||
if err == nil {
|
||||
// 已有激活的主套餐,保持排队状态
|
||||
s.logger.Warn("已有激活主套餐,跳过激活",
|
||||
// 已有生效中或已用完(占位)的主套餐,保持排队状态
|
||||
s.logger.Warn("已有占位主套餐,跳过激活",
|
||||
zap.Uint("usage_id", usage.ID),
|
||||
zap.Uint("active_main_id", activeMain.ID))
|
||||
zap.Uint("active_main_id", activeMain.ID),
|
||||
zap.Int("active_main_status", activeMain.Status))
|
||||
continue
|
||||
}
|
||||
if err != gorm.ErrRecordNotFound {
|
||||
@@ -500,9 +501,10 @@ func (s *ActivationService) getNextPendingMainPackage(ctx context.Context, tx *g
|
||||
|
||||
func (s *ActivationService) hasActiveMainPackage(ctx context.Context, tx *gorm.DB, carrierType string, carrierID uint) (bool, error) {
|
||||
var count int64
|
||||
// 生效中(1)和已用完(2)都属于"占位"状态:已用完的套餐仍在有效期内,不允许提前激活下一个主套餐
|
||||
if err := tx.WithContext(ctx).
|
||||
Model(&model.PackageUsage{}).
|
||||
Where("status = ?", constants.PackageUsageStatusActive).
|
||||
Where("status IN ?", []int{constants.PackageUsageStatusActive, constants.PackageUsageStatusDepleted}).
|
||||
Where("master_usage_id IS NULL").
|
||||
Where(carrierType+"_id = ?", carrierID).
|
||||
Count(&count).Error; err != nil {
|
||||
|
||||
@@ -88,13 +88,22 @@ func (s *UsageService) DeductDataUsage(ctx context.Context, carrierType string,
|
||||
return err
|
||||
}
|
||||
|
||||
today := time.Now().Format("2006-01-02")
|
||||
|
||||
if len(packages) == 0 {
|
||||
// 无生效套餐:尝试找 Depleted 套餐记录溢出流量详单,避免详单断档
|
||||
if recordErr := s.recordOverflowToDepletedPackage(ctx, tx, targetCarrierType, targetCarrierID, deductUsageMB, today); recordErr != nil {
|
||||
s.logger.Warn("记录溢出流量到已耗尽套餐失败",
|
||||
zap.String("carrier_type", targetCarrierType),
|
||||
zap.Uint("carrier_id", targetCarrierID),
|
||||
zap.Int64("usage_mb", deductUsageMB),
|
||||
zap.Error(recordErr))
|
||||
}
|
||||
return errors.New(errors.CodeNoAvailablePackage, "没有可用套餐")
|
||||
}
|
||||
|
||||
// 按优先级扣减流量
|
||||
remainingUsage := deductUsageMB
|
||||
today := time.Now().Format("2006-01-02")
|
||||
|
||||
for index, pkg := range packages {
|
||||
if remainingUsage <= 0 {
|
||||
@@ -446,3 +455,45 @@ func (s *UsageService) triggerSuspensionAfterCommit(carrierType string, carrierI
|
||||
}(cardID)
|
||||
}
|
||||
}
|
||||
|
||||
// recordOverflowToDepletedPackage 在所有套餐已耗尽时,将溢出流量写入最近的 Depleted 套餐日记录。
|
||||
// 目的:保持流量详单连续性,即便套餐用完期间上游仍有流量使用,详单不会断档。
|
||||
// 只写日记录和累加 data_usage_mb,不改变套餐 status,不触发停机。
|
||||
func (s *UsageService) recordOverflowToDepletedPackage(ctx context.Context, tx *gorm.DB, carrierType string, carrierID uint, overflowMB int64, today string) error {
|
||||
query := tx.Where("status = ?", constants.PackageUsageStatusDepleted)
|
||||
switch carrierType {
|
||||
case constants.AssetTypeIotCard:
|
||||
query = query.Where("iot_card_id = ?", carrierID)
|
||||
case constants.AssetTypeDevice:
|
||||
query = query.Where("device_id = ?", carrierID)
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
|
||||
// 优先选主套餐(master_usage_id IS NULL),主套餐不存在时再退而求其次取加油包
|
||||
var pkg model.PackageUsage
|
||||
err := query.Where("master_usage_id IS NULL").Order("id DESC").First(&pkg).Error
|
||||
if err == gorm.ErrRecordNotFound {
|
||||
err = query.Order("id DESC").First(&pkg).Error
|
||||
}
|
||||
if err != nil {
|
||||
// 找不到任何 Depleted 套餐,静默忽略
|
||||
return nil
|
||||
}
|
||||
|
||||
newUsage := pkg.DataUsageMB + overflowMB
|
||||
if err := tx.Model(&pkg).Update("data_usage_mb", newUsage).Error; err != nil {
|
||||
return errors.Wrap(errors.CodeDatabaseError, err, "更新溢出流量失败")
|
||||
}
|
||||
|
||||
if err := s.updateDailyRecord(ctx, tx, pkg.ID, today, overflowMB, newUsage); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
s.logger.Info("记录溢出流量到已耗尽套餐",
|
||||
zap.Uint("package_usage_id", pkg.ID),
|
||||
zap.Int64("overflow_mb", overflowMB),
|
||||
zap.Int64("new_usage_mb", newUsage))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -67,6 +67,16 @@ func (s *DeviceImportTaskStore) UpdateStatus(ctx context.Context, id uint, statu
|
||||
return s.db.WithContext(ctx).Model(&model.DeviceImportTask{}).Where("id = ?", id).Updates(updates).Error
|
||||
}
|
||||
|
||||
// UpdateProgress 实时更新导入进度计数(不更新详情列表,用于批量处理中间状态)
|
||||
func (s *DeviceImportTaskStore) UpdateProgress(ctx context.Context, id uint, successCount, skipCount, failCount int) error {
|
||||
return s.db.WithContext(ctx).Model(&model.DeviceImportTask{}).Where("id = ?", id).Updates(map[string]any{
|
||||
"success_count": successCount,
|
||||
"skip_count": skipCount,
|
||||
"fail_count": failCount,
|
||||
"updated_at": time.Now(),
|
||||
}).Error
|
||||
}
|
||||
|
||||
func (s *DeviceImportTaskStore) UpdateResult(ctx context.Context, id uint, totalCount, successCount, skipCount, failCount, warningCount int, skippedItems, failedItems, warningItems model.ImportResultItems) error {
|
||||
updates := map[string]any{
|
||||
"total_count": totalCount,
|
||||
|
||||
@@ -72,6 +72,16 @@ func (s *IotCardImportTaskStore) UpdateStatus(ctx context.Context, id uint, stat
|
||||
return s.db.WithContext(ctx).Model(&model.IotCardImportTask{}).Where("id = ?", id).Updates(updates).Error
|
||||
}
|
||||
|
||||
// UpdateProgress 实时更新导入进度计数(不更新详情列表,用于批量处理中间状态)
|
||||
func (s *IotCardImportTaskStore) UpdateProgress(ctx context.Context, id uint, successCount, skipCount, failCount int) error {
|
||||
return s.db.WithContext(ctx).Model(&model.IotCardImportTask{}).Where("id = ?", id).Updates(map[string]interface{}{
|
||||
"success_count": successCount,
|
||||
"skip_count": skipCount,
|
||||
"fail_count": failCount,
|
||||
"updated_at": time.Now(),
|
||||
}).Error
|
||||
}
|
||||
|
||||
func (s *IotCardImportTaskStore) UpdateResult(ctx context.Context, id uint, successCount, skipCount, failCount int, skippedItems, failedItems model.ImportResultItems) error {
|
||||
updates := map[string]interface{}{
|
||||
"success_count": successCount,
|
||||
|
||||
@@ -301,6 +301,40 @@ func (s *PackageUsageStore) ListByCarrier(ctx context.Context, carrierType strin
|
||||
return usages, nil
|
||||
}
|
||||
|
||||
// GetCurrentGenerationUsagesForSummary 查询当前世代内所有已激活过的套餐记录(含加油包),用于流量汇总。
|
||||
// 当前世代定义:status∈{1,2,3,4} 中 generation 最大值所在的那一批记录。
|
||||
// 返回计算所需字段:DataUsageMB、DataLimitMB、VirtualTotalMBSnapshot、DisplayGainRatioSnapshot、EnableVirtualDataSnapshot。
|
||||
func (s *PackageUsageStore) GetCurrentGenerationUsagesForSummary(ctx context.Context, carrierType string, carrierID uint) ([]*model.PackageUsage, error) {
|
||||
var usages []*model.PackageUsage
|
||||
|
||||
// 子查询取当前世代编号
|
||||
subQuery := s.db.WithContext(ctx).Model(&model.PackageUsage{}).
|
||||
Select("MAX(generation)").
|
||||
Where("status IN ?", []int{
|
||||
constants.PackageUsageStatusActive,
|
||||
constants.PackageUsageStatusDepleted,
|
||||
constants.PackageUsageStatusExpired,
|
||||
constants.PackageUsageStatusInvalidated,
|
||||
})
|
||||
subQuery = applyPackageUsageCarrierFilter(subQuery, carrierType, carrierID)
|
||||
|
||||
query := s.db.WithContext(ctx).Model(&model.PackageUsage{}).
|
||||
Select("data_usage_mb, data_limit_mb, virtual_total_mb_snapshot, display_gain_ratio_snapshot, enable_virtual_data_snapshot").
|
||||
Where("status IN ?", []int{
|
||||
constants.PackageUsageStatusActive,
|
||||
constants.PackageUsageStatusDepleted,
|
||||
constants.PackageUsageStatusExpired,
|
||||
constants.PackageUsageStatusInvalidated,
|
||||
}).
|
||||
Where("generation = (?)", subQuery)
|
||||
query = applyPackageUsageCarrierFilter(query, carrierType, carrierID)
|
||||
|
||||
if err := query.Find(&usages).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return usages, nil
|
||||
}
|
||||
|
||||
func applyPackageUsageCarrierFilter(query *gorm.DB, carrierType string, carrierID uint) *gorm.DB {
|
||||
if carrierType == "iot_card" {
|
||||
return query.Where("iot_card_id = ?", carrierID)
|
||||
|
||||
@@ -85,7 +85,18 @@ func (h *DeviceImportHandler) HandleDeviceImport(ctx context.Context, task *asyn
|
||||
return asynq.SkipRetry
|
||||
}
|
||||
|
||||
if importTask.Status != model.ImportTaskStatusPending {
|
||||
switch importTask.Status {
|
||||
case model.ImportTaskStatusPending:
|
||||
// 正常首次处理
|
||||
case model.ImportTaskStatusProcessing:
|
||||
// 上次 worker 中途中断(重启/崩溃),重置计数重新处理
|
||||
// 已入库的设备会被 ExistsByDeviceNoBatch 识别为 skip,不会重复写入
|
||||
h.logger.Warn("检测到导入任务上次处理中途中断,重置进度重新处理",
|
||||
zap.Uint("task_id", payload.TaskID),
|
||||
)
|
||||
h.importTaskStore.UpdateProgress(ctx, importTask.ID, 0, 0, 0)
|
||||
default:
|
||||
// 已完成或失败,不重复处理
|
||||
h.logger.Info("导入任务已处理,跳过",
|
||||
zap.Uint("task_id", payload.TaskID),
|
||||
zap.Int("status", importTask.Status),
|
||||
@@ -185,6 +196,8 @@ func (h *DeviceImportHandler) processImport(ctx context.Context, task *model.Dev
|
||||
end := min(i+deviceBatchSize, len(rows))
|
||||
batch := rows[i:end]
|
||||
h.processBatch(ctx, task, batch, result)
|
||||
// 每批完成后实时更新进度计数,让前端可以看到处理进度
|
||||
h.importTaskStore.UpdateProgress(ctx, task.ID, result.successCount, result.skipCount, result.failCount)
|
||||
}
|
||||
|
||||
return result
|
||||
|
||||
@@ -41,15 +41,14 @@ type PollingCallback interface {
|
||||
}
|
||||
|
||||
type IotCardImportHandler struct {
|
||||
db *gorm.DB
|
||||
redis *redis.Client
|
||||
importTaskStore *postgres.IotCardImportTaskStore
|
||||
iotCardStore *postgres.IotCardStore
|
||||
assetWalletStore *postgres.AssetWalletStore
|
||||
assetIdentifierStore *postgres.AssetIdentifierStore
|
||||
storageService *storage.Service
|
||||
pollingCallback PollingCallback
|
||||
logger *zap.Logger
|
||||
db *gorm.DB
|
||||
redis *redis.Client
|
||||
importTaskStore *postgres.IotCardImportTaskStore
|
||||
iotCardStore *postgres.IotCardStore
|
||||
assetWalletStore *postgres.AssetWalletStore
|
||||
storageService *storage.Service
|
||||
pollingCallback PollingCallback
|
||||
logger *zap.Logger
|
||||
}
|
||||
|
||||
func NewIotCardImportHandler(
|
||||
@@ -58,21 +57,19 @@ func NewIotCardImportHandler(
|
||||
importTaskStore *postgres.IotCardImportTaskStore,
|
||||
iotCardStore *postgres.IotCardStore,
|
||||
assetWalletStore *postgres.AssetWalletStore,
|
||||
assetIdentifierStore *postgres.AssetIdentifierStore,
|
||||
storageSvc *storage.Service,
|
||||
pollingCallback PollingCallback,
|
||||
logger *zap.Logger,
|
||||
) *IotCardImportHandler {
|
||||
return &IotCardImportHandler{
|
||||
db: db,
|
||||
redis: redis,
|
||||
importTaskStore: importTaskStore,
|
||||
iotCardStore: iotCardStore,
|
||||
assetWalletStore: assetWalletStore,
|
||||
assetIdentifierStore: assetIdentifierStore,
|
||||
storageService: storageSvc,
|
||||
pollingCallback: pollingCallback,
|
||||
logger: logger,
|
||||
db: db,
|
||||
redis: redis,
|
||||
importTaskStore: importTaskStore,
|
||||
iotCardStore: iotCardStore,
|
||||
assetWalletStore: assetWalletStore,
|
||||
storageService: storageSvc,
|
||||
pollingCallback: pollingCallback,
|
||||
logger: logger,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -95,7 +92,18 @@ func (h *IotCardImportHandler) HandleIotCardImport(ctx context.Context, task *as
|
||||
return asynq.SkipRetry
|
||||
}
|
||||
|
||||
if importTask.Status != model.ImportTaskStatusPending {
|
||||
switch importTask.Status {
|
||||
case model.ImportTaskStatusPending:
|
||||
// 正常首次处理
|
||||
case model.ImportTaskStatusProcessing:
|
||||
// 上次 worker 中途中断(重启/崩溃),重置计数重新处理
|
||||
// 已入库的卡会被 ExistsByICCIDBatch 识别为 skip,不会重复写入
|
||||
h.logger.Warn("检测到导入任务上次处理中途中断,重置进度重新处理",
|
||||
zap.Uint("task_id", payload.TaskID),
|
||||
)
|
||||
h.importTaskStore.UpdateProgress(ctx, importTask.ID, 0, 0, 0)
|
||||
default:
|
||||
// 已完成或失败,不重复处理
|
||||
h.logger.Info("导入任务已处理,跳过",
|
||||
zap.Uint("task_id", payload.TaskID),
|
||||
zap.Int("status", importTask.Status),
|
||||
@@ -218,6 +226,8 @@ func (h *IotCardImportHandler) processImport(ctx context.Context, task *model.Io
|
||||
end := min(i+batchSize, len(cards))
|
||||
batch := cards[i:end]
|
||||
h.processBatch(ctx, task, batch, i+1, result)
|
||||
// 每批完成后实时更新进度计数,让前端可以看到处理进度
|
||||
h.importTaskStore.UpdateProgress(ctx, task.ID, result.successCount, result.skipCount, result.failCount)
|
||||
}
|
||||
|
||||
return result
|
||||
@@ -341,7 +351,14 @@ func (h *IotCardImportHandler) processBatch(ctx context.Context, task *model.Iot
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
createdCards := make([]*model.IotCard, 0, len(finalCards))
|
||||
|
||||
// 构建待批量写入的卡列表,同时记录原始信息用于失败回溯
|
||||
type pendingEntry struct {
|
||||
item model.CardItem
|
||||
meta cardMeta
|
||||
}
|
||||
pending := make([]pendingEntry, 0, len(finalCards))
|
||||
iotCards := make([]*model.IotCard, 0, len(finalCards))
|
||||
|
||||
for _, card := range finalCards {
|
||||
meta := cardMetaMap[card.ICCID]
|
||||
@@ -386,43 +403,63 @@ func (h *IotCardImportHandler) processBatch(ctx context.Context, task *model.Iot
|
||||
iotCard.CreatedAt = now
|
||||
iotCard.UpdatedAt = now
|
||||
|
||||
txErr := h.db.Transaction(func(tx *gorm.DB) error {
|
||||
if err := tx.Create(iotCard).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
txIdentifierStore := postgres.NewAssetIdentifierStore(tx)
|
||||
if err := txIdentifierStore.Register(ctx, card.ICCID, model.AssetTypeIotCard, iotCard.ID); err != nil {
|
||||
return fmt.Errorf("ICCID 已被占用: %s", card.ICCID)
|
||||
}
|
||||
if card.VirtualNo != "" {
|
||||
if err := txIdentifierStore.Register(ctx, card.VirtualNo, model.AssetTypeIotCard, iotCard.ID); err != nil {
|
||||
return fmt.Errorf("虚拟号已被占用: %s", card.VirtualNo)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if txErr != nil {
|
||||
h.logger.Error("创建 IoT 卡失败",
|
||||
zap.String("iccid", card.ICCID),
|
||||
zap.Error(txErr),
|
||||
)
|
||||
result.failedItems = append(result.failedItems, model.ImportResultItem{
|
||||
Line: meta.line,
|
||||
ICCID: card.ICCID,
|
||||
MSISDN: meta.msisdn,
|
||||
Reason: txErr.Error(),
|
||||
})
|
||||
result.failCount++
|
||||
continue
|
||||
}
|
||||
createdCards = append(createdCards, iotCard)
|
||||
result.successCount++
|
||||
iotCards = append(iotCards, iotCard)
|
||||
pending = append(pending, pendingEntry{item: card, meta: meta})
|
||||
}
|
||||
|
||||
h.batchCreateWallets(ctx, createdCards)
|
||||
if len(iotCards) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
// 整批一个事务:批量写入 iot_card + asset_identifier
|
||||
// 相比逐卡事务,将 N 个事务压缩为 1 个,大幅减少数据库往返
|
||||
txErr := h.db.Transaction(func(tx *gorm.DB) error {
|
||||
// 批量插入 iot_card,GORM 会通过 RETURNING 回填所有 ID
|
||||
if err := tx.CreateInBatches(&iotCards, 500).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
// 用回填的 ID 构建 asset_identifier 列表
|
||||
identifiers := make([]*model.AssetIdentifier, 0, len(iotCards)*2)
|
||||
for _, c := range iotCards {
|
||||
identifiers = append(identifiers, &model.AssetIdentifier{
|
||||
Identifier: c.ICCID,
|
||||
AssetType: model.AssetTypeIotCard,
|
||||
AssetID: c.ID,
|
||||
})
|
||||
if c.VirtualNo != "" {
|
||||
identifiers = append(identifiers, &model.AssetIdentifier{
|
||||
Identifier: c.VirtualNo,
|
||||
AssetType: model.AssetTypeIotCard,
|
||||
AssetID: c.ID,
|
||||
})
|
||||
}
|
||||
}
|
||||
return tx.CreateInBatches(&identifiers, 500).Error
|
||||
})
|
||||
|
||||
if txErr != nil {
|
||||
h.logger.Error("批量创建 IoT 卡失败",
|
||||
zap.Error(txErr),
|
||||
zap.Int("batch_size", len(iotCards)),
|
||||
)
|
||||
for _, p := range pending {
|
||||
result.failedItems = append(result.failedItems, model.ImportResultItem{
|
||||
Line: p.meta.line,
|
||||
ICCID: p.item.ICCID,
|
||||
MSISDN: p.meta.msisdn,
|
||||
Reason: "批量写入失败: " + txErr.Error(),
|
||||
})
|
||||
result.failCount++
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
result.successCount += len(iotCards)
|
||||
|
||||
h.batchCreateWallets(ctx, iotCards)
|
||||
|
||||
if h.pollingCallback != nil {
|
||||
h.pollingCallback.OnBatchCardsCreated(ctx, createdCards)
|
||||
h.pollingCallback.OnBatchCardsCreated(ctx, iotCards)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
ALTER TABLE tb_agent_recharge_record
|
||||
DROP COLUMN IF EXISTS rejection_reason;
|
||||
@@ -0,0 +1,5 @@
|
||||
-- 为代理预充值表新增驳回原因字段,支持管理员驳回待支付订单
|
||||
ALTER TABLE tb_agent_recharge_record
|
||||
ADD COLUMN IF NOT EXISTS rejection_reason VARCHAR(500) NULL;
|
||||
|
||||
COMMENT ON COLUMN tb_agent_recharge_record.rejection_reason IS '驳回原因,仅驳回时写入';
|
||||
@@ -0,0 +1,6 @@
|
||||
-- 回滚:将状态约束恢复为不包含驳回状态(6)
|
||||
ALTER TABLE tb_agent_recharge_record
|
||||
DROP CONSTRAINT IF EXISTS chk_agent_recharge_status;
|
||||
|
||||
ALTER TABLE tb_agent_recharge_record
|
||||
ADD CONSTRAINT chk_agent_recharge_status CHECK (status IN (1, 2, 3, 4, 5));
|
||||
@@ -0,0 +1,6 @@
|
||||
-- 更新代理预充值状态检查约束,新增驳回状态(6)
|
||||
ALTER TABLE tb_agent_recharge_record
|
||||
DROP CONSTRAINT IF EXISTS chk_agent_recharge_status;
|
||||
|
||||
ALTER TABLE tb_agent_recharge_record
|
||||
ADD CONSTRAINT chk_agent_recharge_status CHECK (status IN (1, 2, 3, 4, 5, 6));
|
||||
@@ -88,7 +88,6 @@ func (h *Handler) registerIotCardImportHandler() {
|
||||
h.workerResult.Stores.IotCardImportTask,
|
||||
h.workerResult.Stores.IotCard,
|
||||
h.workerResult.Stores.AssetWallet,
|
||||
h.workerResult.Stores.AssetIdentifier,
|
||||
h.storage,
|
||||
h.pollingCallback,
|
||||
h.logger,
|
||||
|
||||
@@ -6,12 +6,29 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from decimal import Decimal
|
||||
from typing import Any, Iterable, Iterator, Optional
|
||||
|
||||
|
||||
BATCH_SIZE = 200
|
||||
# 重构后分阶段索引查询已无大 JOIN,1000 为安全默认值
|
||||
BATCH_SIZE = 1000
|
||||
|
||||
|
||||
class _StageTimer:
|
||||
"""阶段耗时计时器,记录迁移各阶段用时。"""
|
||||
|
||||
def __init__(self, func_name: str) -> None:
|
||||
self._func = func_name
|
||||
self._t = time.perf_counter()
|
||||
|
||||
def log(self, stage: str, **kw: Any) -> None:
|
||||
elapsed = time.perf_counter() - self._t
|
||||
parts = " ".join(f"{k}={v}" for k, v in kw.items())
|
||||
suffix = f" {parts}" if parts else ""
|
||||
print(f"[legacy] {self._func}.{stage}:{suffix} elapsed={elapsed:.1f}s")
|
||||
self._t = time.perf_counter()
|
||||
|
||||
|
||||
def _normalize_iccid_pairs(iccid_pairs: Iterable[tuple]) -> list[tuple[str, str, bool]]:
|
||||
@@ -236,14 +253,35 @@ def _to_mb_decimal(value) -> Decimal:
|
||||
return Decimal("0")
|
||||
|
||||
|
||||
def _build_card_lookup(
|
||||
pairs: list[tuple[str, str, bool]],
|
||||
) -> dict[str, tuple[str, int]]:
|
||||
"""构建 tbl_card 查询键 → (完整 ICCID, rank) 映射。
|
||||
|
||||
rank=0 表示完整精确命中,rank=1 表示 19 位兜底命中。
|
||||
"""
|
||||
by_full: dict[str, tuple[str, int]] = {}
|
||||
for i19, i20, allow_i19_lookup in pairs:
|
||||
card_key = i20 or i19
|
||||
by_full[card_key] = (card_key, 0)
|
||||
if i20 and allow_i19_lookup:
|
||||
by_full[i19] = (card_key, 1)
|
||||
return by_full
|
||||
|
||||
|
||||
# ---------------- 查询函数 ----------------
|
||||
|
||||
|
||||
def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPackage]:
|
||||
def fetch_current_packages(
|
||||
conn,
|
||||
iccids: Iterable[str],
|
||||
batch_size: int = BATCH_SIZE,
|
||||
) -> dict[str, LegacyPackage]:
|
||||
"""查每张卡的当前生效主套餐:status=1、type!=1 且 expire_date 最大。
|
||||
|
||||
返回: iccid → LegacyPackage(找不到的 iccid 不在结果中)
|
||||
"""
|
||||
timer = _StageTimer("fetch_current_packages")
|
||||
iccid_list = sorted({i for i in iccids if i})
|
||||
out: dict[str, LegacyPackage] = {}
|
||||
if not iccid_list:
|
||||
@@ -266,11 +304,15 @@ def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPacka
|
||||
ORDER BY iccid_mark ASC, expire_date DESC
|
||||
"""
|
||||
seen: set[str] = set()
|
||||
rows = 0
|
||||
batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(iccid_list):
|
||||
for batch in _chunks(iccid_list, batch_size):
|
||||
batches += 1
|
||||
cur.execute(sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
iccid = row["iccid_mark"]
|
||||
rows += 1
|
||||
if not iccid or iccid in seen:
|
||||
continue
|
||||
seen.add(iccid)
|
||||
@@ -284,15 +326,21 @@ def fetch_current_packages(conn, iccids: Iterable[str]) -> dict[str, LegacyPacka
|
||||
flow_size_mb=int(row.get("flow_size") or 0),
|
||||
status=int(row.get("status") or 0),
|
||||
)
|
||||
timer.log("tbl_card_life", keys=len(iccid_list), batches=batches, rows=rows, found=len(out))
|
||||
return out
|
||||
|
||||
|
||||
def fetch_package_lifecycles(conn, iccid_pairs: Iterable[tuple]) -> dict[str, list[LegacyPackageLifecycle]]:
|
||||
def fetch_package_lifecycles(
|
||||
conn,
|
||||
iccid_pairs: Iterable[tuple],
|
||||
batch_size: int = BATCH_SIZE,
|
||||
) -> dict[str, list[LegacyPackageLifecycle]]:
|
||||
"""查当前生效和未生效待生效正式套餐生命周期。
|
||||
|
||||
只返回可迁移范围(active/pending)和需要在审核文件中说明的跳过记录。
|
||||
加油包(type=1)和已过期记录不生成套餐 SQL,但保留 skip_reason 供审核产物输出。
|
||||
"""
|
||||
timer = _StageTimer("fetch_package_lifecycles")
|
||||
pairs = _normalize_iccid_pairs(iccid_pairs)
|
||||
out: dict[str, list[LegacyPackageLifecycle]] = {}
|
||||
if not pairs:
|
||||
@@ -328,8 +376,11 @@ def fetch_package_lifecycles(conn, iccid_pairs: Iterable[tuple]) -> dict[str, li
|
||||
COALESCE(expire_date, '9999-12-31') ASC,
|
||||
id ASC
|
||||
"""
|
||||
rows = 0
|
||||
batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(iccid_list):
|
||||
for batch in _chunks(iccid_list, batch_size):
|
||||
batches += 1
|
||||
cur.execute(sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
legacy_iccid = row.get("iccid_mark") or ""
|
||||
@@ -338,6 +389,7 @@ def fetch_package_lifecycles(conn, iccid_pairs: Iterable[tuple]) -> dict[str, li
|
||||
iccid = query_to_full.get(legacy_iccid)
|
||||
if not iccid:
|
||||
continue
|
||||
rows += 1
|
||||
start = row.get("start_date")
|
||||
expire = row.get("expire_date")
|
||||
start_str = start.strftime("%Y-%m-%d %H:%M:%S") if start else None
|
||||
@@ -363,6 +415,7 @@ def fetch_package_lifecycles(conn, iccid_pairs: Iterable[tuple]) -> dict[str, li
|
||||
skip_reason=skip_reason,
|
||||
)
|
||||
out.setdefault(iccid, []).append(item)
|
||||
timer.log("tbl_card_life", keys=len(iccid_list), batches=batches, rows=rows)
|
||||
|
||||
next_sql = """
|
||||
SELECT
|
||||
@@ -384,8 +437,11 @@ def fetch_package_lifecycles(conn, iccid_pairs: Iterable[tuple]) -> dict[str, li
|
||||
COALESCE(effect_complete_time, '9999-12-31') ASC,
|
||||
id ASC
|
||||
"""
|
||||
next_rows = 0
|
||||
next_batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(iccid_list):
|
||||
for batch in _chunks(iccid_list, batch_size):
|
||||
next_batches += 1
|
||||
cur.execute(next_sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
legacy_iccid = row.get("iccid_mark") or ""
|
||||
@@ -394,6 +450,7 @@ def fetch_package_lifecycles(conn, iccid_pairs: Iterable[tuple]) -> dict[str, li
|
||||
iccid = query_to_full.get(legacy_iccid)
|
||||
if not iccid:
|
||||
continue
|
||||
next_rows += 1
|
||||
start = row.get("start_date")
|
||||
expire = row.get("expire_date")
|
||||
start_str = start.strftime("%Y-%m-%d %H:%M:%S") if start else None
|
||||
@@ -417,6 +474,7 @@ def fetch_package_lifecycles(conn, iccid_pairs: Iterable[tuple]) -> dict[str, li
|
||||
migration_status=migration_status,
|
||||
skip_reason=skip_reason,
|
||||
))
|
||||
timer.log("tbl_next_month_card_life", keys=len(iccid_list), batches=next_batches, rows=next_rows)
|
||||
return out
|
||||
|
||||
|
||||
@@ -448,13 +506,18 @@ def _classify_next_month_lifecycle(status: int) -> tuple[str, str]:
|
||||
return "skipped", f"奇成次月待生效状态 {status} 不在迁移范围"
|
||||
|
||||
|
||||
def fetch_card_usage(conn, iccid_pairs: Iterable[tuple]) -> dict[str, LegacyCardUsage]:
|
||||
"""查每张卡的累计已用流量,并同步取当前生效正式套餐的 flow_add_discount。
|
||||
def fetch_card_usage(
|
||||
conn,
|
||||
iccid_pairs: Iterable[tuple],
|
||||
batch_size: int = BATCH_SIZE,
|
||||
) -> dict[str, LegacyCardUsage]:
|
||||
"""查每张卡的累计已用流量,分阶段索引查询替代大 JOIN。
|
||||
|
||||
奇成 ``tbl_card.total_bytes_cnt`` 是虚用量(已被套餐虚比加成)。
|
||||
这里 JOIN 当前生效正式套餐(``tbl_card_life`` status=1,type!=1 且 ``expire_date`` 最大)
|
||||
+ ``tbl_set_meal`` 取 ``flow_add_discount``,放进 ``LegacyCardUsage``;
|
||||
上层通过 ``real_used_mb_floor`` 反算真用量写入新系统。
|
||||
阶段:
|
||||
1. tbl_card — iccid_mark + total_bytes_cnt
|
||||
2. tbl_card_life — 当前生效正式套餐 meal_id(Python 选 expire_date 最大)
|
||||
3. tbl_set_meal — flow_add_discount(按 meal_id 精确查)
|
||||
4. Python 组装 LegacyCardUsage
|
||||
|
||||
Args:
|
||||
iccid_pairs: (iccid_19, iccid_20, allow_iccid_19_lookup) 列表。
|
||||
@@ -462,88 +525,119 @@ def fetch_card_usage(conn, iccid_pairs: Iterable[tuple]) -> dict[str, LegacyCard
|
||||
Returns:
|
||||
{完整 ICCID: LegacyCardUsage}。
|
||||
"""
|
||||
timer = _StageTimer("fetch_card_usage")
|
||||
pairs = _normalize_iccid_pairs(iccid_pairs)
|
||||
out: dict[str, LegacyCardUsage] = {}
|
||||
if not pairs:
|
||||
return out
|
||||
|
||||
query_to_full: dict[str, tuple[str, int]] = {}
|
||||
related_keys_by_full: dict[str, set[str]] = {}
|
||||
for i19, i20, allow_i19_lookup in pairs:
|
||||
card_key = i20 or i19
|
||||
# tbl_card 主表同时查优先键和兜底键,结果选择时按 rank:
|
||||
# 0=业务方完整 ICCID 精确命中,1=20 位卡在奇成主表只存 19 位时兜底。
|
||||
query_to_full[card_key] = (card_key, 0)
|
||||
if i20 and allow_i19_lookup:
|
||||
query_to_full[i19] = (card_key, 1)
|
||||
if not i20 or allow_i19_lookup:
|
||||
related_keys_by_full.setdefault(card_key, set()).add(i19)
|
||||
else:
|
||||
related_keys_by_full.setdefault(card_key, set())
|
||||
if i20:
|
||||
related_keys_by_full[card_key].add(i20)
|
||||
full_set = sorted(query_to_full.keys())
|
||||
by_full = _build_card_lookup(pairs)
|
||||
full_set = sorted(by_full.keys())
|
||||
|
||||
# tbl_card 主表用 iccid_mark IN 全长精确匹配;从表只在主表本身是 19 位时
|
||||
# 按前缀兜底,避免同前缀 20 位卡互相串套餐。
|
||||
sql = """
|
||||
# ── 阶段1: tbl_card ───────────────────────────────────────────────────────
|
||||
card_sql = """
|
||||
SELECT
|
||||
c.iccid_mark,
|
||||
c.total_bytes_cnt,
|
||||
COALESCE(sm.flow_add_discount, 0) AS flow_add_discount,
|
||||
CASE WHEN cl.iccid_mark IS NOT NULL THEN 1 ELSE 0 END AS has_active_package
|
||||
FROM tbl_card c
|
||||
LEFT JOIN (
|
||||
SELECT cl1.iccid_mark, cl1.meal_id
|
||||
FROM tbl_card_life cl1
|
||||
INNER JOIN (
|
||||
SELECT iccid_mark, MAX(expire_date) AS max_expire
|
||||
FROM tbl_card_life
|
||||
WHERE status = 1
|
||||
AND COALESCE(CAST(type AS CHAR), '') <> '1'
|
||||
AND iccid_mark IN %s
|
||||
GROUP BY iccid_mark
|
||||
) latest
|
||||
ON latest.iccid_mark = cl1.iccid_mark
|
||||
AND latest.max_expire = cl1.expire_date
|
||||
WHERE cl1.status = 1
|
||||
AND COALESCE(CAST(cl1.type AS CHAR), '') <> '1'
|
||||
) cl ON (
|
||||
cl.iccid_mark = c.iccid_mark
|
||||
OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(cl.iccid_mark, 19) = c.iccid_mark)
|
||||
)
|
||||
LEFT JOIN tbl_set_meal sm ON sm.id = cl.meal_id
|
||||
WHERE c.iccid_mark IN %s
|
||||
iccid_mark,
|
||||
COALESCE(total_bytes_cnt, 0) AS total_bytes_cnt
|
||||
FROM tbl_card
|
||||
WHERE iccid_mark IN %s
|
||||
"""
|
||||
hits: dict[str, list[tuple[int, dict]]] = {}
|
||||
card_rows = 0
|
||||
card_batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(full_set):
|
||||
batch_keys = {query_to_full[x][0] for x in batch}
|
||||
related_batch = sorted({x for key in batch_keys for x in related_keys_by_full[key]})
|
||||
cur.execute(sql, (tuple(related_batch), tuple(batch)))
|
||||
for batch in _chunks(full_set, batch_size):
|
||||
card_batches += 1
|
||||
cur.execute(card_sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
legacy_iccid = row["iccid_mark"]
|
||||
if not legacy_iccid:
|
||||
continue
|
||||
matched = query_to_full.get(legacy_iccid)
|
||||
matched = by_full.get(legacy_iccid)
|
||||
if matched is None:
|
||||
continue
|
||||
card_key, rank = matched
|
||||
hits.setdefault(card_key, []).append((rank, row))
|
||||
card_rows += 1
|
||||
timer.log("tbl_card", keys=len(full_set), batches=card_batches, rows=card_rows)
|
||||
|
||||
# 取最优命中行,构建 legacy_iccid 列表
|
||||
card_data: dict[str, dict] = {} # card_key → best tbl_card row
|
||||
for card_key, rows in hits.items():
|
||||
best_rank = min(rank for rank, _row in rows)
|
||||
row = next(row for rank, row in rows if rank == best_rank)
|
||||
best_rank = min(r for r, _ in rows)
|
||||
card_data[card_key] = next(row for r, row in rows if r == best_rank)
|
||||
|
||||
if not card_data:
|
||||
return out
|
||||
|
||||
legacy_iccids = sorted({row["iccid_mark"] for row in card_data.values()})
|
||||
|
||||
# ── 阶段2: tbl_card_life 当前生效正式套餐(精确匹配,Python 选最新) ─────────
|
||||
life_sql = """
|
||||
SELECT
|
||||
iccid_mark,
|
||||
COALESCE(meal_id, '') AS meal_id
|
||||
FROM tbl_card_life
|
||||
WHERE iccid_mark IN %s
|
||||
AND status = 1
|
||||
AND COALESCE(CAST(type AS CHAR), '') <> '1'
|
||||
ORDER BY iccid_mark ASC, expire_date DESC, id DESC
|
||||
"""
|
||||
current_meal_by_iccid: dict[str, str] = {} # legacy_iccid → meal_id
|
||||
life_rows = 0
|
||||
life_batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(legacy_iccids, batch_size):
|
||||
life_batches += 1
|
||||
cur.execute(life_sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
iccid = row["iccid_mark"]
|
||||
life_rows += 1
|
||||
if iccid not in current_meal_by_iccid:
|
||||
current_meal_by_iccid[iccid] = row.get("meal_id") or ""
|
||||
timer.log("tbl_card_life", keys=len(legacy_iccids), batches=life_batches, rows=life_rows)
|
||||
|
||||
# ── 阶段3: tbl_set_meal flow_add_discount(按 distinct meal_id 查) ─────────
|
||||
meal_ids = sorted({mid for mid in current_meal_by_iccid.values() if mid})
|
||||
discount_by_meal: dict[str, Decimal] = {}
|
||||
if meal_ids:
|
||||
meal_sql = "SELECT id, COALESCE(flow_add_discount, 0) AS flow_add_discount FROM tbl_set_meal WHERE id IN %s"
|
||||
meal_batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(meal_ids, batch_size):
|
||||
meal_batches += 1
|
||||
cur.execute(meal_sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
mid = str(row["id"] or "")
|
||||
if mid:
|
||||
discount_by_meal[mid] = _to_mb_decimal(row.get("flow_add_discount"))
|
||||
timer.log("tbl_set_meal", meal_ids=len(meal_ids), batches=meal_batches)
|
||||
else:
|
||||
timer.log("tbl_set_meal", meal_ids=0, skipped=True)
|
||||
|
||||
# ── 阶段4: Python 组装 ────────────────────────────────────────────────────
|
||||
for card_key, base_row in card_data.items():
|
||||
legacy_iccid = base_row["iccid_mark"]
|
||||
meal_id = current_meal_by_iccid.get(legacy_iccid, "")
|
||||
has_active = bool(meal_id or legacy_iccid in current_meal_by_iccid)
|
||||
discount = discount_by_meal.get(str(meal_id), Decimal("0")) if meal_id else Decimal("0")
|
||||
out[card_key] = LegacyCardUsage(
|
||||
iccid=card_key,
|
||||
virtual_used_mb=_to_mb_decimal(row.get("total_bytes_cnt")),
|
||||
flow_add_discount_pct=_to_mb_decimal(row.get("flow_add_discount")),
|
||||
has_active_package=bool(row.get("has_active_package")),
|
||||
virtual_used_mb=_to_mb_decimal(base_row.get("total_bytes_cnt")),
|
||||
flow_add_discount_pct=discount,
|
||||
has_active_package=has_active,
|
||||
)
|
||||
timer.log("assemble", cards=len(out))
|
||||
return out
|
||||
|
||||
|
||||
def fetch_commission_accounts(conn, agent_ids: Iterable[str]) -> dict[str, LegacyCommissionAccount]:
|
||||
def fetch_commission_accounts(
|
||||
conn,
|
||||
agent_ids: Iterable[str],
|
||||
batch_size: int = BATCH_SIZE,
|
||||
) -> dict[str, LegacyCommissionAccount]:
|
||||
"""查代理佣金账户。"""
|
||||
timer = _StageTimer("fetch_commission_accounts")
|
||||
agent_list = sorted({a for a in agent_ids if a})
|
||||
out: dict[str, LegacyCommissionAccount] = {}
|
||||
if not agent_list:
|
||||
@@ -557,11 +651,15 @@ def fetch_commission_accounts(conn, agent_ids: Iterable[str]) -> dict[str, Legac
|
||||
FROM tbl_agent_commission_account
|
||||
WHERE agent_id IN %s
|
||||
"""
|
||||
rows = 0
|
||||
batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(agent_list):
|
||||
for batch in _chunks(agent_list, batch_size):
|
||||
batches += 1
|
||||
cur.execute(sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
aid = row["agent_id"]
|
||||
rows += 1
|
||||
if not aid:
|
||||
continue
|
||||
out[aid] = LegacyCommissionAccount(
|
||||
@@ -572,107 +670,69 @@ def fetch_commission_accounts(conn, agent_ids: Iterable[str]) -> dict[str, Legac
|
||||
total_commission_amount_fen=_to_fen(row.get("total_commision_amount")),
|
||||
total_draw_amount_fen=_to_fen(row.get("total_draw_amount")),
|
||||
)
|
||||
timer.log("tbl_agent_commission_account", agents=len(agent_list), batches=batches, rows=rows)
|
||||
return out
|
||||
|
||||
|
||||
def fetch_card_meta(
|
||||
conn,
|
||||
iccid_pairs: Iterable[tuple],
|
||||
batch_size: int = BATCH_SIZE,
|
||||
) -> tuple[dict[str, LegacyCardMeta], set[str]]:
|
||||
"""查每张卡的元数据(卡本体 + 当前生效套餐 + 虚拟号)。
|
||||
"""查每张卡的元数据,分阶段索引查询替代大 JOIN。
|
||||
|
||||
阶段:
|
||||
1. tbl_card — 卡基础字段 + 去重检测
|
||||
2. tbl_vendor_category — 运营商分类名称(按 distinct category 查)
|
||||
3. tbl_card_life — 当前生效正式套餐(Python 选 expire_date 最大,精确匹配)
|
||||
4. tbl_set_meal + tbl_set_meal_series — 套餐系列(按 distinct meal_id 查)
|
||||
5. tbl_virtual_number — 虚拟号(精确匹配,不做 LEFT() 前缀扫描)
|
||||
6. Python 组装 LegacyCardMeta
|
||||
|
||||
匹配策略:
|
||||
- tbl_card 主表同时查优先键和显式允许的兜底键:有 iccid_20 时优先取
|
||||
20 位精确命中;只有业务方也显式填了 iccid_19 时,才允许退回 19 位。
|
||||
只填 iccid_20 的固定 20 位卡绝不拿 19 位前缀查主表。
|
||||
- tbl_card_life / tbl_virtual_number 从表在主表本身是 19 位时才用
|
||||
LEFT(iccid_mark, 19) 兜底,避免同前缀 20 位卡互相串数据
|
||||
- tbl_card 主表同时查优先键和显式允许的兜底键;20 位精确命中优先
|
||||
- 从表(card_life / virtual_number)用 tbl_card 实际返回的 iccid_mark 精确查询
|
||||
- 不再使用 LEFT(vn.iccid_mark, 19) 前缀匹配,避免全表扫描
|
||||
|
||||
Args:
|
||||
iccid_pairs: (iccid_19, iccid_20, allow_iccid_19_lookup) 列表。
|
||||
|
||||
Returns:
|
||||
(metas, duplicate_iccids):
|
||||
- metas: {完整 ICCID: LegacyCardMeta} 单匹结果
|
||||
- duplicate_iccids: 一份业务方完整 ICCID 命中奇成 tbl_card 多条记录的 set,
|
||||
这些卡不进 metas,sql_builder 会写 errors.csv 阻断
|
||||
- metas: {完整 ICCID: LegacyCardMeta}
|
||||
- duplicate_iccids: 一份业务方 ICCID 命中多条 tbl_card 记录的 set
|
||||
"""
|
||||
timer = _StageTimer("fetch_card_meta")
|
||||
pairs = _normalize_iccid_pairs(iccid_pairs)
|
||||
if not pairs:
|
||||
return {}, set()
|
||||
|
||||
by_full: dict[str, tuple[str, int]] = {} # tbl_card 查询键 → (完整 ICCID, rank)
|
||||
related_keys_by_full: dict[str, set[str]] = {}
|
||||
for i19, i20, allow_i19_lookup in pairs:
|
||||
card_key = i20 or i19
|
||||
by_full[card_key] = (card_key, 0)
|
||||
if i20 and allow_i19_lookup:
|
||||
by_full[i19] = (card_key, 1)
|
||||
if not i20 or allow_i19_lookup:
|
||||
related_keys_by_full.setdefault(card_key, set()).add(i19)
|
||||
else:
|
||||
related_keys_by_full.setdefault(card_key, set())
|
||||
if i20:
|
||||
related_keys_by_full[card_key].add(i20)
|
||||
by_full = _build_card_lookup(pairs)
|
||||
full_set = sorted(by_full.keys())
|
||||
|
||||
sql = """
|
||||
# ── 阶段1: tbl_card ───────────────────────────────────────────────────────
|
||||
card_sql = """
|
||||
SELECT
|
||||
c.id AS card_id,
|
||||
c.iccid_mark AS iccid,
|
||||
COALESCE(c.account_id, '') AS account_id,
|
||||
COALESCE(c.account_name, '') AS account_name,
|
||||
COALESCE(c.category, 0) AS category,
|
||||
COALESCE(vc.category_name, '') AS category_name,
|
||||
COALESCE(c.agent_id, '') AS agent_id,
|
||||
COALESCE(c.agent_name, '') AS agent_name,
|
||||
COALESCE(c.phone, '') AS phone,
|
||||
COALESCE(vn.vcode, '') AS vcode,
|
||||
COALESCE(cl.meal_id, '') AS meal_id,
|
||||
COALESCE(cl.meal_name, '') AS meal_name,
|
||||
COALESCE(CAST(cl.meal_type AS CHAR), '') AS meal_type,
|
||||
cl.expire_date AS expire_date,
|
||||
COALESCE(sm.meal_series_id, '') AS series_id,
|
||||
COALESCE(sms.name, '') AS series_name
|
||||
FROM tbl_card c
|
||||
LEFT JOIN tbl_vendor_category vc ON vc.category = c.category
|
||||
LEFT JOIN tbl_virtual_number vn ON (
|
||||
vn.iccid_mark = c.iccid_mark
|
||||
OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(vn.iccid_mark, 19) = c.iccid_mark)
|
||||
)
|
||||
LEFT JOIN (
|
||||
SELECT cl1.iccid_mark, cl1.meal_id, cl1.meal_name, cl1.type AS meal_type, cl1.expire_date
|
||||
FROM tbl_card_life cl1
|
||||
INNER JOIN (
|
||||
SELECT iccid_mark, MAX(expire_date) AS max_expire
|
||||
FROM tbl_card_life
|
||||
WHERE status = 1
|
||||
AND COALESCE(CAST(type AS CHAR), '') <> '1'
|
||||
AND iccid_mark IN %s
|
||||
GROUP BY iccid_mark
|
||||
) latest
|
||||
ON latest.iccid_mark = cl1.iccid_mark
|
||||
AND latest.max_expire = cl1.expire_date
|
||||
WHERE cl1.status = 1
|
||||
AND COALESCE(CAST(cl1.type AS CHAR), '') <> '1'
|
||||
) cl ON (
|
||||
cl.iccid_mark = c.iccid_mark
|
||||
OR (CHAR_LENGTH(c.iccid_mark) = 19 AND LEFT(cl.iccid_mark, 19) = c.iccid_mark)
|
||||
)
|
||||
LEFT JOIN tbl_set_meal sm ON sm.id = cl.meal_id
|
||||
LEFT JOIN tbl_set_meal_series sms ON sms.id = sm.meal_series_id
|
||||
WHERE c.iccid_mark IN %s
|
||||
id AS card_id,
|
||||
iccid_mark,
|
||||
COALESCE(account_id, '') AS account_id,
|
||||
COALESCE(account_name, '') AS account_name,
|
||||
COALESCE(category, 0) AS category,
|
||||
COALESCE(agent_id, '') AS agent_id,
|
||||
COALESCE(agent_name, '') AS agent_name,
|
||||
COALESCE(phone, '') AS phone
|
||||
FROM tbl_card
|
||||
WHERE iccid_mark IN %s
|
||||
"""
|
||||
|
||||
# 收集每个完整 ICCID 命中的奇成行(可能多条 = 脏数据)
|
||||
hits: dict[str, list[tuple[int, dict]]] = {}
|
||||
card_rows = 0
|
||||
card_batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(full_set):
|
||||
batch_keys = {by_full[x][0] for x in batch}
|
||||
related_batch = sorted({x for key in batch_keys for x in related_keys_by_full[key]})
|
||||
cur.execute(sql, (tuple(related_batch), tuple(batch)))
|
||||
for batch in _chunks(full_set, batch_size):
|
||||
card_batches += 1
|
||||
cur.execute(card_sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
legacy_iccid = row["iccid"]
|
||||
legacy_iccid = row["iccid_mark"]
|
||||
if not legacy_iccid:
|
||||
continue
|
||||
matched = by_full.get(legacy_iccid)
|
||||
@@ -680,45 +740,162 @@ def fetch_card_meta(
|
||||
continue
|
||||
card_key, rank = matched
|
||||
hits.setdefault(card_key, []).append((rank, row))
|
||||
card_rows += 1
|
||||
timer.log("tbl_card", keys=len(full_set), batches=card_batches, rows=card_rows)
|
||||
|
||||
metas: dict[str, LegacyCardMeta] = {}
|
||||
# 去重:同一完整 ICCID 命中多条不同 card_id → duplicates
|
||||
metas_base: dict[str, dict] = {} # card_key → best tbl_card row
|
||||
duplicates: set[str] = set()
|
||||
categories: set[int] = set()
|
||||
|
||||
for card_key, rows in hits.items():
|
||||
best_rank = min(rank for rank, _row in rows)
|
||||
best_rows = [row for rank, row in rows if rank == best_rank]
|
||||
# 同一张 tbl_card 可能因 card_life / virtual_number 前缀兜底 JOIN 出多行,
|
||||
# 这不是主表重复;只有不同 card_id 才算同一业务 ICCID 命中多张卡。
|
||||
best_rank = min(r for r, _ in rows)
|
||||
best_rows = [row for r, row in rows if r == best_rank]
|
||||
rows_by_card_id: dict[str, dict] = {}
|
||||
for row in best_rows:
|
||||
rows_by_card_id[str(row.get("card_id") or row.get("iccid") or "")] = row
|
||||
rows_by_card_id[str(row.get("card_id") or row.get("iccid_mark") or "")] = row
|
||||
if len(rows_by_card_id) > 1:
|
||||
duplicates.add(card_key)
|
||||
continue
|
||||
row = next(iter(rows_by_card_id.values()))
|
||||
expire = row.get("expire_date")
|
||||
metas_base[card_key] = row
|
||||
categories.add(int(row.get("category") or 0))
|
||||
|
||||
if not metas_base:
|
||||
return {}, duplicates
|
||||
|
||||
# legacy_iccids: tbl_card 实际存储的 iccid_mark 值(用于从表精确查询)
|
||||
legacy_iccids = sorted({row["iccid_mark"] for row in metas_base.values()})
|
||||
|
||||
# ── 阶段2: tbl_vendor_category(distinct category,通常极少几十个) ─────────
|
||||
category_names: dict[int, str] = {}
|
||||
cat_batches = 0
|
||||
if categories:
|
||||
cat_sql = """
|
||||
SELECT category, COALESCE(category_name, '') AS category_name
|
||||
FROM tbl_vendor_category
|
||||
WHERE category IN %s
|
||||
"""
|
||||
cat_list = sorted(categories)
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(cat_list, batch_size):
|
||||
cat_batches += 1
|
||||
cur.execute(cat_sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
category_names[int(row["category"])] = row.get("category_name") or ""
|
||||
timer.log("tbl_vendor_category", categories=len(categories), batches=cat_batches)
|
||||
|
||||
# ── 阶段3: tbl_card_life 当前生效正式套餐(精确匹配,Python 选最新) ─────────
|
||||
# 精确用 legacy_iccid 查询;不再做 LEFT(iccid_mark, 19) 前缀扫描
|
||||
life_sql = """
|
||||
SELECT
|
||||
iccid_mark,
|
||||
COALESCE(meal_id, '') AS meal_id,
|
||||
COALESCE(meal_name, '') AS meal_name,
|
||||
COALESCE(CAST(type AS CHAR), '') AS meal_type,
|
||||
expire_date
|
||||
FROM tbl_card_life
|
||||
WHERE iccid_mark IN %s
|
||||
AND status = 1
|
||||
AND COALESCE(CAST(type AS CHAR), '') <> '1'
|
||||
ORDER BY iccid_mark ASC, expire_date DESC, id DESC
|
||||
"""
|
||||
current_meals: dict[str, dict] = {} # legacy_iccid → 当前生效套餐行
|
||||
meal_ids: set[str] = set()
|
||||
life_rows = 0
|
||||
life_batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(legacy_iccids, batch_size):
|
||||
life_batches += 1
|
||||
cur.execute(life_sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
iccid = row["iccid_mark"]
|
||||
life_rows += 1
|
||||
if iccid not in current_meals:
|
||||
current_meals[iccid] = row
|
||||
if row.get("meal_id"):
|
||||
meal_ids.add(row["meal_id"])
|
||||
timer.log("tbl_card_life", keys=len(legacy_iccids), batches=life_batches, rows=life_rows, found=len(current_meals))
|
||||
|
||||
# ── 阶段4: tbl_set_meal + tbl_set_meal_series(按 distinct meal_id) ───────
|
||||
meal_series: dict[str, tuple[str, str]] = {} # meal_id → (series_id, series_name)
|
||||
meal_batches = 0
|
||||
if meal_ids:
|
||||
meal_sql = """
|
||||
SELECT
|
||||
sm.id AS meal_id,
|
||||
COALESCE(sm.meal_series_id, '') AS series_id,
|
||||
COALESCE(sms.name, '') AS series_name
|
||||
FROM tbl_set_meal sm
|
||||
LEFT JOIN tbl_set_meal_series sms ON sms.id = sm.meal_series_id
|
||||
WHERE sm.id IN %s
|
||||
"""
|
||||
meal_id_list = sorted(meal_ids)
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(meal_id_list, batch_size):
|
||||
meal_batches += 1
|
||||
cur.execute(meal_sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
mid = str(row["meal_id"] or "")
|
||||
if mid:
|
||||
meal_series[mid] = (row.get("series_id") or "", row.get("series_name") or "")
|
||||
timer.log("tbl_set_meal", meal_ids=len(meal_ids), batches=meal_batches)
|
||||
|
||||
# ── 阶段5: tbl_virtual_number(精确匹配,不做前缀扫描) ─────────────────────
|
||||
virtual_nos: dict[str, str] = {} # legacy_iccid → vcode
|
||||
vn_rows = 0
|
||||
vn_batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(legacy_iccids, batch_size):
|
||||
vn_batches += 1
|
||||
cur.execute(
|
||||
"SELECT iccid_mark, COALESCE(vcode, '') AS vcode FROM tbl_virtual_number WHERE iccid_mark IN %s",
|
||||
(tuple(batch),),
|
||||
)
|
||||
for row in cur.fetchall():
|
||||
iccid = row["iccid_mark"]
|
||||
vn_rows += 1
|
||||
if iccid:
|
||||
virtual_nos[iccid] = row.get("vcode") or ""
|
||||
timer.log("tbl_virtual_number", keys=len(legacy_iccids), batches=vn_batches, rows=vn_rows)
|
||||
|
||||
# ── 阶段6: Python 组装 ────────────────────────────────────────────────────
|
||||
metas: dict[str, LegacyCardMeta] = {}
|
||||
for card_key, base_row in metas_base.items():
|
||||
legacy_iccid = base_row["iccid_mark"]
|
||||
meal_row = current_meals.get(legacy_iccid, {})
|
||||
expire = meal_row.get("expire_date")
|
||||
expire_str = expire.strftime("%Y-%m-%d %H:%M:%S") if expire else None
|
||||
meal_id = meal_row.get("meal_id") or ""
|
||||
series_id, series_name = meal_series.get(str(meal_id), ("", ""))
|
||||
metas[card_key] = LegacyCardMeta(
|
||||
legacy_raw_iccid=row["iccid"],
|
||||
account_id=row.get("account_id") or "",
|
||||
account_name=row.get("account_name") or "",
|
||||
category=int(row.get("category") or 0),
|
||||
category_name=row.get("category_name") or "",
|
||||
agent_id=row.get("agent_id") or "",
|
||||
agent_name=row.get("agent_name") or "",
|
||||
msisdn=row.get("phone") or "",
|
||||
virtual_no=row.get("vcode") or "",
|
||||
current_meal_id=row.get("meal_id") or "",
|
||||
current_meal_name=row.get("meal_name") or "",
|
||||
current_meal_type=str(row.get("meal_type") or ""),
|
||||
legacy_raw_iccid=legacy_iccid,
|
||||
account_id=base_row.get("account_id") or "",
|
||||
account_name=base_row.get("account_name") or "",
|
||||
category=int(base_row.get("category") or 0),
|
||||
category_name=category_names.get(int(base_row.get("category") or 0), ""),
|
||||
agent_id=base_row.get("agent_id") or "",
|
||||
agent_name=base_row.get("agent_name") or "",
|
||||
msisdn=base_row.get("phone") or "",
|
||||
virtual_no=virtual_nos.get(legacy_iccid, ""),
|
||||
current_meal_id=meal_id,
|
||||
current_meal_name=meal_row.get("meal_name") or "",
|
||||
current_meal_type=str(meal_row.get("meal_type") or ""),
|
||||
current_expire_date=expire_str,
|
||||
current_series_id=row.get("series_id") or "",
|
||||
current_series_name=row.get("series_name") or "",
|
||||
current_series_id=series_id,
|
||||
current_series_name=series_name,
|
||||
)
|
||||
timer.log("assemble", cards=len(metas), duplicates=len(duplicates))
|
||||
return metas, duplicates
|
||||
|
||||
|
||||
def fetch_meal_meta(conn, meal_ids: Iterable[str]) -> dict[str, LegacyMealMeta]:
|
||||
def fetch_meal_meta(
|
||||
conn,
|
||||
meal_ids: Iterable[str],
|
||||
batch_size: int = BATCH_SIZE,
|
||||
) -> dict[str, LegacyMealMeta]:
|
||||
"""查套餐 + 系列元数据(供 scan_legacy 收集映射 legacy_* 字段)。"""
|
||||
timer = _StageTimer("fetch_meal_meta")
|
||||
id_list = sorted({m for m in meal_ids if m})
|
||||
out: dict[str, LegacyMealMeta] = {}
|
||||
if not id_list:
|
||||
@@ -733,11 +910,15 @@ def fetch_meal_meta(conn, meal_ids: Iterable[str]) -> dict[str, LegacyMealMeta]:
|
||||
LEFT JOIN tbl_set_meal_series sms ON sms.id = sm.meal_series_id
|
||||
WHERE sm.id IN %s
|
||||
"""
|
||||
rows = 0
|
||||
batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(id_list):
|
||||
for batch in _chunks(id_list, batch_size):
|
||||
batches += 1
|
||||
cur.execute(sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
mid = row["meal_id"]
|
||||
rows += 1
|
||||
if not mid:
|
||||
continue
|
||||
out[mid] = LegacyMealMeta(
|
||||
@@ -746,11 +927,17 @@ def fetch_meal_meta(conn, meal_ids: Iterable[str]) -> dict[str, LegacyMealMeta]:
|
||||
series_id=row.get("series_id") or "",
|
||||
series_name=row.get("series_name") or "",
|
||||
)
|
||||
timer.log("tbl_set_meal", meal_ids=len(id_list), batches=batches, rows=rows)
|
||||
return out
|
||||
|
||||
|
||||
def fetch_agent_balances(conn, agent_ids: Iterable[str]) -> dict[str, LegacyAgentBalance]:
|
||||
def fetch_agent_balances(
|
||||
conn,
|
||||
agent_ids: Iterable[str],
|
||||
batch_size: int = BATCH_SIZE,
|
||||
) -> dict[str, LegacyAgentBalance]:
|
||||
"""查代理主钱包余额(tbl_agent.balance_money,元单位)。"""
|
||||
timer = _StageTimer("fetch_agent_balances")
|
||||
agent_list = sorted({a for a in agent_ids if a})
|
||||
out: dict[str, LegacyAgentBalance] = {}
|
||||
if not agent_list:
|
||||
@@ -761,12 +948,17 @@ def fetch_agent_balances(conn, agent_ids: Iterable[str]) -> dict[str, LegacyAgen
|
||||
FROM tbl_agent
|
||||
WHERE id IN %s
|
||||
"""
|
||||
rows = 0
|
||||
batches = 0
|
||||
with conn.cursor() as cur:
|
||||
for batch in _chunks(agent_list):
|
||||
for batch in _chunks(agent_list, batch_size):
|
||||
batches += 1
|
||||
cur.execute(sql, (tuple(batch),))
|
||||
for row in cur.fetchall():
|
||||
aid = row["id"]
|
||||
rows += 1
|
||||
if not aid:
|
||||
continue
|
||||
out[aid] = LegacyAgentBalance(agent_id=aid, balance_fen=_to_fen(row.get("balance_money")))
|
||||
timer.log("tbl_agent", agents=len(agent_list), batches=batches, rows=rows)
|
||||
return out
|
||||
|
||||
@@ -139,6 +139,7 @@ class AgentMapping:
|
||||
legacy_agent_id: str
|
||||
legacy_agent_name: str
|
||||
target_shop_code: Optional[str] # legacy_agent 模式下允许 null,表示该代理资产进平台库存
|
||||
no_downtime: bool = False # 迁移时免停机:有生效套餐的卡直接以 network_status=1 导入
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -166,6 +167,11 @@ class Mapping:
|
||||
return None
|
||||
return self.series.get(str(legacy_series_id).strip())
|
||||
|
||||
def is_no_downtime_agent(self, legacy_agent_id: str) -> bool:
|
||||
"""判断代理是否配置了免停机迁移。"""
|
||||
agent = self.agents.get(str(legacy_agent_id or "").strip())
|
||||
return agent.no_downtime if agent else False
|
||||
|
||||
def lookup_agent_shop_code(self, legacy_agent_id: str) -> Optional[str]:
|
||||
"""按 agent_id 查 target_shop_code,空字符串视为 None(进平台库存)。"""
|
||||
if not legacy_agent_id:
|
||||
@@ -260,6 +266,7 @@ def load_mapping(config_dir: Path) -> Mapping:
|
||||
legacy_agent_id=str(item["legacy_agent_id"]).strip(),
|
||||
legacy_agent_name=str(item.get("legacy_agent_name") or ""),
|
||||
target_shop_code=_opt_shop_code(item.get("target_shop_code"), "agents[].target_shop_code"),
|
||||
no_downtime=bool(item.get("no_downtime", False)),
|
||||
)
|
||||
m.agents[a.legacy_agent_id] = a
|
||||
|
||||
@@ -536,6 +543,7 @@ def _append_agents(lines: list[str], mapping: Mapping) -> None:
|
||||
f" - legacy_agent_id: {_yaml_scalar(item.legacy_agent_id)}",
|
||||
f" legacy_agent_name: {_yaml_scalar(item.legacy_agent_name)}",
|
||||
f" target_shop_code: {_yaml_scalar(item.target_shop_code)}",
|
||||
f" no_downtime: {_yaml_scalar(item.no_downtime)}",
|
||||
])
|
||||
|
||||
|
||||
|
||||
@@ -621,6 +621,20 @@ def _write_iot_cards(
|
||||
card_status = 2 if shop_code else 1
|
||||
shop_id_expr = _shop_id_sub(shop_code)
|
||||
|
||||
# 免停机迁移:代理配置 no_downtime=true 且卡在奇成有当前生效正式套餐
|
||||
# finalize 保证最后执行,此时 step2 套餐已落库,轮询系统不会在此之前触碰这张卡
|
||||
is_no_downtime = bool(meta.current_meal_id) and mapping.is_no_downtime_agent(meta.agent_id)
|
||||
if is_no_downtime:
|
||||
activation_status = 1
|
||||
real_name_status = 1 # 卡在线说明已实名,写 1 防止 EvaluateAndAct 因 not_realname 停机
|
||||
network_status = 1
|
||||
stop_reason_expr = "NULL"
|
||||
else:
|
||||
activation_status = 0
|
||||
real_name_status = 0
|
||||
network_status = 0
|
||||
stop_reason_expr = _sql_str(_initial_polling_stop_reason(c))
|
||||
|
||||
f.write(
|
||||
"INSERT INTO tb_iot_card (\n"
|
||||
" iccid, iccid_19, iccid_20,\n"
|
||||
@@ -636,9 +650,9 @@ def _write_iot_cards(
|
||||
f" {_sql_str(c.iccid_full)}, {_sql_str(c.iccid_19)}, {iccid_20_expr},\n"
|
||||
f" {carrier.target_carrier_id}, {_sql_str(carrier.target_carrier_type)}, {_sql_str(carrier.target_carrier_name)},\n"
|
||||
f" {_sql_str(card_category)}, {_sql_str(mapping.migration_batch_no)},\n"
|
||||
f" {card_status}, 0, 0, 0,\n"
|
||||
f" {card_status}, {activation_status}, {real_name_status}, {network_status},\n"
|
||||
f" 1, 1, {is_standalone},\n"
|
||||
f" 'after_order', {_sql_str(_initial_polling_stop_reason(c))},\n"
|
||||
f" 'after_order', {stop_reason_expr},\n"
|
||||
f" {_sql_str(meta.msisdn) if meta.msisdn else 'NULL'}, "
|
||||
f"{_sql_str(card_virtual_no) if card_virtual_no else 'NULL'}, "
|
||||
f"{shop_id_expr}, {series_id_expr},\n"
|
||||
@@ -1077,7 +1091,6 @@ def write_step2(
|
||||
cards: list[CardRow],
|
||||
devices: list[DeviceRow],
|
||||
mapping: Mapping,
|
||||
card_metas: dict[str, LegacyCardMeta],
|
||||
usage_snapshots: dict[str, LegacyCardUsage],
|
||||
package_lifecycles: dict[str, list[LegacyPackageLifecycle]],
|
||||
commission_snapshots: dict[str, LegacyCommissionAccount],
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
工作流:
|
||||
1. 读 mapping.yaml(必须先跑过 scan_legacy.py)
|
||||
2. 读 cards.csv / devices.csv
|
||||
3. 连奇成查每张卡的元数据(category/agent_id/msisdn/virtual_no)
|
||||
3. 连奇成查每张卡的元数据(category/agent_id/msisdn/virtual_no/当前生效套餐)
|
||||
4. 应用 mapping 把 legacy_* 翻译成 target_*,自动决定归属
|
||||
5. 生成 output/step1_*.sql + errors.csv + summary.txt
|
||||
|
||||
@@ -14,11 +14,13 @@
|
||||
|
||||
用法:
|
||||
python3 migrate_assets.py [--config-dir config] [--resources-dir resources] [--output-dir output]
|
||||
[--batch-size 1000]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
@@ -31,6 +33,7 @@ def main() -> int:
|
||||
parser.add_argument("--config-dir", default="config", help="yaml 配置目录")
|
||||
parser.add_argument("--resources-dir", default="resources", help="CSV 资源目录")
|
||||
parser.add_argument("--output-dir", default="output", help="SQL/CSV 输出目录")
|
||||
parser.add_argument("--batch-size", type=int, default=legacy_query.BATCH_SIZE, help="老库批量查询大小")
|
||||
args = parser.parse_args()
|
||||
|
||||
base = Path(__file__).resolve().parent
|
||||
@@ -38,16 +41,26 @@ def main() -> int:
|
||||
resources_dir = (base / args.resources_dir).resolve()
|
||||
output_dir = (base / args.output_dir).resolve()
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
batch_size = args.batch_size
|
||||
|
||||
t0 = time.perf_counter()
|
||||
mapping = mapping_loader.load_mapping(config_dir)
|
||||
cards, devices, errors = csv_loader.load_assets(resources_dir, mapping)
|
||||
print(f"[migrate] 加载 CSV: cards={len(cards)} devices={len(devices)} elapsed={time.perf_counter()-t0:.1f}s")
|
||||
|
||||
# 连奇成查卡元数据(运营商分类、agent_id、msisdn、virtual_no、当前生效套餐)
|
||||
iccid_pairs = [(c.iccid_19, c.iccid_20, c.allow_iccid_19_lookup) for c in cards]
|
||||
|
||||
t1 = time.perf_counter()
|
||||
dsn = mapping_loader.load_legacy_dsn(config_dir)
|
||||
with legacy_query.connect_readonly(dsn) as conn:
|
||||
card_metas, dup_iccids = legacy_query.fetch_card_meta(conn, iccid_pairs)
|
||||
print(f"[migrate] 连接老库: elapsed={time.perf_counter()-t1:.1f}s")
|
||||
|
||||
t2 = time.perf_counter()
|
||||
card_metas, dup_iccids = legacy_query.fetch_card_meta(conn, iccid_pairs, batch_size=batch_size)
|
||||
print(f"[migrate] fetch_card_meta: cards={len(card_metas)} duplicates={len(dup_iccids)} elapsed={time.perf_counter()-t2:.1f}s")
|
||||
|
||||
t3 = time.perf_counter()
|
||||
sql_builder.write_step1(
|
||||
output_dir,
|
||||
cards=cards,
|
||||
@@ -57,8 +70,10 @@ def main() -> int:
|
||||
duplicate_iccids=dup_iccids,
|
||||
errors=errors,
|
||||
)
|
||||
print(f"[migrate] SQL 生成: elapsed={time.perf_counter()-t3:.1f}s")
|
||||
|
||||
print(f"完成:卡 {len(cards)} 张,设备 {len(devices)} 台,异常 {len(errors)} 行")
|
||||
total = time.perf_counter() - t0
|
||||
print(f"完成:卡 {len(cards)} 张,设备 {len(devices)} 台,异常 {len(errors)} 行,总耗时 {total:.1f}s")
|
||||
print(f"SQL 输出: {output_dir}")
|
||||
return 0
|
||||
|
||||
|
||||
@@ -5,17 +5,19 @@
|
||||
|
||||
工作流:
|
||||
1. 读 mapping.yaml + cards.csv
|
||||
2. 连奇成查:卡元数据(card_meta) + 累计流量(card_usage) + 代理佣金/主钱包余额
|
||||
2. 连奇成查:累计流量(card_usage) + 套餐生命周期 + 代理佣金/主钱包余额
|
||||
3. 卡 → 当前生效套餐 → mapping.packages → target_package_id
|
||||
4. 生成 step2_*.sql
|
||||
|
||||
用法:
|
||||
python3 migrate_runtime.py [--config-dir config] [--resources-dir resources] [--output-dir output]
|
||||
[--batch-size 1000]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
@@ -28,6 +30,7 @@ def main() -> int:
|
||||
parser.add_argument("--config-dir", default="config", help="yaml 配置目录")
|
||||
parser.add_argument("--resources-dir", default="resources", help="CSV 资源目录")
|
||||
parser.add_argument("--output-dir", default="output", help="SQL/CSV 输出目录")
|
||||
parser.add_argument("--batch-size", type=int, default=legacy_query.BATCH_SIZE, help="老库批量查询大小")
|
||||
args = parser.parse_args()
|
||||
|
||||
base = Path(__file__).resolve().parent
|
||||
@@ -35,9 +38,13 @@ def main() -> int:
|
||||
resources_dir = (base / args.resources_dir).resolve()
|
||||
output_dir = (base / args.output_dir).resolve()
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
batch_size = args.batch_size
|
||||
|
||||
t0 = time.perf_counter()
|
||||
mapping = mapping_loader.load_mapping(config_dir)
|
||||
cards, devices, errors = csv_loader.load_assets(resources_dir, mapping)
|
||||
print(f"[migrate] 加载 CSV: cards={len(cards)} devices={len(devices)} elapsed={time.perf_counter()-t0:.1f}s")
|
||||
|
||||
if errors:
|
||||
sql_builder.write_runtime_input_errors(output_dir, errors)
|
||||
print(f"cards.csv/devices.csv 存在 {len(errors)} 个基础错误,已写入 {output_dir / 'errors.csv'}", file=sys.stderr)
|
||||
@@ -46,27 +53,39 @@ def main() -> int:
|
||||
iccid_pairs = [(c.iccid_19, c.iccid_20, c.allow_iccid_19_lookup) for c in cards]
|
||||
agent_ids = [a.legacy_agent_id for a in mapping.agents.values() if (a.target_shop_code or "").strip()]
|
||||
|
||||
t1 = time.perf_counter()
|
||||
dsn = mapping_loader.load_legacy_dsn(config_dir)
|
||||
with legacy_query.connect_readonly(dsn) as conn:
|
||||
card_metas, _dup_iccids = legacy_query.fetch_card_meta(conn, iccid_pairs)
|
||||
usage_snapshots = legacy_query.fetch_card_usage(conn, iccid_pairs)
|
||||
package_lifecycles = legacy_query.fetch_package_lifecycles(conn, iccid_pairs)
|
||||
commission_snapshots = legacy_query.fetch_commission_accounts(conn, agent_ids) if agent_ids else {}
|
||||
agent_balances = legacy_query.fetch_agent_balances(conn, agent_ids) if agent_ids else {}
|
||||
print(f"[migrate] 连接老库: elapsed={time.perf_counter()-t1:.1f}s")
|
||||
|
||||
t2 = time.perf_counter()
|
||||
usage_snapshots = legacy_query.fetch_card_usage(conn, iccid_pairs, batch_size=batch_size)
|
||||
print(f"[migrate] fetch_card_usage: cards={len(usage_snapshots)} elapsed={time.perf_counter()-t2:.1f}s")
|
||||
|
||||
t3 = time.perf_counter()
|
||||
package_lifecycles = legacy_query.fetch_package_lifecycles(conn, iccid_pairs, batch_size=batch_size)
|
||||
print(f"[migrate] fetch_package_lifecycles: cards={len(package_lifecycles)} elapsed={time.perf_counter()-t3:.1f}s")
|
||||
|
||||
t4 = time.perf_counter()
|
||||
commission_snapshots = legacy_query.fetch_commission_accounts(conn, agent_ids, batch_size=batch_size) if agent_ids else {}
|
||||
agent_balances = legacy_query.fetch_agent_balances(conn, agent_ids, batch_size=batch_size) if agent_ids else {}
|
||||
print(f"[migrate] fetch_agent_data: agents={len(agent_ids)} elapsed={time.perf_counter()-t4:.1f}s")
|
||||
|
||||
t5 = time.perf_counter()
|
||||
warnings = sql_builder.write_step2(
|
||||
output_dir,
|
||||
cards=cards,
|
||||
devices=devices,
|
||||
mapping=mapping,
|
||||
card_metas=card_metas,
|
||||
usage_snapshots=usage_snapshots,
|
||||
package_lifecycles=package_lifecycles,
|
||||
commission_snapshots=commission_snapshots,
|
||||
agent_balances=agent_balances,
|
||||
)
|
||||
print(f"[migrate] SQL 生成: elapsed={time.perf_counter()-t5:.1f}s")
|
||||
|
||||
print(f"完成:卡 {len(cards)} 张,代理 {len(agent_ids)} 个,警告 {len(warnings)} 条")
|
||||
total = time.perf_counter() - t0
|
||||
print(f"完成:卡 {len(cards)} 张,代理 {len(agent_ids)} 个,警告 {len(warnings)} 条,总耗时 {total:.1f}s")
|
||||
print(f"SQL 输出: {output_dir}")
|
||||
return 0
|
||||
|
||||
|
||||
186
scripts/migration/remove_cards.py
Normal file
186
scripts/migration/remove_cards.py
Normal file
@@ -0,0 +1,186 @@
|
||||
#!/usr/bin/env python3
|
||||
"""卡删除脚本:读取 ICCID 列表,生成数据库删除 SQL 和 Redis 轮询清理命令。
|
||||
|
||||
用法:
|
||||
python3 remove_cards.py --input <csv文件> [--output-dir output]
|
||||
|
||||
CSV 格式: 纯文本,每行一个 ICCID,首行可有 "iccid" 表头(自动跳过)
|
||||
|
||||
输出:
|
||||
output/remove_cards_db.sql -- 数据库软删 + 标识符硬删 SQL(事务封装)
|
||||
output/remove_cards_redis_gen.sql -- 在数据库执行后把输出通过 redis-cli 清理轮询队列
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_iccids(csv_path: Path) -> list[str]:
|
||||
"""读取 CSV 文件中的 ICCID 列表,自动跳过空行和表头。"""
|
||||
iccids: list[str] = []
|
||||
with csv_path.open(encoding="utf-8") as f:
|
||||
for lineno, raw in enumerate(f, 1):
|
||||
line = raw.strip()
|
||||
if not line:
|
||||
continue
|
||||
# 跳过纯字母表头(如 "iccid")
|
||||
if lineno == 1 and not line.isdigit() and not line.startswith("8"):
|
||||
print(f"[remove] 跳过表头: {line!r}", file=sys.stderr)
|
||||
continue
|
||||
iccids.append(line)
|
||||
return iccids
|
||||
|
||||
|
||||
def sql_in_list(iccids: list[str]) -> str:
|
||||
"""生成 SQL IN 列表字符串,如 '898...', '898...'"""
|
||||
return ", ".join(f"'{i}'" for i in iccids)
|
||||
|
||||
|
||||
def generate_db_sql(iccids: list[str], generated_at: str) -> str:
|
||||
in_list = sql_in_list(iccids)
|
||||
count = len(iccids)
|
||||
return f"""\
|
||||
-- ============================================================
|
||||
-- 迁移卡删除脚本 (数据库部分)
|
||||
-- 生成时间: {generated_at}
|
||||
-- 卡数量 : {count}
|
||||
-- 执行前请先用 remove_cards_redis_gen.sql 生成 Redis 清理命令并执行
|
||||
-- 建议执行: psql ... -1 -f <此文件>
|
||||
-- ============================================================
|
||||
|
||||
BEGIN;
|
||||
|
||||
-- 临时表:本次要删除的卡
|
||||
CREATE TEMP TABLE tmp_remove_cards AS
|
||||
SELECT id, iccid
|
||||
FROM tb_iot_card
|
||||
WHERE iccid IN ({in_list});
|
||||
|
||||
DO $$ BEGIN
|
||||
RAISE NOTICE '本次将删除 % 张卡(共输入 {count} 个 ICCID)', (SELECT COUNT(*) FROM tmp_remove_cards);
|
||||
END $$;
|
||||
|
||||
-- 1. 删除套餐使用记录
|
||||
DELETE FROM tb_package_usage
|
||||
WHERE iot_card_id IN (SELECT id FROM tmp_remove_cards);
|
||||
|
||||
-- 2. 删除订单
|
||||
DELETE FROM tb_order
|
||||
WHERE iot_card_id IN (SELECT id FROM tmp_remove_cards);
|
||||
|
||||
-- 3. 删除分配记录
|
||||
DELETE FROM tb_asset_allocation_record
|
||||
WHERE asset_type = 'iot_card'
|
||||
AND asset_id IN (SELECT id FROM tmp_remove_cards);
|
||||
|
||||
-- 4. 删除资产钱包
|
||||
DELETE FROM tb_asset_wallet
|
||||
WHERE resource_type = 'iot_card'
|
||||
AND resource_id IN (SELECT id FROM tmp_remove_cards);
|
||||
|
||||
-- 5. 删除设备绑卡关系
|
||||
DELETE FROM tb_device_sim_binding
|
||||
WHERE iot_card_id IN (SELECT id FROM tmp_remove_cards);
|
||||
|
||||
-- 6. 删除资产标识符
|
||||
DELETE FROM tb_asset_identifier
|
||||
WHERE asset_type = 'iot_card'
|
||||
AND asset_id IN (SELECT id FROM tmp_remove_cards);
|
||||
|
||||
-- 7. 删除卡本体(最后执行)
|
||||
DELETE FROM tb_iot_card
|
||||
WHERE id IN (SELECT id FROM tmp_remove_cards);
|
||||
|
||||
COMMIT;
|
||||
"""
|
||||
|
||||
|
||||
def generate_redis_gen_sql(iccids: list[str], generated_at: str) -> str:
|
||||
in_list = sql_in_list(iccids)
|
||||
count = len(iccids)
|
||||
return f"""\
|
||||
-- ============================================================
|
||||
-- 轮询队列清理命令生成 SQL
|
||||
-- 生成时间: {generated_at}
|
||||
-- 卡数量 : {count}
|
||||
-- 执行方式:
|
||||
-- psql ... -t -A -c "$(cat remove_cards_redis_gen.sql)" | redis-cli --pipe
|
||||
-- 或:
|
||||
-- psql ... -t -A -f remove_cards_redis_gen.sql | redis-cli
|
||||
-- ============================================================
|
||||
|
||||
WITH cards AS (
|
||||
SELECT id FROM tb_iot_card
|
||||
WHERE iccid IN ({in_list})
|
||||
),
|
||||
task_types(task_type) AS (
|
||||
VALUES
|
||||
('polling:realname'),
|
||||
('polling:carddata'),
|
||||
('polling:package'),
|
||||
('polling:protect'),
|
||||
('polling:card_status')
|
||||
)
|
||||
SELECT format('ZREM polling:shard:%s:queue:%s %s', id % 16, task_type, id)
|
||||
FROM cards CROSS JOIN task_types
|
||||
UNION ALL
|
||||
SELECT format('DEL polling:card:%s', id)
|
||||
FROM cards
|
||||
UNION ALL
|
||||
SELECT format('LREM polling:manual:%s 0 %s', task_type, id)
|
||||
FROM cards CROSS JOIN task_types
|
||||
UNION ALL
|
||||
SELECT format('SREM polling:manual:dedupe:%s %s', task_type, id)
|
||||
FROM cards CROSS JOIN task_types;
|
||||
"""
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="迁移卡删除脚本:生成删除 SQL 和 Redis 清理命令")
|
||||
parser.add_argument("--input", required=True, help="ICCID 列表 CSV 文件路径")
|
||||
parser.add_argument("--output-dir", default="output", help="输出目录(默认 output)")
|
||||
args = parser.parse_args()
|
||||
|
||||
base = Path(__file__).resolve().parent
|
||||
input_path = Path(args.input)
|
||||
if not input_path.is_absolute():
|
||||
input_path = base / input_path
|
||||
output_dir = (base / args.output_dir).resolve()
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if not input_path.exists():
|
||||
print(f"[remove] 错误: 找不到输入文件 {input_path}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
iccids = load_iccids(input_path)
|
||||
if not iccids:
|
||||
print("[remove] 错误: CSV 中没有找到有效的 ICCID", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"[remove] 读取到 {len(iccids)} 个 ICCID,开始生成 SQL...")
|
||||
|
||||
generated_at = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
db_sql_path = output_dir / "remove_cards_db.sql"
|
||||
redis_sql_path = output_dir / "remove_cards_redis_gen.sql"
|
||||
|
||||
db_sql_path.write_text(generate_db_sql(iccids, generated_at), encoding="utf-8")
|
||||
redis_sql_path.write_text(generate_redis_gen_sql(iccids, generated_at), encoding="utf-8")
|
||||
|
||||
print(f"[remove] 生成完成:")
|
||||
print(f" 数据库删除 SQL : {db_sql_path}")
|
||||
print(f" Redis 清理 SQL : {redis_sql_path}")
|
||||
print()
|
||||
print("执行步骤:")
|
||||
print(" 1. 先执行 Redis 清理(在删除卡数据之前,以便 SQL 还能查到卡 ID):")
|
||||
print(f" psql <连接参数> -t -A -f {redis_sql_path} | redis-cli")
|
||||
print(" 2. 再执行数据库删除:")
|
||||
print(f" psql <连接参数> -1 -f {db_sql_path}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user