feat: 添加环境变量管理工具和部署配置改版
All checks were successful
构建并部署到测试环境(无 SSH) / build-and-deploy (push) Successful in 5m33s
All checks were successful
构建并部署到测试环境(无 SSH) / build-and-deploy (push) Successful in 5m33s
主要改动: - 新增交互式环境配置脚本 (scripts/setup-env.sh) - 新增本地启动快捷脚本 (scripts/run-local.sh) - 新增环境变量模板文件 (.env.example) - 部署模式改版:使用嵌入式配置 + 环境变量覆盖 - 添加对象存储功能支持 - 改进 IoT 卡片导入任务 - 优化 OpenAPI 文档生成 - 删除旧的配置文件,改用嵌入式默认配置
This commit is contained in:
163
docs/object-storage/使用指南.md
Normal file
163
docs/object-storage/使用指南.md
Normal file
@@ -0,0 +1,163 @@
|
||||
# 对象存储使用指南
|
||||
|
||||
本文档介绍如何在后端代码中使用对象存储服务。
|
||||
|
||||
## 配置
|
||||
|
||||
通过环境变量配置对象存储:
|
||||
|
||||
```bash
|
||||
# 存储提供商
|
||||
export JUNHONG_STORAGE_PROVIDER="s3"
|
||||
|
||||
# S3 配置
|
||||
export JUNHONG_STORAGE_S3_ENDPOINT="http://obs-helf.cucloud.cn"
|
||||
export JUNHONG_STORAGE_S3_REGION="cn-langfang-2"
|
||||
export JUNHONG_STORAGE_S3_BUCKET="cmp"
|
||||
export JUNHONG_STORAGE_S3_ACCESS_KEY_ID="YOUR_ACCESS_KEY"
|
||||
export JUNHONG_STORAGE_S3_SECRET_ACCESS_KEY="YOUR_SECRET_KEY"
|
||||
export JUNHONG_STORAGE_S3_USE_SSL="false"
|
||||
export JUNHONG_STORAGE_S3_PATH_STYLE="true"
|
||||
|
||||
# 预签名 URL 配置
|
||||
export JUNHONG_STORAGE_PRESIGN_UPLOAD_EXPIRES="15m"
|
||||
export JUNHONG_STORAGE_PRESIGN_DOWNLOAD_EXPIRES="24h"
|
||||
|
||||
# 临时文件目录
|
||||
export JUNHONG_STORAGE_TEMP_DIR="/tmp/junhong-storage"
|
||||
```
|
||||
|
||||
详细配置说明见 [环境变量配置文档](../environment-variables.md)
|
||||
|
||||
## StorageService 使用
|
||||
|
||||
### 获取预签名上传 URL
|
||||
|
||||
```go
|
||||
result, err := storageService.GetUploadURL(ctx, "iot_import", "cards.csv", "text/csv")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// result.URL - 预签名上传 URL
|
||||
// result.FileKey - 文件路径(用于后续业务接口)
|
||||
// result.ExpiresIn - URL 有效期(秒)
|
||||
```
|
||||
|
||||
### 下载文件到临时目录
|
||||
|
||||
```go
|
||||
localPath, cleanup, err := storageService.DownloadToTemp(ctx, fileKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer cleanup() // 处理完成后自动删除临时文件
|
||||
|
||||
// 使用 localPath 读取文件内容
|
||||
f, _ := os.Open(localPath)
|
||||
defer f.Close()
|
||||
```
|
||||
|
||||
### 直接上传文件
|
||||
|
||||
```go
|
||||
reader := bytes.NewReader(content)
|
||||
err := storageService.Provider().Upload(ctx, fileKey, reader, "text/csv")
|
||||
```
|
||||
|
||||
### 检查文件是否存在
|
||||
|
||||
```go
|
||||
exists, err := storageService.Provider().Exists(ctx, fileKey)
|
||||
```
|
||||
|
||||
### 删除文件
|
||||
|
||||
```go
|
||||
err := storageService.Provider().Delete(ctx, fileKey)
|
||||
```
|
||||
|
||||
## Purpose 类型
|
||||
|
||||
| Purpose | 说明 | 生成路径 | ContentType |
|
||||
|---------|------|---------|-------------|
|
||||
| iot_import | ICCID 导入 | imports/YYYY/MM/DD/uuid.csv | text/csv |
|
||||
| export | 数据导出 | exports/YYYY/MM/DD/uuid.xlsx | application/vnd.openxmlformats... |
|
||||
| attachment | 附件上传 | attachments/YYYY/MM/DD/uuid.ext | 自动检测 |
|
||||
|
||||
## 错误处理
|
||||
|
||||
存储相关错误码定义在 `pkg/errors/codes.go`:
|
||||
|
||||
| 错误码 | 说明 |
|
||||
|-------|------|
|
||||
| 1090 | 对象存储服务未配置 |
|
||||
| 1091 | 文件上传失败 |
|
||||
| 1092 | 文件下载失败 |
|
||||
| 1093 | 文件不存在 |
|
||||
| 1094 | 不支持的文件用途 |
|
||||
| 1095 | 不支持的文件类型 |
|
||||
|
||||
## 在 Handler 中使用
|
||||
|
||||
```go
|
||||
type MyHandler struct {
|
||||
storageService *storage.Service
|
||||
}
|
||||
|
||||
func (h *MyHandler) Upload(c *fiber.Ctx) error {
|
||||
var req dto.GetUploadURLRequest
|
||||
if err := c.BodyParser(&req); err != nil {
|
||||
return errors.New(errors.CodeInvalidParam, "参数解析失败")
|
||||
}
|
||||
|
||||
result, err := h.storageService.GetUploadURL(
|
||||
c.UserContext(),
|
||||
req.Purpose,
|
||||
req.FileName,
|
||||
req.ContentType,
|
||||
)
|
||||
if err != nil {
|
||||
return errors.New(errors.CodeStorageUploadFailed, err.Error())
|
||||
}
|
||||
|
||||
return response.Success(c, result)
|
||||
}
|
||||
```
|
||||
|
||||
## 在 Worker 中使用
|
||||
|
||||
```go
|
||||
func (h *TaskHandler) HandleTask(ctx context.Context, task *asynq.Task) error {
|
||||
// 从任务记录获取文件路径
|
||||
fileKey := importTask.StorageKey
|
||||
|
||||
// 下载到临时文件
|
||||
localPath, cleanup, err := h.storageService.DownloadToTemp(ctx, fileKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer cleanup()
|
||||
|
||||
// 解析文件
|
||||
f, _ := os.Open(localPath)
|
||||
defer f.Close()
|
||||
|
||||
// 处理文件内容...
|
||||
}
|
||||
```
|
||||
|
||||
## 测试验证
|
||||
|
||||
运行对象存储功能测试:
|
||||
|
||||
```bash
|
||||
go run scripts/test_storage.go
|
||||
```
|
||||
|
||||
测试内容包括:
|
||||
1. 生成预签名上传 URL
|
||||
2. 上传测试文件
|
||||
3. 检查文件是否存在
|
||||
4. 下载到临时文件
|
||||
5. 删除测试文件
|
||||
250
docs/object-storage/前端接入指南.md
Normal file
250
docs/object-storage/前端接入指南.md
Normal file
@@ -0,0 +1,250 @@
|
||||
# 对象存储前端接入指南
|
||||
|
||||
## 文件上传流程
|
||||
|
||||
```
|
||||
前端 后端 API 对象存储
|
||||
│ │ │
|
||||
│ 1. POST /storage/upload-url │
|
||||
│ {file_name, content_type, purpose} │
|
||||
│ ─────────────────────────► │
|
||||
│ │ │
|
||||
│ 2. 返回 {upload_url, file_key, expires_in} │
|
||||
│ ◄───────────────────────── │
|
||||
│ │ │
|
||||
│ 3. PUT upload_url (文件内容) │
|
||||
│ ─────────────────────────────────────────────────► │
|
||||
│ │ │
|
||||
│ 4. 上传成功 (200 OK) │
|
||||
│ ◄───────────────────────────────────────────────── │
|
||||
│ │ │
|
||||
│ 5. POST /iot-cards/import │
|
||||
│ {carrier_id, batch_no, file_key} │
|
||||
│ ─────────────────────────► │
|
||||
│ │ │
|
||||
│ 6. 返回任务创建成功 │
|
||||
│ ◄───────────────────────── │
|
||||
```
|
||||
|
||||
## 获取预签名 URL 接口
|
||||
|
||||
### 请求
|
||||
|
||||
```http
|
||||
POST /api/admin/storage/upload-url
|
||||
Content-Type: application/json
|
||||
Authorization: Bearer {token}
|
||||
|
||||
{
|
||||
"file_name": "cards.csv",
|
||||
"content_type": "text/csv",
|
||||
"purpose": "iot_import"
|
||||
}
|
||||
```
|
||||
|
||||
### 响应
|
||||
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "成功",
|
||||
"data": {
|
||||
"upload_url": "http://obs-helf.cucloud.cn/cmp/imports/2025/01/24/abc123.csv?X-Amz-Algorithm=...",
|
||||
"file_key": "imports/2025/01/24/abc123.csv",
|
||||
"expires_in": 900
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### purpose 可选值
|
||||
|
||||
| 值 | 说明 | 生成路径 |
|
||||
|---|------|---------|
|
||||
| iot_import | ICCID 导入 | imports/YYYY/MM/DD/uuid.csv |
|
||||
| export | 数据导出 | exports/YYYY/MM/DD/uuid.xlsx |
|
||||
| attachment | 附件上传 | attachments/YYYY/MM/DD/uuid.ext |
|
||||
|
||||
## 使用预签名 URL 上传文件
|
||||
|
||||
获取到 `upload_url` 后,直接使用 PUT 请求上传文件到对象存储:
|
||||
|
||||
```javascript
|
||||
const response = await fetch(upload_url, {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
'Content-Type': content_type
|
||||
},
|
||||
body: file
|
||||
});
|
||||
|
||||
if (response.ok) {
|
||||
console.log('上传成功');
|
||||
} else {
|
||||
console.error('上传失败:', response.status);
|
||||
}
|
||||
```
|
||||
|
||||
## ICCID 导入接口变更(BREAKING CHANGE)
|
||||
|
||||
### 变更前
|
||||
|
||||
```http
|
||||
POST /api/admin/iot-cards/import
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
carrier_id=1
|
||||
batch_no=BATCH-2025-01
|
||||
file=@cards.csv
|
||||
```
|
||||
|
||||
### 变更后
|
||||
|
||||
```http
|
||||
POST /api/admin/iot-cards/import
|
||||
Content-Type: application/json
|
||||
Authorization: Bearer {token}
|
||||
|
||||
{
|
||||
"carrier_id": 1,
|
||||
"batch_no": "BATCH-2025-01",
|
||||
"file_key": "imports/2025/01/24/abc123.csv"
|
||||
}
|
||||
```
|
||||
|
||||
## 完整代码示例(TypeScript)
|
||||
|
||||
```typescript
|
||||
interface UploadURLResponse {
|
||||
upload_url: string;
|
||||
file_key: string;
|
||||
expires_in: number;
|
||||
}
|
||||
|
||||
async function uploadAndImportCards(
|
||||
file: File,
|
||||
carrierId: number,
|
||||
batchNo: string
|
||||
): Promise<void> {
|
||||
// 1. 获取预签名上传 URL
|
||||
const urlResponse = await fetch('/api/admin/storage/upload-url', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': `Bearer ${getToken()}`
|
||||
},
|
||||
body: JSON.stringify({
|
||||
file_name: file.name,
|
||||
content_type: file.type || 'text/csv',
|
||||
purpose: 'iot_import'
|
||||
})
|
||||
});
|
||||
|
||||
if (!urlResponse.ok) {
|
||||
throw new Error('获取上传 URL 失败');
|
||||
}
|
||||
|
||||
const { data } = await urlResponse.json();
|
||||
const { upload_url, file_key } = data as UploadURLResponse;
|
||||
|
||||
// 2. 上传文件到对象存储
|
||||
const uploadResponse = await fetch(upload_url, {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
'Content-Type': file.type || 'text/csv'
|
||||
},
|
||||
body: file
|
||||
});
|
||||
|
||||
if (!uploadResponse.ok) {
|
||||
throw new Error('文件上传失败');
|
||||
}
|
||||
|
||||
// 3. 调用导入接口
|
||||
const importResponse = await fetch('/api/admin/iot-cards/import', {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Authorization': `Bearer ${getToken()}`
|
||||
},
|
||||
body: JSON.stringify({
|
||||
carrier_id: carrierId,
|
||||
batch_no: batchNo,
|
||||
file_key: file_key
|
||||
})
|
||||
});
|
||||
|
||||
if (!importResponse.ok) {
|
||||
throw new Error('导入任务创建失败');
|
||||
}
|
||||
|
||||
console.log('导入任务已创建');
|
||||
}
|
||||
```
|
||||
|
||||
## 错误处理和重试策略
|
||||
|
||||
### 预签名 URL 过期
|
||||
|
||||
预签名 URL 有效期为 15 分钟。如果上传时 URL 已过期,需要重新获取:
|
||||
|
||||
```typescript
|
||||
async function uploadWithRetry(file: File, purpose: string, maxRetries = 3) {
|
||||
for (let i = 0; i < maxRetries; i++) {
|
||||
const { upload_url, file_key } = await getUploadURL(file.name, file.type, purpose);
|
||||
|
||||
try {
|
||||
await uploadFile(upload_url, file);
|
||||
return file_key;
|
||||
} catch (error) {
|
||||
if (i === maxRetries - 1) throw error;
|
||||
console.warn(`上传失败,重试 ${i + 1}/${maxRetries}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 网络错误
|
||||
|
||||
对象存储上传可能因网络问题失败,建议实现重试机制:
|
||||
|
||||
```typescript
|
||||
async function uploadFile(url: string, file: File, retries = 3) {
|
||||
for (let i = 0; i < retries; i++) {
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': file.type },
|
||||
body: file
|
||||
});
|
||||
|
||||
if (response.ok) return;
|
||||
|
||||
if (response.status >= 500) {
|
||||
// 服务端错误,可重试
|
||||
continue;
|
||||
}
|
||||
|
||||
throw new Error(`上传失败: ${response.status}`);
|
||||
} catch (error) {
|
||||
if (i === retries - 1) throw error;
|
||||
await new Promise(r => setTimeout(r, 1000 * (i + 1)));
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 常见问题
|
||||
|
||||
### Q: 上传时报 CORS 错误
|
||||
|
||||
确保对象存储已配置 CORS 规则允许前端域名访问。
|
||||
|
||||
### Q: 预签名 URL 无法使用
|
||||
|
||||
1. 检查 URL 是否过期(15 分钟有效期)
|
||||
2. 确保 Content-Type 与获取 URL 时指定的一致
|
||||
3. 检查文件大小是否超过限制
|
||||
|
||||
### Q: file_key 可以重复使用吗
|
||||
|
||||
可以。file_key 一旦上传成功就永久有效,可以在多个业务接口中使用同一个 file_key。
|
||||
Reference in New Issue
Block a user