meteor_detection_system/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Distributed meteor monitoring network with microservices architecture. Four main services communicate via REST APIs and SQS queues.

## Core Services

| Service | Technology | Entry Point | Port |
|---------|------------|-------------|------|
| meteor-frontend | Next.js 15, React 19, TailwindCSS 4 | `src/app/layout.tsx` | 3000 |
| meteor-web-backend | NestJS, TypeORM, PostgreSQL | `src/main.ts` | 3001 |
| meteor-compute-service | Go 1.24, AWS SDK | `cmd/meteor-compute-service/main.go` | - |
| meteor-edge-client | Rust, Tokio, OpenCV | `src/main.rs` → `src/app.rs` | - |

## Development Commands

```bash
# Full stack development
npm run dev                    # Start backend (3001) + frontend (3000) concurrently
npm run dev:frontend           # Next.js only
npm run dev:backend            # NestJS only

# Building
npm run build                  # Build all services
npm run build:frontend
npm run build:backend

# Testing
npm run test                   # Unit tests (frontend + backend)
npm run test:frontend          # Jest + Testing Library
npm run test:backend           # Jest for NestJS
npm run test:integration       # Backend integration with real DB
npm run test:e2e               # Playwright E2E tests
npm run test:fullstack         # Complete test suite

# Single test execution
cd meteor-web-backend && npm test -- --testPathPattern=events.service
cd meteor-frontend && npm test -- --testPathPattern=gallery
cd meteor-frontend && npx playwright test --grep="Gallery page"

# Test environment
./test-setup.sh                # Start LocalStack + test PostgreSQL (port 5433)
npm run clean:test             # Teardown test containers

# Linting
npm run lint                   # Lint all
npm run lint:frontend          # ESLint for Next.js
npm run lint:backend           # ESLint + Prettier for NestJS

# Database migrations (from meteor-web-backend/)
npm run migrate:up
npm run migrate:down
npm run migrate:create <name>
```

### Edge Client (Rust)

```bash
cd meteor-edge-client
cargo build --release
cargo run -- run --camera sim:pattern:meteor    # Simulated camera
cargo run -- run --camera file:video.mp4        # File replay
./build.sh                                       # Cross-compile for Raspberry Pi ARM64

# Vida Meteor Detection Testing
./target/debug/meteor-edge-client test-vida <video.mp4>     # Test Vida detection on video
./target/debug/meteor-edge-client run --camera file:video.mp4  # Run with video file

# Advanced Memory Management Testing
./target/debug/meteor-edge-client test          # Core frame pool tests
./target/debug/meteor-edge-client test-adaptive # Adaptive pool management
./target/debug/meteor-edge-client test-integration # Complete integration tests
./target/debug/meteor-edge-client test-ring-buffer # Ring buffer & memory mapping
./target/debug/meteor-edge-client test-hierarchical-cache # Hierarchical cache system

# Production Monitoring & Optimization
./target/debug/meteor-edge-client monitor       # Production monitoring system

# Phase 5: End-to-End Integration & Deployment
./target/debug/meteor-edge-client test-integrated-system    # Integrated memory system
./target/debug/meteor-edge-client test-camera-integration   # Camera memory integration
./target/debug/meteor-edge-client test-meteor-detection     # Real-time meteor detection

./demo_integration_test.sh                      # Integration test
```

### Compute Service (Go)

```bash
cd meteor-compute-service
go run cmd/meteor-compute-service/main.go
go test ./...
```

## Data Flow

1. **Edge Client** captures frames via camera, detects meteors, uploads events to backend
2. **Backend API** receives uploads, stores in S3, pushes to SQS for processing
3. **Compute Service** consumes SQS, validates events, writes analysis results to PostgreSQL
4. **Frontend** displays gallery, dashboard, and analysis views via React Query

## Database Schema

Uses node-pg-migrate. Entities in `meteor-web-backend/src/entities/`:
- **Users**: `UserProfile` + `UserIdentity` (multiple auth methods)
- **Devices**: `Device` + `InventoryDevice` + `DeviceRegistration` + `DeviceCertificate`
- **Events**: `RawEvent` → `ValidatedEvent` → `AnalysisResult`
- **Subscriptions**: `SubscriptionPlan` + `UserSubscription` + `PaymentRecord`
- **Weather**: `WeatherStation` + `WeatherObservation` + `WeatherForecast`

## API Structure

- Base path: `/api/v1/`
- JWT Bearer token authentication
- Key modules: `AuthModule`, `DevicesModule`, `EventsModule`, `PaymentsModule`, `SubscriptionModule`

## Frontend Architecture

- App Router in `src/app/` (dashboard, gallery, analysis, devices, settings)
- Components in `src/components/` organized by domain (charts, device-registration, ui)
- API services in `src/services/`
- Auth context in `src/contexts/auth-context.tsx`
- React Query for server state, React Hook Form + Zod for forms

## Backend Architecture

- NestJS modules with domain-driven organization
- TypeORM entities with PostgreSQL
- Pino structured logging with correlation IDs
- Prometheus metrics via MetricsModule
- Stripe integration in PaymentsModule

## Coding Standards

- **TypeScript**: ESLint + Prettier, PascalCase for classes, camelCase for functions
- **Rust**: `cargo fmt`, snake_case modules, prefer `?` over `unwrap()`
- **Go**: `go fmt` + `go vet`, standard project layout

## Infrastructure

- Terraform configs in `infrastructure/`
- AWS: S3 (media), SQS (event queue), RDS (PostgreSQL), CloudWatch (logs/metrics)
- LocalStack for local development
- Docker Compose for test environment

## Key Type Considerations

- User type uses `userId` not `id` (defined in `src/contexts/auth-context.tsx`)
- `DeviceStatus` is an enum - use enum values not strings with StatusIndicator
- Recharts has React 19 compatibility issues - type declarations in `src/types/recharts.d.ts`

## Additional Testing Commands

```bash
# Single E2E test
cd meteor-frontend && npx playwright test --grep="Gallery page"

# Integration test for specific feature
cd meteor-web-backend && npm run test:integration -- --testPathPattern=events

# Rust edge client memory management tests
cd meteor-edge-client && cargo test
cd meteor-edge-client && ./target/debug/meteor-edge-client test
cd meteor-edge-client && ./target/debug/meteor-edge-client test-adaptive
cd meteor-edge-client && ./target/debug/meteor-edge-client test-integration
cd meteor-edge-client && ./target/debug/meteor-edge-client test-ring-buffer
```

## Production Deployment

### Docker Support
- Dockerfiles for all services
- Next.js standalone output mode
- Multi-stage builds for optimization

### Infrastructure
- Terraform configurations in `infrastructure/`
- AWS services: RDS, S3, SQS, CloudWatch
- Environment-based configuration

### Observability
- Structured JSON logging throughout stack
- Metrics collection with Prometheus
- Health check endpoints
- Correlation IDs for request tracking

## Advanced Memory Management (Edge Client)

The meteor edge client features a sophisticated 4-phase memory optimization system designed for high-performance astronomical data processing on resource-constrained devices.

### Phase 1: Zero-Copy Architecture
- **Arc-based frame sharing** eliminates unnecessary memory copies
- **RAII pattern** ensures automatic resource cleanup
- **Event-driven processing** with efficient memory propagation

### Phase 2: Hierarchical Frame Pools
- **Multiple pool sizes**: 64KB, 256KB, 900KB, 2MB buffers
- **Adaptive capacity management** based on memory pressure
- **Historical metrics tracking** for intelligent resizing
- **Cross-platform memory pressure detection**

Key Features:
- Automatic pool resizing based on system memory usage (70%/80%/90% thresholds)
- Zero-allocation buffer acquisition with automatic return
- Comprehensive statistics tracking and monitoring
- Memory leak detection and prevention

### Phase 3: Advanced Streaming & Caching

#### Week 1: Lock-Free Ring Buffers & Memory Mapping
- **Lock-free ring buffers** using atomic operations for concurrent access
- **Memory-mapped I/O** for large astronomical datasets
- **Cross-platform implementation** (Unix libc, Windows winapi)
- **Performance benchmarks**: >3M frames/sec throughput

#### Week 2: Hierarchical Cache System
- **Multi-level cache architecture** (L1/L2/L3) with different eviction policies
- **Astronomical data optimization** with metadata support
- **Intelligent prefetching** based on access patterns
- **Memory pressure adaptation** with configurable limits

Cache Performance:
- L1: Hot data, LRU eviction, fastest access
- L2: Warm data, LFU eviction with frequency tracking
- L3: Cold data, time-based eviction for historical access
- Cache hit rates: >80% for typical astronomical workloads

### Phase 4: Production Optimization & Monitoring

#### Real-Time Monitoring System
- **Health check monitoring** with component-level status tracking
- **Performance profiling** with latency histograms and percentiles
- **Alert management** with configurable thresholds and suppression
- **Comprehensive diagnostics** including system resource tracking

#### Key Metrics Tracked:
- Memory usage and efficiency ratios
- Cache hit rates across all levels
- Frame processing latency (P50, P95, P99)
- System resource utilization
- Error rates and alert conditions

#### Production Features:
- Real-time health status reporting
- Configurable alert thresholds
- Performance profiling with microsecond precision
- System diagnostics with resource tracking
- Automated metric aggregation and retention

### Memory Management Testing Commands

```bash
cd meteor-edge-client

# Phase 2 Testing
./target/release/meteor-edge-client test                   # Core frame pools
./target/release/meteor-edge-client test-adaptive          # Adaptive management
./target/release/meteor-edge-client test-integration       # Integration tests

# Phase 3 Testing
./target/release/meteor-edge-client test-ring-buffer       # Ring buffers & memory mapping
./target/release/meteor-edge-client test-hierarchical-cache # Cache system

# Phase 4 Production Monitoring
./target/release/meteor-edge-client monitor                # Live monitoring system

# Phase 5 End-to-End Integration
./target/release/meteor-edge-client test-integrated-system    # Integrated memory system
./target/release/meteor-edge-client test-camera-integration   # Camera memory integration
./target/release/meteor-edge-client test-meteor-detection     # Real-time meteor detection
```

### Phase 5: End-to-End Integration & Deployment

The final phase integrates all memory management components into a cohesive system for real-time meteor detection with camera integration.

#### Integrated Memory System
- **Unified Architecture**: All memory components work together seamlessly
- **Multi-Configuration Support**: Raspberry Pi and high-performance server configurations
- **Auto-Optimization**: Dynamic performance tuning based on system conditions
- **Health Monitoring**: Comprehensive system health reporting with recommendations

Key Components:
- Hierarchical frame pools with adaptive management
- Ring buffer streaming for astronomical frames
- Multi-level caching with prefetching
- Production monitoring with alerts
- Camera integration with memory-optimized capture

#### Camera Memory Integration
- **Memory-Optimized Capture**: Integration with hierarchical frame pools
- **Real-Time Processing**: Zero-copy frame processing pipeline
- **Buffer Management**: Adaptive capture buffer pools with memory pressure handling
- **Performance Monitoring**: Camera-specific metrics and health reporting

Camera Features:
- Multiple configuration support (Pi camera, performance camera)
- Capture buffer pool with automatic optimization
- Real-time statistics collection
- Memory pressure detection and response
- Health monitoring with diagnostic recommendations

#### Real-Time Meteor Detection Pipeline
- **Vida Algorithm**: Scientific meteor detection based on Vida et al. 2016 paper
- **Dual Detection Paths**: FireballDetector (K1=4) and MeteorDetector (K1=1.5)
- **FTP Compression**: 256 frames → 4 statistical images (maxpixel, avepixel, stdpixel, maxframe)
- **Memory-Optimized Processing**: Integrated with zero-copy architecture
- **Real-Time Performance**: ~100ms/block processing latency

#### Production-Ready Features
- **Raspberry Pi Optimization**: Conservative memory usage and CPU utilization
- **Real-Time Constraints**: Guaranteed processing latency limits
- **Error Recovery**: Robust error handling with automatic recovery
- **Performance Metrics**: Comprehensive detection and system metrics
- **Memory Efficiency**: Optimized for resource-constrained environments

### Performance Benchmarks
- **Frame Pool Operations**: >100K allocations/sec with zero memory leaks
- **Ring Buffer Throughput**: >3M frames/sec with concurrent access
- **Cache Performance**: >50K lookups/sec with 80%+ hit rates
- **Memory Efficiency**: <2x growth under sustained load
- **Production Monitoring**: Real-time metrics with <50μs overhead

This advanced memory management system enables the meteor edge client to:
1. Process high-resolution astronomical frames with minimal memory overhead
2. Adapt to varying system memory conditions automatically
3. Provide production-grade observability and monitoring
4. Maintain high performance on resource-constrained Raspberry Pi devices
5. Support real-time meteor detection with sub-millisecond processing latency

## Vida Meteor Detection Algorithm (Edge Client)

The edge client implements the Vida detection algorithm based on *"Open-source meteor detection software for low-cost single-board computers"* (Vida et al., 2016). This is the same algorithm used by the Croatian Meteor Network (CMN) and RMS project.

### Architecture Overview

```
输入帧流 (视频/摄像头)
    ↓
[FrameAccumulator] - 256帧 FTP 压缩
    ↓
[AccumulatedFrame] - maxpixel, avepixel, stdpixel, maxframe
    ├─→ [FireballDetector] - K1=4, 3D点云分析 → 火球检测
    └─→ [MeteorDetector] - K1=1.5, Hough变换 → 流星检测
    ↓
[VidaDetectionController] - 协调和回调
    ↓
[FtpDetectWriter] - FTPdetectinfo 格式输出
```

### Core Modules (`src/detection/vida/`)

| 模块 | 功能 | 代码行数 |
|------|------|----------|
| `frame_accumulator.rs` | FTP 压缩引擎 | ~1200 |
| `accumulated_frame.rs` | FTP 数据结构 | ~700 |
| `fireball_detector.rs` | 火球检测 (K1=4) | ~800 |
| `meteor_detector.rs` | 流星检测 (K1=1.5) | ~1000 |
| `line_detector.rs` | Hough + 3D 线检测 | ~800 |
| `morphology.rs` | 形态学预处理 | ~950 |
| `star_extractor.rs` | 星点提取和天空质量 | ~1000 |
| `calibration.rs` | 测量校准 | ~1000 |
| `ftpdetect.rs` | FTPdetectinfo 输出 | ~450 |
| `controller.rs` | 主控制器 | ~470 |
| `config.rs` | 配置管理 | ~375 |

### Detection Parameters

**FireballDetector** (明亮火球):
- `k1_threshold`: 4.0 (标准差倍数)
- `min_intensity`: 40 (最小像素强度)
- 使用 3D 点云分析

**MeteorDetector** (普通流星):
- `k1_threshold`: 1.5 (标准差倍数，RMS 推荐)
- `j1_offset`: 9.0 (绝对强度偏移)
- `max_white_ratio`: 0.07 (最大白像素比例)
- 使用 Hough 变换 + 时间窗口

### FTP Compression Format

256帧压缩为4个统计图像：
- **maxpixel**: 每像素最大值 (流星轨迹可见)
- **avepixel**: 平均值 (排除前4大值，天空背景)
- **stdpixel**: 标准差 (变化区域)
- **maxframe**: 最大值出现的帧号 (时间信息)

### Detection Pipeline

1. **帧累积**: 收集256帧，实时计算统计
2. **阈值化**: 应用 K1×σ + J1 阈值
3. **形态学处理**: 清理 → 桥接 → 闭合 → 细化
4. **线检测**: Hough变换(2D) 或 点云分析(3D)
5. **时间传播**: 7个重叠窗口验证
6. **质心提取**: 亚像素精度定位
7. **输出**: FTPdetectinfo 格式

### Testing Commands

```bash
cd meteor-edge-client

# 在视频文件上测试 Vida 检测
./target/debug/meteor-edge-client test-vida video.mp4

# 使用摄像头运行
./target/debug/meteor-edge-client run --camera device:0

# 使用视频文件运行
./target/debug/meteor-edge-client run --camera file:video.mp4
```

### Performance

- **处理速度**: ~100ms/block (256帧)
- **误检率**: 0-2 个/block (优化后)
- **内存效率**: 在线统计，无需存储原始帧