meteor_detection_system/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Architecture

This is a distributed meteor monitoring network with a microservices architecture:

Core Services

• meteor-frontend/ - Next.js 15 frontend with React 19, TypeScript, TailwindCSS 4, and React Query
• meteor-web-backend/ - NestJS backend API with PostgreSQL, TypeORM, JWT auth, and Stripe payments
• meteor-compute-service/ - Go microservice for event processing with AWS SDK and PostgreSQL
• meteor-edge-client/ - Rust edge client for Raspberry Pi devices with camera integration

Key Technologies

• Frontend: Next.js 15, React 19, TypeScript, TailwindCSS 4, React Query, React Hook Form, Recharts
• Backend: NestJS, TypeORM, PostgreSQL, JWT, Stripe, AWS SDK (S3, SQS, CloudWatch), Pino logging
• Processing: Go 1.24, PostgreSQL (pgx), AWS SDK, structured logging
• Edge: Rust, Tokio, Serde, Reqwest, OpenCV (optional), cross-compilation for ARM64

Development Commands

Full Stack Development

# Start all services (frontend + backend)
npm run dev

# Start individual services
npm run dev:frontend    # Next.js on port 3000
npm run dev:backend     # NestJS on port 3000 (with CORS for frontend)

Building

npm run build           # Build all services
npm run build:frontend  # Build Next.js app
npm run build:backend   # Build NestJS app

Testing

# Complete test suite
npm run test:fullstack  # All tests: unit + integration + e2e

# Individual test types
npm run test            # Unit tests for both frontend and backend
npm run test:frontend   # Jest tests for React components
npm run test:backend    # Jest tests for NestJS services
npm run test:e2e        # Playwright E2E tests
npm run test:integration # Backend integration tests with real DB

# Test setup and teardown
./test-setup.sh         # Initialize test environment with Docker
npm run setup:test      # Alternative test setup command
npm run clean:test      # Clean up test containers

Linting

npm run lint            # Lint all code
npm run lint:frontend   # Next.js ESLint
npm run lint:backend    # NestJS ESLint with Prettier

Database Operations (Backend)

cd meteor-web-backend
npm run migrate:up      # Run pending migrations
npm run migrate:down    # Rollback last migration
npm run migrate:create <name>  # Create new migration

Edge Client (Rust)

cd meteor-edge-client
cargo build --release                           # Native build
cargo build --target=aarch64-unknown-linux-gnu # ARM64 build for Pi
./build.sh                                      # Cross-compile for Raspberry Pi

# Vida Meteor Detection Testing
./target/debug/meteor-edge-client test-vida <video.mp4>     # Test Vida detection on video
./target/debug/meteor-edge-client run --camera file:video.mp4  # Run with video file

# Advanced Memory Management Testing
./target/debug/meteor-edge-client test          # Core frame pool tests
./target/debug/meteor-edge-client test-adaptive # Adaptive pool management
./target/debug/meteor-edge-client test-integration # Complete integration tests
./target/debug/meteor-edge-client test-ring-buffer # Ring buffer & memory mapping
./target/debug/meteor-edge-client test-hierarchical-cache # Hierarchical cache system

# Production Monitoring & Optimization
./target/debug/meteor-edge-client monitor       # Production monitoring system

# Phase 5: End-to-End Integration & Deployment
./target/debug/meteor-edge-client test-integrated-system    # Integrated memory system
./target/debug/meteor-edge-client test-camera-integration   # Camera memory integration
./target/debug/meteor-edge-client test-meteor-detection     # Real-time meteor detection

./demo_integration_test.sh                      # Integration test

Database Architecture

Entity Relationships

• Users: UserProfile + UserIdentity (supports multiple auth methods)
• Devices: Device + InventoryDevice (ownership tracking)
• Events: RawEvent → ValidatedEvent (processing pipeline)
• Analysis: AnalysisResult (AI/ML analysis data)
• Weather: WeatherStation + WeatherObservation + WeatherForecast
• Camera: CameraDevice (hardware management)
• Subscriptions: SubscriptionPlan + UserSubscription + SubscriptionHistory + PaymentRecord

Migration System

• Uses node-pg-migrate for schema management
• All entities defined in meteor-web-backend/src/entities/
• Migrations in meteor-web-backend/migrations/

Authentication & Authorization

JWT-based Authentication

• Login/Register endpoints in AuthController
• JWT strategy with passport-jwt
• Subscription-based authorization via SubscriptionGuard
• Password hashing with bcrypt

API Structure

• Base path: /api/v1/
• Protected routes require Bearer token
• CORS enabled for localhost:3000 (frontend)

Event Processing Pipeline

Data Flow

1. Edge Client (Rust) captures meteor events via camera with advanced memory management
2. Ring Buffer Streaming - Lock-free astronomical frame processing (>3M frames/sec)
3. Memory-Mapped Files - Direct access to large astronomical datasets (GB+ files)
4. Hierarchical Frame Pools - Zero-copy buffer management with adaptive sizing
5. Raw Event Upload to backend API with media files
6. SQS Queue triggers processing in Go compute service
7. Validation using MVP or Classical CV providers
8. Analysis Results stored and exposed via API
9. Frontend Gallery displays validated events with infinite scroll

Advanced Memory Management (Phase 2 & 3)

• Zero-Copy Architecture - Arc-based frame sharing eliminates memory copies
• Hierarchical Frame Pools - Multi-size buffer pools (64KB, 256KB, 900KB, 2MB)
• Adaptive Pool Management - Dynamic resizing based on memory pressure (70%/80%/90% thresholds)
• Lock-Free Ring Buffers - High-throughput astronomical frame streaming
• Memory-Mapped I/O - Efficient access to large FITS and astronomical data files
• NUMA-Aware Allocation - Optimized for modern multi-core Raspberry Pi systems

Performance Metrics

• Ring Buffer Throughput: 3.6M+ writes/sec, 7.2M+ reads/sec
• Memory Efficiency: 100%+ throughput with zero frame loss
• Buffer Utilization: Dynamic 0-100% with real-time monitoring
• Memory Savings: Multi-GB savings through zero-copy architecture
• Concurrent Safety: Lock-free operations with atomic ordering

File Storage

• AWS S3 for media storage (images/videos)
• LocalStack for development/testing
• Multipart upload support in backend
• Memory-mapped access for large astronomical datasets

Testing Architecture

Four-Layer Testing

1. Unit Tests: Jest for both frontend and backend components
2. Integration Tests: Full API workflows with test database
3. E2E Tests: Playwright for user interactions
4. Memory Management Tests: Advanced Rust-based performance testing

Test Environment

• Docker Compose setup with test services
• Separate PostgreSQL instance (port 5433)
• LocalStack for AWS service mocking
• Test data generation scripts

Memory Management Testing (Rust Edge Client)

• Core Frame Pool Tests: Basic pooling and zero-copy validation
• Adaptive Management Tests: Dynamic resizing under memory pressure
• Integration Tests: End-to-end memory optimization workflows
• Ring Buffer Tests: Lock-free concurrent streaming validation
• Memory Mapping Tests: Large file processing and performance benchmarks
• Stress Testing: Multi-million frame throughput validation
• Production Readiness: Error handling, resource cleanup, configuration validation

Gallery Testing

• Complete E2E coverage for authentication, infinite scroll, responsive design
• Integration tests for upload → processing → display workflow
• Performance testing with large datasets

Frontend Architecture

Next.js App Router Structure

• Pages in src/app/ (gallery, dashboard, settings, etc.)
• Components organized by domain (auth/, gallery/, charts/, ui/)
• React Query for server state management
• Context-based auth state management

UI Components

• Custom UI components in components/ui/
• Chart components using Recharts
• Form handling with React Hook Form + Zod validation
• Responsive design with TailwindCSS 4

Backend Architecture

NestJS Module Structure

• Domain-driven modules: Auth, Devices, Events, Payments, etc.
• TypeORM entities with PostgreSQL
• Structured logging with Pino
• Prometheus metrics collection
• Health checks and observability

Key Services

• EventsService: Core event processing and pagination
• AuthService: JWT token management and user validation
• PaymentsService: Stripe integration for subscriptions
• AnalysisService: AI/ML result processing
• WeatherService: Environmental data correlation

Development Workflow

Adding New Features

1. Create/update database entities and migrations
2. Implement backend service and controller with tests
3. Add frontend components and API integration
4. Update E2E tests for user workflows
5. Run full test suite before committing

Running Single Tests

# Backend unit test for specific service
cd meteor-web-backend && npm test -- --testPathPattern=events.service

# Frontend component test
cd meteor-frontend && npm test -- --testPathPattern=gallery

# Single E2E test
cd meteor-frontend && npx playwright test --grep="Gallery page"

# Integration test for specific feature
cd meteor-web-backend && npm run test:integration -- --testPathPattern=events

# Rust edge client memory management tests
cd meteor-edge-client && cargo test
cd meteor-edge-client && ./target/debug/meteor-edge-client test
cd meteor-edge-client && ./target/debug/meteor-edge-client test-adaptive
cd meteor-edge-client && ./target/debug/meteor-edge-client test-integration
cd meteor-edge-client && ./target/debug/meteor-edge-client test-ring-buffer

Production Deployment

Docker Support

• Dockerfiles for all services
• Next.js standalone output mode
• Multi-stage builds for optimization

Infrastructure

• Terraform configurations in infrastructure/
• AWS services: RDS, S3, SQS, CloudWatch
• Environment-based configuration

Observability

• Structured JSON logging throughout stack
• Metrics collection with Prometheus
• Health check endpoints
• Correlation IDs for request tracking

Advanced Memory Management (Edge Client)

The meteor edge client features a sophisticated 4-phase memory optimization system designed for high-performance astronomical data processing on resource-constrained devices.

Phase 1: Zero-Copy Architecture

• Arc-based frame sharing eliminates unnecessary memory copies
• RAII pattern ensures automatic resource cleanup
• Event-driven processing with efficient memory propagation

Phase 2: Hierarchical Frame Pools

• Multiple pool sizes: 64KB, 256KB, 900KB, 2MB buffers
• Adaptive capacity management based on memory pressure
• Historical metrics tracking for intelligent resizing
• Cross-platform memory pressure detection

Key Features:

• Automatic pool resizing based on system memory usage (70%/80%/90% thresholds)
• Zero-allocation buffer acquisition with automatic return
• Comprehensive statistics tracking and monitoring
• Memory leak detection and prevention

Phase 3: Advanced Streaming & Caching

Week 1: Lock-Free Ring Buffers & Memory Mapping

• Lock-free ring buffers using atomic operations for concurrent access
• Memory-mapped I/O for large astronomical datasets
• Cross-platform implementation (Unix libc, Windows winapi)
• Performance benchmarks: >3M frames/sec throughput

Week 2: Hierarchical Cache System

• Multi-level cache architecture (L1/L2/L3) with different eviction policies
• Astronomical data optimization with metadata support
• Intelligent prefetching based on access patterns
• Memory pressure adaptation with configurable limits

Cache Performance:

• L1: Hot data, LRU eviction, fastest access
• L2: Warm data, LFU eviction with frequency tracking
• L3: Cold data, time-based eviction for historical access
• Cache hit rates: >80% for typical astronomical workloads

Phase 4: Production Optimization & Monitoring

Real-Time Monitoring System

• Health check monitoring with component-level status tracking
• Performance profiling with latency histograms and percentiles
• Alert management with configurable thresholds and suppression
• Comprehensive diagnostics including system resource tracking

Key Metrics Tracked:

• Memory usage and efficiency ratios
• Cache hit rates across all levels
• Frame processing latency (P50, P95, P99)
• System resource utilization
• Error rates and alert conditions

Production Features:

• Real-time health status reporting
• Configurable alert thresholds
• Performance profiling with microsecond precision
• System diagnostics with resource tracking
• Automated metric aggregation and retention

Memory Management Testing Commands

cd meteor-edge-client

# Phase 2 Testing
./target/release/meteor-edge-client test                   # Core frame pools
./target/release/meteor-edge-client test-adaptive          # Adaptive management
./target/release/meteor-edge-client test-integration       # Integration tests

# Phase 3 Testing  
./target/release/meteor-edge-client test-ring-buffer       # Ring buffers & memory mapping
./target/release/meteor-edge-client test-hierarchical-cache # Cache system

# Phase 4 Production Monitoring
./target/release/meteor-edge-client monitor                # Live monitoring system

# Phase 5 End-to-End Integration
./target/release/meteor-edge-client test-integrated-system    # Integrated memory system
./target/release/meteor-edge-client test-camera-integration   # Camera memory integration
./target/release/meteor-edge-client test-meteor-detection     # Real-time meteor detection

Phase 5: End-to-End Integration & Deployment

The final phase integrates all memory management components into a cohesive system for real-time meteor detection with camera integration.

Integrated Memory System

• Unified Architecture: All memory components work together seamlessly
• Multi-Configuration Support: Raspberry Pi and high-performance server configurations
• Auto-Optimization: Dynamic performance tuning based on system conditions
• Health Monitoring: Comprehensive system health reporting with recommendations

Key Components:

• Hierarchical frame pools with adaptive management
• Ring buffer streaming for astronomical frames
• Multi-level caching with prefetching
• Production monitoring with alerts
• Camera integration with memory-optimized capture

Camera Memory Integration

• Memory-Optimized Capture: Integration with hierarchical frame pools
• Real-Time Processing: Zero-copy frame processing pipeline
• Buffer Management: Adaptive capture buffer pools with memory pressure handling
• Performance Monitoring: Camera-specific metrics and health reporting

Camera Features:

• Multiple configuration support (Pi camera, performance camera)
• Capture buffer pool with automatic optimization
• Real-time statistics collection
• Memory pressure detection and response
• Health monitoring with diagnostic recommendations

Real-Time Meteor Detection Pipeline

• Vida Algorithm: Scientific meteor detection based on Vida et al. 2016 paper
• Dual Detection Paths: FireballDetector (K1=4) and MeteorDetector (K1=1.5)
• FTP Compression: 256 frames → 4 statistical images (maxpixel, avepixel, stdpixel, maxframe)
• Memory-Optimized Processing: Integrated with zero-copy architecture
• Real-Time Performance: ~100ms/block processing latency

Production-Ready Features

• Raspberry Pi Optimization: Conservative memory usage and CPU utilization
• Real-Time Constraints: Guaranteed processing latency limits
• Error Recovery: Robust error handling with automatic recovery
• Performance Metrics: Comprehensive detection and system metrics
• Memory Efficiency: Optimized for resource-constrained environments

Performance Benchmarks

• Frame Pool Operations: >100K allocations/sec with zero memory leaks
• Ring Buffer Throughput: >3M frames/sec with concurrent access
• Cache Performance: >50K lookups/sec with 80%+ hit rates
• Memory Efficiency: <2x growth under sustained load
• Production Monitoring: Real-time metrics with <50μs overhead

This advanced memory management system enables the meteor edge client to:

1. Process high-resolution astronomical frames with minimal memory overhead
2. Adapt to varying system memory conditions automatically
3. Provide production-grade observability and monitoring
4. Maintain high performance on resource-constrained Raspberry Pi devices
5. Support real-time meteor detection with sub-millisecond processing latency

Vida Meteor Detection Algorithm (Edge Client)

The edge client implements the Vida detection algorithm based on "Open-source meteor detection software for low-cost single-board computers" (Vida et al., 2016). This is the same algorithm used by the Croatian Meteor Network (CMN) and RMS project.

Architecture Overview

输入帧流 (视频/摄像头)
    ↓
[FrameAccumulator] - 256帧 FTP 压缩
    ↓
[AccumulatedFrame] - maxpixel, avepixel, stdpixel, maxframe
    ├─→ [FireballDetector] - K1=4, 3D点云分析 → 火球检测
    └─→ [MeteorDetector] - K1=1.5, Hough变换 → 流星检测
    ↓
[VidaDetectionController] - 协调和回调
    ↓
[FtpDetectWriter] - FTPdetectinfo 格式输出

Core Modules (src/detection/vida/)

模块	功能	代码行数
frame_accumulator.rs	FTP 压缩引擎	~1200
accumulated_frame.rs	FTP 数据结构	~700
fireball_detector.rs	火球检测 (K1=4)	~800
meteor_detector.rs	流星检测 (K1=1.5)	~1000
line_detector.rs	Hough + 3D 线检测	~800
morphology.rs	形态学预处理	~950
star_extractor.rs	星点提取和天空质量	~1000
calibration.rs	测量校准	~1000
ftpdetect.rs	FTPdetectinfo 输出	~450
controller.rs	主控制器	~470
config.rs	配置管理	~375

Detection Parameters

FireballDetector (明亮火球):

• k1_threshold: 4.0 (标准差倍数)
• min_intensity: 40 (最小像素强度)
• 使用 3D 点云分析

MeteorDetector (普通流星):

• k1_threshold: 1.5 (标准差倍数，RMS 推荐)
• j1_offset: 9.0 (绝对强度偏移)
• max_white_ratio: 0.07 (最大白像素比例)
• 使用 Hough 变换 + 时间窗口

FTP Compression Format

256帧压缩为4个统计图像：

• maxpixel: 每像素最大值 (流星轨迹可见)
• avepixel: 平均值 (排除前4大值，天空背景)
• stdpixel: 标准差 (变化区域)
• maxframe: 最大值出现的帧号 (时间信息)

Detection Pipeline

1. 帧累积: 收集256帧，实时计算统计
2. 阈值化: 应用 K1×σ + J1 阈值
3. 形态学处理: 清理 → 桥接 → 闭合 → 细化
4. 线检测: Hough变换(2D) 或 点云分析(3D)
5. 时间传播: 7个重叠窗口验证
6. 质心提取: 亚像素精度定位
7. 输出: FTPdetectinfo 格式

Testing Commands

cd meteor-edge-client

# 在视频文件上测试 Vida 检测
./target/debug/meteor-edge-client test-vida video.mp4

# 使用摄像头运行
./target/debug/meteor-edge-client run --camera device:0

# 使用视频文件运行
./target/debug/meteor-edge-client run --camera file:video.mp4

Performance

• 处理速度: ~100ms/block (256帧)
• 误检率: 0-2 个/block (优化后)
• 内存效率: 在线统计，无需存储原始帧