grabbit/meteor_detection_system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
meteor_detection_system/CLAUDE.md
grabbit b9c2b7e17d memory optimize
2025-08-12 07:21:41 +08:00

17 KiB
Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Architecture

This is a distributed meteor monitoring network with a microservices architecture:

Core Services

  • meteor-frontend/ - Next.js 15 frontend with React 19, TypeScript, TailwindCSS 4, and React Query
  • meteor-web-backend/ - NestJS backend API with PostgreSQL, TypeORM, JWT auth, and Stripe payments
  • meteor-compute-service/ - Go microservice for event processing with AWS SDK and PostgreSQL
  • meteor-edge-client/ - Rust edge client for Raspberry Pi devices with camera integration

Key Technologies

  • Frontend: Next.js 15, React 19, TypeScript, TailwindCSS 4, React Query, React Hook Form, Recharts
  • Backend: NestJS, TypeORM, PostgreSQL, JWT, Stripe, AWS SDK (S3, SQS, CloudWatch), Pino logging
  • Processing: Go 1.24, PostgreSQL (pgx), AWS SDK, structured logging
  • Edge: Rust, Tokio, Serde, Reqwest, OpenCV (optional), cross-compilation for ARM64

Development Commands

Full Stack Development

# Start all services (frontend + backend)
npm run dev

# Start individual services
npm run dev:frontend    # Next.js on port 3000
npm run dev:backend     # NestJS on port 3000 (with CORS for frontend)

Building

npm run build           # Build all services
npm run build:frontend  # Build Next.js app
npm run build:backend   # Build NestJS app

Testing

# Complete test suite
npm run test:fullstack  # All tests: unit + integration + e2e

# Individual test types
npm run test            # Unit tests for both frontend and backend
npm run test:frontend   # Jest tests for React components
npm run test:backend    # Jest tests for NestJS services
npm run test:e2e        # Playwright E2E tests
npm run test:integration # Backend integration tests with real DB

# Test setup and teardown
./test-setup.sh         # Initialize test environment with Docker
npm run setup:test      # Alternative test setup command
npm run clean:test      # Clean up test containers

Linting

npm run lint            # Lint all code
npm run lint:frontend   # Next.js ESLint
npm run lint:backend    # NestJS ESLint with Prettier

Database Operations (Backend)

cd meteor-web-backend
npm run migrate:up      # Run pending migrations
npm run migrate:down    # Rollback last migration
npm run migrate:create <name>  # Create new migration

Edge Client (Rust)

cd meteor-edge-client
cargo build --release                           # Native build
cargo build --target=aarch64-unknown-linux-gnu # ARM64 build for Pi
./build.sh                                      # Cross-compile for Raspberry Pi

# Advanced Memory Management Testing
./target/debug/meteor-edge-client test          # Core frame pool tests
./target/debug/meteor-edge-client test-adaptive # Adaptive pool management
./target/debug/meteor-edge-client test-integration # Complete integration tests
./target/debug/meteor-edge-client test-ring-buffer # Ring buffer & memory mapping
./target/debug/meteor-edge-client test-hierarchical-cache # Hierarchical cache system

# Production Monitoring & Optimization
./target/debug/meteor-edge-client monitor       # Production monitoring system

# Phase 5: End-to-End Integration & Deployment
./target/debug/meteor-edge-client test-integrated-system    # Integrated memory system
./target/debug/meteor-edge-client test-camera-integration   # Camera memory integration
./target/debug/meteor-edge-client test-meteor-detection     # Real-time meteor detection

./demo_integration_test.sh                      # Integration test

Database Architecture

Entity Relationships

  • Users: UserProfile + UserIdentity (supports multiple auth methods)
  • Devices: Device + InventoryDevice (ownership tracking)
  • Events: RawEvent → ValidatedEvent (processing pipeline)
  • Analysis: AnalysisResult (AI/ML analysis data)
  • Weather: WeatherStation + WeatherObservation + WeatherForecast
  • Camera: CameraDevice (hardware management)
  • Subscriptions: SubscriptionPlan + UserSubscription + SubscriptionHistory + PaymentRecord

Migration System

  • Uses node-pg-migrate for schema management
  • All entities defined in meteor-web-backend/src/entities/
  • Migrations in meteor-web-backend/migrations/

Authentication & Authorization

JWT-based Authentication

  • Login/Register endpoints in AuthController
  • JWT strategy with passport-jwt
  • Subscription-based authorization via SubscriptionGuard
  • Password hashing with bcrypt

API Structure

  • Base path: /api/v1/
  • Protected routes require Bearer token
  • CORS enabled for localhost:3000 (frontend)

Event Processing Pipeline

Data Flow

  1. Edge Client (Rust) captures meteor events via camera with advanced memory management
  2. Ring Buffer Streaming - Lock-free astronomical frame processing (>3M frames/sec)
  3. Memory-Mapped Files - Direct access to large astronomical datasets (GB+ files)
  4. Hierarchical Frame Pools - Zero-copy buffer management with adaptive sizing
  5. Raw Event Upload to backend API with media files
  6. SQS Queue triggers processing in Go compute service
  7. Validation using MVP or Classical CV providers
  8. Analysis Results stored and exposed via API
  9. Frontend Gallery displays validated events with infinite scroll

Advanced Memory Management (Phase 2 & 3)

  • Zero-Copy Architecture - Arc-based frame sharing eliminates memory copies
  • Hierarchical Frame Pools - Multi-size buffer pools (64KB, 256KB, 900KB, 2MB)
  • Adaptive Pool Management - Dynamic resizing based on memory pressure (70%/80%/90% thresholds)
  • Lock-Free Ring Buffers - High-throughput astronomical frame streaming
  • Memory-Mapped I/O - Efficient access to large FITS and astronomical data files
  • NUMA-Aware Allocation - Optimized for modern multi-core Raspberry Pi systems

Performance Metrics

  • Ring Buffer Throughput: 3.6M+ writes/sec, 7.2M+ reads/sec
  • Memory Efficiency: 100%+ throughput with zero frame loss
  • Buffer Utilization: Dynamic 0-100% with real-time monitoring
  • Memory Savings: Multi-GB savings through zero-copy architecture
  • Concurrent Safety: Lock-free operations with atomic ordering

File Storage

  • AWS S3 for media storage (images/videos)
  • LocalStack for development/testing
  • Multipart upload support in backend
  • Memory-mapped access for large astronomical datasets

Testing Architecture

Four-Layer Testing

  1. Unit Tests: Jest for both frontend and backend components
  2. Integration Tests: Full API workflows with test database
  3. E2E Tests: Playwright for user interactions
  4. Memory Management Tests: Advanced Rust-based performance testing

Test Environment

  • Docker Compose setup with test services
  • Separate PostgreSQL instance (port 5433)
  • LocalStack for AWS service mocking
  • Test data generation scripts

Memory Management Testing (Rust Edge Client)

  • Core Frame Pool Tests: Basic pooling and zero-copy validation
  • Adaptive Management Tests: Dynamic resizing under memory pressure
  • Integration Tests: End-to-end memory optimization workflows
  • Ring Buffer Tests: Lock-free concurrent streaming validation
  • Memory Mapping Tests: Large file processing and performance benchmarks
  • Stress Testing: Multi-million frame throughput validation
  • Production Readiness: Error handling, resource cleanup, configuration validation

Gallery Testing

  • Complete E2E coverage for authentication, infinite scroll, responsive design
  • Integration tests for upload → processing → display workflow
  • Performance testing with large datasets

Frontend Architecture

Next.js App Router Structure

  • Pages in src/app/ (gallery, dashboard, settings, etc.)
  • Components organized by domain (auth/, gallery/, charts/, ui/)
  • React Query for server state management
  • Context-based auth state management

UI Components

  • Custom UI components in components/ui/
  • Chart components using Recharts
  • Form handling with React Hook Form + Zod validation
  • Responsive design with TailwindCSS 4

Backend Architecture

NestJS Module Structure

  • Domain-driven modules: Auth, Devices, Events, Payments, etc.
  • TypeORM entities with PostgreSQL
  • Structured logging with Pino
  • Prometheus metrics collection
  • Health checks and observability

Key Services

  • EventsService: Core event processing and pagination
  • AuthService: JWT token management and user validation
  • PaymentsService: Stripe integration for subscriptions
  • AnalysisService: AI/ML result processing
  • WeatherService: Environmental data correlation

Development Workflow

Adding New Features

  1. Create/update database entities and migrations
  2. Implement backend service and controller with tests
  3. Add frontend components and API integration
  4. Update E2E tests for user workflows
  5. Run full test suite before committing

Running Single Tests

# Backend unit test for specific service
cd meteor-web-backend && npm test -- --testPathPattern=events.service

# Frontend component test
cd meteor-frontend && npm test -- --testPathPattern=gallery

# Single E2E test
cd meteor-frontend && npx playwright test --grep="Gallery page"

# Integration test for specific feature
cd meteor-web-backend && npm run test:integration -- --testPathPattern=events

# Rust edge client memory management tests
cd meteor-edge-client && cargo test
cd meteor-edge-client && ./target/debug/meteor-edge-client test
cd meteor-edge-client && ./target/debug/meteor-edge-client test-adaptive
cd meteor-edge-client && ./target/debug/meteor-edge-client test-integration
cd meteor-edge-client && ./target/debug/meteor-edge-client test-ring-buffer

Production Deployment

Docker Support

  • Dockerfiles for all services
  • Next.js standalone output mode
  • Multi-stage builds for optimization

Infrastructure

  • Terraform configurations in infrastructure/
  • AWS services: RDS, S3, SQS, CloudWatch
  • Environment-based configuration

Observability

  • Structured JSON logging throughout stack
  • Metrics collection with Prometheus
  • Health check endpoints
  • Correlation IDs for request tracking

Advanced Memory Management (Edge Client)

The meteor edge client features a sophisticated 4-phase memory optimization system designed for high-performance astronomical data processing on resource-constrained devices.

Phase 1: Zero-Copy Architecture

  • Arc-based frame sharing eliminates unnecessary memory copies
  • RAII pattern ensures automatic resource cleanup
  • Event-driven processing with efficient memory propagation

Phase 2: Hierarchical Frame Pools

  • Multiple pool sizes: 64KB, 256KB, 900KB, 2MB buffers
  • Adaptive capacity management based on memory pressure
  • Historical metrics tracking for intelligent resizing
  • Cross-platform memory pressure detection

Key Features:

  • Automatic pool resizing based on system memory usage (70%/80%/90% thresholds)
  • Zero-allocation buffer acquisition with automatic return
  • Comprehensive statistics tracking and monitoring
  • Memory leak detection and prevention

Phase 3: Advanced Streaming & Caching

Week 1: Lock-Free Ring Buffers & Memory Mapping

  • Lock-free ring buffers using atomic operations for concurrent access
  • Memory-mapped I/O for large astronomical datasets
  • Cross-platform implementation (Unix libc, Windows winapi)
  • Performance benchmarks: >3M frames/sec throughput

Week 2: Hierarchical Cache System

  • Multi-level cache architecture (L1/L2/L3) with different eviction policies
  • Astronomical data optimization with metadata support
  • Intelligent prefetching based on access patterns
  • Memory pressure adaptation with configurable limits

Cache Performance:

  • L1: Hot data, LRU eviction, fastest access
  • L2: Warm data, LFU eviction with frequency tracking
  • L3: Cold data, time-based eviction for historical access
  • Cache hit rates: >80% for typical astronomical workloads

Phase 4: Production Optimization & Monitoring

Real-Time Monitoring System

  • Health check monitoring with component-level status tracking
  • Performance profiling with latency histograms and percentiles
  • Alert management with configurable thresholds and suppression
  • Comprehensive diagnostics including system resource tracking

Key Metrics Tracked:

  • Memory usage and efficiency ratios
  • Cache hit rates across all levels
  • Frame processing latency (P50, P95, P99)
  • System resource utilization
  • Error rates and alert conditions

Production Features:

  • Real-time health status reporting
  • Configurable alert thresholds
  • Performance profiling with microsecond precision
  • System diagnostics with resource tracking
  • Automated metric aggregation and retention

Memory Management Testing Commands

cd meteor-edge-client

# Phase 2 Testing
./target/release/meteor-edge-client test                   # Core frame pools
./target/release/meteor-edge-client test-adaptive          # Adaptive management
./target/release/meteor-edge-client test-integration       # Integration tests

# Phase 3 Testing  
./target/release/meteor-edge-client test-ring-buffer       # Ring buffers & memory mapping
./target/release/meteor-edge-client test-hierarchical-cache # Cache system

# Phase 4 Production Monitoring
./target/release/meteor-edge-client monitor                # Live monitoring system

# Phase 5 End-to-End Integration
./target/release/meteor-edge-client test-integrated-system    # Integrated memory system
./target/release/meteor-edge-client test-camera-integration   # Camera memory integration
./target/release/meteor-edge-client test-meteor-detection     # Real-time meteor detection

Phase 5: End-to-End Integration & Deployment

The final phase integrates all memory management components into a cohesive system for real-time meteor detection with camera integration.

Integrated Memory System

  • Unified Architecture: All memory components work together seamlessly
  • Multi-Configuration Support: Raspberry Pi and high-performance server configurations
  • Auto-Optimization: Dynamic performance tuning based on system conditions
  • Health Monitoring: Comprehensive system health reporting with recommendations

Key Components:

  • Hierarchical frame pools with adaptive management
  • Ring buffer streaming for astronomical frames
  • Multi-level caching with prefetching
  • Production monitoring with alerts
  • Camera integration with memory-optimized capture

Camera Memory Integration

  • Memory-Optimized Capture: Integration with hierarchical frame pools
  • Real-Time Processing: Zero-copy frame processing pipeline
  • Buffer Management: Adaptive capture buffer pools with memory pressure handling
  • Performance Monitoring: Camera-specific metrics and health reporting

Camera Features:

  • Multiple configuration support (Pi camera, performance camera)
  • Capture buffer pool with automatic optimization
  • Real-time statistics collection
  • Memory pressure detection and response
  • Health monitoring with diagnostic recommendations

Real-Time Meteor Detection Pipeline

  • Multi-Algorithm Detection: Brightness, motion, background subtraction algorithms
  • Consensus-Based Detection: Combines multiple algorithms for higher accuracy
  • Memory-Optimized Processing: Integrated with zero-copy architecture
  • Real-Time Performance: Sub-30ms processing latency for real-time detection

Detection Algorithms:

  • Brightness Detector: Threshold-based detection for bright meteors
  • Motion Detector: Optical flow analysis for movement detection
  • Background Subtraction: Adaptive background modeling for change detection
  • Consensus Detector: Weighted algorithm combination for improved accuracy

Production-Ready Features

  • Raspberry Pi Optimization: Conservative memory usage and CPU utilization
  • Real-Time Constraints: Guaranteed processing latency limits
  • Error Recovery: Robust error handling with automatic recovery
  • Performance Metrics: Comprehensive detection and system metrics
  • Memory Efficiency: Optimized for resource-constrained environments

Performance Benchmarks

  • Frame Pool Operations: >100K allocations/sec with zero memory leaks
  • Ring Buffer Throughput: >3M frames/sec with concurrent access
  • Cache Performance: >50K lookups/sec with 80%+ hit rates
  • Memory Efficiency: <2x growth under sustained load
  • Production Monitoring: Real-time metrics with <50μs overhead

This advanced memory management system enables the meteor edge client to:

  1. Process high-resolution astronomical frames with minimal memory overhead
  2. Adapt to varying system memory conditions automatically
  3. Provide production-grade observability and monitoring
  4. Maintain high performance on resource-constrained Raspberry Pi devices
  5. Support real-time meteor detection with sub-millisecond processing latency