meteor_detection_system/meteor-edge-client

Meteor Edge Client

An autonomous meteor detection system for edge devices (Raspberry Pi) with event-driven architecture, advanced memory management, real-time video processing, and cloud integration.

Overview

The Meteor Edge Client is a sophisticated edge computing application that serves as the "eyes" and "frontline sentinel" of the distributed meteor monitoring network. It autonomously performs continuous sky monitoring, meteor event detection, data archiving, and cloud synchronization without human intervention.

Core Features

Event-Driven Architecture

• Modular Design: All components operate as independent modules communicating through a central Event Bus
• Real-time Processing: Frame-by-frame video analysis with configurable detection algorithms
• Asynchronous Operations: Non-blocking event handling for optimal performance

Advanced Memory Management

• Zero-Copy Architecture: Arc-based frame sharing eliminates unnecessary memory copies
• Hierarchical Frame Pools: Multi-size buffer pools (64KB, 256KB, 900KB, 2MB) with adaptive management
• Lock-Free Ring Buffers: High-throughput astronomical frame streaming (>3M frames/sec)
• Memory-Mapped I/O: Efficient access to large astronomical datasets
• Adaptive Pool Management: Dynamic resizing based on memory pressure (70%/80%/90% thresholds)

Camera Simulation System

• Multiple Input Sources: Physical cameras, video files, test patterns, V4L2 loopback (Linux)
• FFmpeg Integration: Support for all major video formats (MP4, AVI, MKV, MOV)
• Test Pattern Generator: Synthetic meteor events for deterministic testing
• CI/CD Friendly: Headless operation for automated testing

Key Capabilities

• Autonomous Operation: Runs continuously without human intervention
• Meteor Detection: Real-time video analysis with multiple detection algorithms
• Event Recording: Automatic video capture and archiving of detected events
• Cloud Synchronization: Secure upload of events to backend API
• Device Registration: JWT-based device registration and authentication
• Structured Logging: JSON-formatted logs with correlation IDs for observability
• Hardware ID Detection: Automatic extraction of unique device identifiers

System Architecture

graph TD
    subgraph "Core"
        App[Application Coordinator]
        EventBus[Event Bus]
    end

    subgraph "Data Sources"
        Camera[Camera Controller]
        GPS[GPS Module - Future]
        Sensors[Environment Sensors - Future]
    end

    subgraph "Processing Pipeline"
        Detection[Detection Engine]
        Storage[Storage Manager]
        Communication[Cloud Communication]
    end

    App --> Camera
    App --> Detection
    App --> Storage
    App --> Communication
    
    Camera --FrameCapturedEvent--> EventBus
    EventBus --> Detection
    Detection --MeteorDetectedEvent--> EventBus
    EventBus --> Storage
    Storage --EventPackageArchivedEvent--> EventBus
    EventBus --> Communication

Module Descriptions

app - Application Coordinator

• Initializes and manages all modules
• Coordinates system lifecycle
• Handles graceful shutdown

events - Event Bus

• Central message passing system
• Enables decoupled module communication
• Supports multiple event types

camera - Camera Controller

• Real-time video frame capture
• Configurable FPS and resolution
• Publishes FrameCapturedEvent with timestamps

detection - Detection Pipeline

• Subscribes to video frames
• Maintains frame buffer for analysis
• Runs pluggable detection algorithms
• Publishes MeteorDetectedEvent on detection

storage - Storage Manager

• Archives detected events with metadata
• Manages local disk space
• Creates event packages for upload

communication - Communication Manager

• Handles cloud API integration
• Uploads event packages
• Manages device heartbeat

logging - Structured Logging

• JSON-formatted log output
• Correlation ID tracking
• Log rotation and upload

Installation

Prerequisites

• Rust 1.70+ (2021 edition)
• Network connectivity to Meteor backend
• Camera device (USB or CSI for Raspberry Pi)
• Sufficient disk space for event storage

Build from Source

# Clone the repository
git clone <repository-url>
cd meteor-edge-client

# Build for native platform
cargo build --release

# Cross-compile for Raspberry Pi (ARM64)
./build.sh

# Binary location
target/release/meteor-edge-client

Configuration

The application uses a unified TOML configuration file that includes both device registration and application settings.

Configuration File Location

• Primary: /etc/meteor-client/config.toml (system-wide)
• User: ~/.config/meteor-client/config.toml (user-specific)
• Fallback: ./meteor-client-config.toml (current directory)

Configuration Structure

# Device Registration Section
[device]
registered = true
hardware_id = "CPU_00000000a1b2c3d4"
device_id = "device-uuid-here"
user_profile_id = "user-uuid-here"
registered_at = "2023-07-30T12:00:00Z"
jwt_token = "eyJ..."

# API Configuration
[api]
base_url = "http://localhost:3000"
upload_endpoint = "/api/v1/events"
timeout_seconds = 30

# Camera Configuration
[camera]
source_type = "device"  # "device", "file", "test_pattern", "v4l2_loopback"
device_index = 0
fps = 30.0
width = 1280
height = 720

# Camera simulation (for testing)
[camera.test_pattern]
type = "simulated_meteor"  # static, noise, bar, meteor, checkerboard, gradient
duration_seconds = 300

[camera.file]
path = "test_data/meteor_capture.mp4"
loop = true
playback_speed = 1.0

[camera.v4l2_loopback]  # Linux only
device = "/dev/video10"
input_file = "test_data/meteor_capture.mp4"

[camera.performance]
enable_memory_optimization = true
max_camera_memory_mb = 64
buffer_pool_size = 20

# Detection Configuration
[detection]
algorithm = "brightness_diff"  # Detection algorithm to use
threshold = 0.3
buffer_frames = 150  # 5 seconds at 30fps

# Storage Configuration
[storage]
base_path = "/var/meteor/events"
max_storage_gb = 10
retention_days = 30
pre_event_seconds = 2
post_event_seconds = 3

# Communication Configuration
[communication]
heartbeat_interval_seconds = 60
upload_batch_size = 5
retry_attempts = 3

# Logging Configuration
[logging]
level = "info"
directory = "/var/log/meteor"
max_file_size_mb = 100
max_files = 10
upload_enabled = true

Usage

Basic Commands

1. Run Autonomous Detection System

# Production mode (requires device registration)
./meteor-edge-client run

# Test mode with camera simulation
cargo run --features camera_sim -- run --camera-source "test:meteor"

# Run with video file
cargo run --features camera_sim -- run --camera-source "file:test_data/meteor_video.mp4"

# Run with specific test pattern
cargo run --features camera_sim -- run --camera-source "test:checkerboard"

2. Camera Simulation Testing

# Available test patterns
--camera-source "test:static"       # Fixed pattern
--camera-source "test:noise"        # Random noise
--camera-source "test:bar"          # Moving bar
--camera-source "test:meteor"       # Meteor simulation with random events
--camera-source "test:checkerboard" # Checkerboard pattern
--camera-source "test:gradient"     # Gradient pattern

# Video file sources
--camera-source "file:video.mp4"       # Relative path
--camera-source "file:/path/to/video.mp4"  # Absolute path

# V4L2 loopback (Linux only)
--camera-source "v4l2:/dev/video10"

3. Memory Management Tests

# Advanced memory management testing commands
./target/release/meteor-edge-client test                 # Core frame pools
./target/release/meteor-edge-client test-adaptive        # Adaptive management
./target/release/meteor-edge-client test-integration     # Integration tests
./target/release/meteor-edge-client test-ring-buffer     # Ring buffers & memory mapping
./target/release/meteor-edge-client test-hierarchical-cache # Cache system
./target/release/meteor-edge-client monitor              # Production monitoring

4. Device Registration

# Register device with user account using JWT token
./meteor-edge-client register <JWT_TOKEN> [--api-url <URL>]

5. Device Status & Health

# Show hardware ID, registration status, and configuration
./meteor-edge-client status

# Verify backend connectivity
./meteor-edge-client health [--api-url <URL>]

# Version information
./meteor-edge-client version

Operational Workflow

Initial Setup

1. User logs into web interface and obtains JWT token
2. SSH into edge device
3. Run registration command with token
4. Verify registration with status command

Autonomous Operation

1. Start application with run command
2. System initializes all modules
3. Camera begins capturing frames
4. Detection algorithm analyzes frame stream
5. On meteor detection:
   • Event package created with video and metadata
   • Package archived to local storage
   • Package uploaded to cloud backend
6. Continuous operation with periodic health checks

Event Processing Pipeline

Data Flow

1. Frame Capture: Camera module captures video at configured FPS
2. Event Detection: Detection algorithm analyzes frame buffer
3. Event Archiving: Detected events saved with pre/post buffers
4. Cloud Upload: Compressed event packages sent to backend
5. Local Cleanup: Old events removed based on retention policy

Event Package Structure

event_<timestamp>_<event_id>/
├── metadata.json       # Event metadata and detection info
├── video.mp4          # Event video with pre/post buffer
├── frames/            # Key frame images
│   ├── trigger.jpg    # Frame that triggered detection
│   └── ...
└── logs/              # Related log entries

Advanced Memory Management

The meteor edge client features a sophisticated 5-phase memory optimization system designed for high-performance astronomical data processing on resource-constrained devices.

Key Features

Zero-Copy Architecture (Phase 1)

• Arc-based frame sharing eliminates unnecessary memory copies
• RAII pattern ensures automatic resource cleanup
• Event-driven processing with efficient memory propagation

Hierarchical Frame Pools (Phase 2)

• Multiple pool sizes: 64KB, 256KB, 900KB, 2MB buffers
• Adaptive capacity management based on memory pressure (70%/80%/90% thresholds)
• Historical metrics tracking for intelligent resizing
• Cross-platform memory pressure detection

Advanced Streaming & Caching (Phase 3)

• Lock-free ring buffers using atomic operations (>3M frames/sec throughput)
• Memory-mapped I/O for large astronomical datasets
• Multi-level cache architecture (L1/L2/L3) with different eviction policies
• Intelligent prefetching based on access patterns

Production Monitoring (Phase 4)

• Real-time health check monitoring with component-level status
• Performance profiling with latency histograms (P50, P95, P99)
• Alert management with configurable thresholds
• Comprehensive diagnostics including system resource tracking

End-to-End Integration (Phase 5)

• Unified memory system across all components
• Camera memory integration with zero-copy capture pipeline
• Real-time meteor detection with sub-30ms processing latency
• Multi-configuration support (Raspberry Pi and high-performance servers)

Performance Benchmarks

• Frame Pool Operations: >100K allocations/sec with zero memory leaks
• Ring Buffer Throughput: 3.6M+ writes/sec, 7.2M+ reads/sec
• Cache Performance: >50K lookups/sec with 80%+ hit rates
• Memory Efficiency: <2x growth under sustained load
• Memory Savings: Multi-GB savings through zero-copy architecture

Development

Running Tests

# Unit tests
cargo test

# Camera simulation tests
cargo test --features camera_sim

# Integration test
./demo_integration_test.sh

# With debug output
cargo test -- --nocapture

# Comprehensive camera simulation test suite
scripts/test_camera_simulation.sh all

Module Structure

• src/main.rs - CLI entry point and command handling
• src/app.rs - Application coordinator
• src/events.rs - Event bus and event types
• src/camera/ - Camera control and frame capture with simulation support
• src/camera_sim/ - Camera simulation backends (test patterns, file reader, FFmpeg, V4L2)
• src/detection.rs - Detection algorithms
• src/storage.rs - Event storage and archiving
• src/communication.rs - Cloud API client
• src/config.rs - Configuration management
• src/frame_pool.rs - Hierarchical frame pool management
• src/memory_monitor.rs - Memory monitoring and optimization
• src/hardware_fingerprint.rs - Hardware ID extraction
• src/logging.rs - Structured logging
• src/api.rs - HTTP client utilities

Production Deployment

Systemd Service

[Unit]
Description=Meteor Edge Detection System
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/meteor-edge-client run
Restart=always
RestartSec=10
User=meteor
Group=meteor
Environment="RUST_LOG=info"

[Install]
WantedBy=multi-user.target

Resource Requirements

• CPU: ARM Cortex-A53 or better
• RAM: 1GB minimum, 2GB recommended
• Storage: 16GB minimum for event buffering
• Network: Stable internet connection for cloud sync

Monitoring

• Structured JSON logs in /var/log/meteor/
• Prometheus metrics endpoint (future)
• Health check endpoint for monitoring tools
• Correlation IDs for request tracing

Troubleshooting

Common Issues

1. Camera not detected

   • Check camera connection (USB or CSI)
   • Verify camera permissions
   • Test with v4l2-ctl --list-devices

2. Detection not triggering

   • Adjust detection threshold in config
   • Check camera exposure settings
   • Verify sufficient lighting contrast

3. Upload failures

   • Check network connectivity
   • Verify backend API health
   • Review JWT token expiration

4. Storage issues

   • Monitor disk space usage
   • Adjust retention policy
   • Check write permissions

Debug Mode

# Run with debug logging
RUST_LOG=debug ./meteor-edge-client run

# Check logs
tail -f /var/log/meteor/meteor-edge-client.log

Camera Simulation for Development & Testing

The edge client includes a comprehensive camera simulation system that enables development and testing without physical camera hardware.

Quick Start with Simulation

# Build with camera simulation support
cargo build --features camera_sim

# Run with meteor simulation
cargo run --features camera_sim -- run --camera-source "test:meteor"

# Run with video file
cargo run --features camera_sim -- run --camera-source "file:test_data/meteor_video.mp4"

Available Simulation Backends

1. Test Pattern Generator - Synthetic patterns for deterministic testing

   • Static, noise, moving bar, checkerboard, gradient
   • Simulated meteor events with realistic brightness variations

2. File Reader - Read video files (MP4, AVI, MKV, MOV)

   • Loop playback support
   • Variable playback speed
   • Resolution and frame rate detection

3. V4L2 Loopback (Linux only) - Virtual V4L2 devices

   • Integration with external streaming tools
   • Multi-camera simulation support

4. FFmpeg Integration (optional) - Enhanced video processing

   • Hardware acceleration support
   • Broader format compatibility
   • Advanced video processing capabilities

Detailed Documentation

For comprehensive camera simulation documentation, see:

• CAMERA_SIMULATION_USAGE.md - Complete usage guide with examples and troubleshooting

CI/CD Integration

The camera simulation system is designed for headless operation in CI/CD pipelines:

# Install dependencies (Ubuntu/Debian)
sudo apt-get install -y ffmpeg libavcodec-dev libavformat-dev

# Run automated tests
cargo test --features camera_sim
scripts/test_camera_simulation.sh all

Future Enhancements

• GPS integration for location tagging
• Environmental sensor support
• RTSP streaming server
• Advanced ML-based detection algorithms
• Multi-camera support
• Real-time web dashboard
• Edge-to-edge communication
• Offline operation mode with sync

Documentation

• README.md - This file (main documentation)
• CAMERA_SIMULATION_USAGE.md - Camera simulation guide
• prd.md - Product requirements document (Chinese)

License

[Specify your license here]

Contributing

[Contribution guidelines if applicable]