grabbit/meteor_detection_system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
meteor_detection_system/meteor-edge-client/README.md
grabbit dd54b81262 lot
2025-11-03 00:11:13 +08:00

545 lines
16 KiB
Markdown
Raw Blame History

# Meteor Edge Client
An autonomous meteor detection system for edge devices (Raspberry Pi) with event-driven architecture, advanced memory management, real-time video processing, and cloud integration.
## Overview
The Meteor Edge Client is a sophisticated edge computing application that serves as the "eyes" and "frontline sentinel" of the distributed meteor monitoring network. It autonomously performs continuous sky monitoring, meteor event detection, data archiving, and cloud synchronization without human intervention.
## Core Features
### Event-Driven Architecture
- **Modular Design**: All components operate as independent modules communicating through a central Event Bus
- **Real-time Processing**: Frame-by-frame video analysis with configurable detection algorithms
- **Asynchronous Operations**: Non-blocking event handling for optimal performance
### Advanced Memory Management
- **Zero-Copy Architecture**: Arc-based frame sharing eliminates unnecessary memory copies
- **Hierarchical Frame Pools**: Multi-size buffer pools (64KB, 256KB, 900KB, 2MB) with adaptive management
- **Lock-Free Ring Buffers**: High-throughput astronomical frame streaming (>3M frames/sec)
- **Memory-Mapped I/O**: Efficient access to large astronomical datasets
- **Adaptive Pool Management**: Dynamic resizing based on memory pressure (70%/80%/90% thresholds)
### Camera Simulation System
- **Multiple Input Sources**: Physical cameras, video files, test patterns, V4L2 loopback (Linux)
- **FFmpeg Integration**: Support for all major video formats (MP4, AVI, MKV, MOV)
- **Test Pattern Generator**: Synthetic meteor events for deterministic testing
- **CI/CD Friendly**: Headless operation for automated testing
### Key Capabilities
- **Autonomous Operation**: Runs continuously without human intervention
- **Meteor Detection**: Real-time video analysis with multiple detection algorithms
- **Event Recording**: Automatic video capture and archiving of detected events
- **Cloud Synchronization**: Secure upload of events to backend API
- **Device Registration**: JWT-based device registration and authentication
- **Structured Logging**: JSON-formatted logs with correlation IDs for observability
- **Hardware ID Detection**: Automatic extraction of unique device identifiers
## System Architecture
```mermaid
graph TD
subgraph "Core"
App[Application Coordinator]
EventBus[Event Bus]
end
subgraph "Data Sources"
Camera[Camera Controller]
GPS[GPS Module - Future]
Sensors[Environment Sensors - Future]
end
subgraph "Processing Pipeline"
Detection[Detection Engine]
Storage[Storage Manager]
Communication[Cloud Communication]
end
App --> Camera
App --> Detection
App --> Storage
App --> Communication
Camera --FrameCapturedEvent--> EventBus
EventBus --> Detection
Detection --MeteorDetectedEvent--> EventBus
EventBus --> Storage
Storage --EventPackageArchivedEvent--> EventBus
EventBus --> Communication
```
## Module Descriptions
### `app` - Application Coordinator
- Initializes and manages all modules
- Coordinates system lifecycle
- Handles graceful shutdown
### `events` - Event Bus
- Central message passing system
- Enables decoupled module communication
- Supports multiple event types
### `camera` - Camera Controller
- Real-time video frame capture
- Configurable FPS and resolution
- Publishes FrameCapturedEvent with timestamps
### `detection` - Detection Pipeline
- Subscribes to video frames
- Maintains frame buffer for analysis
- Runs pluggable detection algorithms
- Publishes MeteorDetectedEvent on detection
### `storage` - Storage Manager
- Archives detected events with metadata
- Manages local disk space
- Creates event packages for upload
### `communication` - Communication Manager
- Handles cloud API integration
- Uploads event packages
- Manages device heartbeat
### `logging` - Structured Logging
- JSON-formatted log output
- Correlation ID tracking
- Log rotation and upload
## Installation
### Prerequisites
- Rust 1.70+ (2021 edition)
- Network connectivity to Meteor backend
- Camera device (USB or CSI for Raspberry Pi)
- Sufficient disk space for event storage
### Build from Source
```bash
# Clone the repository
git clone <repository-url>
cd meteor-edge-client
# Build for native platform
cargo build --release
# Cross-compile for Raspberry Pi (ARM64)
./build.sh
# Binary location
target/release/meteor-edge-client
```
## Configuration
The application uses a unified TOML configuration file that includes both device registration and application settings.
### Configuration File Location
- Primary: `/etc/meteor-client/config.toml` (system-wide)
- User: `~/.config/meteor-client/config.toml` (user-specific)
- Fallback: `./meteor-client-config.toml` (current directory)
### Configuration Structure
```toml
# Device Registration Section
[device]
registered = true
hardware_id = "CPU_00000000a1b2c3d4"
device_id = "device-uuid-here"
user_profile_id = "user-uuid-here"
registered_at = "2023-07-30T12:00:00Z"
jwt_token = "eyJ..."
# API Configuration
[api]
base_url = "http://localhost:3000"
upload_endpoint = "/api/v1/events"
timeout_seconds = 30
# Camera Configuration
[camera]
source_type = "device" # "device", "file", "test_pattern", "v4l2_loopback"
device_index = 0
fps = 30.0
width = 1280
height = 720
# Camera simulation (for testing)
[camera.test_pattern]
type = "simulated_meteor" # static, noise, bar, meteor, checkerboard, gradient
duration_seconds = 300
[camera.file]
path = "test_data/meteor_capture.mp4"
loop = true
playback_speed = 1.0
[camera.v4l2_loopback] # Linux only
device = "/dev/video10"
input_file = "test_data/meteor_capture.mp4"
[camera.performance]
enable_memory_optimization = true
max_camera_memory_mb = 64
buffer_pool_size = 20
# Detection Configuration
[detection]
algorithm = "brightness_diff" # Detection algorithm to use
threshold = 0.3
buffer_frames = 150 # 5 seconds at 30fps
# Storage Configuration
[storage]
base_path = "/var/meteor/events"
max_storage_gb = 10
retention_days = 30
pre_event_seconds = 2
post_event_seconds = 3
# Communication Configuration
[communication]
heartbeat_interval_seconds = 60
upload_batch_size = 5
retry_attempts = 3
# Logging Configuration
[logging]
level = "info"
directory = "/var/log/meteor"
max_file_size_mb = 100
max_files = 10
upload_enabled = true
```
## Usage
### Basic Commands
#### 1. Run Autonomous Detection System
```bash
# Production mode (requires device registration)
./meteor-edge-client run
# Test mode with camera simulation
cargo run --features camera_sim -- run --camera-source "test:meteor"
# Run with video file
cargo run --features camera_sim -- run --camera-source "file:test_data/meteor_video.mp4"
# Run with specific test pattern
cargo run --features camera_sim -- run --camera-source "test:checkerboard"
```
#### 2. Camera Simulation Testing
```bash
# Available test patterns
--camera-source "test:static" # Fixed pattern
--camera-source "test:noise" # Random noise
--camera-source "test:bar" # Moving bar
--camera-source "test:meteor" # Meteor simulation with random events
--camera-source "test:checkerboard" # Checkerboard pattern
--camera-source "test:gradient" # Gradient pattern
# Video file sources
--camera-source "file:video.mp4" # Relative path
--camera-source "file:/path/to/video.mp4" # Absolute path
# V4L2 loopback (Linux only)
--camera-source "v4l2:/dev/video10"
```
#### 3. Memory Management Tests
```bash
# Advanced memory management testing commands
./target/release/meteor-edge-client test # Core frame pools
./target/release/meteor-edge-client test-adaptive # Adaptive management
./target/release/meteor-edge-client test-integration # Integration tests
./target/release/meteor-edge-client test-ring-buffer # Ring buffers & memory mapping
./target/release/meteor-edge-client test-hierarchical-cache # Cache system
./target/release/meteor-edge-client monitor # Production monitoring
```
#### 4. Device Registration
```bash
# Register device with user account using JWT token
./meteor-edge-client register <JWT_TOKEN> [--api-url <URL>]
```
#### 5. Device Status & Health
```bash
# Show hardware ID, registration status, and configuration
./meteor-edge-client status
# Verify backend connectivity
./meteor-edge-client health [--api-url <URL>]
# Version information
./meteor-edge-client version
```
## Operational Workflow
### Initial Setup
1. User logs into web interface and obtains JWT token
2. SSH into edge device
3. Run registration command with token
4. Verify registration with status command
### Autonomous Operation
1. Start application with `run` command
2. System initializes all modules
3. Camera begins capturing frames
4. Detection algorithm analyzes frame stream
5. On meteor detection:
- Event package created with video and metadata
- Package archived to local storage
- Package uploaded to cloud backend
6. Continuous operation with periodic health checks
## Event Processing Pipeline
### Data Flow
1. **Frame Capture**: Camera module captures video at configured FPS
2. **Event Detection**: Detection algorithm analyzes frame buffer
3. **Event Archiving**: Detected events saved with pre/post buffers
4. **Cloud Upload**: Compressed event packages sent to backend
5. **Local Cleanup**: Old events removed based on retention policy
### Event Package Structure
```
event_<timestamp>_<event_id>/
├── metadata.json # Event metadata and detection info
├── video.mp4 # Event video with pre/post buffer
├── frames/ # Key frame images
│ ├── trigger.jpg # Frame that triggered detection
│ └── ...
└── logs/ # Related log entries
```
## Advanced Memory Management
The meteor edge client features a sophisticated 5-phase memory optimization system designed for high-performance astronomical data processing on resource-constrained devices.
### Key Features
#### Zero-Copy Architecture (Phase 1)
- Arc-based frame sharing eliminates unnecessary memory copies
- RAII pattern ensures automatic resource cleanup
- Event-driven processing with efficient memory propagation
#### Hierarchical Frame Pools (Phase 2)
- Multiple pool sizes: 64KB, 256KB, 900KB, 2MB buffers
- Adaptive capacity management based on memory pressure (70%/80%/90% thresholds)
- Historical metrics tracking for intelligent resizing
- Cross-platform memory pressure detection
#### Advanced Streaming & Caching (Phase 3)
- Lock-free ring buffers using atomic operations (>3M frames/sec throughput)
- Memory-mapped I/O for large astronomical datasets
- Multi-level cache architecture (L1/L2/L3) with different eviction policies
- Intelligent prefetching based on access patterns
#### Production Monitoring (Phase 4)
- Real-time health check monitoring with component-level status
- Performance profiling with latency histograms (P50, P95, P99)
- Alert management with configurable thresholds
- Comprehensive diagnostics including system resource tracking
#### End-to-End Integration (Phase 5)
- Unified memory system across all components
- Camera memory integration with zero-copy capture pipeline
- Real-time meteor detection with sub-30ms processing latency
- Multi-configuration support (Raspberry Pi and high-performance servers)
### Performance Benchmarks
- **Frame Pool Operations**: >100K allocations/sec with zero memory leaks
- **Ring Buffer Throughput**: 3.6M+ writes/sec, 7.2M+ reads/sec
- **Cache Performance**: >50K lookups/sec with 80%+ hit rates
- **Memory Efficiency**: <2x growth under sustained load
- **Memory Savings**: Multi-GB savings through zero-copy architecture
## Development
### Running Tests
```bash
# Unit tests
cargo test
# Camera simulation tests
cargo test --features camera_sim
# Integration test
./demo_integration_test.sh
# With debug output
cargo test -- --nocapture
# Comprehensive camera simulation test suite
scripts/test_camera_simulation.sh all
```
### Module Structure
- `src/main.rs` - CLI entry point and command handling
- `src/app.rs` - Application coordinator
- `src/events.rs` - Event bus and event types
- `src/camera/` - Camera control and frame capture with simulation support
- `src/camera_sim/` - Camera simulation backends (test patterns, file reader, FFmpeg, V4L2)
- `src/detection.rs` - Detection algorithms
- `src/storage.rs` - Event storage and archiving
- `src/communication.rs` - Cloud API client
- `src/config.rs` - Configuration management
- `src/frame_pool.rs` - Hierarchical frame pool management
- `src/memory_monitor.rs` - Memory monitoring and optimization
- `src/hardware_fingerprint.rs` - Hardware ID extraction
- `src/logging.rs` - Structured logging
- `src/api.rs` - HTTP client utilities
## Production Deployment
### Systemd Service
```ini
[Unit]
Description=Meteor Edge Detection System
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/meteor-edge-client run
Restart=always
RestartSec=10
User=meteor
Group=meteor
Environment="RUST_LOG=info"
[Install]
WantedBy=multi-user.target
```
### Resource Requirements
- **CPU**: ARM Cortex-A53 or better
- **RAM**: 1GB minimum, 2GB recommended
- **Storage**: 16GB minimum for event buffering
- **Network**: Stable internet connection for cloud sync
### Monitoring
- Structured JSON logs in `/var/log/meteor/`
- Prometheus metrics endpoint (future)
- Health check endpoint for monitoring tools
- Correlation IDs for request tracing
## Troubleshooting
### Common Issues
1. **Camera not detected**
- Check camera connection (USB or CSI)
- Verify camera permissions
- Test with `v4l2-ctl --list-devices`
2. **Detection not triggering**
- Adjust detection threshold in config
- Check camera exposure settings
- Verify sufficient lighting contrast
3. **Upload failures**
- Check network connectivity
- Verify backend API health
- Review JWT token expiration
4. **Storage issues**
- Monitor disk space usage
- Adjust retention policy
- Check write permissions
### Debug Mode
```bash
# Run with debug logging
RUST_LOG=debug ./meteor-edge-client run
# Check logs
tail -f /var/log/meteor/meteor-edge-client.log
```
## Camera Simulation for Development & Testing
The edge client includes a comprehensive camera simulation system that enables development and testing without physical camera hardware.
### Quick Start with Simulation
```bash
# Build with camera simulation support
cargo build --features camera_sim
# Run with meteor simulation
cargo run --features camera_sim -- run --camera-source "test:meteor"
# Run with video file
cargo run --features camera_sim -- run --camera-source "file:test_data/meteor_video.mp4"
```
### Available Simulation Backends
1. **Test Pattern Generator** - Synthetic patterns for deterministic testing
- Static, noise, moving bar, checkerboard, gradient
- Simulated meteor events with realistic brightness variations
2. **File Reader** - Read video files (MP4, AVI, MKV, MOV)
- Loop playback support
- Variable playback speed
- Resolution and frame rate detection
3. **V4L2 Loopback** (Linux only) - Virtual V4L2 devices
- Integration with external streaming tools
- Multi-camera simulation support
4. **FFmpeg Integration** (optional) - Enhanced video processing
- Hardware acceleration support
- Broader format compatibility
- Advanced video processing capabilities
### Detailed Documentation
For comprehensive camera simulation documentation, see:
- [CAMERA_SIMULATION_USAGE.md](CAMERA_SIMULATION_USAGE.md) - Complete usage guide with examples and troubleshooting
### CI/CD Integration
The camera simulation system is designed for headless operation in CI/CD pipelines:
```bash
# Install dependencies (Ubuntu/Debian)
sudo apt-get install -y ffmpeg libavcodec-dev libavformat-dev
# Run automated tests
cargo test --features camera_sim
scripts/test_camera_simulation.sh all
```
## Future Enhancements
- GPS integration for location tagging
- Environmental sensor support
- RTSP streaming server
- Advanced ML-based detection algorithms
- Multi-camera support
- Real-time web dashboard
- Edge-to-edge communication
- Offline operation mode with sync
## Documentation
- [README.md](README.md) - This file (main documentation)
- [CAMERA_SIMULATION_USAGE.md](CAMERA_SIMULATION_USAGE.md) - Camera simulation guide
- [prd.md](prd.md) - Product requirements document (Chinese)
## License
[Specify your license here]
## Contributing
[Contribution guidelines if applicable]
Reference in New Issue View Git Blame Copy Permalink