meteor_detection_system/meteor-web-backend/EVENT_UPLOAD_README.md

Event Data Upload API - Story 1.10 Implementation

This document describes the implementation of the Event Data Upload API that allows registered edge devices to securely upload event data (files and metadata) to the cloud platform.

Overview

The Event Upload API provides a secure, asynchronous endpoint for edge devices to upload event data including:

• Media files (images, videos)
• Event metadata (timestamp, type, device information)
• Automatic cloud storage and processing queue integration

API Endpoints

POST /api/v1/events/upload

Uploads an event file with metadata from a registered device.

Authentication: JWT Bearer token required Content-Type: multipart/form-data Response: 202 Accepted

Request Format

POST /api/v1/events/upload
Authorization: Bearer <JWT_TOKEN>
Content-Type: multipart/form-data

Form Fields:
- file: Binary file data (image/video)
- eventData: JSON string containing event metadata

Event Data JSON Structure

{
  "eventType": "motion",
  "eventTimestamp": "2023-07-31T10:00:00Z",
  "metadata": {
    "deviceId": "device-uuid-here",
    "location": "front_door",
    "confidence": 0.95,
    "temperature": 22.5
  }
}

Response Format

{
  "rawEventId": "event-uuid-here",
  "message": "Event uploaded successfully and queued for processing"
}

Error Responses

• 401 Unauthorized: Invalid or missing JWT token
• 400 Bad Request: Missing file, invalid event data, unsupported file type
• 404 Not Found: Device not found or doesn't belong to user
• 413 Payload Too Large: File exceeds 100MB limit
• 500 Internal Server Error: AWS or database failure

GET /api/v1/events/user

Retrieves events for the authenticated user.

Authentication: JWT Bearer token required

Query Parameters

• limit (optional): Maximum number of events to return (default: 50)
• offset (optional): Number of events to skip (default: 0)

Response Format

{
  "events": [
    {
      "id": "event-uuid",
      "deviceId": "device-uuid",
      "eventType": "motion",
      "eventTimestamp": "2023-07-31T10:00:00Z",
      "filePath": "events/device-123/motion/2023-07-31_uuid.jpg",
      "processingStatus": "pending",
      "metadata": {
        "location": "front_door",
        "confidence": 0.95
      },
      "createdAt": "2023-07-31T10:00:00Z"
    }
  ],
  "total": 1
}

GET /api/v1/events/device/:deviceId

Retrieves events for a specific device.

Authentication: JWT Bearer token required

Response Format

Same as /api/v1/events/user but filtered by device.

GET /api/v1/events/health/check

Health check endpoint for the events service.

Response Format

{
  "status": "healthy",
  "checks": {
    "database": true,
    "aws": true
  }
}

Database Schema

raw_events Table

Column	Type	Description
id	uuid	Primary key
device_id	uuid	Foreign key to devices table
user_profile_id	uuid	Foreign key to user_profiles table
file_path	text	S3 file path
file_size	bigint	File size in bytes
file_type	varchar(100)	MIME type
original_filename	varchar(255)	Original filename
event_type	varchar(50)	Type of event
event_timestamp	timestamptz	When event occurred on device
metadata	jsonb	Additional event metadata
processing_status	varchar(20)	pending/processing/completed/failed
sqs_message_id	varchar(255)	SQS message ID for tracking
processed_at	timestamptz	When processing completed
created_at	timestamptz	Record creation time
updated_at	timestamptz	Record update time

AWS Integration

S3 File Storage

Files are stored in S3 with the following structure:

events/{deviceId}/{eventType}/{timestamp}_{uuid}.{extension}

Example: events/device-123/motion/2023-07-31T10-00-00_a1b2c3d4.jpg

SQS Message Queue

After successful upload, a message is sent to SQS for async processing:

{
  "rawEventId": "event-uuid",
  "deviceId": "device-uuid", 
  "userProfileId": "user-uuid",
  "eventType": "motion",
  "timestamp": "2023-07-31T10:00:00Z"
}

Configuration

Environment Variables

# AWS Configuration (required)
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_S3_BUCKET_NAME=meteor-events-bucket
AWS_SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/meteor-events-queue

File Upload Limits

• Maximum file size: 100MB
• Supported file types:
  • Images: JPEG, PNG, GIF, WebP
  • Videos: MP4, AVI, QuickTime, X-MSVIDEO

Security Features

1. JWT Authentication: All endpoints require valid JWT tokens
2. Device Ownership Validation: Users can only upload for their registered devices
3. File Type Validation: Only approved media types are accepted
4. File Size Limits: Prevents abuse with large file uploads
5. Input Sanitization: All inputs are validated and sanitized

Error Handling

The API implements comprehensive error handling:

• Validation Errors: Client-side input validation with detailed messages
• Authentication Errors: Clear JWT-related error responses
• AWS Failures: Graceful handling of S3/SQS service failures
• Database Errors: Transaction rollbacks and appropriate error responses
• File Processing Errors: Proper cleanup of partial uploads

Testing

Unit Tests

Run unit tests for the EventsService:

npm run test -- --testPathPattern=events.service.spec.ts

Integration Tests

Run end-to-end tests:

npm run test:e2e -- --testPathPattern=events.e2e-spec.ts

Manual Testing

Use the provided test script:

# Install dependencies
npm install node-fetch@2 form-data

# Run test script
node test-event-upload.js

Usage Example

Edge Device Implementation

const FormData = require('form-data');
const fs = require('fs');

async function uploadEvent(filePath, eventData, jwtToken) {
  const form = new FormData();
  
  // Add file
  form.append('file', fs.createReadStream(filePath));
  
  // Add event metadata
  form.append('eventData', JSON.stringify({
    eventType: 'motion',
    eventTimestamp: new Date().toISOString(),
    metadata: {
      deviceId: 'your-device-id',
      location: 'front_door',
      confidence: 0.95
    }
  }));

  const response = await fetch('http://localhost:3000/api/v1/events/upload', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${jwtToken}`,
    },
    body: form
  });

  if (response.ok) {
    const result = await response.json();
    console.log('Event uploaded:', result.rawEventId);
  } else {
    console.error('Upload failed:', await response.text());
  }
}

cURL Example

curl -X POST http://localhost:3000/api/v1/events/upload \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -F "file=@motion_capture.jpg" \
  -F 'eventData={"eventType":"motion","eventTimestamp":"2023-07-31T10:00:00Z","metadata":{"deviceId":"device-uuid","location":"front_door"}}'

Performance Considerations

• Streaming Uploads: Files are processed as streams to avoid memory issues
• Async Processing: Upload returns immediately; processing happens asynchronously
• Database Indexing: Optimized indexes for common query patterns
• Connection Pooling: Database connections are pooled for efficiency

Monitoring and Observability

• Health Checks: /api/v1/events/health/check endpoint for monitoring
• Structured Logging: Comprehensive logging for debugging and monitoring
• Error Tracking: Detailed error logs with context information
• Metrics: Processing status tracking for analytics

Deployment Notes

1. AWS Setup: Ensure S3 bucket and SQS queue are created and accessible
2. IAM Permissions: Configure appropriate IAM roles for S3 and SQS access
3. Environment Variables: Set all required AWS configuration variables
4. Database Migration: Run migrations to create the raw_events table
5. File Upload Limits: Configure reverse proxy (nginx) for large file uploads

Future Enhancements

Potential improvements for future iterations:

1. File Compression: Automatic compression of uploaded files
2. Batch Uploads: Support for uploading multiple files at once
3. Resumable Uploads: Support for resuming interrupted uploads
4. Image Processing: Automatic thumbnail generation and image optimization
5. Video Transcoding: Automatic video format conversion and optimization
6. Content Analysis: AI-powered content analysis and tagging
7. Retention Policies: Automatic cleanup of old files based on policies

Support

For issues or questions regarding the Event Upload API:

1. Check the application logs for detailed error information
2. Verify AWS credentials and service availability
3. Ensure database connectivity and table existence
4. Review file format and size requirements