# Event Data Upload API - Story 1.10 Implementation

This document describes the implementation of the Event Data Upload API that allows registered edge devices to securely upload event data (files and metadata) to the cloud platform.

## Overview

The Event Upload API provides a secure, asynchronous endpoint for edge devices to upload event data including:
- Media files (images, videos)
- Event metadata (timestamp, type, device information)
- Automatic cloud storage and processing queue integration

## API Endpoints

### POST /api/v1/events/upload

Uploads an event file with metadata from a registered device.

**Authentication**: JWT Bearer token required
**Content-Type**: multipart/form-data
**Response**: 202 Accepted

#### Request Format

```http
POST /api/v1/events/upload
Authorization: Bearer <JWT_TOKEN>
Content-Type: multipart/form-data

Form Fields:
- file: Binary file data (image/video)
- eventData: JSON string containing event metadata
```

#### Event Data JSON Structure

```json
{
  "eventType": "motion",
  "eventTimestamp": "2023-07-31T10:00:00Z",
  "metadata": {
    "deviceId": "device-uuid-here",
    "location": "front_door",
    "confidence": 0.95,
    "temperature": 22.5
  }
}
```

#### Response Format

```json
{
  "rawEventId": "event-uuid-here",
  "message": "Event uploaded successfully and queued for processing"
}
```

#### Error Responses

- `401 Unauthorized`: Invalid or missing JWT token
- `400 Bad Request`: Missing file, invalid event data, unsupported file type
- `404 Not Found`: Device not found or doesn't belong to user
- `413 Payload Too Large`: File exceeds 100MB limit
- `500 Internal Server Error`: AWS or database failure

### GET /api/v1/events/user

Retrieves events for the authenticated user.

**Authentication**: JWT Bearer token required

#### Query Parameters

- `limit` (optional): Maximum number of events to return (default: 50)
- `offset` (optional): Number of events to skip (default: 0)

#### Response Format

```json
{
  "events": [
    {
      "id": "event-uuid",
      "deviceId": "device-uuid",
      "eventType": "motion",
      "eventTimestamp": "2023-07-31T10:00:00Z",
      "filePath": "events/device-123/motion/2023-07-31_uuid.jpg",
      "processingStatus": "pending",
      "metadata": {
        "location": "front_door",
        "confidence": 0.95
      },
      "createdAt": "2023-07-31T10:00:00Z"
    }
  ],
  "total": 1
}
```

### GET /api/v1/events/device/:deviceId

Retrieves events for a specific device.

**Authentication**: JWT Bearer token required

#### Response Format

Same as `/api/v1/events/user` but filtered by device.

### GET /api/v1/events/health/check

Health check endpoint for the events service.

#### Response Format

```json
{
  "status": "healthy",
  "checks": {
    "database": true,
    "aws": true
  }
}
```

## Database Schema

### raw_events Table

| Column | Type | Description |
|--------|------|-------------|
| id | uuid | Primary key |
| device_id | uuid | Foreign key to devices table |
| user_profile_id | uuid | Foreign key to user_profiles table |
| file_path | text | S3 file path |
| file_size | bigint | File size in bytes |
| file_type | varchar(100) | MIME type |
| original_filename | varchar(255) | Original filename |
| event_type | varchar(50) | Type of event |
| event_timestamp | timestamptz | When event occurred on device |
| metadata | jsonb | Additional event metadata |
| processing_status | varchar(20) | pending/processing/completed/failed |
| sqs_message_id | varchar(255) | SQS message ID for tracking |
| processed_at | timestamptz | When processing completed |
| created_at | timestamptz | Record creation time |
| updated_at | timestamptz | Record update time |

## AWS Integration

### S3 File Storage

Files are stored in S3 with the following structure:
```
events/{deviceId}/{eventType}/{timestamp}_{uuid}.{extension}
```

Example: `events/device-123/motion/2023-07-31T10-00-00_a1b2c3d4.jpg`

### SQS Message Queue

After successful upload, a message is sent to SQS for async processing:

```json
{
  "rawEventId": "event-uuid",
  "deviceId": "device-uuid", 
  "userProfileId": "user-uuid",
  "eventType": "motion",
  "timestamp": "2023-07-31T10:00:00Z"
}
```

## Configuration

### Environment Variables

```bash
# AWS Configuration (required)
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_S3_BUCKET_NAME=meteor-events-bucket
AWS_SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/meteor-events-queue
```

### File Upload Limits

- Maximum file size: 100MB
- Supported file types:
  - Images: JPEG, PNG, GIF, WebP
  - Videos: MP4, AVI, QuickTime, X-MSVIDEO

## Security Features

1. **JWT Authentication**: All endpoints require valid JWT tokens
2. **Device Ownership Validation**: Users can only upload for their registered devices
3. **File Type Validation**: Only approved media types are accepted
4. **File Size Limits**: Prevents abuse with large file uploads
5. **Input Sanitization**: All inputs are validated and sanitized

## Error Handling

The API implements comprehensive error handling:

- **Validation Errors**: Client-side input validation with detailed messages
- **Authentication Errors**: Clear JWT-related error responses
- **AWS Failures**: Graceful handling of S3/SQS service failures
- **Database Errors**: Transaction rollbacks and appropriate error responses
- **File Processing Errors**: Proper cleanup of partial uploads

## Testing

### Unit Tests

Run unit tests for the EventsService:
```bash
npm run test -- --testPathPattern=events.service.spec.ts
```

### Integration Tests

Run end-to-end tests:
```bash
npm run test:e2e -- --testPathPattern=events.e2e-spec.ts
```

### Manual Testing

Use the provided test script:
```bash
# Install dependencies
npm install node-fetch@2 form-data

# Run test script
node test-event-upload.js
```

## Usage Example

### Edge Device Implementation

```javascript
const FormData = require('form-data');
const fs = require('fs');

async function uploadEvent(filePath, eventData, jwtToken) {
  const form = new FormData();
  
  // Add file
  form.append('file', fs.createReadStream(filePath));
  
  // Add event metadata
  form.append('eventData', JSON.stringify({
    eventType: 'motion',
    eventTimestamp: new Date().toISOString(),
    metadata: {
      deviceId: 'your-device-id',
      location: 'front_door',
      confidence: 0.95
    }
  }));

  const response = await fetch('http://localhost:3000/api/v1/events/upload', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${jwtToken}`,
    },
    body: form
  });

  if (response.ok) {
    const result = await response.json();
    console.log('Event uploaded:', result.rawEventId);
  } else {
    console.error('Upload failed:', await response.text());
  }
}
```

### cURL Example

```bash
curl -X POST http://localhost:3000/api/v1/events/upload \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -F "file=@motion_capture.jpg" \
  -F 'eventData={"eventType":"motion","eventTimestamp":"2023-07-31T10:00:00Z","metadata":{"deviceId":"device-uuid","location":"front_door"}}'
```

## Performance Considerations

- **Streaming Uploads**: Files are processed as streams to avoid memory issues
- **Async Processing**: Upload returns immediately; processing happens asynchronously
- **Database Indexing**: Optimized indexes for common query patterns
- **Connection Pooling**: Database connections are pooled for efficiency

## Monitoring and Observability

- **Health Checks**: `/api/v1/events/health/check` endpoint for monitoring
- **Structured Logging**: Comprehensive logging for debugging and monitoring
- **Error Tracking**: Detailed error logs with context information
- **Metrics**: Processing status tracking for analytics

## Deployment Notes

1. **AWS Setup**: Ensure S3 bucket and SQS queue are created and accessible
2. **IAM Permissions**: Configure appropriate IAM roles for S3 and SQS access
3. **Environment Variables**: Set all required AWS configuration variables
4. **Database Migration**: Run migrations to create the raw_events table
5. **File Upload Limits**: Configure reverse proxy (nginx) for large file uploads

## Future Enhancements

Potential improvements for future iterations:

1. **File Compression**: Automatic compression of uploaded files
2. **Batch Uploads**: Support for uploading multiple files at once
3. **Resumable Uploads**: Support for resuming interrupted uploads
4. **Image Processing**: Automatic thumbnail generation and image optimization
5. **Video Transcoding**: Automatic video format conversion and optimization
6. **Content Analysis**: AI-powered content analysis and tagging
7. **Retention Policies**: Automatic cleanup of old files based on policies

## Support

For issues or questions regarding the Event Upload API:

1. Check the application logs for detailed error information
2. Verify AWS credentials and service availability
3. Ensure database connectivity and table existence
4. Review file format and size requirements