meteor_detection_system/infrastructure

Meteor Fullstack Infrastructure

This directory contains Terraform configuration for the Meteor fullstack application AWS infrastructure.

Overview

The infrastructure includes:

• S3 bucket for storing meteor event files and media
• SQS queue for processing meteor events with dead letter queue
• CloudWatch dashboard for comprehensive monitoring
• IAM policies and roles for service permissions
• Optional RDS PostgreSQL instance
• Optional VPC and Fargate configuration for containerized deployment

Quick Start

1. Install Terraform (version >= 1.0)

2. Configure AWS credentials:

   aws configure
   # OR set environment variables:
   export AWS_ACCESS_KEY_ID="your-access-key"
   export AWS_SECRET_ACCESS_KEY="your-secret-key"
   export AWS_DEFAULT_REGION="us-east-1"
   

3. Copy and customize variables:

   cp terraform.tfvars.example terraform.tfvars
   # Edit terraform.tfvars with your desired configuration
   

4. Initialize and apply:

   terraform init
   terraform plan
   terraform apply
   

Configuration Options

Basic Setup (Default)

• Creates S3 bucket and SQS queue only
• Uses external database and container deployment
• Minimal cost option

With RDS Database

enable_rds = true
rds_instance_class = "db.t3.micro"  # or larger for production

With VPC and Fargate

enable_fargate = true
web_backend_cpu = 256
web_backend_memory = 512
compute_service_cpu = 256
compute_service_memory = 512

Environment Variables

After applying Terraform, configure your applications with these environment variables:

# From terraform output
AWS_REGION=$(terraform output -raw aws_region)
AWS_S3_BUCKET_NAME=$(terraform output -raw s3_bucket_name)
AWS_SQS_QUEUE_URL=$(terraform output -raw sqs_queue_url)

# If using RDS
DATABASE_URL=$(terraform output -raw rds_endpoint)

# If using IAM user (not Fargate)
AWS_ACCESS_KEY_ID=$(terraform output -raw app_access_key_id)
AWS_SECRET_ACCESS_KEY=$(terraform output -raw app_secret_access_key)

CloudWatch Dashboard

The infrastructure creates a comprehensive monitoring dashboard at:

https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=meteor-dev-monitoring-dashboard

Dashboard Includes:

• Application metrics: Request volume, response times, error rates
• Business metrics: Event processing, validation performance
• Infrastructure metrics: SQS queue depth, RDS performance, Fargate utilization
• Custom metrics: From your NestJS and Go services

Metrics Integration

Your applications are already configured to send metrics to CloudWatch:

NestJS Web Backend

• Namespace: MeteorApp/WebBackend
• Metrics: RequestCount, RequestDuration, ErrorCount, AuthOperationCount, etc.

Go Compute Service

• Namespace: MeteorApp/ComputeService
• Metrics: MessageProcessingCount, ValidationCount, DatabaseOperationCount, etc.

Cost Optimization

Development Environment

environment = "dev"
enable_rds = false                    # Use external database
enable_fargate = false                # Use external containers
cloudwatch_log_retention_days = 7     # Shorter retention

Production Environment

environment = "prod"
enable_rds = true
rds_instance_class = "db.t3.small"    # Appropriate size
enable_fargate = true                 # High availability
cloudwatch_log_retention_days = 30    # Longer retention

File Structure

infrastructure/
├── main.tf              # Provider and common configuration
├── variables.tf         # Input variables
├── outputs.tf           # Output values
├── s3.tf               # S3 bucket for event storage
├── sqs.tf              # SQS queues for processing
├── cloudwatch.tf       # Monitoring dashboard and alarms
├── iam.tf              # IAM roles and policies
├── rds.tf              # Optional PostgreSQL database
├── vpc.tf              # Optional VPC for Fargate
├── terraform.tfvars.example  # Example configuration
└── README.md           # This file

Deployment Integration

Docker Compose

Update your docker-compose.yml with Terraform outputs:

environment:
  - AWS_REGION=${AWS_REGION}
  - AWS_S3_BUCKET_NAME=${AWS_S3_BUCKET_NAME}
  - AWS_SQS_QUEUE_URL=${AWS_SQS_QUEUE_URL}

GitHub Actions

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v1
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: us-east-1

- name: Deploy infrastructure
  run: |
    cd infrastructure
    terraform init
    terraform apply -auto-approve

Security Best Practices

1. IAM Permissions: Follow principle of least privilege
2. S3 Security: All buckets have public access blocked
3. Encryption: S3 server-side encryption enabled
4. VPC: Private subnets for database and compute resources
5. Secrets: RDS passwords stored in AWS Secrets Manager

Monitoring and Alerts

The infrastructure includes CloudWatch alarms for:

• High error rates in web backend and compute service
• High response times
• SQS message age and dead letter queue messages
• RDS CPU utilization (when enabled)

To add notifications:

1. Create an SNS topic
2. Add the topic ARN to alarm actions in cloudwatch.tf

Cleanup

To destroy all resources:

terraform destroy

Warning: This will delete all data in S3 and databases. For production, ensure you have backups.

Troubleshooting

Common Issues

1. S3 bucket name conflicts: Bucket names must be globally unique

   • Solution: Change project_name or environment in variables

2. RDS subnet group errors: Requires subnets in different AZs

   • Solution: Ensure enable_fargate = true when using RDS

3. IAM permission errors: Check AWS credentials and permissions

   • Solution: Ensure your AWS account has admin access or required permissions

4. CloudWatch dashboard empty: Wait for applications to send metrics

   • Solution: Deploy and run your applications to generate metrics

Getting Help

1. Check Terraform documentation: https://registry.terraform.io/providers/hashicorp/aws/latest/docs
2. Review AWS service limits and quotas
3. Check AWS CloudFormation events for detailed error messages