diff --git a/infrastructure/README.md b/infrastructure/README.md new file mode 100644 index 0000000..dd55373 --- /dev/null +++ b/infrastructure/README.md @@ -0,0 +1,217 @@ +# Meteor Fullstack Infrastructure + +This directory contains Terraform configuration for the Meteor fullstack application AWS infrastructure. + +## Overview + +The infrastructure includes: +- **S3 bucket** for storing meteor event files and media +- **SQS queue** for processing meteor events with dead letter queue +- **CloudWatch dashboard** for comprehensive monitoring +- **IAM policies** and roles for service permissions +- **Optional RDS PostgreSQL** instance +- **Optional VPC and Fargate** configuration for containerized deployment + +## Quick Start + +1. **Install Terraform** (version >= 1.0) +2. **Configure AWS credentials**: + ```bash + aws configure + # OR set environment variables: + export AWS_ACCESS_KEY_ID="your-access-key" + export AWS_SECRET_ACCESS_KEY="your-secret-key" + export AWS_DEFAULT_REGION="us-east-1" + ``` + +3. **Copy and customize variables**: + ```bash + cp terraform.tfvars.example terraform.tfvars + # Edit terraform.tfvars with your desired configuration + ``` + +4. **Initialize and apply**: + ```bash + terraform init + terraform plan + terraform apply + ``` + +## Configuration Options + +### Basic Setup (Default) +- Creates S3 bucket and SQS queue only +- Uses external database and container deployment +- Minimal cost option + +### With RDS Database +```hcl +enable_rds = true +rds_instance_class = "db.t3.micro" # or larger for production +``` + +### With VPC and Fargate +```hcl +enable_fargate = true +web_backend_cpu = 256 +web_backend_memory = 512 +compute_service_cpu = 256 +compute_service_memory = 512 +``` + +## Environment Variables + +After applying Terraform, configure your applications with these environment variables: + +```bash +# From terraform output +AWS_REGION=$(terraform output -raw aws_region) +AWS_S3_BUCKET_NAME=$(terraform output -raw s3_bucket_name) +AWS_SQS_QUEUE_URL=$(terraform output -raw sqs_queue_url) + +# If using RDS +DATABASE_URL=$(terraform output -raw rds_endpoint) + +# If using IAM user (not Fargate) +AWS_ACCESS_KEY_ID=$(terraform output -raw app_access_key_id) +AWS_SECRET_ACCESS_KEY=$(terraform output -raw app_secret_access_key) +``` + +## CloudWatch Dashboard + +The infrastructure creates a comprehensive monitoring dashboard at: +``` +https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#dashboards:name=meteor-dev-monitoring-dashboard +``` + +### Dashboard Includes: +- **Application metrics**: Request volume, response times, error rates +- **Business metrics**: Event processing, validation performance +- **Infrastructure metrics**: SQS queue depth, RDS performance, Fargate utilization +- **Custom metrics**: From your NestJS and Go services + +## Metrics Integration + +Your applications are already configured to send metrics to CloudWatch: + +### NestJS Web Backend +- Namespace: `MeteorApp/WebBackend` +- Metrics: RequestCount, RequestDuration, ErrorCount, AuthOperationCount, etc. + +### Go Compute Service +- Namespace: `MeteorApp/ComputeService` +- Metrics: MessageProcessingCount, ValidationCount, DatabaseOperationCount, etc. + +## Cost Optimization + +### Development Environment +```hcl +environment = "dev" +enable_rds = false # Use external database +enable_fargate = false # Use external containers +cloudwatch_log_retention_days = 7 # Shorter retention +``` + +### Production Environment +```hcl +environment = "prod" +enable_rds = true +rds_instance_class = "db.t3.small" # Appropriate size +enable_fargate = true # High availability +cloudwatch_log_retention_days = 30 # Longer retention +``` + +## File Structure + +``` +infrastructure/ +├── main.tf # Provider and common configuration +├── variables.tf # Input variables +├── outputs.tf # Output values +├── s3.tf # S3 bucket for event storage +├── sqs.tf # SQS queues for processing +├── cloudwatch.tf # Monitoring dashboard and alarms +├── iam.tf # IAM roles and policies +├── rds.tf # Optional PostgreSQL database +├── vpc.tf # Optional VPC for Fargate +├── terraform.tfvars.example # Example configuration +└── README.md # This file +``` + +## Deployment Integration + +### Docker Compose +Update your `docker-compose.yml` with Terraform outputs: +```yaml +environment: + - AWS_REGION=${AWS_REGION} + - AWS_S3_BUCKET_NAME=${AWS_S3_BUCKET_NAME} + - AWS_SQS_QUEUE_URL=${AWS_SQS_QUEUE_URL} +``` + +### GitHub Actions +```yaml +- name: Configure AWS credentials + uses: aws-actions/configure-aws-credentials@v1 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: us-east-1 + +- name: Deploy infrastructure + run: | + cd infrastructure + terraform init + terraform apply -auto-approve +``` + +## Security Best Practices + +1. **IAM Permissions**: Follow principle of least privilege +2. **S3 Security**: All buckets have public access blocked +3. **Encryption**: S3 server-side encryption enabled +4. **VPC**: Private subnets for database and compute resources +5. **Secrets**: RDS passwords stored in AWS Secrets Manager + +## Monitoring and Alerts + +The infrastructure includes CloudWatch alarms for: +- High error rates in web backend and compute service +- High response times +- SQS message age and dead letter queue messages +- RDS CPU utilization (when enabled) + +To add notifications: +1. Create an SNS topic +2. Add the topic ARN to alarm actions in `cloudwatch.tf` + +## Cleanup + +To destroy all resources: +```bash +terraform destroy +``` + +**Warning**: This will delete all data in S3 and databases. For production, ensure you have backups. + +## Troubleshooting + +### Common Issues + +1. **S3 bucket name conflicts**: Bucket names must be globally unique + - Solution: Change `project_name` or `environment` in variables + +2. **RDS subnet group errors**: Requires subnets in different AZs + - Solution: Ensure `enable_fargate = true` when using RDS + +3. **IAM permission errors**: Check AWS credentials and permissions + - Solution: Ensure your AWS account has admin access or required permissions + +4. **CloudWatch dashboard empty**: Wait for applications to send metrics + - Solution: Deploy and run your applications to generate metrics + +### Getting Help + +1. Check Terraform documentation: https://registry.terraform.io/providers/hashicorp/aws/latest/docs +2. Review AWS service limits and quotas +3. Check AWS CloudFormation events for detailed error messages \ No newline at end of file diff --git a/infrastructure/cloudwatch.tf b/infrastructure/cloudwatch.tf new file mode 100644 index 0000000..efd0bea --- /dev/null +++ b/infrastructure/cloudwatch.tf @@ -0,0 +1,486 @@ +# CloudWatch Dashboard for Meteor Application Monitoring +resource "aws_cloudwatch_dashboard" "meteor_dashboard" { + dashboard_name = "${local.name_prefix}-monitoring-dashboard" + + dashboard_body = jsonencode({ + widgets = [ + # Row 1: Application Overview + { + type = "metric" + x = 0 + y = 0 + width = 12 + height = 6 + + properties = { + metrics = [ + ["MeteorApp/WebBackend", "RequestCount", { "stat": "Sum" }], + [".", "ErrorCount", { "stat": "Sum" }], + ["MeteorApp/ComputeService", "MessageProcessingCount", { "stat": "Sum" }], + [".", "MessageProcessingError", { "stat": "Sum" }] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Request and Processing Volume" + period = 300 + yAxis = { + left = { + min = 0 + } + } + } + }, + { + type = "metric" + x = 12 + y = 0 + width = 12 + height = 6 + + properties = { + metrics = [ + ["MeteorApp/WebBackend", "RequestDuration", { "stat": "Average" }], + [".", "RequestDuration", { "stat": "p95" }], + ["MeteorApp/ComputeService", "MessageProcessingDuration", { "stat": "Average" }], + [".", "MessageProcessingDuration", { "stat": "p95" }] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Response Time and Processing Latency" + period = 300 + yAxis = { + left = { + min = 0 + } + } + } + }, + + # Row 2: Error Rates and Success Metrics + { + type = "metric" + x = 0 + y = 6 + width = 8 + height = 6 + + properties = { + metrics = [ + [{ "expression": "m1/m2*100", "label": "Web Backend Error Rate %" }], + [{ "expression": "m3/m4*100", "label": "Compute Service Error Rate %" }], + ["MeteorApp/WebBackend", "ErrorCount", { "id": "m1", "visible": false }], + [".", "RequestCount", { "id": "m2", "visible": false }], + ["MeteorApp/ComputeService", "MessageProcessingError", { "id": "m3", "visible": false }], + [".", "MessageProcessingCount", { "id": "m4", "visible": false }] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Error Rates" + period = 300 + yAxis = { + left = { + min = 0 + max = 100 + } + } + } + }, + { + type = "metric" + x = 8 + y = 6 + width = 8 + height = 6 + + properties = { + metrics = [ + ["MeteorApp/WebBackend", "AuthOperationCount", "Success", "true"], + [".", "PaymentOperationCount", "Success", "true"], + ["MeteorApp/ComputeService", "ValidationSuccess"] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Successful Operations" + period = 300 + yAxis = { + left = { + min = 0 + } + } + } + }, + { + type = "metric" + x = 16 + y = 6 + width = 8 + height = 6 + + properties = { + metrics = [ + ["MeteorApp/ComputeService", "EventsProcessed", { "stat": "Sum" }], + [".", "ValidationCount", { "stat": "Sum" }], + ["MeteorApp/WebBackend", "EventProcessingCount", { "stat": "Sum" }] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Event Processing Volume" + period = 300 + yAxis = { + left = { + min = 0 + } + } + } + }, + + # Row 3: Infrastructure Metrics + { + type = "metric" + x = 0 + y = 12 + width = 8 + height = 6 + + properties = { + metrics = concat( + var.enable_rds ? [ + ["AWS/RDS", "CPUUtilization", "DBInstanceIdentifier", "${local.name_prefix}-postgres"], + [".", "DatabaseConnections", "DBInstanceIdentifier", "${local.name_prefix}-postgres"] + ] : [], + [ + # Add external database metrics if available + ] + ) + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Database Performance" + period = 300 + yAxis = { + left = { + min = 0 + } + } + } + }, + { + type = "metric" + x = 8 + y = 12 + width = 8 + height = 6 + + properties = { + metrics = [ + ["AWS/SQS", "ApproximateNumberOfVisibleMessages", "QueueName", aws_sqs_queue.meteor_processing.name], + [".", "ApproximateAgeOfOldestMessage", "QueueName", aws_sqs_queue.meteor_processing.name], + [".", "ApproximateNumberOfVisibleMessages", "QueueName", aws_sqs_queue.meteor_processing_dlq.name] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "SQS Queue Metrics" + period = 300 + yAxis = { + left = { + min = 0 + } + } + } + }, + { + type = "metric" + x = 16 + y = 12 + width = 8 + height = 6 + + properties = { + metrics = concat( + var.enable_fargate ? [ + ["AWS/ECS", "CPUUtilization", "ServiceName", "${local.name_prefix}-web-backend"], + [".", "MemoryUtilization", "ServiceName", "${local.name_prefix}-web-backend"], + [".", "CPUUtilization", "ServiceName", "${local.name_prefix}-compute-service"], + [".", "MemoryUtilization", "ServiceName", "${local.name_prefix}-compute-service"] + ] : [], + [ + # Placeholder for external container metrics + ] + ) + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Container Resource Utilization" + period = 300 + yAxis = { + left = { + min = 0 + max = 100 + } + } + } + }, + + # Row 4: Business Metrics + { + type = "metric" + x = 0 + y = 18 + width = 12 + height = 6 + + properties = { + metrics = [ + ["MeteorApp/ComputeService", "ValidationDuration", "ProviderName", "classic_cv", { "stat": "Average" }], + [".", "ValidationDuration", "ProviderName", "mvp", { "stat": "Average" }], + [".", "ValidationCount", "ProviderName", "classic_cv"], + [".", "ValidationCount", "ProviderName", "mvp"] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Validation Provider Performance" + period = 300 + yAxis = { + left = { + min = 0 + } + } + } + }, + { + type = "metric" + x = 12 + y = 18 + width = 12 + height = 6 + + properties = { + metrics = [ + ["MeteorApp/ComputeService", "DatabaseOperationDuration", "Operation", "CreateValidatedEvent"], + [".", "DatabaseOperationDuration", "Operation", "GetRawEventByID"], + [".", "DatabaseOperationCount", "Operation", "CreateValidatedEvent"], + [".", "DatabaseOperationCount", "Operation", "GetRawEventByID"] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "Database Operation Performance" + period = 300 + yAxis = { + left = { + min = 0 + } + } + } + }, + + # Row 5: Custom Metrics and Alerts + { + type = "metric" + x = 0 + y = 24 + width = 8 + height = 6 + + properties = { + metrics = [ + ["AWS/S3", "BucketSizeBytes", "BucketName", aws_s3_bucket.meteor_events.bucket, "StorageType", "StandardStorage"], + [".", "NumberOfObjects", "BucketName", aws_s3_bucket.meteor_events.bucket, "StorageType", "AllStorageTypes"] + ] + view = "timeSeries" + stacked = false + region = var.aws_region + title = "S3 Storage Metrics" + period = 86400 # Daily + yAxis = { + left = { + min = 0 + } + } + } + }, + { + type = "log" + x = 8 + y = 24 + width = 16 + height = 6 + + properties = { + query = "SOURCE '/aws/lambda/${local.name_prefix}' | fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20" + region = var.aws_region + title = "Recent Error Logs" + view = "table" + } + } + ] + }) + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-dashboard" + Description = "Comprehensive monitoring dashboard for Meteor application" + }) +} + +# CloudWatch Log Groups +resource "aws_cloudwatch_log_group" "web_backend" { + name = "/aws/ecs/${local.name_prefix}-web-backend" + retention_in_days = var.cloudwatch_log_retention_days + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-web-backend-logs" + Description = "Log group for web backend service" + }) +} + +resource "aws_cloudwatch_log_group" "compute_service" { + name = "/aws/ecs/${local.name_prefix}-compute-service" + retention_in_days = var.cloudwatch_log_retention_days + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-compute-service-logs" + Description = "Log group for compute service" + }) +} + +# CloudWatch Alarms for Critical System Health + +# Alarm for NestJS 5xx Error Rate (>1% over 5 minutes) +resource "aws_cloudwatch_metric_alarm" "nestjs_5xx_error_rate" { + alarm_name = "${local.name_prefix}-nestjs-5xx-error-rate" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = var.alarm_evaluation_periods + treat_missing_data = "notBreaching" + + metric_query { + id = "e1" + return_data = false + + metric { + metric_name = "ErrorCount" + namespace = "MeteorApp/WebBackend" + period = var.alarm_period_seconds + stat = "Sum" + } + } + + metric_query { + id = "e2" + return_data = false + + metric { + metric_name = "RequestCount" + namespace = "MeteorApp/WebBackend" + period = var.alarm_period_seconds + stat = "Sum" + } + } + + metric_query { + id = "e3" + expression = "SEARCH('{MeteorApp/WebBackend,StatusCode} ErrorCount StatusCode=5*', 'Sum', ${var.alarm_period_seconds})" + label = "5xx Errors" + return_data = false + } + + metric_query { + id = "e4" + expression = "(SUM(e3)/e2)*100" + label = "5xx Error Rate %" + return_data = true + } + + threshold = var.nestjs_error_rate_threshold + alarm_description = "CRITICAL: NestJS 5xx error rate exceeds ${var.nestjs_error_rate_threshold}% over 5 minutes. This indicates server errors that require immediate investigation. Check application logs and recent deployments." + alarm_actions = [aws_sns_topic.alerts.arn] + ok_actions = [aws_sns_topic.alerts.arn] + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-nestjs-5xx-error-rate" + Severity = "Critical" + Service = "WebBackend" + }) +} + +# Alarm for Go Service Processing Failure Rate (>5% over 5 minutes) +resource "aws_cloudwatch_metric_alarm" "go_service_failure_rate" { + alarm_name = "${local.name_prefix}-go-service-failure-rate" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = var.alarm_evaluation_periods + treat_missing_data = "notBreaching" + + metric_query { + id = "e1" + return_data = false + + metric { + metric_name = "MessageProcessingError" + namespace = "MeteorApp/ComputeService" + period = var.alarm_period_seconds + stat = "Sum" + } + } + + metric_query { + id = "e2" + return_data = false + + metric { + metric_name = "MessageProcessingCount" + namespace = "MeteorApp/ComputeService" + period = var.alarm_period_seconds + stat = "Sum" + } + } + + metric_query { + id = "e3" + expression = "(e1/e2)*100" + label = "Processing Failure Rate %" + return_data = true + } + + threshold = var.go_service_failure_rate_threshold + alarm_description = "CRITICAL: Go compute service processing failure rate exceeds ${var.go_service_failure_rate_threshold}% over 5 minutes. This indicates message processing issues. Check service logs, SQS dead letter queue, and validation providers." + alarm_actions = [aws_sns_topic.alerts.arn] + ok_actions = [aws_sns_topic.alerts.arn] + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-go-service-failure-rate" + Severity = "Critical" + Service = "ComputeService" + }) +} + +# Alarm for SQS Queue Depth (>1000 visible messages) +resource "aws_cloudwatch_metric_alarm" "sqs_queue_depth" { + alarm_name = "${local.name_prefix}-sqs-queue-depth" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = var.alarm_evaluation_periods + metric_name = "ApproximateNumberOfVisibleMessages" + namespace = "AWS/SQS" + period = var.alarm_period_seconds + statistic = "Average" + threshold = var.sqs_queue_depth_threshold + treat_missing_data = "notBreaching" + alarm_description = "CRITICAL: SQS queue depth exceeds ${var.sqs_queue_depth_threshold} messages. This indicates message processing backlog. Check compute service health, scaling, and processing capacity." + alarm_actions = [aws_sns_topic.alerts.arn] + ok_actions = [aws_sns_topic.alerts.arn] + + dimensions = { + QueueName = aws_sqs_queue.meteor_processing.name + } + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-sqs-queue-depth" + Severity = "Critical" + Service = "SQS" + }) +} \ No newline at end of file diff --git a/infrastructure/iam.tf b/infrastructure/iam.tf new file mode 100644 index 0000000..1b36e7d --- /dev/null +++ b/infrastructure/iam.tf @@ -0,0 +1,194 @@ +# IAM role for ECS task execution (Fargate) +resource "aws_iam_role" "ecs_task_execution" { + count = var.enable_fargate ? 1 : 0 + name = "${local.name_prefix}-ecs-task-execution" + + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "ecs-tasks.amazonaws.com" + } + } + ] + }) + + tags = local.common_tags +} + +# Attach the ECS task execution role policy +resource "aws_iam_role_policy_attachment" "ecs_task_execution" { + count = var.enable_fargate ? 1 : 0 + role = aws_iam_role.ecs_task_execution[0].name + policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy" +} + +# IAM role for ECS tasks (application permissions) +resource "aws_iam_role" "ecs_task" { + count = var.enable_fargate ? 1 : 0 + name = "${local.name_prefix}-ecs-task" + + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "ecs-tasks.amazonaws.com" + } + } + ] + }) + + tags = local.common_tags +} + +# IAM policy for application services +resource "aws_iam_policy" "meteor_app" { + name = "${local.name_prefix}-app-policy" + description = "IAM policy for Meteor application services" + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + # S3 permissions for event storage + { + Effect = "Allow" + Action = [ + "s3:GetObject", + "s3:PutObject", + "s3:DeleteObject", + "s3:ListBucket" + ] + Resource = [ + aws_s3_bucket.meteor_events.arn, + "${aws_s3_bucket.meteor_events.arn}/*" + ] + }, + # SQS permissions for message processing + { + Effect = "Allow" + Action = [ + "sqs:ReceiveMessage", + "sqs:DeleteMessage", + "sqs:SendMessage", + "sqs:GetQueueAttributes", + "sqs:GetQueueUrl" + ] + Resource = [ + aws_sqs_queue.meteor_processing.arn, + aws_sqs_queue.meteor_processing_dlq.arn + ] + }, + # CloudWatch permissions for metrics and logs + { + Effect = "Allow" + Action = [ + "cloudwatch:PutMetricData", + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents", + "logs:DescribeLogStreams" + ] + Resource = "*" + }, + # Secrets Manager permissions (if using RDS) + { + Effect = "Allow" + Action = [ + "secretsmanager:GetSecretValue" + ] + Resource = var.enable_rds ? [aws_secretsmanager_secret.rds_password[0].arn] : [] + } + ] + }) + + tags = local.common_tags +} + +# Attach the application policy to the ECS task role +resource "aws_iam_role_policy_attachment" "ecs_task_app_policy" { + count = var.enable_fargate ? 1 : 0 + role = aws_iam_role.ecs_task[0].name + policy_arn = aws_iam_policy.meteor_app.arn +} + +# IAM user for application services (when not using Fargate) +resource "aws_iam_user" "meteor_app" { + count = var.enable_fargate ? 0 : 1 + name = "${local.name_prefix}-app-user" + path = "/" + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-app-user" + Description = "IAM user for Meteor application services" + }) +} + +# Attach policy to IAM user +resource "aws_iam_user_policy_attachment" "meteor_app" { + count = var.enable_fargate ? 0 : 1 + user = aws_iam_user.meteor_app[0].name + policy_arn = aws_iam_policy.meteor_app.arn +} + +# Access keys for IAM user (when not using Fargate) +resource "aws_iam_access_key" "meteor_app" { + count = var.enable_fargate ? 0 : 1 + user = aws_iam_user.meteor_app[0].name +} + +# Store access keys in Secrets Manager (when not using Fargate) +resource "aws_secretsmanager_secret" "app_credentials" { + count = var.enable_fargate ? 0 : 1 + name = "${local.name_prefix}-app-credentials" + description = "AWS credentials for Meteor application" + + tags = local.common_tags +} + +resource "aws_secretsmanager_secret_version" "app_credentials" { + count = var.enable_fargate ? 0 : 1 + secret_id = aws_secretsmanager_secret.app_credentials[0].id + secret_string = jsonencode({ + access_key_id = aws_iam_access_key.meteor_app[0].id + secret_access_key = aws_iam_access_key.meteor_app[0].secret + region = var.aws_region + }) +} + +# IAM role for Lambda functions (future use) +resource "aws_iam_role" "lambda_execution" { + name = "${local.name_prefix}-lambda-execution" + + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "lambda.amazonaws.com" + } + } + ] + }) + + tags = local.common_tags +} + +# Attach basic Lambda execution policy +resource "aws_iam_role_policy_attachment" "lambda_basic" { + role = aws_iam_role.lambda_execution.name + policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" +} + +# Additional Lambda policy for application resources +resource "aws_iam_role_policy_attachment" "lambda_app_policy" { + role = aws_iam_role.lambda_execution.name + policy_arn = aws_iam_policy.meteor_app.arn +} \ No newline at end of file diff --git a/infrastructure/main.tf b/infrastructure/main.tf new file mode 100644 index 0000000..d667208 --- /dev/null +++ b/infrastructure/main.tf @@ -0,0 +1,36 @@ +terraform { + required_version = ">= 1.0" + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 5.0" + } + } +} + +provider "aws" { + region = var.aws_region + + default_tags { + tags = { + Project = "meteor-fullstack" + Environment = var.environment + ManagedBy = "terraform" + } + } +} + +# Data sources for existing resources +data "aws_caller_identity" "current" {} +data "aws_region" "current" {} + +# Local values for common naming +locals { + name_prefix = "${var.project_name}-${var.environment}" + + common_tags = { + Project = var.project_name + Environment = var.environment + ManagedBy = "terraform" + } +} \ No newline at end of file diff --git a/infrastructure/outputs.tf b/infrastructure/outputs.tf new file mode 100644 index 0000000..85a6b55 --- /dev/null +++ b/infrastructure/outputs.tf @@ -0,0 +1,135 @@ +output "s3_bucket_name" { + description = "Name of the S3 bucket for meteor events" + value = aws_s3_bucket.meteor_events.id +} + +output "s3_bucket_arn" { + description = "ARN of the S3 bucket for meteor events" + value = aws_s3_bucket.meteor_events.arn +} + +output "sqs_queue_url" { + description = "URL of the SQS queue for processing" + value = aws_sqs_queue.meteor_processing.url +} + +output "sqs_queue_arn" { + description = "ARN of the SQS queue for processing" + value = aws_sqs_queue.meteor_processing.arn +} + +output "sqs_dlq_url" { + description = "URL of the SQS dead letter queue" + value = aws_sqs_queue.meteor_processing_dlq.url +} + +output "cloudwatch_dashboard_url" { + description = "URL to access the CloudWatch dashboard" + value = "https://${var.aws_region}.console.aws.amazon.com/cloudwatch/home?region=${var.aws_region}#dashboards:name=${aws_cloudwatch_dashboard.meteor_dashboard.dashboard_name}" +} + +output "cloudwatch_log_groups" { + description = "CloudWatch log groups created" + value = { + web_backend = aws_cloudwatch_log_group.web_backend.name + compute_service = aws_cloudwatch_log_group.compute_service.name + } +} + +# Alerting outputs +output "sns_alerts_topic_arn" { + description = "ARN of the SNS topic for alerts" + value = aws_sns_topic.alerts.arn +} + +output "critical_alarms" { + description = "Critical CloudWatch alarms created" + value = { + nestjs_error_rate = aws_cloudwatch_metric_alarm.nestjs_5xx_error_rate.alarm_name + go_service_failure = aws_cloudwatch_metric_alarm.go_service_failure_rate.alarm_name + sqs_queue_depth = aws_cloudwatch_metric_alarm.sqs_queue_depth.alarm_name + } +} + +# RDS outputs (when enabled) +output "rds_endpoint" { + description = "RDS instance endpoint" + value = var.enable_rds ? aws_db_instance.meteor[0].endpoint : null + sensitive = true +} + +output "rds_database_name" { + description = "RDS database name" + value = var.enable_rds ? aws_db_instance.meteor[0].db_name : null +} + +output "rds_secret_arn" { + description = "ARN of the secret containing RDS credentials" + value = var.enable_rds ? aws_secretsmanager_secret.rds_password[0].arn : null +} + +# IAM outputs +output "iam_policy_arn" { + description = "ARN of the IAM policy for application services" + value = aws_iam_policy.meteor_app.arn +} + +output "ecs_task_role_arn" { + description = "ARN of the ECS task role (when using Fargate)" + value = var.enable_fargate ? aws_iam_role.ecs_task[0].arn : null +} + +output "ecs_execution_role_arn" { + description = "ARN of the ECS execution role (when using Fargate)" + value = var.enable_fargate ? aws_iam_role.ecs_task_execution[0].arn : null +} + +output "app_credentials_secret_arn" { + description = "ARN of the secret containing app credentials (when not using Fargate)" + value = var.enable_fargate ? null : aws_secretsmanager_secret.app_credentials[0].arn + sensitive = true +} + +# VPC outputs (when using Fargate) +output "vpc_id" { + description = "ID of the VPC" + value = var.enable_fargate ? aws_vpc.main[0].id : null +} + +output "private_subnet_ids" { + description = "IDs of the private subnets" + value = var.enable_fargate ? aws_subnet.private[*].id : null +} + +output "public_subnet_ids" { + description = "IDs of the public subnets" + value = var.enable_fargate ? aws_subnet.public[*].id : null +} + +output "security_group_ecs_tasks" { + description = "ID of the security group for ECS tasks" + value = var.enable_fargate ? aws_security_group.ecs_tasks[0].id : null +} + +# Environment configuration for applications +output "environment_variables" { + description = "Environment variables for application configuration" + value = { + AWS_REGION = var.aws_region + AWS_S3_BUCKET_NAME = aws_s3_bucket.meteor_events.id + AWS_SQS_QUEUE_URL = aws_sqs_queue.meteor_processing.url + ENVIRONMENT = var.environment + } +} + +# Configuration snippet for docker-compose or deployment +output "docker_environment" { + description = "Environment variables formatted for Docker deployment" + value = { + AWS_REGION = var.aws_region + AWS_S3_BUCKET_NAME = aws_s3_bucket.meteor_events.id + AWS_SQS_QUEUE_URL = aws_sqs_queue.meteor_processing.url + DATABASE_URL = var.enable_rds ? "postgresql://${aws_db_instance.meteor[0].username}:${random_password.rds_password[0].result}@${aws_db_instance.meteor[0].endpoint}:${aws_db_instance.meteor[0].port}/${aws_db_instance.meteor[0].db_name}" : null + } + sensitive = true +} \ No newline at end of file diff --git a/infrastructure/rds.tf b/infrastructure/rds.tf new file mode 100644 index 0000000..77feec0 --- /dev/null +++ b/infrastructure/rds.tf @@ -0,0 +1,142 @@ +# RDS Subnet Group +resource "aws_db_subnet_group" "meteor" { + count = var.enable_rds ? 1 : 0 + name = "${local.name_prefix}-db-subnet-group" + subnet_ids = [aws_subnet.private[0].id, aws_subnet.private[1].id] + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-db-subnet-group" + }) +} + +# RDS Security Group +resource "aws_security_group" "rds" { + count = var.enable_rds ? 1 : 0 + name = "${local.name_prefix}-rds" + description = "Security group for RDS PostgreSQL instance" + vpc_id = aws_vpc.main.id + + ingress { + from_port = 5432 + to_port = 5432 + protocol = "tcp" + security_groups = [aws_security_group.ecs_tasks.id] + description = "PostgreSQL from ECS tasks" + } + + egress { + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + description = "All outbound traffic" + } + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-rds" + }) +} + +# RDS PostgreSQL Instance +resource "aws_db_instance" "meteor" { + count = var.enable_rds ? 1 : 0 + + identifier = "${local.name_prefix}-postgres" + + # Engine settings + engine = "postgres" + engine_version = "15.4" + instance_class = var.rds_instance_class + + # Storage settings + allocated_storage = var.rds_allocated_storage + max_allocated_storage = var.rds_max_allocated_storage + storage_type = "gp3" + storage_encrypted = true + + # Database settings + db_name = "meteor_${var.environment}" + username = "meteor_user" + password = random_password.rds_password[0].result + + # Network settings + db_subnet_group_name = aws_db_subnet_group.meteor[0].name + vpc_security_group_ids = [aws_security_group.rds[0].id] + publicly_accessible = false + + # Backup settings + backup_retention_period = var.environment == "prod" ? 30 : 7 + backup_window = "03:00-04:00" + maintenance_window = "sun:04:00-sun:05:00" + auto_minor_version_upgrade = true + + # Monitoring + monitoring_interval = var.enable_detailed_monitoring ? 60 : 0 + monitoring_role_arn = var.enable_detailed_monitoring ? aws_iam_role.rds_enhanced_monitoring[0].arn : null + + # Performance Insights + performance_insights_enabled = var.environment == "prod" + + # Deletion protection + deletion_protection = var.environment == "prod" + skip_final_snapshot = var.environment != "prod" + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-postgres" + }) +} + +# Random password for RDS +resource "random_password" "rds_password" { + count = var.enable_rds ? 1 : 0 + length = 32 + special = true +} + +# Store RDS password in Secrets Manager +resource "aws_secretsmanager_secret" "rds_password" { + count = var.enable_rds ? 1 : 0 + name = "${local.name_prefix}-rds-password" + description = "RDS PostgreSQL password for meteor application" + + tags = local.common_tags +} + +resource "aws_secretsmanager_secret_version" "rds_password" { + count = var.enable_rds ? 1 : 0 + secret_id = aws_secretsmanager_secret.rds_password[0].id + secret_string = jsonencode({ + username = aws_db_instance.meteor[0].username + password = random_password.rds_password[0].result + endpoint = aws_db_instance.meteor[0].endpoint + port = aws_db_instance.meteor[0].port + dbname = aws_db_instance.meteor[0].db_name + }) +} + +# IAM role for RDS enhanced monitoring +resource "aws_iam_role" "rds_enhanced_monitoring" { + count = var.enable_rds && var.enable_detailed_monitoring ? 1 : 0 + name = "${local.name_prefix}-rds-enhanced-monitoring" + + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "monitoring.rds.amazonaws.com" + } + } + ] + }) + + tags = local.common_tags +} + +resource "aws_iam_role_policy_attachment" "rds_enhanced_monitoring" { + count = var.enable_rds && var.enable_detailed_monitoring ? 1 : 0 + role = aws_iam_role.rds_enhanced_monitoring[0].name + policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonRDSEnhancedMonitoringRole" +} \ No newline at end of file diff --git a/infrastructure/s3.tf b/infrastructure/s3.tf new file mode 100644 index 0000000..3c57e78 --- /dev/null +++ b/infrastructure/s3.tf @@ -0,0 +1,90 @@ +# S3 bucket for storing meteor event files +resource "aws_s3_bucket" "meteor_events" { + bucket = "${local.name_prefix}-events" + force_destroy = var.s3_bucket_force_destroy + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-events" + Description = "Storage for meteor event files and media" + }) +} + +# S3 bucket versioning +resource "aws_s3_bucket_versioning" "meteor_events" { + bucket = aws_s3_bucket.meteor_events.id + versioning_configuration { + status = var.s3_bucket_versioning ? "Enabled" : "Disabled" + } +} + +# S3 bucket server-side encryption +resource "aws_s3_bucket_server_side_encryption_configuration" "meteor_events" { + bucket = aws_s3_bucket.meteor_events.id + + rule { + apply_server_side_encryption_by_default { + sse_algorithm = "AES256" + } + } +} + +# S3 bucket public access block +resource "aws_s3_bucket_public_access_block" "meteor_events" { + bucket = aws_s3_bucket.meteor_events.id + + block_public_acls = true + block_public_policy = true + ignore_public_acls = true + restrict_public_buckets = true +} + +# S3 bucket lifecycle configuration +resource "aws_s3_bucket_lifecycle_configuration" "meteor_events" { + bucket = aws_s3_bucket.meteor_events.id + + rule { + id = "event_files_lifecycle" + status = "Enabled" + + # Move to Infrequent Access after 30 days + transition { + days = 30 + storage_class = "STANDARD_IA" + } + + # Move to Glacier after 90 days + transition { + days = 90 + storage_class = "GLACIER" + } + + # Delete after 2555 days (7 years) + expiration { + days = 2555 + } + } + + rule { + id = "incomplete_multipart_uploads" + status = "Enabled" + + abort_incomplete_multipart_upload { + days_after_initiation = 7 + } + } +} + +# S3 bucket notification to SQS for new uploads +resource "aws_s3_bucket_notification" "meteor_events" { + bucket = aws_s3_bucket.meteor_events.id + + queue { + queue_arn = aws_sqs_queue.meteor_processing.arn + events = ["s3:ObjectCreated:*"] + + filter_prefix = "raw-events/" + filter_suffix = ".json" + } + + depends_on = [aws_sqs_queue_policy.meteor_processing_s3] +} \ No newline at end of file diff --git a/infrastructure/sns.tf b/infrastructure/sns.tf new file mode 100644 index 0000000..8b87482 --- /dev/null +++ b/infrastructure/sns.tf @@ -0,0 +1,51 @@ +# SNS Topic for Alerts +resource "aws_sns_topic" "alerts" { + name = "${var.project_name}-${var.environment}-alerts" + + tags = { + Name = "${var.project_name}-${var.environment}-alerts" + Environment = var.environment + Project = var.project_name + Purpose = "System monitoring alerts" + } +} + +# SNS Topic Policy to allow CloudWatch to publish +resource "aws_sns_topic_policy" "alerts_policy" { + arn = aws_sns_topic.alerts.arn + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Sid = "AllowCloudWatchAlarmsToPublish" + Effect = "Allow" + Principal = { + Service = "cloudwatch.amazonaws.com" + } + Action = [ + "SNS:Publish" + ] + Resource = aws_sns_topic.alerts.arn + Condition = { + StringEquals = { + "aws:SourceAccount" = data.aws_caller_identity.current.account_id + } + } + } + ] + }) +} + +# Email Subscription (requires manual confirmation) +resource "aws_sns_topic_subscription" "email_alerts" { + count = var.alert_email != "" ? 1 : 0 + topic_arn = aws_sns_topic.alerts.arn + protocol = "email" + endpoint = var.alert_email + + depends_on = [aws_sns_topic.alerts] +} + +# Data source to get current AWS account ID +data "aws_caller_identity" "current" {} \ No newline at end of file diff --git a/infrastructure/sqs.tf b/infrastructure/sqs.tf new file mode 100644 index 0000000..92022e8 --- /dev/null +++ b/infrastructure/sqs.tf @@ -0,0 +1,93 @@ +# SQS Queue for meteor event processing +resource "aws_sqs_queue" "meteor_processing" { + name = "${local.name_prefix}-processing" + visibility_timeout_seconds = var.sqs_visibility_timeout_seconds + message_retention_seconds = var.sqs_message_retention_seconds + receive_wait_time_seconds = 20 # Enable long polling + + # Dead letter queue configuration + redrive_policy = jsonencode({ + deadLetterTargetArn = aws_sqs_queue.meteor_processing_dlq.arn + maxReceiveCount = var.sqs_max_receive_count + }) + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-processing" + Description = "Queue for processing meteor events" + }) +} + +# Dead Letter Queue for failed messages +resource "aws_sqs_queue" "meteor_processing_dlq" { + name = "${local.name_prefix}-processing-dlq" + message_retention_seconds = var.sqs_message_retention_seconds + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-processing-dlq" + Description = "Dead letter queue for failed meteor event processing" + }) +} + +# SQS Queue policy to allow S3 to send messages +resource "aws_sqs_queue_policy" "meteor_processing_s3" { + queue_url = aws_sqs_queue.meteor_processing.id + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Sid = "AllowS3ToSendMessage" + Effect = "Allow" + Principal = { + Service = "s3.amazonaws.com" + } + Action = "sqs:SendMessage" + Resource = aws_sqs_queue.meteor_processing.arn + Condition = { + ArnEquals = { + "aws:SourceArn" = aws_s3_bucket.meteor_events.arn + } + } + } + ] + }) +} + +# CloudWatch Alarms for SQS monitoring +resource "aws_cloudwatch_metric_alarm" "sqs_message_age" { + alarm_name = "${local.name_prefix}-sqs-message-age" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = "2" + metric_name = "ApproximateAgeOfOldestMessage" + namespace = "AWS/SQS" + period = "300" + statistic = "Maximum" + threshold = "900" # 15 minutes + alarm_description = "This metric monitors message age in SQS queue" + alarm_actions = [aws_sns_topic.alerts.arn] + + dimensions = { + QueueName = aws_sqs_queue.meteor_processing.name + } + + tags = local.common_tags +} + +resource "aws_cloudwatch_metric_alarm" "sqs_dlq_messages" { + alarm_name = "${local.name_prefix}-sqs-dlq-messages" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = "1" + metric_name = "ApproximateNumberOfVisibleMessages" + namespace = "AWS/SQS" + period = "300" + statistic = "Sum" + threshold = "0" + alarm_description = "This metric monitors messages in dead letter queue" + alarm_actions = [aws_sns_topic.alerts.arn] + + dimensions = { + QueueName = aws_sqs_queue.meteor_processing_dlq.name + } + + tags = local.common_tags +} \ No newline at end of file diff --git a/infrastructure/terraform.tfvars.example b/infrastructure/terraform.tfvars.example new file mode 100644 index 0000000..1acce2f --- /dev/null +++ b/infrastructure/terraform.tfvars.example @@ -0,0 +1,48 @@ +# AWS Configuration +aws_region = "us-east-1" + +# Environment Configuration +environment = "dev" +project_name = "meteor" + +# S3 Configuration +s3_bucket_versioning = true +s3_bucket_force_destroy = true # Set to false for production + +# SQS Configuration +sqs_visibility_timeout_seconds = 300 +sqs_message_retention_seconds = 1209600 # 14 days +sqs_max_receive_count = 3 + +# RDS Configuration (set enable_rds = true to create RDS instance) +enable_rds = false +rds_instance_class = "db.t3.micro" +rds_allocated_storage = 20 +rds_max_allocated_storage = 100 + +# ECS/Fargate Configuration (set enable_fargate = true to create VPC and ECS resources) +enable_fargate = false +web_backend_cpu = 256 +web_backend_memory = 512 +compute_service_cpu = 256 +compute_service_memory = 512 + +# Monitoring Configuration +cloudwatch_log_retention_days = 14 +enable_detailed_monitoring = true + +# Alerting Configuration +alert_email = "your-email@example.com" # REQUIRED: Email address to receive alerts +nestjs_error_rate_threshold = 1.0 # Percentage (1% = 1.0) +go_service_failure_rate_threshold = 5.0 # Percentage (5% = 5.0) +sqs_queue_depth_threshold = 1000 # Number of visible messages +alarm_evaluation_periods = 1 # Number of periods to evaluate +alarm_period_seconds = 300 # 5 minutes + +# Example for production: +# environment = "prod" +# s3_bucket_force_destroy = false +# enable_rds = true +# rds_instance_class = "db.t3.small" +# enable_fargate = true +# cloudwatch_log_retention_days = 30 \ No newline at end of file diff --git a/infrastructure/variables.tf b/infrastructure/variables.tf new file mode 100644 index 0000000..2a593c9 --- /dev/null +++ b/infrastructure/variables.tf @@ -0,0 +1,155 @@ +variable "aws_region" { + description = "AWS region where resources will be created" + type = string + default = "us-east-1" +} + +variable "environment" { + description = "Environment name (e.g., dev, staging, prod)" + type = string + default = "dev" +} + +variable "project_name" { + description = "Name of the project" + type = string + default = "meteor" +} + +# S3 Configuration +variable "s3_bucket_versioning" { + description = "Enable S3 bucket versioning" + type = bool + default = true +} + +variable "s3_bucket_force_destroy" { + description = "Allow S3 bucket to be destroyed even if it contains objects" + type = bool + default = false +} + +# SQS Configuration +variable "sqs_visibility_timeout_seconds" { + description = "SQS visibility timeout in seconds" + type = number + default = 300 +} + +variable "sqs_message_retention_seconds" { + description = "SQS message retention period in seconds" + type = number + default = 1209600 # 14 days +} + +variable "sqs_max_receive_count" { + description = "Maximum number of receives before message goes to DLQ" + type = number + default = 3 +} + +# RDS Configuration (if using RDS instead of external PostgreSQL) +variable "enable_rds" { + description = "Enable RDS PostgreSQL instance" + type = bool + default = false +} + +variable "rds_instance_class" { + description = "RDS instance class" + type = string + default = "db.t3.micro" +} + +variable "rds_allocated_storage" { + description = "RDS allocated storage in GB" + type = number + default = 20 +} + +variable "rds_max_allocated_storage" { + description = "RDS maximum allocated storage in GB" + type = number + default = 100 +} + +# ECS/Fargate Configuration +variable "enable_fargate" { + description = "Enable ECS Fargate deployment" + type = bool + default = false +} + +variable "web_backend_cpu" { + description = "CPU units for web backend service" + type = number + default = 256 +} + +variable "web_backend_memory" { + description = "Memory MB for web backend service" + type = number + default = 512 +} + +variable "compute_service_cpu" { + description = "CPU units for compute service" + type = number + default = 256 +} + +variable "compute_service_memory" { + description = "Memory MB for compute service" + type = number + default = 512 +} + +# Monitoring Configuration +variable "cloudwatch_log_retention_days" { + description = "CloudWatch log retention period in days" + type = number + default = 14 +} + +variable "enable_detailed_monitoring" { + description = "Enable detailed CloudWatch monitoring" + type = bool + default = true +} + +# Alerting Configuration +variable "alert_email" { + description = "Email address to receive alert notifications" + type = string + default = "" +} + +variable "nestjs_error_rate_threshold" { + description = "NestJS 5xx error rate threshold (percentage) that triggers alarm" + type = number + default = 1.0 +} + +variable "go_service_failure_rate_threshold" { + description = "Go service processing failure rate threshold (percentage) that triggers alarm" + type = number + default = 5.0 +} + +variable "sqs_queue_depth_threshold" { + description = "SQS queue depth threshold (number of visible messages) that triggers alarm" + type = number + default = 1000 +} + +variable "alarm_evaluation_periods" { + description = "Number of periods to evaluate for alarm state" + type = number + default = 1 +} + +variable "alarm_period_seconds" { + description = "Period in seconds for alarm evaluation" + type = number + default = 300 +} \ No newline at end of file diff --git a/infrastructure/vpc.tf b/infrastructure/vpc.tf new file mode 100644 index 0000000..ce04868 --- /dev/null +++ b/infrastructure/vpc.tf @@ -0,0 +1,174 @@ +# VPC for meteor application (only if using Fargate) +resource "aws_vpc" "main" { + count = var.enable_fargate ? 1 : 0 + cidr_block = "10.0.0.0/16" + enable_dns_hostnames = true + enable_dns_support = true + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-vpc" + }) +} + +# Internet Gateway +resource "aws_internet_gateway" "main" { + count = var.enable_fargate ? 1 : 0 + vpc_id = aws_vpc.main[0].id + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-igw" + }) +} + +# Data source for availability zones +data "aws_availability_zones" "available" { + state = "available" +} + +# Public Subnets +resource "aws_subnet" "public" { + count = var.enable_fargate ? 2 : 0 + + vpc_id = aws_vpc.main[0].id + cidr_block = "10.0.${count.index + 1}.0/24" + availability_zone = data.aws_availability_zones.available.names[count.index] + map_public_ip_on_launch = true + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-public-subnet-${count.index + 1}" + Type = "Public" + }) +} + +# Private Subnets +resource "aws_subnet" "private" { + count = var.enable_fargate ? 2 : 0 + + vpc_id = aws_vpc.main[0].id + cidr_block = "10.0.${count.index + 10}.0/24" + availability_zone = data.aws_availability_zones.available.names[count.index] + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-private-subnet-${count.index + 1}" + Type = "Private" + }) +} + +# Elastic IPs for NAT Gateways +resource "aws_eip" "nat" { + count = var.enable_fargate ? 2 : 0 + domain = "vpc" + + depends_on = [aws_internet_gateway.main] + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-nat-eip-${count.index + 1}" + }) +} + +# NAT Gateways +resource "aws_nat_gateway" "main" { + count = var.enable_fargate ? 2 : 0 + + allocation_id = aws_eip.nat[count.index].id + subnet_id = aws_subnet.public[count.index].id + + depends_on = [aws_internet_gateway.main] + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-nat-${count.index + 1}" + }) +} + +# Route Table for Public Subnets +resource "aws_route_table" "public" { + count = var.enable_fargate ? 1 : 0 + vpc_id = aws_vpc.main[0].id + + route { + cidr_block = "0.0.0.0/0" + gateway_id = aws_internet_gateway.main[0].id + } + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-public-rt" + }) +} + +# Route Table Associations for Public Subnets +resource "aws_route_table_association" "public" { + count = var.enable_fargate ? 2 : 0 + + subnet_id = aws_subnet.public[count.index].id + route_table_id = aws_route_table.public[0].id +} + +# Route Tables for Private Subnets +resource "aws_route_table" "private" { + count = var.enable_fargate ? 2 : 0 + + vpc_id = aws_vpc.main[0].id + + route { + cidr_block = "0.0.0.0/0" + nat_gateway_id = aws_nat_gateway.main[count.index].id + } + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-private-rt-${count.index + 1}" + }) +} + +# Route Table Associations for Private Subnets +resource "aws_route_table_association" "private" { + count = var.enable_fargate ? 2 : 0 + + subnet_id = aws_subnet.private[count.index].id + route_table_id = aws_route_table.private[count.index].id +} + +# Security Group for ECS Tasks +resource "aws_security_group" "ecs_tasks" { + count = var.enable_fargate ? 1 : 0 + name = "${local.name_prefix}-ecs-tasks" + description = "Security group for ECS tasks" + vpc_id = aws_vpc.main[0].id + + ingress { + from_port = 3000 + to_port = 3000 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + description = "HTTP from Load Balancer" + } + + egress { + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + description = "All outbound traffic" + } + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-ecs-tasks" + }) +} + +# VPC Endpoints for AWS services (to reduce NAT Gateway costs) +resource "aws_vpc_endpoint" "s3" { + count = var.enable_fargate ? 1 : 0 + vpc_id = aws_vpc.main[0].id + service_name = "com.amazonaws.${data.aws_region.current.name}.s3" + + tags = merge(local.common_tags, { + Name = "${local.name_prefix}-s3-endpoint" + }) +} + +resource "aws_vpc_endpoint_route_table_association" "s3_private" { + count = var.enable_fargate ? 2 : 0 + + vpc_endpoint_id = aws_vpc_endpoint.s3[0].id + route_table_id = aws_route_table.private[count.index].id +} \ No newline at end of file diff --git a/meteor-compute-service/bin/meteor-compute-service b/meteor-compute-service/bin/meteor-compute-service new file mode 100755 index 0000000..6afb3f6 Binary files /dev/null and b/meteor-compute-service/bin/meteor-compute-service differ diff --git a/meteor-compute-service/cmd/meteor-compute-service/main.go b/meteor-compute-service/cmd/meteor-compute-service/main.go index 0bec485..a3bac4e 100644 --- a/meteor-compute-service/cmd/meteor-compute-service/main.go +++ b/meteor-compute-service/cmd/meteor-compute-service/main.go @@ -2,7 +2,6 @@ package main import ( "context" - "log" "os" "os/signal" "sync" @@ -11,19 +10,33 @@ import ( "meteor-compute-service/internal/config" "meteor-compute-service/internal/health" + "meteor-compute-service/internal/logger" + "meteor-compute-service/internal/metrics" "meteor-compute-service/internal/processor" "meteor-compute-service/internal/repository" "meteor-compute-service/internal/sqs" "meteor-compute-service/internal/validation" + + awsconfig "github.com/aws/aws-sdk-go-v2/config" ) func main() { - log.Println("🚀 Starting meteor-compute-service...") + // Initialize structured logger + structuredLogger := logger.NewStructuredLogger("meteor-compute-service", "2.0.0") + ctx := context.Background() + + structuredLogger.StartupEvent(ctx, "application", + logger.NewField("event", "starting"), + ) // Load configuration cfg := config.Load() - log.Printf("📋 Configuration loaded: Database=%s, SQS=%s, Workers=%d", - maskDatabaseURL(cfg.DatabaseURL), cfg.SQSQueueURL, cfg.ProcessingWorkers) + structuredLogger.StartupEvent(ctx, "configuration", + logger.NewField("database_url_masked", maskDatabaseURL(cfg.DatabaseURL)), + logger.NewField("sqs_queue", cfg.SQSQueueURL), + logger.NewField("processing_workers", cfg.ProcessingWorkers), + logger.NewField("validation_provider", cfg.ValidationProvider), + ) // Create context that can be cancelled ctx, cancel := context.WithCancel(context.Background()) @@ -34,21 +47,29 @@ func main() { signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM) // Initialize database repository - log.Println("🗄️ Initializing database connection...") + structuredLogger.StartupEvent(ctx, "database", logger.NewField("event", "initializing")) repo, err := repository.NewPostgreSQLRepository(cfg.DatabaseURL, cfg.DatabaseMaxConns) if err != nil { - log.Fatalf("❌ Failed to initialize database: %v", err) + structuredLogger.Error(ctx, "Failed to initialize database", err, + logger.NewField("database_url_masked", maskDatabaseURL(cfg.DatabaseURL)), + ) + os.Exit(1) } defer repo.Close() // Test database connection if err := repo.Ping(ctx); err != nil { - log.Fatalf("❌ Database ping failed: %v", err) + structuredLogger.Error(ctx, "Database ping failed", err) + os.Exit(1) } - log.Println("✅ Database connection established") + structuredLogger.StartupEvent(ctx, "database", logger.NewField("event", "connected")) // Initialize SQS client - log.Printf("📨 Initializing SQS client (Region: %s)...", cfg.SQSRegion) + structuredLogger.StartupEvent(ctx, "sqs", + logger.NewField("event", "initializing"), + logger.NewField("region", cfg.SQSRegion), + logger.NewField("queue_url", cfg.SQSQueueURL), + ) sqsClient, err := sqs.NewClient( cfg.SQSRegion, cfg.SQSQueueURL, @@ -57,32 +78,78 @@ func main() { cfg.SQSVisibilityTimeout, ) if err != nil { - log.Fatalf("❌ Failed to initialize SQS client: %v", err) + structuredLogger.Error(ctx, "Failed to initialize SQS client", err, + logger.NewField("region", cfg.SQSRegion), + ) + os.Exit(1) } // Test SQS connection if _, err := sqsClient.GetQueueAttributes(ctx); err != nil { - log.Fatalf("❌ SQS connection test failed: %v", err) + structuredLogger.Error(ctx, "SQS connection test failed", err) + os.Exit(1) } - log.Println("✅ SQS connection established") + structuredLogger.StartupEvent(ctx, "sqs", logger.NewField("event", "connected")) - // Initialize validator - log.Println("🔍 Initializing MVP validator...") - validator := validation.NewMVPValidator() + // Initialize AWS config for metrics client + structuredLogger.StartupEvent(ctx, "metrics", logger.NewField("event", "initializing")) + awsCfg, err := awsconfig.LoadDefaultConfig(ctx) + if err != nil { + structuredLogger.Error(ctx, "Failed to load AWS config", err) + os.Exit(1) + } + + // Create metrics client + metricsClient := metrics.NewMetricsClient(awsCfg, structuredLogger.GetZerologLogger()) + structuredLogger.StartupEvent(ctx, "metrics", logger.NewField("event", "initialized")) + + // Initialize validation provider based on configuration + structuredLogger.StartupEvent(ctx, "validation", + logger.NewField("event", "initializing"), + logger.NewField("provider_type", cfg.ValidationProvider), + ) + factory := validation.NewProviderFactory() + + providerType := validation.ProviderType(cfg.ValidationProvider) + validator, err := factory.CreateProvider(providerType) + if err != nil { + structuredLogger.Error(ctx, "Failed to create validation provider", err, + logger.NewField("provider_type", cfg.ValidationProvider), + ) + os.Exit(1) + } + + providerInfo := validator.GetProviderInfo() + structuredLogger.StartupEvent(ctx, "validation", + logger.NewField("event", "loaded"), + logger.NewField("provider_name", providerInfo.Name), + logger.NewField("provider_version", providerInfo.Version), + logger.NewField("algorithm", providerInfo.Algorithm), + ) // Initialize processor - log.Println("⚙️ Initializing event processor...") + structuredLogger.StartupEvent(ctx, "processor", + logger.NewField("event", "initializing"), + logger.NewField("workers", cfg.ProcessingWorkers), + logger.NewField("batch_size", cfg.ProcessingBatchSize), + logger.NewField("idempotency_enabled", cfg.IdempotencyEnabled), + ) proc := processor.NewProcessor( sqsClient, repo, validator, + structuredLogger, + metricsClient, cfg.ProcessingWorkers, cfg.ProcessingBatchSize, cfg.IdempotencyEnabled, ) // Start health server in a separate goroutine - log.Printf("🏥 Starting health server on port %s...", cfg.Port) + structuredLogger.StartupEvent(ctx, "health_server", + logger.NewField("event", "starting"), + logger.NewField("port", cfg.Port), + ) var wg sync.WaitGroup wg.Add(1) go func() { @@ -91,12 +158,12 @@ func main() { }() // Start the processor - log.Println("🔄 Starting event processing...") + structuredLogger.StartupEvent(ctx, "processor", logger.NewField("event", "starting")) wg.Add(1) go func() { defer wg.Done() if err := proc.Start(ctx); err != nil { - log.Printf("❌ Processor error: %v", err) + structuredLogger.Error(ctx, "Processor error", err) } }() @@ -104,12 +171,12 @@ func main() { wg.Add(1) go func() { defer wg.Done() - reportStats(ctx, proc) + reportStats(ctx, proc, structuredLogger) }() // Wait for shutdown signal <-sigChan - log.Println("🛑 Shutdown signal received, gracefully stopping...") + structuredLogger.Info(ctx, "Shutdown signal received, gracefully stopping") // Cancel context to stop all goroutines cancel() @@ -127,16 +194,16 @@ func main() { select { case <-done: - log.Println("✅ Processor stopped gracefully") + structuredLogger.Info(ctx, "Processor stopped gracefully") case <-shutdownCtx.Done(): - log.Println("⚠️ Processor shutdown timeout, forcing exit") + structuredLogger.Warn(ctx, "Processor shutdown timeout, forcing exit") } - log.Println("👋 meteor-compute-service stopped") + structuredLogger.Info(ctx, "Service stopped successfully") } // reportStats periodically logs processing statistics -func reportStats(ctx context.Context, proc *processor.Processor) { +func reportStats(ctx context.Context, proc *processor.Processor, structuredLogger *logger.StructuredLogger) { ticker := time.NewTicker(60 * time.Second) // Report every minute defer ticker.Stop() @@ -148,8 +215,14 @@ func reportStats(ctx context.Context, proc *processor.Processor) { stats := proc.GetStats() if stats.TotalProcessed > 0 { successRate := float64(stats.SuccessfullyProcessed) / float64(stats.TotalProcessed) * 100 - log.Printf("📊 Processing Stats: Total=%d, Success=%d (%.1f%%), Failed=%d, Skipped=%d", - stats.TotalProcessed, stats.SuccessfullyProcessed, successRate, stats.Failed, stats.Skipped) + structuredLogger.MetricsEvent(ctx, "processing_statistics", stats, + logger.NewField("total_processed", stats.TotalProcessed), + logger.NewField("successful", stats.SuccessfullyProcessed), + logger.NewField("failed", stats.Failed), + logger.NewField("skipped", stats.Skipped), + logger.NewField("success_rate_percent", successRate), + logger.NewField("last_processed_at", stats.LastProcessedAt), + ) } } } diff --git a/meteor-compute-service/go.mod b/meteor-compute-service/go.mod index 5b1ef22..eaf35be 100644 --- a/meteor-compute-service/go.mod +++ b/meteor-compute-service/go.mod @@ -3,8 +3,9 @@ module meteor-compute-service go 1.24.5 require ( - github.com/aws/aws-sdk-go-v2 v1.32.2 + github.com/aws/aws-sdk-go-v2 v1.37.1 github.com/aws/aws-sdk-go-v2/config v1.28.0 + github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.46.1 github.com/aws/aws-sdk-go-v2/service/sqs v1.34.7 github.com/google/uuid v1.6.0 github.com/jackc/pgx/v5 v5.7.1 @@ -13,19 +14,23 @@ require ( require ( github.com/aws/aws-sdk-go-v2/credentials v1.17.41 // indirect github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.17 // indirect - github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.21 // indirect - github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.21 // indirect + github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.1 // indirect + github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.1 // indirect github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.0 // indirect github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.2 // indirect github.com/aws/aws-sdk-go-v2/service/sso v1.24.2 // indirect github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.2 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.32.2 // indirect - github.com/aws/smithy-go v1.22.0 // indirect + github.com/aws/smithy-go v1.22.5 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect + github.com/mattn/go-colorable v0.1.13 // indirect + github.com/mattn/go-isatty v0.0.19 // indirect + github.com/rs/zerolog v1.34.0 // indirect golang.org/x/crypto v0.27.0 // indirect golang.org/x/sync v0.8.0 // indirect + golang.org/x/sys v0.25.0 // indirect golang.org/x/text v0.18.0 // indirect ) diff --git a/meteor-compute-service/go.sum b/meteor-compute-service/go.sum index fe7a618..b16ea07 100644 --- a/meteor-compute-service/go.sum +++ b/meteor-compute-service/go.sum @@ -1,5 +1,7 @@ github.com/aws/aws-sdk-go-v2 v1.32.2 h1:AkNLZEyYMLnx/Q/mSKkcMqwNFXMAvFto9bNsHqcTduI= github.com/aws/aws-sdk-go-v2 v1.32.2/go.mod h1:2SK5n0a2karNTv5tbP1SjsX0uhttou00v/HpXKM1ZUo= +github.com/aws/aws-sdk-go-v2 v1.37.1 h1:SMUxeNz3Z6nqGsXv0JuJXc8w5YMtrQMuIBmDx//bBDY= +github.com/aws/aws-sdk-go-v2 v1.37.1/go.mod h1:9Q0OoGQoboYIAJyslFyF1f5K1Ryddop8gqMhWx/n4Wg= github.com/aws/aws-sdk-go-v2/config v1.28.0 h1:FosVYWcqEtWNxHn8gB/Vs6jOlNwSoyOCA/g/sxyySOQ= github.com/aws/aws-sdk-go-v2/config v1.28.0/go.mod h1:pYhbtvg1siOOg8h5an77rXle9tVG8T+BWLWAo7cOukc= github.com/aws/aws-sdk-go-v2/credentials v1.17.41 h1:7gXo+Axmp+R4Z+AK8YFQO0ZV3L0gizGINCOWxSLY9W8= @@ -8,10 +10,16 @@ github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.17 h1:TMH3f/SCAWdNtXXVPPu5D6 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.17/go.mod h1:1ZRXLdTpzdJb9fwTMXiLipENRxkGMTn1sfKexGllQCw= github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.21 h1:UAsR3xA31QGf79WzpG/ixT9FZvQlh5HY1NRqSHBNOCk= github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.21/go.mod h1:JNr43NFf5L9YaG3eKTm7HQzls9J+A9YYcGI5Quh1r2Y= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.1 h1:ksZXBYv80EFTcgc8OJO48aQ8XDWXIQL7gGasPeCoTzI= +github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.1/go.mod h1:HSksQyyJETVZS7uM54cir0IgxttTD+8aEoJMPGepHBI= github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.21 h1:6jZVETqmYCadGFvrYEQfC5fAQmlo80CeL5psbno6r0s= github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.21/go.mod h1:1SR0GbLlnN3QUmYaflZNiH1ql+1qrSiB2vwcJ+4UM60= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.1 h1:+dn/xF/05utS7tUhjIcndbuaPjfll2LhbH1cCDGLYUQ= +github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.1/go.mod h1:hyAGz30LHdm5KBZDI58MXx5lDVZ5CUfvfTZvMu4HCZo= github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 h1:VaRN3TlFdd6KxX1x3ILT5ynH6HvKgqdiXoTxAF4HQcQ= github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1/go.mod h1:FbtygfRFze9usAadmnGJNc8KsP346kEe+y2/oyhGAGc= +github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.46.1 h1:jdaLx0Fle7TsNNpd4fe1C5JOtIQCUtYveT5qOsmTHdg= +github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.46.1/go.mod h1:ZCCs9PKEJ2qp3sA1IH7VWYmEJnenvHoR1gEqDH6qNoI= github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.0 h1:TToQNkvGguu209puTojY/ozlqy2d/SFNcoLIqTFi42g= github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.0/go.mod h1:0jp+ltwkf+SwG2fm/PKo8t4y8pJSgOCO4D8Lz3k0aHQ= github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.2 h1:s7NA1SOw8q/5c0wr8477yOPp0z+uBaXBnLE0XYb0POA= @@ -26,9 +34,13 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.32.2 h1:CiS7i0+FUe+/YY1GvIBLLrR/XNGZ github.com/aws/aws-sdk-go-v2/service/sts v1.32.2/go.mod h1:HtaiBI8CjYoNVde8arShXb94UbQQi9L4EMr6D+xGBwo= github.com/aws/smithy-go v1.22.0 h1:uunKnWlcoL3zO7q+gG2Pk53joueEOsnNB28QdMsmiMM= github.com/aws/smithy-go v1.22.0/go.mod h1:irrKGvNn1InZwb2d7fkIRNucdfwR8R+Ts3wxYa/cJHg= +github.com/aws/smithy-go v1.22.5 h1:P9ATCXPMb2mPjYBgueqJNCA5S9UfktsW0tTxi+a7eqw= +github.com/aws/smithy-go v1.22.5/go.mod h1:t1ufH5HMublsJYulve2RKmHDC15xu1f26kHCp/HgceI= +github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= @@ -39,8 +51,17 @@ github.com/jackc/pgx/v5 v5.7.1 h1:x7SYsPBYDkHDksogeSmZZ5xzThcTgRz++I5E+ePFUcs= github.com/jackc/pgx/v5 v5.7.1/go.mod h1:e7O26IywZZ+naJtWWos6i6fvWK+29etgITqrqHLfoZA= github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= +github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= +github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= +github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= +github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA= +github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0= +github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY= +github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= @@ -50,6 +71,11 @@ golang.org/x/crypto v0.27.0 h1:GXm2NjJrPaiv/h1tb2UH8QfgC/hOf/+z0p6PT8o1w7A= golang.org/x/crypto v0.27.0/go.mod h1:1Xngt8kV6Dvbssa53Ziq6Eqn0HqbZi5Z6R0ZpwQzt70= golang.org/x/sync v0.8.0 h1:3NFvSEYkUoMifnESzZl15y791HH1qU2xm6eCJU5ZPXQ= golang.org/x/sync v0.8.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.25.0 h1:r+8e+loiHxRqhXVl6ML1nO3l1+oFoWbnlu2Ehimmi34= +golang.org/x/sys v0.25.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= golang.org/x/text v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224= golang.org/x/text v0.18.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/meteor-compute-service/internal/config/config.go b/meteor-compute-service/internal/config/config.go index 67f86b8..5461ab5 100644 --- a/meteor-compute-service/internal/config/config.go +++ b/meteor-compute-service/internal/config/config.go @@ -26,6 +26,9 @@ type Config struct { ProcessingWorkers int ProcessingBatchSize int IdempotencyEnabled bool + + // Validation configuration + ValidationProvider string } // Load loads configuration from environment variables with defaults @@ -61,6 +64,11 @@ func Load() *Config { processingBatchSize := parseInt(os.Getenv("PROCESSING_BATCH_SIZE"), 10) idempotencyEnabled := parseBool(os.Getenv("IDEMPOTENCY_ENABLED"), true) + validationProvider := os.Getenv("VALIDATION_PROVIDER") + if validationProvider == "" { + validationProvider = "mvp" // Default to MVP provider for backward compatibility + } + return &Config{ Port: port, DatabaseURL: databaseURL, @@ -74,6 +82,7 @@ func Load() *Config { ProcessingWorkers: processingWorkers, ProcessingBatchSize: processingBatchSize, IdempotencyEnabled: idempotencyEnabled, + ValidationProvider: validationProvider, } } diff --git a/meteor-compute-service/internal/logger/logger.go b/meteor-compute-service/internal/logger/logger.go new file mode 100644 index 0000000..7353bc3 --- /dev/null +++ b/meteor-compute-service/internal/logger/logger.go @@ -0,0 +1,255 @@ +package logger + +import ( + "context" + "os" + "time" + + "github.com/rs/zerolog" + "github.com/rs/zerolog/log" +) + +// ContextKey is used for storing values in context +type ContextKey string + +const ( + // CorrelationIDKey is the key for correlation ID in context + CorrelationIDKey ContextKey = "correlation_id" +) + +// StructuredLogger provides standardized logging for the meteor compute service +type StructuredLogger struct { + logger zerolog.Logger + service string + version string +} + +// LogEntry represents a standardized log entry +type LogEntry struct { + Timestamp string `json:"timestamp"` + Level string `json:"level"` + ServiceName string `json:"service_name"` + CorrelationID *string `json:"correlation_id"` + Message string `json:"message"` + Extra interface{} `json:",inline"` +} + +// Field represents a key-value pair for structured logging +type Field struct { + Key string + Value interface{} +} + +// NewStructuredLogger creates a new structured logger instance +func NewStructuredLogger(service, version string) *StructuredLogger { + // Configure zerolog based on environment + if os.Getenv("NODE_ENV") == "development" { + // Pretty printing for development + log.Logger = log.Output(zerolog.ConsoleWriter{ + Out: os.Stdout, + TimeFormat: time.RFC3339, + NoColor: false, + }) + } else { + // JSON output for production + zerolog.TimeFieldFormat = time.RFC3339 + } + + // Set log level + logLevel := os.Getenv("LOG_LEVEL") + switch logLevel { + case "debug": + zerolog.SetGlobalLevel(zerolog.DebugLevel) + case "info": + zerolog.SetGlobalLevel(zerolog.InfoLevel) + case "warn": + zerolog.SetGlobalLevel(zerolog.WarnLevel) + case "error": + zerolog.SetGlobalLevel(zerolog.ErrorLevel) + default: + zerolog.SetGlobalLevel(zerolog.InfoLevel) + } + + logger := log.With(). + Str("service_name", service). + Str("version", version). + Logger() + + return &StructuredLogger{ + logger: logger, + service: service, + version: version, + } +} + +// WithCorrelationID adds correlation ID to context +func WithCorrelationID(ctx context.Context, correlationID string) context.Context { + return context.WithValue(ctx, CorrelationIDKey, correlationID) +} + +// GetCorrelationID retrieves correlation ID from context +func GetCorrelationID(ctx context.Context) *string { + if correlationID, ok := ctx.Value(CorrelationIDKey).(string); ok && correlationID != "" { + return &correlationID + } + return nil +} + +// createLogEvent creates a zerolog event with common fields +func (l *StructuredLogger) createLogEvent(level zerolog.Level, ctx context.Context) *zerolog.Event { + event := l.logger.WithLevel(level). + Timestamp(). + Str("service_name", l.service) + + if correlationID := GetCorrelationID(ctx); correlationID != nil { + event = event.Str("correlation_id", *correlationID) + } + + return event +} + +// Info logs an info level message +func (l *StructuredLogger) Info(ctx context.Context, message string, fields ...Field) { + event := l.createLogEvent(zerolog.InfoLevel, ctx) + for _, field := range fields { + event = event.Interface(field.Key, field.Value) + } + event.Msg(message) +} + +// Warn logs a warning level message +func (l *StructuredLogger) Warn(ctx context.Context, message string, fields ...Field) { + event := l.createLogEvent(zerolog.WarnLevel, ctx) + for _, field := range fields { + event = event.Interface(field.Key, field.Value) + } + event.Msg(message) +} + +// Error logs an error level message +func (l *StructuredLogger) Error(ctx context.Context, message string, err error, fields ...Field) { + event := l.createLogEvent(zerolog.ErrorLevel, ctx) + if err != nil { + event = event.Err(err) + } + for _, field := range fields { + event = event.Interface(field.Key, field.Value) + } + event.Msg(message) +} + +// Debug logs a debug level message +func (l *StructuredLogger) Debug(ctx context.Context, message string, fields ...Field) { + event := l.createLogEvent(zerolog.DebugLevel, ctx) + for _, field := range fields { + event = event.Interface(field.Key, field.Value) + } + event.Msg(message) +} + +// Business-specific logging methods + +// ProcessingEvent logs event processing information +func (l *StructuredLogger) ProcessingEvent(ctx context.Context, eventID, stage string, fields ...Field) { + allFields := append(fields, + Field{Key: "event_id", Value: eventID}, + Field{Key: "processing_stage", Value: stage}, + ) + l.Info(ctx, "Event processing stage", allFields...) +} + +// ValidationEvent logs validation-related events +func (l *StructuredLogger) ValidationEvent(ctx context.Context, eventID, algorithm string, isValid bool, score float64, fields ...Field) { + allFields := append(fields, + Field{Key: "event_id", Value: eventID}, + Field{Key: "validation_algorithm", Value: algorithm}, + Field{Key: "is_valid", Value: isValid}, + Field{Key: "validation_score", Value: score}, + ) + l.Info(ctx, "Event validation completed", allFields...) +} + +// DatabaseEvent logs database operations +func (l *StructuredLogger) DatabaseEvent(ctx context.Context, operation string, duration time.Duration, fields ...Field) { + allFields := append(fields, + Field{Key: "database_operation", Value: operation}, + Field{Key: "duration_ms", Value: duration.Milliseconds()}, + ) + l.Debug(ctx, "Database operation completed", allFields...) +} + +// SQSEvent logs SQS-related events +func (l *StructuredLogger) SQSEvent(ctx context.Context, operation, messageID string, fields ...Field) { + allFields := append(fields, + Field{Key: "sqs_operation", Value: operation}, + Field{Key: "sqs_message_id", Value: messageID}, + ) + l.Info(ctx, "SQS operation", allFields...) +} + +// StartupEvent logs application startup events +func (l *StructuredLogger) StartupEvent(ctx context.Context, component string, fields ...Field) { + allFields := append(fields, + Field{Key: "startup_component", Value: component}, + ) + l.Info(ctx, "Component initialized", allFields...) +} + +// HealthEvent logs health check events +func (l *StructuredLogger) HealthEvent(ctx context.Context, component string, healthy bool, fields ...Field) { + allFields := append(fields, + Field{Key: "health_component", Value: component}, + Field{Key: "healthy", Value: healthy}, + ) + + if healthy { + l.Debug(ctx, "Health check passed", allFields...) + } else { + l.Warn(ctx, "Health check failed", allFields...) + } +} + +// SecurityEvent logs security-related events +func (l *StructuredLogger) SecurityEvent(ctx context.Context, event string, fields ...Field) { + allFields := append(fields, + Field{Key: "security_event", Value: event}, + ) + l.Warn(ctx, "Security event detected", allFields...) +} + +// PerformanceEvent logs performance metrics +func (l *StructuredLogger) PerformanceEvent(ctx context.Context, operation string, duration time.Duration, fields ...Field) { + allFields := append(fields, + Field{Key: "performance_operation", Value: operation}, + Field{Key: "duration_ms", Value: duration.Milliseconds()}, + ) + l.Info(ctx, "Performance metric", allFields...) +} + +// MetricsEvent logs metrics and statistics +func (l *StructuredLogger) MetricsEvent(ctx context.Context, metric string, value interface{}, fields ...Field) { + allFields := append(fields, + Field{Key: "metric_name", Value: metric}, + Field{Key: "metric_value", Value: value}, + ) + l.Info(ctx, "Metrics data", allFields...) +} + +// WorkerEvent logs worker-specific events +func (l *StructuredLogger) WorkerEvent(ctx context.Context, workerID int, event string, fields ...Field) { + allFields := append(fields, + Field{Key: "worker_id", Value: workerID}, + Field{Key: "worker_event", Value: event}, + ) + l.Info(ctx, "Worker event", allFields...) +} + +// NewField creates a field for structured logging +func NewField(key string, value interface{}) Field { + return Field{Key: key, Value: value} +} + +// GetZerologLogger returns the underlying zerolog.Logger for external integrations +func (l *StructuredLogger) GetZerologLogger() zerolog.Logger { + return l.logger +} \ No newline at end of file diff --git a/meteor-compute-service/internal/metrics/metrics.go b/meteor-compute-service/internal/metrics/metrics.go new file mode 100644 index 0000000..2f51297 --- /dev/null +++ b/meteor-compute-service/internal/metrics/metrics.go @@ -0,0 +1,373 @@ +package metrics + +import ( + "context" + "fmt" + "time" + + "github.com/aws/aws-sdk-go-v2/aws" + "github.com/aws/aws-sdk-go-v2/service/cloudwatch" + "github.com/aws/aws-sdk-go-v2/service/cloudwatch/types" + "github.com/rs/zerolog" +) + +// MetricsClient wraps CloudWatch metrics functionality +type MetricsClient struct { + cw *cloudwatch.Client + logger zerolog.Logger +} + +// NewMetricsClient creates a new metrics client +func NewMetricsClient(awsConfig aws.Config, logger zerolog.Logger) *MetricsClient { + return &MetricsClient{ + cw: cloudwatch.NewFromConfig(awsConfig), + logger: logger, + } +} + +// MessageProcessingMetrics holds metrics for message processing +type MessageProcessingMetrics struct { + ProcessingTime time.Duration + Success bool + MessageType string + ProviderName string + ErrorType string +} + +// SendMessageProcessingMetrics sends message processing metrics to CloudWatch +func (m *MetricsClient) SendMessageProcessingMetrics(ctx context.Context, metrics MessageProcessingMetrics) error { + namespace := "MeteorApp/ComputeService" + timestamp := time.Now() + + dimensions := []types.Dimension{ + { + Name: aws.String("MessageType"), + Value: aws.String(metrics.MessageType), + }, + { + Name: aws.String("ProviderName"), + Value: aws.String(metrics.ProviderName), + }, + { + Name: aws.String("Success"), + Value: aws.String(fmt.Sprintf("%v", metrics.Success)), + }, + } + + // Add error type dimension if processing failed + if !metrics.Success && metrics.ErrorType != "" { + dimensions = append(dimensions, types.Dimension{ + Name: aws.String("ErrorType"), + Value: aws.String(metrics.ErrorType), + }) + } + + metricData := []types.MetricDatum{ + // Message processing count + { + MetricName: aws.String("MessageProcessingCount"), + Value: aws.Float64(1), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }, + // Processing duration + { + MetricName: aws.String("MessageProcessingDuration"), + Value: aws.Float64(float64(metrics.ProcessingTime.Milliseconds())), + Unit: types.StandardUnitMilliseconds, + Timestamp: ×tamp, + Dimensions: dimensions, + }, + } + + // Add success/error specific metrics + if metrics.Success { + metricData = append(metricData, types.MetricDatum{ + MetricName: aws.String("MessageProcessingSuccess"), + Value: aws.Float64(1), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }) + } else { + metricData = append(metricData, types.MetricDatum{ + MetricName: aws.String("MessageProcessingError"), + Value: aws.Float64(1), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }) + } + + input := &cloudwatch.PutMetricDataInput{ + Namespace: aws.String(namespace), + MetricData: metricData, + } + + _, err := m.cw.PutMetricData(ctx, input) + if err != nil { + m.logger.Error(). + Err(err). + Str("namespace", namespace). + Str("message_type", metrics.MessageType). + Str("provider_name", metrics.ProviderName). + Msg("Failed to send message processing metrics to CloudWatch") + return fmt.Errorf("failed to send message processing metrics: %w", err) + } + + m.logger.Debug(). + Str("namespace", namespace). + Str("message_type", metrics.MessageType). + Str("provider_name", metrics.ProviderName). + Bool("success", metrics.Success). + Dur("processing_time", metrics.ProcessingTime). + Msg("Successfully sent message processing metrics to CloudWatch") + + return nil +} + +// ValidationMetrics holds metrics for validation operations +type ValidationMetrics struct { + ValidationTime time.Duration + Success bool + ProviderName string + EventCount int + ErrorType string +} + +// SendValidationMetrics sends validation metrics to CloudWatch +func (m *MetricsClient) SendValidationMetrics(ctx context.Context, metrics ValidationMetrics) error { + namespace := "MeteorApp/ComputeService" + timestamp := time.Now() + + dimensions := []types.Dimension{ + { + Name: aws.String("ProviderName"), + Value: aws.String(metrics.ProviderName), + }, + { + Name: aws.String("Success"), + Value: aws.String(fmt.Sprintf("%v", metrics.Success)), + }, + } + + if !metrics.Success && metrics.ErrorType != "" { + dimensions = append(dimensions, types.Dimension{ + Name: aws.String("ErrorType"), + Value: aws.String(metrics.ErrorType), + }) + } + + metricData := []types.MetricDatum{ + // Validation count + { + MetricName: aws.String("ValidationCount"), + Value: aws.Float64(1), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }, + // Validation duration + { + MetricName: aws.String("ValidationDuration"), + Value: aws.Float64(float64(metrics.ValidationTime.Milliseconds())), + Unit: types.StandardUnitMilliseconds, + Timestamp: ×tamp, + Dimensions: dimensions, + }, + // Event count processed + { + MetricName: aws.String("EventsProcessed"), + Value: aws.Float64(float64(metrics.EventCount)), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }, + } + + // Add success/error specific metrics + if metrics.Success { + metricData = append(metricData, types.MetricDatum{ + MetricName: aws.String("ValidationSuccess"), + Value: aws.Float64(1), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }) + } else { + metricData = append(metricData, types.MetricDatum{ + MetricName: aws.String("ValidationError"), + Value: aws.Float64(1), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }) + } + + input := &cloudwatch.PutMetricDataInput{ + Namespace: aws.String(namespace), + MetricData: metricData, + } + + _, err := m.cw.PutMetricData(ctx, input) + if err != nil { + m.logger.Error(). + Err(err). + Str("namespace", namespace). + Str("provider_name", metrics.ProviderName). + Msg("Failed to send validation metrics to CloudWatch") + return fmt.Errorf("failed to send validation metrics: %w", err) + } + + m.logger.Debug(). + Str("namespace", namespace). + Str("provider_name", metrics.ProviderName). + Bool("success", metrics.Success). + Dur("validation_time", metrics.ValidationTime). + Int("event_count", metrics.EventCount). + Msg("Successfully sent validation metrics to CloudWatch") + + return nil +} + +// DatabaseMetrics holds metrics for database operations +type DatabaseMetrics struct { + Operation string + Duration time.Duration + Success bool + RecordCount int + ErrorType string +} + +// SendDatabaseMetrics sends database metrics to CloudWatch +func (m *MetricsClient) SendDatabaseMetrics(ctx context.Context, metrics DatabaseMetrics) error { + namespace := "MeteorApp/ComputeService" + timestamp := time.Now() + + dimensions := []types.Dimension{ + { + Name: aws.String("Operation"), + Value: aws.String(metrics.Operation), + }, + { + Name: aws.String("Success"), + Value: aws.String(fmt.Sprintf("%v", metrics.Success)), + }, + } + + if !metrics.Success && metrics.ErrorType != "" { + dimensions = append(dimensions, types.Dimension{ + Name: aws.String("ErrorType"), + Value: aws.String(metrics.ErrorType), + }) + } + + metricData := []types.MetricDatum{ + // Database operation count + { + MetricName: aws.String("DatabaseOperationCount"), + Value: aws.Float64(1), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }, + // Operation duration + { + MetricName: aws.String("DatabaseOperationDuration"), + Value: aws.Float64(float64(metrics.Duration.Milliseconds())), + Unit: types.StandardUnitMilliseconds, + Timestamp: ×tamp, + Dimensions: dimensions, + }, + } + + // Add record count if applicable + if metrics.RecordCount > 0 { + metricData = append(metricData, types.MetricDatum{ + MetricName: aws.String("DatabaseRecordsProcessed"), + Value: aws.Float64(float64(metrics.RecordCount)), + Unit: types.StandardUnitCount, + Timestamp: ×tamp, + Dimensions: dimensions, + }) + } + + input := &cloudwatch.PutMetricDataInput{ + Namespace: aws.String(namespace), + MetricData: metricData, + } + + _, err := m.cw.PutMetricData(ctx, input) + if err != nil { + m.logger.Error(). + Err(err). + Str("namespace", namespace). + Str("operation", metrics.Operation). + Msg("Failed to send database metrics to CloudWatch") + return fmt.Errorf("failed to send database metrics: %w", err) + } + + m.logger.Debug(). + Str("namespace", namespace). + Str("operation", metrics.Operation). + Bool("success", metrics.Success). + Dur("duration", metrics.Duration). + Int("record_count", metrics.RecordCount). + Msg("Successfully sent database metrics to CloudWatch") + + return nil +} + +// CustomMetric holds custom metric data +type CustomMetric struct { + Name string + Value float64 + Unit types.StandardUnit + Dimensions map[string]string +} + +// SendCustomMetric sends a custom metric to CloudWatch +func (m *MetricsClient) SendCustomMetric(ctx context.Context, metric CustomMetric) error { + namespace := "MeteorApp/ComputeService" + timestamp := time.Now() + + dimensions := make([]types.Dimension, 0, len(metric.Dimensions)) + for key, value := range metric.Dimensions { + dimensions = append(dimensions, types.Dimension{ + Name: aws.String(key), + Value: aws.String(value), + }) + } + + input := &cloudwatch.PutMetricDataInput{ + Namespace: aws.String(namespace), + MetricData: []types.MetricDatum{ + { + MetricName: aws.String(metric.Name), + Value: aws.Float64(metric.Value), + Unit: metric.Unit, + Timestamp: ×tamp, + Dimensions: dimensions, + }, + }, + } + + _, err := m.cw.PutMetricData(ctx, input) + if err != nil { + m.logger.Error(). + Err(err). + Str("namespace", namespace). + Str("metric_name", metric.Name). + Msg("Failed to send custom metric to CloudWatch") + return fmt.Errorf("failed to send custom metric: %w", err) + } + + m.logger.Debug(). + Str("namespace", namespace). + Str("metric_name", metric.Name). + Float64("value", metric.Value). + Msg("Successfully sent custom metric to CloudWatch") + + return nil +} \ No newline at end of file diff --git a/meteor-compute-service/internal/processor/processor.go b/meteor-compute-service/internal/processor/processor.go index 41006fd..e21393f 100644 --- a/meteor-compute-service/internal/processor/processor.go +++ b/meteor-compute-service/internal/processor/processor.go @@ -5,6 +5,8 @@ import ( "errors" "fmt" "log" + "meteor-compute-service/internal/logger" + "meteor-compute-service/internal/metrics" "meteor-compute-service/internal/models" "meteor-compute-service/internal/repository" "meteor-compute-service/internal/sqs" @@ -24,7 +26,8 @@ type ProcessingStats struct { ProcessingErrors []string `json:"recent_errors"` } -// Validator interface for event validation +// Validator interface for event validation (maintained for backward compatibility) +// The actual validation is now done through ValidationProvider interface type Validator interface { Validate(ctx context.Context, rawEvent *models.RawEvent) (*models.ValidationResult, error) } @@ -34,6 +37,8 @@ type Processor struct { sqsClient sqs.SQSClient repository repository.Repository validator Validator + logger *logger.StructuredLogger + metricsClient *metrics.MetricsClient workers int batchSize int idempotency bool @@ -54,17 +59,21 @@ func NewProcessor( sqsClient sqs.SQSClient, repo repository.Repository, validator Validator, + structuredLogger *logger.StructuredLogger, + metricsClient *metrics.MetricsClient, workers int, batchSize int, idempotency bool, ) *Processor { return &Processor{ - sqsClient: sqsClient, - repository: repo, - validator: validator, - workers: workers, - batchSize: batchSize, - idempotency: idempotency, + sqsClient: sqsClient, + repository: repo, + validator: validator, + logger: structuredLogger, + metricsClient: metricsClient, + workers: workers, + batchSize: batchSize, + idempotency: idempotency, messagesChan: make(chan *sqs.Message, batchSize*2), errorsChan: make(chan error, 10), stopChan: make(chan struct{}), @@ -153,8 +162,18 @@ func (p *Processor) worker(ctx context.Context, workerID int) { // processMessage handles a single SQS message func (p *Processor) processMessage(ctx context.Context, workerID int, message *sqs.Message) { startTime := time.Now() - log.Printf("Worker %d processing message %s for raw_event_id %s", - workerID, message.ID, message.RawEventID) + success := false + var errorType string + + // Add correlation ID to context if available + if message.CorrelationID != nil { + ctx = logger.WithCorrelationID(ctx, *message.CorrelationID) + } + + p.logger.WorkerEvent(ctx, workerID, "message_processing_start", + logger.NewField("sqs_message_id", message.ID), + logger.NewField("raw_event_id", message.RawEventID), + ) // Update stats p.updateStats(func(stats *ProcessingStats) { @@ -165,29 +184,57 @@ func (p *Processor) processMessage(ctx context.Context, workerID int, message *s // Parse raw event ID rawEventID, err := uuid.Parse(message.RawEventID) if err != nil { - p.handleProcessingError(fmt.Sprintf("Invalid UUID in message %s: %v", message.ID, err)) + errorType = "invalid_uuid" + p.logger.Error(ctx, "Invalid UUID in SQS message", err, + logger.NewField("sqs_message_id", message.ID), + logger.NewField("raw_event_id", message.RawEventID), + logger.NewField("worker_id", workerID), + ) p.updateStats(func(stats *ProcessingStats) { stats.Failed++ }) + + // Send metrics for failed processing + processingTime := time.Since(startTime) + go p.sendMessageProcessingMetrics(ctx, processingTime, false, errorType, "unknown") return } // Process the event if err := p.processEvent(ctx, rawEventID, message); err != nil { - p.handleProcessingError(fmt.Sprintf("Failed to process event %s: %v", rawEventID, err)) + errorType = p.categorizeError(err) + p.logger.Error(ctx, "Failed to process event", err, + logger.NewField("raw_event_id", rawEventID.String()), + logger.NewField("sqs_message_id", message.ID), + logger.NewField("worker_id", workerID), + ) p.updateStats(func(stats *ProcessingStats) { stats.Failed++ }) + + // Send metrics for failed processing + processingTime := time.Since(startTime) + go p.sendMessageProcessingMetrics(ctx, processingTime, false, errorType, p.getProviderName()) return } // Delete message from SQS after successful processing if err := p.sqsClient.DeleteMessage(ctx, message.ReceiptHandle); err != nil { - log.Printf("Warning: Failed to delete message %s after successful processing: %v", message.ID, err) + p.logger.Warn(ctx, "Failed to delete SQS message after successful processing", + logger.NewField("sqs_message_id", message.ID), + logger.NewField("error", err.Error()), + ) // Don't count this as a failure since the event was processed successfully } + success = true processingTime := time.Since(startTime) - log.Printf("Worker %d successfully processed message %s in %v", - workerID, message.ID, processingTime) + p.logger.WorkerEvent(ctx, workerID, "message_processing_complete", + logger.NewField("sqs_message_id", message.ID), + logger.NewField("raw_event_id", message.RawEventID), + logger.NewField("processing_time_ms", processingTime.Milliseconds()), + ) p.updateStats(func(stats *ProcessingStats) { stats.SuccessfullyProcessed++ }) + + // Send metrics for successful processing + go p.sendMessageProcessingMetrics(ctx, processingTime, success, "", p.getProviderName()) } // processEvent handles the core business logic for processing a single event @@ -302,4 +349,68 @@ func (p *Processor) HealthCheck(ctx context.Context) error { } return nil +} + +// sendMessageProcessingMetrics sends message processing metrics to CloudWatch +func (p *Processor) sendMessageProcessingMetrics(ctx context.Context, processingTime time.Duration, success bool, errorType, providerName string) { + if p.metricsClient == nil { + return + } + + metrics := metrics.MessageProcessingMetrics{ + ProcessingTime: processingTime, + Success: success, + MessageType: "sqs_message", + ProviderName: providerName, + ErrorType: errorType, + } + + if err := p.metricsClient.SendMessageProcessingMetrics(ctx, metrics); err != nil { + p.logger.Warn(ctx, "Failed to send message processing metrics", + logger.NewField("error", err.Error()), + logger.NewField("success", success), + logger.NewField("provider_name", providerName), + ) + } +} + +// categorizeError categorizes errors for metrics reporting +func (p *Processor) categorizeError(err error) string { + if err == nil { + return "" + } + + errorStr := err.Error() + + // Database errors + if errors.Is(err, repository.ErrRawEventNotFound) { + return "raw_event_not_found" + } + if errors.Is(err, repository.ErrValidatedEventExists) { + return "validated_event_exists" + } + + // Validation errors + if fmt.Sprintf("%T", err) == "validation.ValidationError" { + return "validation_error" + } + + // Generic categorization based on error message + switch { + case fmt.Sprintf("%s", errorStr) == "context canceled": + return "context_canceled" + case fmt.Sprintf("%s", errorStr) == "context deadline exceeded": + return "timeout" + default: + return "unknown_error" + } +} + +// getProviderName gets the validation provider name for metrics +func (p *Processor) getProviderName() string { + // Try to extract provider name from validator if it has the method + if provider, ok := p.validator.(interface{ GetProviderName() string }); ok { + return provider.GetProviderName() + } + return "unknown" } \ No newline at end of file diff --git a/meteor-compute-service/internal/sqs/client.go b/meteor-compute-service/internal/sqs/client.go index 044dd9f..8c4f2fc 100644 --- a/meteor-compute-service/internal/sqs/client.go +++ b/meteor-compute-service/internal/sqs/client.go @@ -20,6 +20,7 @@ type Message struct { Body string ReceiptHandle string RawEventID string + CorrelationID *string // Optional correlation ID from message attributes } // RawEventMessage represents the expected structure of SQS message body @@ -116,11 +117,26 @@ func (c *Client) parseMessage(sqsMsg types.Message) (*Message, error) { return nil, errors.New("raw_event_id is missing from message body") } + // Extract correlation_id from message attributes if present + var correlationID *string + if sqsMsg.MessageAttributes != nil { + if attr, ok := sqsMsg.MessageAttributes["correlation_id"]; ok && attr.StringValue != nil { + correlationID = attr.StringValue + } + // Also check for x-correlation-id (alternative naming) + if correlationID == nil { + if attr, ok := sqsMsg.MessageAttributes["x-correlation-id"]; ok && attr.StringValue != nil { + correlationID = attr.StringValue + } + } + } + return &Message{ ID: *sqsMsg.MessageId, Body: *sqsMsg.Body, ReceiptHandle: *sqsMsg.ReceiptHandle, RawEventID: rawEventMsg.RawEventID, + CorrelationID: correlationID, }, nil } diff --git a/meteor-compute-service/internal/validation/classic_cv_provider.go b/meteor-compute-service/internal/validation/classic_cv_provider.go new file mode 100644 index 0000000..02671a9 --- /dev/null +++ b/meteor-compute-service/internal/validation/classic_cv_provider.go @@ -0,0 +1,910 @@ +package validation + +import ( + "context" + "encoding/json" + "fmt" + "image" + "image/color" + "math" + "meteor-compute-service/internal/models" + "time" +) + +// ClassicCvProvider implements computer vision-based meteor validation +// Based on Vida et al. (2016) and Jenniskens et al. (2011) research +type ClassicCvProvider struct { + info ProviderInfo + // Configuration parameters from research papers + k1Parameter float64 // K1=1.7 from paper + j1Parameter float64 // J1=9 from paper + minFrames int // Minimum 4 frames for valid detection + maxNoiseArea int // Maximum noise area in pixels +} + +// NewClassicCvProvider creates a new classic computer vision validation provider +func NewClassicCvProvider() *ClassicCvProvider { + return &ClassicCvProvider{ + info: ProviderInfo{ + Name: "Classic Computer Vision Provider", + Version: "2.0.0", + Description: "Computer vision-based meteor validation using classic CV algorithms", + Algorithm: "classic_cv_v2", + }, + k1Parameter: 1.7, // From Vida et al. (2016) + j1Parameter: 9.0, // From Vida et al. (2016) + minFrames: 4, // Minimum frames for valid meteor + maxNoiseArea: 100, // Maximum noise area threshold + } +} + +// GetProviderInfo returns metadata about this validation provider +func (c *ClassicCvProvider) GetProviderInfo() ProviderInfo { + return c.info +} + +// Validate performs computer vision validation on a raw event +func (c *ClassicCvProvider) Validate(ctx context.Context, rawEvent *models.RawEvent) (*models.ValidationResult, error) { + startTime := time.Now() + + // Initialize validation details + details := ValidationDetails{ + Algorithm: c.info.Algorithm, + Version: c.info.Version, + ValidationSteps: []ValidationStep{}, + Metadata: map[string]interface{}{ + "k1_parameter": c.k1Parameter, + "j1_parameter": c.j1Parameter, + "min_frames": c.minFrames, + "max_noise_area": c.maxNoiseArea, + "processing_time": nil, // Will be filled at the end + }, + } + + // Step 1: Load and validate video frames + frames, step1 := c.loadVideoFrames(rawEvent) + details.ValidationSteps = append(details.ValidationSteps, step1) + if !step1.Passed { + return c.createFailedResult(&details, "Failed to load video frames") + } + + // Step 2: Generate four-frame compression (FF) + fourFrames, step2 := c.generateFourFrameCompression(frames) + details.ValidationSteps = append(details.ValidationSteps, step2) + if !step2.Passed { + return c.createFailedResult(&details, "Failed to generate four-frame compression") + } + + // Step 3: Star field validity check + step3 := c.validateStarField(fourFrames.AvgPixel) + details.ValidationSteps = append(details.ValidationSteps, step3) + if !step3.Passed { + return c.createFailedResult(&details, "Star field validation failed - poor weather conditions") + } + + // Step 4: Statistical threshold segmentation + binaryMask, step4 := c.performThresholdSegmentation(fourFrames) + details.ValidationSteps = append(details.ValidationSteps, step4) + if !step4.Passed { + return c.createFailedResult(&details, "Threshold segmentation failed") + } + + // Step 5: Morphological processing + processedMask, step5 := c.performMorphologicalProcessing(binaryMask) + details.ValidationSteps = append(details.ValidationSteps, step5) + if !step5.Passed { + return c.createFailedResult(&details, "Morphological processing failed") + } + + // Step 6: Line detection using KHT + detectedLines, step6 := c.performLineDetection(processedMask) + details.ValidationSteps = append(details.ValidationSteps, step6) + if !step6.Passed { + return c.createFailedResult(&details, "Line detection failed") + } + + // Step 7: Time dimension validation + validationResult, step7 := c.performTimeValidation(detectedLines, fourFrames.MaxFrame, frames) + details.ValidationSteps = append(details.ValidationSteps, step7) + + // Calculate final score and validity + passedSteps := 0 + for _, step := range details.ValidationSteps { + if step.Passed { + passedSteps++ + } + } + + totalSteps := len(details.ValidationSteps) + score := float64(passedSteps) / float64(totalSteps) + isValid := step7.Passed && score >= 0.85 // High threshold for CV validation + + // Add processing time + processingTime := time.Since(startTime) + details.Metadata["processing_time"] = processingTime.Seconds() + details.Metadata["total_steps"] = totalSteps + details.Metadata["passed_steps"] = passedSteps + details.Metadata["final_score"] = score + + // Serialize details + detailsJSON, err := json.Marshal(details) + if err != nil { + return nil, fmt.Errorf("failed to marshal validation details: %w", err) + } + + reason := c.generateReason(isValid, validationResult, passedSteps, totalSteps) + + return &models.ValidationResult{ + IsValid: isValid, + Score: score, + Algorithm: c.info.Algorithm, + Details: detailsJSON, + ProcessedAt: time.Now().UTC(), + Reason: reason, + }, nil +} + +// FourFrameData represents the four compressed frames from the algorithm +type FourFrameData struct { + MaxPixel *image.Gray // Maximum pixel values + AvgPixel *image.Gray // Average pixel values (excluding max) + StdPixel *image.Gray // Standard deviation of pixel values + MaxFrame *image.Gray // Frame numbers where max occurred + Width int // Image width + Height int // Image height +} + +// TimeValidationResult contains results from time dimension validation +type TimeValidationResult struct { + ContinuousTrajectories int `json:"continuous_trajectories"` + LongestTrajectory int `json:"longest_trajectory"` + AverageTrajectoryLength float64 `json:"average_trajectory_length"` + ValidMeteorDetected bool `json:"valid_meteor_detected"` +} + +// loadVideoFrames loads and validates video frames from the raw event +func (c *ClassicCvProvider) loadVideoFrames(rawEvent *models.RawEvent) ([]*image.Gray, ValidationStep) { + step := ValidationStep{ + Name: "load_video_frames", + Description: "Load and validate 256 video frames from raw event data", + Details: make(map[string]interface{}), + } + + // For MVP implementation, we'll simulate loading frames + // In production, this would decode the actual video file + expectedFrames := 256 + step.Details["expected_frames"] = expectedFrames + + // Simulate frame loading - in real implementation this would: + // 1. Download video file from S3 using rawEvent.FilePath + // 2. Decode video using ffmpeg or similar + // 3. Extract exactly 256 frames + // 4. Convert to grayscale + + frames := make([]*image.Gray, expectedFrames) + width, height := 640, 480 // Standard resolution + + // Create mock grayscale frames for testing + for i := 0; i < expectedFrames; i++ { + frame := image.NewGray(image.Rect(0, 0, width, height)) + // Fill with some test pattern + for y := 0; y < height; y++ { + for x := 0; x < width; x++ { + // Create a simple test pattern with some variation + value := uint8((x + y + i) % 256) + frame.SetGray(x, y, color.Gray{Y: value}) + } + } + frames[i] = frame + } + + step.Details["loaded_frames"] = len(frames) + step.Details["frame_width"] = width + step.Details["frame_height"] = height + step.Details["total_pixels"] = width * height + step.Passed = len(frames) == expectedFrames + + if !step.Passed { + step.Error = fmt.Sprintf("Expected %d frames, got %d", expectedFrames, len(frames)) + } + + return frames, step +} + +// generateFourFrameCompression implements the four-frame compression algorithm +func (c *ClassicCvProvider) generateFourFrameCompression(frames []*image.Gray) (*FourFrameData, ValidationStep) { + step := ValidationStep{ + Name: "four_frame_compression", + Description: "Generate maxpixel, avepixel, stdpixel, and maxframe images", + Details: make(map[string]interface{}), + } + + if len(frames) == 0 { + step.Error = "No frames provided for compression" + step.Passed = false + return nil, step + } + + bounds := frames[0].Bounds() + width, height := bounds.Dx(), bounds.Dy() + + // Initialize output images + maxPixel := image.NewGray(bounds) + avgPixel := image.NewGray(bounds) + stdPixel := image.NewGray(bounds) + maxFrame := image.NewGray(bounds) + + step.Details["frame_count"] = len(frames) + step.Details["width"] = width + step.Details["height"] = height + + // For each pixel position (x, y) + for y := bounds.Min.Y; y < bounds.Max.Y; y++ { + for x := bounds.Min.X; x < bounds.Max.X; x++ { + // Collect all pixel values for this position across all frames + values := make([]float64, len(frames)) + maxVal := float64(0) + maxFrameIdx := 0 + + for frameIdx, frame := range frames { + pixelVal := float64(frame.GrayAt(x, y).Y) + values[frameIdx] = pixelVal + + // Track maximum value and its frame + if pixelVal > maxVal { + maxVal = pixelVal + maxFrameIdx = frameIdx + } + } + + // Set maxpixel value + maxPixel.SetGray(x, y, color.Gray{Y: uint8(maxVal)}) + + // Set maxframe value (frame index where max occurred) + maxFrame.SetGray(x, y, color.Gray{Y: uint8(maxFrameIdx)}) + + // Calculate average excluding the maximum value + sum := float64(0) + count := 0 + for _, val := range values { + if val != maxVal { + sum += val + count++ + } + } + + var avgVal float64 + if count > 0 { + avgVal = sum / float64(count) + } + avgPixel.SetGray(x, y, color.Gray{Y: uint8(avgVal)}) + + // Calculate standard deviation excluding the maximum value + if count > 1 { + sumSquaredDiff := float64(0) + for _, val := range values { + if val != maxVal { + diff := val - avgVal + sumSquaredDiff += diff * diff + } + } + stdDev := math.Sqrt(sumSquaredDiff / float64(count-1)) + stdPixel.SetGray(x, y, color.Gray{Y: uint8(math.Min(stdDev, 255))}) + } else { + stdPixel.SetGray(x, y, color.Gray{Y: 0}) + } + } + } + + fourFrames := &FourFrameData{ + MaxPixel: maxPixel, + AvgPixel: avgPixel, + StdPixel: stdPixel, + MaxFrame: maxFrame, + Width: width, + Height: height, + } + + step.Details["compression_completed"] = true + step.Passed = true + + return fourFrames, step +} + +// validateStarField checks if the star field is valid for meteor detection +func (c *ClassicCvProvider) validateStarField(avgPixelImage *image.Gray) ValidationStep { + step := ValidationStep{ + Name: "star_field_validation", + Description: "Validate star field quality for meteor detection", + Details: make(map[string]interface{}), + } + + bounds := avgPixelImage.Bounds() + width, height := bounds.Dx(), bounds.Dy() + + // Simple star detection using local maxima + starCount := 0 + threshold := uint8(50) // Minimum brightness for star detection + minDistance := 5 // Minimum distance between stars + + step.Details["detection_threshold"] = threshold + step.Details["min_star_distance"] = minDistance + + // Find local maxima that could be stars + for y := minDistance; y < height-minDistance; y++ { + for x := minDistance; x < width-minDistance; x++ { + centerVal := avgPixelImage.GrayAt(x, y).Y + + if centerVal < threshold { + continue + } + + // Check if this is a local maximum + isLocalMax := true + for dy := -minDistance; dy <= minDistance && isLocalMax; dy++ { + for dx := -minDistance; dx <= minDistance && isLocalMax; dx++ { + if dx == 0 && dy == 0 { + continue + } + neighborVal := avgPixelImage.GrayAt(x+dx, y+dy).Y + if neighborVal >= centerVal { + isLocalMax = false + } + } + } + + if isLocalMax { + starCount++ + } + } + } + + // Minimum number of stars required for valid sky conditions + minStarsRequired := 20 + step.Details["detected_stars"] = starCount + step.Details["min_stars_required"] = minStarsRequired + step.Details["star_density"] = float64(starCount) / float64(width*height) * 1000000 // stars per million pixels + + step.Passed = starCount >= minStarsRequired + + if !step.Passed { + step.Error = fmt.Sprintf("Insufficient stars detected: %d (required: %d) - possible cloudy conditions", + starCount, minStarsRequired) + } + + return step +} + +// performThresholdSegmentation applies statistical threshold segmentation +func (c *ClassicCvProvider) performThresholdSegmentation(fourFrames *FourFrameData) (*image.Gray, ValidationStep) { + step := ValidationStep{ + Name: "threshold_segmentation", + Description: "Apply statistical threshold: max > avg + K*stddev + J", + Details: map[string]interface{}{ + "k1_parameter": c.k1Parameter, + "j1_parameter": c.j1Parameter, + }, + } + + bounds := fourFrames.MaxPixel.Bounds() + binaryMask := image.NewGray(bounds) + detectedPixels := 0 + totalPixels := bounds.Dx() * bounds.Dy() + + // Apply threshold formula: max > avg + K1*stddev + J1 + for y := bounds.Min.Y; y < bounds.Max.Y; y++ { + for x := bounds.Min.X; x < bounds.Max.X; x++ { + maxVal := float64(fourFrames.MaxPixel.GrayAt(x, y).Y) + avgVal := float64(fourFrames.AvgPixel.GrayAt(x, y).Y) + stdVal := float64(fourFrames.StdPixel.GrayAt(x, y).Y) + + threshold := avgVal + c.k1Parameter*stdVal + c.j1Parameter + + if maxVal > threshold { + binaryMask.SetGray(x, y, color.Gray{Y: 255}) // White for detected pixel + detectedPixels++ + } else { + binaryMask.SetGray(x, y, color.Gray{Y: 0}) // Black for background + } + } + } + + detectionRate := float64(detectedPixels) / float64(totalPixels) + step.Details["detected_pixels"] = detectedPixels + step.Details["total_pixels"] = totalPixels + step.Details["detection_rate"] = detectionRate + + // Reasonable detection rate (not too high, not too low) + minDetectionRate := 0.001 // 0.1% + maxDetectionRate := 0.05 // 5% + + step.Passed = detectionRate >= minDetectionRate && detectionRate <= maxDetectionRate + + if !step.Passed { + if detectionRate < minDetectionRate { + step.Error = fmt.Sprintf("Detection rate too low: %.4f%% (min: %.4f%%)", + detectionRate*100, minDetectionRate*100) + } else { + step.Error = fmt.Sprintf("Detection rate too high: %.4f%% (max: %.4f%%) - possible noise", + detectionRate*100, maxDetectionRate*100) + } + } + + return binaryMask, step +} + +// performMorphologicalProcessing cleans up the binary mask +func (c *ClassicCvProvider) performMorphologicalProcessing(binaryMask *image.Gray) (*image.Gray, ValidationStep) { + step := ValidationStep{ + Name: "morphological_processing", + Description: "Clean noise, bridge gaps, and thin lines in binary mask", + Details: make(map[string]interface{}), + } + + bounds := binaryMask.Bounds() + processed := image.NewGray(bounds) + + // Copy original image first + for y := bounds.Min.Y; y < bounds.Max.Y; y++ { + for x := bounds.Min.X; x < bounds.Max.X; x++ { + processed.SetGray(x, y, binaryMask.GrayAt(x, y)) + } + } + + // Step 1: Noise removal (opening operation) + temp1 := c.morphologicalOpening(processed, 1) + step.Details["noise_removal"] = "applied" + + // Step 2: Gap bridging (closing operation) + temp2 := c.morphologicalClosing(temp1, 2) + step.Details["gap_bridging"] = "applied" + + // Step 3: Line thinning + final := c.morphologicalThinning(temp2) + step.Details["line_thinning"] = "applied" + + // Count remaining pixels + remainingPixels := 0 + for y := bounds.Min.Y; y < bounds.Max.Y; y++ { + for x := bounds.Min.X; x < bounds.Max.X; x++ { + if final.GrayAt(x, y).Y > 0 { + remainingPixels++ + } + } + } + + step.Details["remaining_pixels"] = remainingPixels + step.Passed = remainingPixels > 0 && remainingPixels < bounds.Dx()*bounds.Dy()/10 // Reasonable amount + + if !step.Passed { + if remainingPixels == 0 { + step.Error = "No pixels remaining after morphological processing" + } else { + step.Error = "Too many pixels remaining - possible excessive noise" + } + } + + return final, step +} + +// performLineDetection implements KHT-based line detection +func (c *ClassicCvProvider) performLineDetection(processedMask *image.Gray) ([]Line, ValidationStep) { + step := ValidationStep{ + Name: "line_detection", + Description: "Detect lines using Kernel-based Hough Transform (KHT)", + Details: make(map[string]interface{}), + } + + lines := c.kernelHoughTransform(processedMask) + + step.Details["detected_lines"] = len(lines) + step.Details["line_details"] = lines + + // We expect to find at least one significant line for a meteor + minLines := 1 + maxLines := 10 // Too many lines might indicate noise + + step.Passed = len(lines) >= minLines && len(lines) <= maxLines + + if !step.Passed { + if len(lines) < minLines { + step.Error = fmt.Sprintf("Insufficient lines detected: %d (min: %d)", len(lines), minLines) + } else { + step.Error = fmt.Sprintf("Too many lines detected: %d (max: %d) - possible noise", len(lines), maxLines) + } + } + + return lines, step +} + +// performTimeValidation validates temporal continuity using maxframe data +func (c *ClassicCvProvider) performTimeValidation(lines []Line, maxFrameImage *image.Gray, originalFrames []*image.Gray) (*TimeValidationResult, ValidationStep) { + step := ValidationStep{ + Name: "time_validation", + Description: "Validate 3D spatio-temporal continuity of detected lines", + Details: make(map[string]interface{}), + } + + result := &TimeValidationResult{} + + if len(lines) == 0 { + step.Error = "No lines provided for time validation" + step.Passed = false + return result, step + } + + // For each detected line, check temporal continuity + validTrajectories := 0 + totalTrajectoryLength := 0 + longestTrajectory := 0 + + for i, line := range lines { + trajectory := c.extractTrajectoryFromLine(line, maxFrameImage) + trajectoryLength := len(trajectory) + + if trajectoryLength >= c.minFrames { + validTrajectories++ + } + + totalTrajectoryLength += trajectoryLength + if trajectoryLength > longestTrajectory { + longestTrajectory = trajectoryLength + } + + step.Details[fmt.Sprintf("line_%d_trajectory_length", i)] = trajectoryLength + } + + avgTrajectoryLength := float64(0) + if len(lines) > 0 { + avgTrajectoryLength = float64(totalTrajectoryLength) / float64(len(lines)) + } + + result.ContinuousTrajectories = validTrajectories + result.LongestTrajectory = longestTrajectory + result.AverageTrajectoryLength = avgTrajectoryLength + result.ValidMeteorDetected = validTrajectories > 0 && longestTrajectory >= c.minFrames + + step.Details["valid_trajectories"] = validTrajectories + step.Details["longest_trajectory"] = longestTrajectory + step.Details["average_trajectory_length"] = avgTrajectoryLength + step.Details["min_frames_required"] = c.minFrames + + step.Passed = result.ValidMeteorDetected + + if !step.Passed { + step.Error = fmt.Sprintf("No valid meteor trajectories found (min %d frames required)", c.minFrames) + } + + return result, step +} + +// Helper functions for morphological operations + +func (c *ClassicCvProvider) morphologicalOpening(img *image.Gray, kernelSize int) *image.Gray { + // Erosion followed by dilation + eroded := c.morphologicalErosion(img, kernelSize) + return c.morphologicalDilation(eroded, kernelSize) +} + +func (c *ClassicCvProvider) morphologicalClosing(img *image.Gray, kernelSize int) *image.Gray { + // Dilation followed by erosion + dilated := c.morphologicalDilation(img, kernelSize) + return c.morphologicalErosion(dilated, kernelSize) +} + +func (c *ClassicCvProvider) morphologicalErosion(img *image.Gray, kernelSize int) *image.Gray { + bounds := img.Bounds() + result := image.NewGray(bounds) + + for y := bounds.Min.Y; y < bounds.Max.Y; y++ { + for x := bounds.Min.X; x < bounds.Max.X; x++ { + minVal := uint8(255) + + for dy := -kernelSize; dy <= kernelSize; dy++ { + for dx := -kernelSize; dx <= kernelSize; dx++ { + nx, ny := x+dx, y+dy + if nx >= bounds.Min.X && nx < bounds.Max.X && ny >= bounds.Min.Y && ny < bounds.Max.Y { + val := img.GrayAt(nx, ny).Y + if val < minVal { + minVal = val + } + } + } + } + + result.SetGray(x, y, color.Gray{Y: minVal}) + } + } + + return result +} + +func (c *ClassicCvProvider) morphologicalDilation(img *image.Gray, kernelSize int) *image.Gray { + bounds := img.Bounds() + result := image.NewGray(bounds) + + for y := bounds.Min.Y; y < bounds.Max.Y; y++ { + for x := bounds.Min.X; x < bounds.Max.X; x++ { + maxVal := uint8(0) + + for dy := -kernelSize; dy <= kernelSize; dy++ { + for dx := -kernelSize; dx <= kernelSize; dx++ { + nx, ny := x+dx, y+dy + if nx >= bounds.Min.X && nx < bounds.Max.X && ny >= bounds.Min.Y && ny < bounds.Max.Y { + val := img.GrayAt(nx, ny).Y + if val > maxVal { + maxVal = val + } + } + } + } + + result.SetGray(x, y, color.Gray{Y: maxVal}) + } + } + + return result +} + +func (c *ClassicCvProvider) morphologicalThinning(img *image.Gray) *image.Gray { + // Simplified thinning operation + bounds := img.Bounds() + result := image.NewGray(bounds) + + // Copy the image + for y := bounds.Min.Y; y < bounds.Max.Y; y++ { + for x := bounds.Min.X; x < bounds.Max.X; x++ { + result.SetGray(x, y, img.GrayAt(x, y)) + } + } + + // Apply simple thinning - remove pixels that have too many neighbors + for y := bounds.Min.Y+1; y < bounds.Max.Y-1; y++ { + for x := bounds.Min.X+1; x < bounds.Max.X-1; x++ { + if img.GrayAt(x, y).Y > 0 { + // Count neighbors + neighbors := 0 + for dy := -1; dy <= 1; dy++ { + for dx := -1; dx <= 1; dx++ { + if dx == 0 && dy == 0 { + continue + } + if img.GrayAt(x+dx, y+dy).Y > 0 { + neighbors++ + } + } + } + + // Remove pixels with too many neighbors (not on a line) + if neighbors > 2 { + result.SetGray(x, y, color.Gray{Y: 0}) + } + } + } + } + + return result +} + +// Line represents a detected line segment +type Line struct { + X1, Y1, X2, Y2 int `json:"coordinates"` + Length float64 `json:"length"` + Angle float64 `json:"angle"` + Strength float64 `json:"strength"` +} + +// kernelHoughTransform implements a simplified KHT algorithm +func (c *ClassicCvProvider) kernelHoughTransform(img *image.Gray) []Line { + bounds := img.Bounds() + lines := []Line{} + + // Find edge pixels + edgePixels := []image.Point{} + for y := bounds.Min.Y; y < bounds.Max.Y; y++ { + for x := bounds.Min.X; x < bounds.Max.X; x++ { + if img.GrayAt(x, y).Y > 0 { + edgePixels = append(edgePixels, image.Point{X: x, Y: y}) + } + } + } + + // Group nearby pixels into potential lines + minLineLength := 10 + maxDistance := 3 + + for i := 0; i < len(edgePixels); i++ { + for j := i + minLineLength; j < len(edgePixels); j++ { + p1, p2 := edgePixels[i], edgePixels[j] + + // Calculate line parameters + dx := float64(p2.X - p1.X) + dy := float64(p2.Y - p1.Y) + length := math.Sqrt(dx*dx + dy*dy) + + if length < float64(minLineLength) { + continue + } + + // Check if pixels between p1 and p2 are also edges + steps := int(length) + supportCount := 0 + + for step := 0; step <= steps; step++ { + t := float64(step) / float64(steps) + x := int(float64(p1.X) + t*dx) + y := int(float64(p1.Y) + t*dy) + + if x >= bounds.Min.X && x < bounds.Max.X && y >= bounds.Min.Y && y < bounds.Max.Y { + // Check if there's an edge pixel nearby + found := false + for _, edgePixel := range edgePixels { + dist := math.Sqrt(float64((edgePixel.X-x)*(edgePixel.X-x) + (edgePixel.Y-y)*(edgePixel.Y-y))) + if dist <= float64(maxDistance) { + found = true + break + } + } + if found { + supportCount++ + } + } + } + + // Calculate line strength + strength := float64(supportCount) / float64(steps+1) + + // Only keep lines with good support + if strength > 0.7 && length > float64(minLineLength) { + angle := math.Atan2(dy, dx) * 180 / math.Pi + line := Line{ + X1: p1.X, + Y1: p1.Y, + X2: p2.X, + Y2: p2.Y, + Length: length, + Angle: angle, + Strength: strength, + } + lines = append(lines, line) + } + } + } + + // Remove duplicate lines + return c.removeDuplicateLines(lines) +} + +func (c *ClassicCvProvider) removeDuplicateLines(lines []Line) []Line { + if len(lines) <= 1 { + return lines + } + + filtered := []Line{} + + for i, line1 := range lines { + isDuplicate := false + + for j := i + 1; j < len(lines); j++ { + line2 := lines[j] + + // Check if lines are similar + dist1 := math.Sqrt(float64((line1.X1-line2.X1)*(line1.X1-line2.X1) + (line1.Y1-line2.Y1)*(line1.Y1-line2.Y1))) + dist2 := math.Sqrt(float64((line1.X2-line2.X2)*(line1.X2-line2.X2) + (line1.Y2-line2.Y2)*(line1.Y2-line2.Y2))) + angleDiff := math.Abs(line1.Angle - line2.Angle) + + if dist1 < 10 && dist2 < 10 && angleDiff < 15 { + isDuplicate = true + break + } + } + + if !isDuplicate { + filtered = append(filtered, line1) + } + } + + return filtered +} + +// extractTrajectoryFromLine extracts frame sequence for a line using maxframe data +func (c *ClassicCvProvider) extractTrajectoryFromLine(line Line, maxFrameImage *image.Gray) []int { + // Extract frame numbers along the line + frameNumbers := []int{} + + dx := line.X2 - line.X1 + dy := line.Y2 - line.Y1 + steps := int(math.Max(math.Abs(float64(dx)), math.Abs(float64(dy)))) + + if steps == 0 { + return frameNumbers + } + + for step := 0; step <= steps; step++ { + t := float64(step) / float64(steps) + x := int(float64(line.X1) + t*float64(dx)) + y := int(float64(line.Y1) + t*float64(dy)) + + bounds := maxFrameImage.Bounds() + if x >= bounds.Min.X && x < bounds.Max.X && y >= bounds.Min.Y && y < bounds.Max.Y { + frameNum := int(maxFrameImage.GrayAt(x, y).Y) + frameNumbers = append(frameNumbers, frameNum) + } + } + + // Count consecutive frame sequence + if len(frameNumbers) == 0 { + return []int{} + } + + // Find the longest consecutive sequence + longestSeq := []int{} + currentSeq := []int{frameNumbers[0]} + + for i := 1; i < len(frameNumbers); i++ { + if frameNumbers[i] == frameNumbers[i-1]+1 { + currentSeq = append(currentSeq, frameNumbers[i]) + } else { + if len(currentSeq) > len(longestSeq) { + longestSeq = make([]int, len(currentSeq)) + copy(longestSeq, currentSeq) + } + currentSeq = []int{frameNumbers[i]} + } + } + + if len(currentSeq) > len(longestSeq) { + longestSeq = currentSeq + } + + return longestSeq +} + +// createFailedResult creates a validation result for failed validation +func (c *ClassicCvProvider) createFailedResult(details *ValidationDetails, reason string) (*models.ValidationResult, error) { + // Calculate partial score + passedSteps := 0 + for _, step := range details.ValidationSteps { + if step.Passed { + passedSteps++ + } + } + + totalSteps := len(details.ValidationSteps) + score := float64(0) + if totalSteps > 0 { + score = float64(passedSteps) / float64(totalSteps) + } + + details.Metadata["final_score"] = score + details.Metadata["failure_reason"] = reason + + detailsJSON, err := json.Marshal(details) + if err != nil { + return nil, fmt.Errorf("failed to marshal validation details: %w", err) + } + + return &models.ValidationResult{ + IsValid: false, + Score: score, + Algorithm: c.info.Algorithm, + Details: detailsJSON, + ProcessedAt: time.Now().UTC(), + Reason: reason, + }, nil +} + +// generateReason creates a human-readable reason for the validation result +func (c *ClassicCvProvider) generateReason(isValid bool, timeResult *TimeValidationResult, passedSteps, totalSteps int) string { + if isValid { + return fmt.Sprintf("Valid meteor detected: %d continuous trajectories, longest: %d frames (passed %d/%d validation steps)", + timeResult.ContinuousTrajectories, timeResult.LongestTrajectory, passedSteps, totalSteps) + } + + if timeResult != nil { + return fmt.Sprintf("No valid meteor detected: %d trajectories, longest: %d frames (min: %d required)", + timeResult.ContinuousTrajectories, timeResult.LongestTrajectory, c.minFrames) + } + + return fmt.Sprintf("Validation failed: passed %d/%d steps", passedSteps, totalSteps) +} \ No newline at end of file diff --git a/meteor-compute-service/internal/validation/mvp_provider.go b/meteor-compute-service/internal/validation/mvp_provider.go new file mode 100644 index 0000000..d31ad0c --- /dev/null +++ b/meteor-compute-service/internal/validation/mvp_provider.go @@ -0,0 +1,300 @@ +package validation + +import ( + "context" + "encoding/json" + "fmt" + "meteor-compute-service/internal/models" + "time" + + "github.com/google/uuid" +) + +// MVPValidationProvider implements a basic pass-through validation for MVP +// This will be replaced with more sophisticated algorithms in Epic 3 +type MVPValidationProvider struct { + info ProviderInfo +} + +// NewMVPValidationProvider creates a new MVP validation provider instance +func NewMVPValidationProvider() *MVPValidationProvider { + return &MVPValidationProvider{ + info: ProviderInfo{ + Name: "MVP Validation Provider", + Version: "1.0.0", + Description: "Basic pass-through validation for MVP phase", + Algorithm: "mvp_pass_through", + }, + } +} + +// GetProviderInfo returns metadata about this validation provider +func (v *MVPValidationProvider) GetProviderInfo() ProviderInfo { + return v.info +} + +// Validate performs basic validation on a raw event +// For MVP, this is a simple pass-through that marks all events as valid +func (v *MVPValidationProvider) Validate(ctx context.Context, rawEvent *models.RawEvent) (*models.ValidationResult, error) { + // Basic validation details that will be stored + details := ValidationDetails{ + Algorithm: v.info.Algorithm, + Version: v.info.Version, + ValidationSteps: []ValidationStep{}, + Metadata: make(map[string]interface{}), + } + + // Step 1: Basic data completeness check + step1 := v.validateDataCompleteness(rawEvent) + details.ValidationSteps = append(details.ValidationSteps, step1) + + // Step 2: Event type validation + step2 := v.validateEventType(rawEvent) + details.ValidationSteps = append(details.ValidationSteps, step2) + + // Step 3: File validation + step3 := v.validateFile(rawEvent) + details.ValidationSteps = append(details.ValidationSteps, step3) + + // Step 4: Metadata validation + step4 := v.validateMetadata(rawEvent) + details.ValidationSteps = append(details.ValidationSteps, step4) + + // For MVP, calculate a simple score based on completed validation steps + totalSteps := len(details.ValidationSteps) + passedSteps := 0 + for _, step := range details.ValidationSteps { + if step.Passed { + passedSteps++ + } + } + + score := float64(passedSteps) / float64(totalSteps) + isValid := score >= 0.8 // 80% threshold for MVP + + // Add summary to metadata + details.Metadata["total_steps"] = totalSteps + details.Metadata["passed_steps"] = passedSteps + details.Metadata["score"] = score + details.Metadata["threshold"] = 0.8 + + // Serialize details to JSON + detailsJSON, err := json.Marshal(details) + if err != nil { + return nil, fmt.Errorf("failed to marshal validation details: %w", err) + } + + return &models.ValidationResult{ + IsValid: isValid, + Score: score, + Algorithm: v.info.Algorithm, + Details: detailsJSON, + ProcessedAt: time.Now().UTC(), + Reason: v.generateReason(isValid, passedSteps, totalSteps), + }, nil +} + +// validateDataCompleteness checks if required fields are present +func (v *MVPValidationProvider) validateDataCompleteness(rawEvent *models.RawEvent) ValidationStep { + step := ValidationStep{ + Name: "data_completeness", + Description: "Checks if required fields are present and valid", + Details: make(map[string]interface{}), + } + + issues := []string{} + + // Check required UUID fields + if rawEvent.ID == (uuid.UUID{}) { + issues = append(issues, "missing_id") + } + if rawEvent.DeviceID == (uuid.UUID{}) { + issues = append(issues, "missing_device_id") + } + if rawEvent.UserProfileID == (uuid.UUID{}) { + issues = append(issues, "missing_user_profile_id") + } + + // Check required string fields + if rawEvent.FilePath == "" { + issues = append(issues, "missing_file_path") + } + if rawEvent.EventType == "" { + issues = append(issues, "missing_event_type") + } + + // Check timestamp + if rawEvent.EventTimestamp.IsZero() { + issues = append(issues, "missing_event_timestamp") + } + + step.Details["issues"] = issues + step.Details["issues_count"] = len(issues) + step.Passed = len(issues) == 0 + + if len(issues) > 0 { + step.Error = fmt.Sprintf("Found %d data completeness issues", len(issues)) + } + + return step +} + +// validateEventType checks if the event type is supported +func (v *MVPValidationProvider) validateEventType(rawEvent *models.RawEvent) ValidationStep { + step := ValidationStep{ + Name: "event_type_validation", + Description: "Validates that the event type is supported", + Details: make(map[string]interface{}), + } + + supportedTypes := []string{ + models.EventTypeMotion, + models.EventTypeAlert, + models.EventTypeMeteor, + } + + step.Details["event_type"] = rawEvent.EventType + step.Details["supported_types"] = supportedTypes + + // Check if event type is supported + isSupported := false + for _, supportedType := range supportedTypes { + if rawEvent.EventType == supportedType { + isSupported = true + break + } + } + + step.Passed = isSupported + step.Details["is_supported"] = isSupported + + if !isSupported { + step.Error = fmt.Sprintf("Unsupported event type: %s", rawEvent.EventType) + } + + return step +} + +// validateFile checks basic file information +func (v *MVPValidationProvider) validateFile(rawEvent *models.RawEvent) ValidationStep { + step := ValidationStep{ + Name: "file_validation", + Description: "Validates file information and properties", + Details: make(map[string]interface{}), + } + + issues := []string{} + + // Check file path format (basic validation) + if len(rawEvent.FilePath) < 3 { + issues = append(issues, "file_path_too_short") + } + + // Check file size if provided + if rawEvent.FileSize != nil { + step.Details["file_size"] = *rawEvent.FileSize + if *rawEvent.FileSize <= 0 { + issues = append(issues, "invalid_file_size") + } + // Check for reasonable file size limits (e.g., not more than 100MB for video files) + if *rawEvent.FileSize > 100*1024*1024 { + issues = append(issues, "file_size_too_large") + } + } + + // Check file type if provided + if rawEvent.FileType != nil { + step.Details["file_type"] = *rawEvent.FileType + // Basic MIME type validation for common formats + supportedMimeTypes := []string{ + "video/mp4", + "video/quicktime", + "video/x-msvideo", + "image/jpeg", + "image/png", + "application/gzip", + "application/x-tar", + } + + isSupportedMime := false + for _, mimeType := range supportedMimeTypes { + if *rawEvent.FileType == mimeType { + isSupportedMime = true + break + } + } + + if !isSupportedMime { + issues = append(issues, "unsupported_file_type") + } + step.Details["supported_mime_types"] = supportedMimeTypes + } + + step.Details["issues"] = issues + step.Details["issues_count"] = len(issues) + step.Passed = len(issues) == 0 + + if len(issues) > 0 { + step.Error = fmt.Sprintf("Found %d file validation issues", len(issues)) + } + + return step +} + +// validateMetadata performs basic metadata validation +func (v *MVPValidationProvider) validateMetadata(rawEvent *models.RawEvent) ValidationStep { + step := ValidationStep{ + Name: "metadata_validation", + Description: "Validates event metadata structure and content", + Details: make(map[string]interface{}), + } + + issues := []string{} + + // Check if metadata is valid JSON + if rawEvent.Metadata != nil { + var metadata map[string]interface{} + if err := json.Unmarshal(rawEvent.Metadata, &metadata); err != nil { + issues = append(issues, "invalid_json_metadata") + step.Details["json_error"] = err.Error() + } else { + step.Details["metadata_keys"] = getKeys(metadata) + step.Details["metadata_size"] = len(rawEvent.Metadata) + + // Check for reasonable metadata size (not more than 10KB) + if len(rawEvent.Metadata) > 10*1024 { + issues = append(issues, "metadata_too_large") + } + } + } else { + // Metadata is optional, so this is not an error + step.Details["metadata_present"] = false + } + + step.Details["issues"] = issues + step.Details["issues_count"] = len(issues) + step.Passed = len(issues) == 0 + + if len(issues) > 0 { + step.Error = fmt.Sprintf("Found %d metadata validation issues", len(issues)) + } + + return step +} + +// generateReason creates a human-readable reason for the validation result +func (v *MVPValidationProvider) generateReason(isValid bool, passedSteps, totalSteps int) string { + if isValid { + return fmt.Sprintf("Event passed validation with %d/%d steps completed successfully", passedSteps, totalSteps) + } + return fmt.Sprintf("Event failed validation with only %d/%d steps completed successfully (required: 80%%)", passedSteps, totalSteps) +} + +// getKeys extracts keys from a map +func getKeys(m map[string]interface{}) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + return keys +} \ No newline at end of file diff --git a/meteor-compute-service/internal/validation/provider.go b/meteor-compute-service/internal/validation/provider.go new file mode 100644 index 0000000..3b9b742 --- /dev/null +++ b/meteor-compute-service/internal/validation/provider.go @@ -0,0 +1,60 @@ +package validation + +import ( + "context" + "fmt" + "meteor-compute-service/internal/models" +) + +// ValidationProvider defines the pluggable interface for event validation algorithms +type ValidationProvider interface { + // Validate performs validation on a raw event and returns a validation result + Validate(ctx context.Context, rawEvent *models.RawEvent) (*models.ValidationResult, error) + + // GetProviderInfo returns metadata about this validation provider + GetProviderInfo() ProviderInfo +} + +// ProviderInfo contains metadata about a validation provider +type ProviderInfo struct { + Name string `json:"name"` + Version string `json:"version"` + Description string `json:"description"` + Algorithm string `json:"algorithm"` +} + +// ProviderType represents the available validation provider types +type ProviderType string + +const ( + ProviderTypeMVP ProviderType = "mvp" + ProviderTypeClassicCV ProviderType = "classic_cv" +) + +// ProviderFactory creates validation providers based on configuration +type ProviderFactory struct{} + +// NewProviderFactory creates a new provider factory instance +func NewProviderFactory() *ProviderFactory { + return &ProviderFactory{} +} + +// CreateProvider creates a validation provider based on the specified type +func (f *ProviderFactory) CreateProvider(providerType ProviderType) (ValidationProvider, error) { + switch providerType { + case ProviderTypeMVP: + return NewMVPValidationProvider(), nil + case ProviderTypeClassicCV: + return NewClassicCvProvider(), nil + default: + return nil, fmt.Errorf("unknown validation provider type: %s", providerType) + } +} + +// GetAvailableProviders returns a list of all available provider types +func (f *ProviderFactory) GetAvailableProviders() []ProviderType { + return []ProviderType{ + ProviderTypeMVP, + ProviderTypeClassicCV, + } +} \ No newline at end of file diff --git a/meteor-compute-service/internal/validation/validator.go b/meteor-compute-service/internal/validation/validator.go index a348ede..8045caf 100644 --- a/meteor-compute-service/internal/validation/validator.go +++ b/meteor-compute-service/internal/validation/validator.go @@ -2,96 +2,38 @@ package validation import ( "context" - "encoding/json" - "fmt" "meteor-compute-service/internal/models" - "time" - - "github.com/google/uuid" ) // Validator interface defines the contract for event validation +// DEPRECATED: Use ValidationProvider interface instead type Validator interface { Validate(ctx context.Context, rawEvent *models.RawEvent) (*models.ValidationResult, error) } // MVPValidator implements a basic pass-through validation for MVP -// This will be replaced with more sophisticated algorithms in Epic 3 +// DEPRECATED: Use MVPValidationProvider through the provider factory instead type MVPValidator struct { - algorithmName string - version string + provider ValidationProvider } // NewMVPValidator creates a new MVP validator instance +// DEPRECATED: Use NewMVPValidationProvider() through the provider factory instead func NewMVPValidator() *MVPValidator { return &MVPValidator{ - algorithmName: "mvp_pass_through", - version: "1.0.0", + provider: NewMVPValidationProvider(), } } // Validate performs basic validation on a raw event -// For MVP, this is a simple pass-through that marks all events as valid +// DEPRECATED: This method now delegates to the new ValidationProvider system func (v *MVPValidator) Validate(ctx context.Context, rawEvent *models.RawEvent) (*models.ValidationResult, error) { - // Basic validation details that will be stored - details := ValidationDetails{ - Algorithm: v.algorithmName, - Version: v.version, - ValidationSteps: []ValidationStep{}, - Metadata: make(map[string]interface{}), - } - - // Step 1: Basic data completeness check - step1 := v.validateDataCompleteness(rawEvent) - details.ValidationSteps = append(details.ValidationSteps, step1) - - // Step 2: Event type validation - step2 := v.validateEventType(rawEvent) - details.ValidationSteps = append(details.ValidationSteps, step2) - - // Step 3: File validation - step3 := v.validateFile(rawEvent) - details.ValidationSteps = append(details.ValidationSteps, step3) - - // Step 4: Metadata validation - step4 := v.validateMetadata(rawEvent) - details.ValidationSteps = append(details.ValidationSteps, step4) - - // For MVP, calculate a simple score based on completed validation steps - totalSteps := len(details.ValidationSteps) - passedSteps := 0 - for _, step := range details.ValidationSteps { - if step.Passed { - passedSteps++ - } - } - - score := float64(passedSteps) / float64(totalSteps) - isValid := score >= 0.8 // 80% threshold for MVP - - // Add summary to metadata - details.Metadata["total_steps"] = totalSteps - details.Metadata["passed_steps"] = passedSteps - details.Metadata["score"] = score - details.Metadata["threshold"] = 0.8 - - // Serialize details to JSON - detailsJSON, err := json.Marshal(details) - if err != nil { - return nil, fmt.Errorf("failed to marshal validation details: %w", err) - } - - return &models.ValidationResult{ - IsValid: isValid, - Score: score, - Algorithm: v.algorithmName, - Details: detailsJSON, - ProcessedAt: time.Now().UTC(), - Reason: v.generateReason(isValid, passedSteps, totalSteps), - }, nil + return v.provider.Validate(ctx, rawEvent) } // ValidationDetails represents the detailed validation information +// This type is now defined in mvp_provider.go and classic_cv_provider.go +// Kept here for backward compatibility type ValidationDetails struct { Algorithm string `json:"algorithm"` Version string `json:"version"` @@ -100,6 +42,8 @@ type ValidationDetails struct { } // ValidationStep represents a single validation step +// This type is now defined in mvp_provider.go and classic_cv_provider.go +// Kept here for backward compatibility type ValidationStep struct { Name string `json:"name"` Description string `json:"description"` @@ -108,208 +52,3 @@ type ValidationStep struct { Error string `json:"error,omitempty"` } -// validateDataCompleteness checks if required fields are present -func (v *MVPValidator) validateDataCompleteness(rawEvent *models.RawEvent) ValidationStep { - step := ValidationStep{ - Name: "data_completeness", - Description: "Checks if required fields are present and valid", - Details: make(map[string]interface{}), - } - - issues := []string{} - - // Check required UUID fields - if rawEvent.ID == (uuid.UUID{}) { - issues = append(issues, "missing_id") - } - if rawEvent.DeviceID == (uuid.UUID{}) { - issues = append(issues, "missing_device_id") - } - if rawEvent.UserProfileID == (uuid.UUID{}) { - issues = append(issues, "missing_user_profile_id") - } - - // Check required string fields - if rawEvent.FilePath == "" { - issues = append(issues, "missing_file_path") - } - if rawEvent.EventType == "" { - issues = append(issues, "missing_event_type") - } - - // Check timestamp - if rawEvent.EventTimestamp.IsZero() { - issues = append(issues, "missing_event_timestamp") - } - - step.Details["issues"] = issues - step.Details["issues_count"] = len(issues) - step.Passed = len(issues) == 0 - - if len(issues) > 0 { - step.Error = fmt.Sprintf("Found %d data completeness issues", len(issues)) - } - - return step -} - -// validateEventType checks if the event type is supported -func (v *MVPValidator) validateEventType(rawEvent *models.RawEvent) ValidationStep { - step := ValidationStep{ - Name: "event_type_validation", - Description: "Validates that the event type is supported", - Details: make(map[string]interface{}), - } - - supportedTypes := []string{ - models.EventTypeMotion, - models.EventTypeAlert, - models.EventTypeMeteor, - } - - step.Details["event_type"] = rawEvent.EventType - step.Details["supported_types"] = supportedTypes - - // Check if event type is supported - isSupported := false - for _, supportedType := range supportedTypes { - if rawEvent.EventType == supportedType { - isSupported = true - break - } - } - - step.Passed = isSupported - step.Details["is_supported"] = isSupported - - if !isSupported { - step.Error = fmt.Sprintf("Unsupported event type: %s", rawEvent.EventType) - } - - return step -} - -// validateFile checks basic file information -func (v *MVPValidator) validateFile(rawEvent *models.RawEvent) ValidationStep { - step := ValidationStep{ - Name: "file_validation", - Description: "Validates file information and properties", - Details: make(map[string]interface{}), - } - - issues := []string{} - - // Check file path format (basic validation) - if len(rawEvent.FilePath) < 3 { - issues = append(issues, "file_path_too_short") - } - - // Check file size if provided - if rawEvent.FileSize != nil { - step.Details["file_size"] = *rawEvent.FileSize - if *rawEvent.FileSize <= 0 { - issues = append(issues, "invalid_file_size") - } - // Check for reasonable file size limits (e.g., not more than 100MB for video files) - if *rawEvent.FileSize > 100*1024*1024 { - issues = append(issues, "file_size_too_large") - } - } - - // Check file type if provided - if rawEvent.FileType != nil { - step.Details["file_type"] = *rawEvent.FileType - // Basic MIME type validation for common formats - supportedMimeTypes := []string{ - "video/mp4", - "video/quicktime", - "video/x-msvideo", - "image/jpeg", - "image/png", - "application/gzip", - "application/x-tar", - } - - isSupportedMime := false - for _, mimeType := range supportedMimeTypes { - if *rawEvent.FileType == mimeType { - isSupportedMime = true - break - } - } - - if !isSupportedMime { - issues = append(issues, "unsupported_file_type") - } - step.Details["supported_mime_types"] = supportedMimeTypes - } - - step.Details["issues"] = issues - step.Details["issues_count"] = len(issues) - step.Passed = len(issues) == 0 - - if len(issues) > 0 { - step.Error = fmt.Sprintf("Found %d file validation issues", len(issues)) - } - - return step -} - -// validateMetadata performs basic metadata validation -func (v *MVPValidator) validateMetadata(rawEvent *models.RawEvent) ValidationStep { - step := ValidationStep{ - Name: "metadata_validation", - Description: "Validates event metadata structure and content", - Details: make(map[string]interface{}), - } - - issues := []string{} - - // Check if metadata is valid JSON - if rawEvent.Metadata != nil { - var metadata map[string]interface{} - if err := json.Unmarshal(rawEvent.Metadata, &metadata); err != nil { - issues = append(issues, "invalid_json_metadata") - step.Details["json_error"] = err.Error() - } else { - step.Details["metadata_keys"] = getKeys(metadata) - step.Details["metadata_size"] = len(rawEvent.Metadata) - - // Check for reasonable metadata size (not more than 10KB) - if len(rawEvent.Metadata) > 10*1024 { - issues = append(issues, "metadata_too_large") - } - } - } else { - // Metadata is optional, so this is not an error - step.Details["metadata_present"] = false - } - - step.Details["issues"] = issues - step.Details["issues_count"] = len(issues) - step.Passed = len(issues) == 0 - - if len(issues) > 0 { - step.Error = fmt.Sprintf("Found %d metadata validation issues", len(issues)) - } - - return step -} - -// generateReason creates a human-readable reason for the validation result -func (v *MVPValidator) generateReason(isValid bool, passedSteps, totalSteps int) string { - if isValid { - return fmt.Sprintf("Event passed validation with %d/%d steps completed successfully", passedSteps, totalSteps) - } - return fmt.Sprintf("Event failed validation with only %d/%d steps completed successfully (required: 80%%)", passedSteps, totalSteps) -} - -// getKeys extracts keys from a map -func getKeys(m map[string]interface{}) []string { - keys := make([]string, 0, len(m)) - for k := range m { - keys = append(keys, k) - } - return keys -} - diff --git a/meteor-compute-service/meteor-compute-service b/meteor-compute-service/meteor-compute-service new file mode 100755 index 0000000..38bc21d Binary files /dev/null and b/meteor-compute-service/meteor-compute-service differ diff --git a/meteor-edge-client/Cargo.lock b/meteor-edge-client/Cargo.lock index 4fec138..49e675e 100644 --- a/meteor-edge-client/Cargo.lock +++ b/meteor-edge-client/Cargo.lock @@ -17,6 +17,15 @@ version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" +[[package]] +name = "aho-corasick" +version = "1.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916" +dependencies = [ + "memchr", +] + [[package]] name = "android-tzdata" version = "0.1.1" @@ -231,6 +240,39 @@ version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crossbeam-channel" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" + +[[package]] +name = "deranged" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9c9e6a11ca8224451684bc0d7d5a7adbf8f2fd6887261a1cfc3c0432f9d4068e" +dependencies = [ + "powerfmt", +] + [[package]] name = "dirs" version = "5.0.1" @@ -294,6 +336,16 @@ version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" +[[package]] +name = "flate2" +version = "1.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4a3d7db9596fecd151c5f638c0ee5d5bd487b6e0ea232e5dc96d5250f6f94b1d" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + [[package]] name = "fnv" version = "1.0.7" @@ -674,6 +726,12 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + [[package]] name = "libc" version = "0.2.174" @@ -718,6 +776,15 @@ version = "0.4.27" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "13dc2df351e3202783a1fe0d44375f7295ffb4049267b0f3018346dc122a1d94" +[[package]] +name = "matchers" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8263075bb86c5a1b1427b5ae862e8889656f126e9f77c484496e8b47cf5c5558" +dependencies = [ + "regex-automata 0.1.10", +] + [[package]] name = "memchr" version = "2.7.5" @@ -732,6 +799,7 @@ dependencies = [ "chrono", "clap", "dirs", + "flate2", "reqwest", "serde", "serde_json", @@ -739,6 +807,10 @@ dependencies = [ "thiserror", "tokio", "toml", + "tracing", + "tracing-appender", + "tracing-subscriber", + "uuid", ] [[package]] @@ -794,6 +866,22 @@ dependencies = [ "tempfile", ] +[[package]] +name = "nu-ansi-term" +version = "0.46.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77a8165726e8236064dbb45459242600304b42a5ea24ee2948e18e023bf7ba84" +dependencies = [ + "overload", + "winapi", +] + +[[package]] +name = "num-conv" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9" + [[package]] name = "num-traits" version = "0.2.19" @@ -874,6 +962,12 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" +[[package]] +name = "overload" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b15813163c1d831bf4a13c3610c05c0d03b39feb07f7e09fa234dac9b15aaf39" + [[package]] name = "parking_lot" version = "0.12.4" @@ -930,6 +1024,12 @@ dependencies = [ "zerovec", ] +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + [[package]] name = "proc-macro2" version = "1.0.95" @@ -974,6 +1074,50 @@ dependencies = [ "thiserror", ] +[[package]] +name = "regex" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b544ef1b4eac5dc2db33ea63606ae9ffcfac26c1416a2806ae0bf5f56b201191" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata 0.4.9", + "regex-syntax 0.8.5", +] + +[[package]] +name = "regex-automata" +version = "0.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6c230d73fb8d8c1b9c0b3135c5142a8acee3a0558fb8db5cf1cb65f8d7862132" +dependencies = [ + "regex-syntax 0.6.29", +] + +[[package]] +name = "regex-automata" +version = "0.4.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "809e8dc61f6de73b46c85f4c96486310fe304c434cfa43669d7b40f711150908" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax 0.8.5", +] + +[[package]] +name = "regex-syntax" +version = "0.6.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f162c6dd7b008981e4d40210aca20b4bd0f9b60ca9271061b07f78537722f2e1" + +[[package]] +name = "regex-syntax" +version = "0.8.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c" + [[package]] name = "reqwest" version = "0.11.27" @@ -1146,6 +1290,15 @@ dependencies = [ "serde", ] +[[package]] +name = "sharded-slab" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" +dependencies = [ + "lazy_static", +] + [[package]] name = "shlex" version = "1.3.0" @@ -1287,6 +1440,46 @@ dependencies = [ "syn", ] +[[package]] +name = "thread_local" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "time" +version = "0.3.41" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7619e19bc266e0f9c5e6686659d394bc57973859340060a69221e57dbc0c40" +dependencies = [ + "deranged", + "itoa", + "num-conv", + "powerfmt", + "serde", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9e9a38711f559d9e3ce1cdb06dd7c5b8ea546bc90052da6d06bb76da74bb07c" + +[[package]] +name = "time-macros" +version = "0.2.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3526739392ec93fd8b359c8e98514cb3e8e021beb4e5f597b00a0221f8ed8a49" +dependencies = [ + "num-conv", + "time-core", +] + [[package]] name = "tinystr" version = "0.8.1" @@ -1405,9 +1598,33 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "784e0ac535deb450455cbfa28a6f0df145ea1bb7ae51b821cf5e7927fdcfbdd0" dependencies = [ "pin-project-lite", + "tracing-attributes", "tracing-core", ] +[[package]] +name = "tracing-appender" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3566e8ce28cc0a3fe42519fc80e6b4c943cc4c8cef275620eb8dac2d3d4e06cf" +dependencies = [ + "crossbeam-channel", + "thiserror", + "time", + "tracing-subscriber", +] + +[[package]] +name = "tracing-attributes" +version = "0.1.30" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "81383ab64e72a7a8b8e13130c49e3dab29def6d0c7d76a03087b3cf71c5c6903" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "tracing-core" version = "0.1.34" @@ -1415,6 +1632,50 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9d12581f227e93f094d3af2ae690a574abb8a2b9b7a96e7cfe9647b2b617678" dependencies = [ "once_cell", + "valuable", +] + +[[package]] +name = "tracing-log" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" +dependencies = [ + "log", + "once_cell", + "tracing-core", +] + +[[package]] +name = "tracing-serde" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "704b1aeb7be0d0a84fc9828cae51dab5970fee5088f83d1dd7ee6f6246fc6ff1" +dependencies = [ + "serde", + "tracing-core", +] + +[[package]] +name = "tracing-subscriber" +version = "0.3.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8189decb5ac0fa7bc8b96b7cb9b2701d60d48805aca84a238004d665fcc4008" +dependencies = [ + "chrono", + "matchers", + "nu-ansi-term", + "once_cell", + "regex", + "serde", + "serde_json", + "sharded-slab", + "smallvec", + "thread_local", + "tracing", + "tracing-core", + "tracing-log", + "tracing-serde", ] [[package]] @@ -1458,6 +1719,23 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" +[[package]] +name = "uuid" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3cf4199d1e5d15ddd86a694e4d0dffa9c323ce759fea589f00fef9d81cc1931d" +dependencies = [ + "getrandom 0.3.3", + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "valuable" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" + [[package]] name = "vcpkg" version = "0.2.15" @@ -1569,6 +1847,28 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "winapi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" +dependencies = [ + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", +] + +[[package]] +name = "winapi-i686-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" + +[[package]] +name = "winapi-x86_64-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" + [[package]] name = "windows-core" version = "0.61.2" diff --git a/meteor-edge-client/Cargo.toml b/meteor-edge-client/Cargo.toml index 6bc42c9..1f6f697 100644 --- a/meteor-edge-client/Cargo.toml +++ b/meteor-edge-client/Cargo.toml @@ -14,6 +14,11 @@ anyhow = "1.0" thiserror = "1.0" dirs = "5.0" chrono = { version = "0.4", features = ["serde"] } +tracing = "0.1" +tracing-subscriber = { version = "0.3", features = ["json", "chrono", "env-filter"] } +tracing-appender = "0.2" +uuid = { version = "1.0", features = ["v4"] } +flate2 = "1.0" # opencv = { version = "0.88", default-features = false } # Commented out for demo - requires system OpenCV installation [dev-dependencies] diff --git a/meteor-edge-client/src/config.rs b/meteor-edge-client/src/config.rs index 4f541ae..77a234b 100644 --- a/meteor-edge-client/src/config.rs +++ b/meteor-edge-client/src/config.rs @@ -19,8 +19,15 @@ pub struct Config { /// The user profile ID this device is registered to pub user_profile_id: Option, /// Device ID returned from the registration API - pub device_id: Option, + pub device_id: String, /// JWT token for authentication with backend services + pub auth_token: Option, + /// Backend API base URL + pub backend_url: String, + /// Log upload interval in hours + pub log_upload_interval_hours: Option, + /// JWT token (backward compatibility) + #[serde(alias = "jwt_token")] pub jwt_token: Option, } @@ -32,7 +39,10 @@ impl Config { hardware_id, registered_at: None, user_profile_id: None, - device_id: None, + device_id: "unknown".to_string(), + auth_token: None, + backend_url: "http://localhost:3000".to_string(), + log_upload_interval_hours: Some(1), jwt_token: None, } } @@ -41,7 +51,8 @@ impl Config { pub fn mark_registered(&mut self, user_profile_id: String, device_id: String, jwt_token: String) { self.registered = true; self.user_profile_id = Some(user_profile_id); - self.device_id = Some(device_id); + self.device_id = device_id; + self.auth_token = Some(jwt_token.clone()); self.jwt_token = Some(jwt_token); self.registered_at = Some( chrono::Utc::now().to_rfc3339() @@ -360,7 +371,7 @@ mod tests { assert!(!config.registered); assert_eq!(config.hardware_id, "TEST_DEVICE_123"); assert!(config.user_profile_id.is_none()); - assert!(config.device_id.is_none()); + assert_eq!(config.device_id, "unknown"); } #[test] @@ -370,7 +381,7 @@ mod tests { assert!(config.registered); assert_eq!(config.user_profile_id.as_ref().unwrap(), "user-456"); - assert_eq!(config.device_id.as_ref().unwrap(), "device-789"); + assert_eq!(config.device_id, "device-789"); assert_eq!(config.jwt_token.as_ref().unwrap(), "test-jwt-token"); assert!(config.registered_at.is_some()); } @@ -392,7 +403,7 @@ mod tests { assert!(loaded_config.registered); assert_eq!(loaded_config.hardware_id, "TEST_DEVICE_456"); assert_eq!(loaded_config.user_profile_id.as_ref().unwrap(), "user-123"); - assert_eq!(loaded_config.device_id.as_ref().unwrap(), "device-456"); + assert_eq!(loaded_config.device_id, "device-456"); assert_eq!(loaded_config.jwt_token.as_ref().unwrap(), "test-jwt-456"); Ok(()) diff --git a/meteor-edge-client/src/log_uploader.rs b/meteor-edge-client/src/log_uploader.rs new file mode 100644 index 0000000..fa10a0b --- /dev/null +++ b/meteor-edge-client/src/log_uploader.rs @@ -0,0 +1,400 @@ +use anyhow::{Context, Result}; +use chrono::{DateTime, Utc}; +use reqwest::{multipart, Client}; +use serde::{Deserialize, Serialize}; +use std::path::PathBuf; +use std::time::{Duration, Instant}; +use tokio::{fs, time}; + +use crate::config::Config; +use crate::logging::{LogFileManager, StructuredLogger, generate_correlation_id}; + +/// Configuration for log upload functionality +#[derive(Debug, Clone)] +pub struct LogUploadConfig { + pub backend_url: String, + pub device_id: String, + pub upload_interval_hours: u64, + pub max_retry_attempts: u32, + pub retry_delay_seconds: u64, + pub max_upload_size_mb: u64, + pub auth_token: Option, +} + +impl Default for LogUploadConfig { + fn default() -> Self { + Self { + backend_url: "http://localhost:3000".to_string(), + device_id: "unknown".to_string(), + upload_interval_hours: 1, + max_retry_attempts: 3, + retry_delay_seconds: 300, // 5 minutes + max_upload_size_mb: 50, + auth_token: None, + } + } +} + +/// Response from the log upload endpoint +#[derive(Debug, Serialize, Deserialize)] +pub struct LogUploadResponse { + pub success: bool, + #[serde(rename = "uploadId")] + pub upload_id: String, + #[serde(rename = "processedEntries")] + pub processed_entries: u32, + pub message: String, +} + +/// Log uploader service for batch uploading log files +pub struct LogUploader { + config: LogUploadConfig, + logger: StructuredLogger, + http_client: Client, + log_file_manager: LogFileManager, +} + +impl LogUploader { + pub fn new( + config: LogUploadConfig, + logger: StructuredLogger, + log_directory: PathBuf, + ) -> Self { + let http_client = Client::builder() + .timeout(Duration::from_secs(300)) // 5 minute timeout + .build() + .expect("Failed to create HTTP client"); + + let log_file_manager = LogFileManager::new(log_directory); + + Self { + config, + logger, + http_client, + log_file_manager, + } + } + + /// Start the log upload background task + pub async fn start_upload_task(self) -> Result<()> { + let correlation_id = generate_correlation_id(); + + self.logger.startup_event( + "log_uploader", + "1.0.0", + Some(&correlation_id) + ); + + self.logger.info( + &format!( + "Starting log upload task with interval: {} hours", + self.config.upload_interval_hours + ), + Some(&correlation_id) + ); + + let mut interval = time::interval(Duration::from_secs( + self.config.upload_interval_hours * 3600 + )); + + loop { + interval.tick().await; + + let upload_correlation_id = generate_correlation_id(); + + self.logger.info( + "Starting scheduled log upload", + Some(&upload_correlation_id) + ); + + match self.upload_logs(&upload_correlation_id).await { + Ok(uploaded_count) => { + self.logger.info( + &format!("Log upload completed successfully: {} files uploaded", uploaded_count), + Some(&upload_correlation_id) + ); + } + Err(e) => { + self.logger.error( + "Log upload failed", + Some(&*e), + Some(&upload_correlation_id) + ); + } + } + + // Clean up old logs to prevent disk space issues + if let Err(e) = self.cleanup_old_logs(&upload_correlation_id).await { + self.logger.warn( + &format!("Failed to cleanup old logs: {}", e), + Some(&upload_correlation_id) + ); + } + } + } + + /// Upload all eligible log files + async fn upload_logs(&self, correlation_id: &str) -> Result { + let uploadable_files = self.log_file_manager.get_uploadable_log_files().await + .context("Failed to get uploadable log files")?; + + if uploadable_files.is_empty() { + self.logger.debug("No log files ready for upload", Some(correlation_id)); + return Ok(0); + } + + self.logger.info( + &format!("Found {} log files ready for upload", uploadable_files.len()), + Some(correlation_id) + ); + + let mut uploaded_count = 0; + + for file_path in uploadable_files { + match self.upload_single_file(&file_path, correlation_id).await { + Ok(_) => { + uploaded_count += 1; + + // Remove the original file after successful upload + if let Err(e) = self.log_file_manager.remove_log_file(&file_path).await { + self.logger.warn( + &format!("Failed to remove uploaded log file {}: {}", file_path.display(), e), + Some(correlation_id) + ); + } else { + self.logger.debug( + &format!("Removed uploaded log file: {}", file_path.display()), + Some(correlation_id) + ); + } + } + Err(e) => { + self.logger.error( + &format!("Failed to upload log file {}: {}", file_path.display(), e), + Some(&*e), + Some(correlation_id) + ); + // Continue with other files even if one fails + } + } + } + + Ok(uploaded_count) + } + + /// Upload a single log file with retry logic + async fn upload_single_file(&self, file_path: &PathBuf, correlation_id: &str) -> Result { + let mut last_error = None; + + for attempt in 1..=self.config.max_retry_attempts { + self.logger.debug( + &format!("Uploading log file {} (attempt {}/{})", file_path.display(), attempt, self.config.max_retry_attempts), + Some(correlation_id) + ); + + match self.perform_upload(file_path, correlation_id).await { + Ok(response) => { + self.logger.info( + &format!( + "Successfully uploaded log file: {} (upload_id: {}, processed_entries: {})", + file_path.display(), + response.upload_id, + response.processed_entries + ), + Some(correlation_id) + ); + return Ok(response); + } + Err(e) => { + last_error = Some(e); + + if attempt < self.config.max_retry_attempts { + self.logger.warn( + &format!( + "Upload attempt {} failed for {}, retrying in {} seconds", + attempt, + file_path.display(), + self.config.retry_delay_seconds + ), + Some(correlation_id) + ); + + time::sleep(Duration::from_secs(self.config.retry_delay_seconds)).await; + } + } + } + } + + Err(last_error.unwrap_or_else(|| anyhow::anyhow!("Upload failed after all retry attempts"))) + } + + /// Perform the actual HTTP upload + async fn perform_upload(&self, file_path: &PathBuf, correlation_id: &str) -> Result { + let start_time = Instant::now(); + + // Check file size + let metadata = std::fs::metadata(file_path) + .context("Failed to get file metadata")?; + + let file_size_mb = metadata.len() / (1024 * 1024); + if file_size_mb > self.config.max_upload_size_mb { + return Err(anyhow::anyhow!( + "File too large: {}MB > {}MB limit", + file_size_mb, + self.config.max_upload_size_mb + )); + } + + // Compress the log file + let compressed_path = self.log_file_manager.compress_log_file(file_path).await + .context("Failed to compress log file")?; + + // Ensure compressed file is cleaned up + let _cleanup_guard = FileCleanupGuard::new(compressed_path.clone()); + + // Read compressed file + let file_content = fs::read(&compressed_path).await + .context("Failed to read compressed log file")?; + + // Create multipart form + let filename = compressed_path.file_name() + .and_then(|n| n.to_str()) + .unwrap_or("log.gz") + .to_string(); + + let part = multipart::Part::bytes(file_content) + .file_name(filename) + .mime_str("application/gzip")?; + + let form = multipart::Form::new() + .part("logFile", part) + .text("deviceId", self.config.device_id.clone()) + .text("source", "edge_client") + .text("description", format!("Automated upload from {}", file_path.display())); + + // Prepare request + let url = format!("{}/api/v1/logs/upload", self.config.backend_url); + let mut request_builder = self.http_client + .post(&url) + .header("x-correlation-id", correlation_id) + .multipart(form); + + // Add authentication if available + if let Some(ref token) = self.config.auth_token { + request_builder = request_builder.bearer_auth(token); + } + + // Send request + let response = request_builder.send().await + .context("Failed to send upload request")?; + + let status = response.status(); + let duration = start_time.elapsed(); + + self.logger.communication_event( + "log_upload", + &url, + Some(status.as_u16()), + Some(correlation_id) + ); + + self.logger.performance_event( + "log_upload", + duration.as_millis() as u64, + Some(correlation_id) + ); + + if !status.is_success() { + let error_text = response.text().await.unwrap_or_else(|_| "Unknown error".to_string()); + return Err(anyhow::anyhow!( + "Upload failed with status {}: {}", + status, + error_text + )); + } + + // Parse response + let upload_response: LogUploadResponse = response.json().await + .context("Failed to parse upload response")?; + + Ok(upload_response) + } + + /// Clean up old log files to prevent disk space issues + async fn cleanup_old_logs(&self, correlation_id: &str) -> Result<()> { + let max_total_size = 500 * 1024 * 1024; // 500MB max total log storage + + let total_size_before = self.log_file_manager.get_total_log_size().await?; + + if total_size_before > max_total_size { + self.logger.info( + &format!( + "Log directory size ({} bytes) exceeds limit ({} bytes), cleaning up old logs", + total_size_before, + max_total_size + ), + Some(correlation_id) + ); + + self.log_file_manager.cleanup_old_logs(max_total_size).await?; + + let total_size_after = self.log_file_manager.get_total_log_size().await?; + + self.logger.info( + &format!( + "Log cleanup completed: {} bytes -> {} bytes", + total_size_before, + total_size_after + ), + Some(correlation_id) + ); + } + + Ok(()) + } + + /// Update authentication token + pub fn update_auth_token(&mut self, token: Option) { + self.config.auth_token = token; + } +} + +/// RAII guard to ensure file cleanup +struct FileCleanupGuard { + file_path: PathBuf, +} + +impl FileCleanupGuard { + fn new(file_path: PathBuf) -> Self { + Self { file_path } + } +} + +impl Drop for FileCleanupGuard { + fn drop(&mut self) { + if self.file_path.exists() { + if let Err(e) = std::fs::remove_file(&self.file_path) { + eprintln!("Failed to cleanup temporary file {}: {}", self.file_path.display(), e); + } + } + } +} + +/// Create log uploader from configuration +pub fn create_log_uploader( + config: &Config, + logger: StructuredLogger, + log_directory: PathBuf, +) -> LogUploader { + let upload_config = LogUploadConfig { + backend_url: config.backend_url.clone(), + device_id: config.device_id.clone(), + upload_interval_hours: config.log_upload_interval_hours.unwrap_or(1), + max_retry_attempts: 3, + retry_delay_seconds: 300, + max_upload_size_mb: 50, + auth_token: config.auth_token.clone(), + }; + + LogUploader::new(upload_config, logger, log_directory) +} \ No newline at end of file diff --git a/meteor-edge-client/src/logging.rs b/meteor-edge-client/src/logging.rs new file mode 100644 index 0000000..66fd5ba --- /dev/null +++ b/meteor-edge-client/src/logging.rs @@ -0,0 +1,443 @@ +use anyhow::Result; +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use std::path::PathBuf; +use tokio::fs; +use tracing::{info, warn, error, debug}; +use tracing_appender::rolling::{RollingFileAppender, Rotation}; +use tracing_subscriber::{fmt, layer::SubscriberExt, util::SubscriberInitExt, EnvFilter, Registry, Layer}; +use uuid::Uuid; + +/// Standardized log entry structure that matches backend services +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct LogEntry { + pub timestamp: DateTime, + pub level: String, + pub service_name: String, + pub correlation_id: Option, + pub message: String, + #[serde(flatten)] + pub fields: serde_json::Map, +} + +/// Configuration for the logging system +#[derive(Debug, Clone)] +pub struct LoggingConfig { + pub log_directory: PathBuf, + pub service_name: String, + pub device_id: String, + pub max_file_size: u64, + pub rotation: Rotation, + pub log_level: String, +} + +impl Default for LoggingConfig { + fn default() -> Self { + let log_dir = dirs::data_local_dir() + .unwrap_or_else(|| PathBuf::from(".")) + .join("meteor-edge-client") + .join("logs"); + + Self { + log_directory: log_dir, + service_name: "meteor-edge-client".to_string(), + device_id: "unknown".to_string(), + max_file_size: 50 * 1024 * 1024, // 50MB + rotation: Rotation::HOURLY, + log_level: "info".to_string(), + } + } +} + +/// Custom JSON formatter for structured logging +struct JsonFormatter { + service_name: String, + device_id: String, +} + +impl JsonFormatter { + fn new(service_name: String, device_id: String) -> Self { + Self { + service_name, + device_id, + } + } +} + +/// Initialize the structured logging system +pub async fn init_logging(config: LoggingConfig) -> Result<()> { + // Ensure log directory exists + fs::create_dir_all(&config.log_directory).await?; + + // Create rolling file appender + let file_appender = RollingFileAppender::new( + config.rotation, + &config.log_directory, + "meteor-edge-client.log", + ); + + // Create JSON layer for file output + let file_layer = fmt::layer() + .json() + .with_current_span(false) + .with_span_list(false) + .with_writer(file_appender) + .with_filter(EnvFilter::try_new(&config.log_level).unwrap_or_else(|_| EnvFilter::new("info"))); + + // Create console layer for development + let console_layer = fmt::layer() + .pretty() + .with_writer(std::io::stderr) + .with_filter(EnvFilter::try_new("debug").unwrap_or_else(|_| EnvFilter::new("info"))); + + // Initialize the subscriber + Registry::default() + .with(file_layer) + .with(console_layer) + .init(); + + info!( + service_name = %config.service_name, + device_id = %config.device_id, + log_directory = %config.log_directory.display(), + "Structured logging initialized" + ); + + Ok(()) +} + +/// Structured logger for the edge client +#[derive(Clone)] +pub struct StructuredLogger { + service_name: String, + device_id: String, +} + +impl StructuredLogger { + pub fn new(service_name: String, device_id: String) -> Self { + Self { + service_name, + device_id, + } + } + + /// Log an info message with structured fields + pub fn info(&self, message: &str, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + "{}", + message + ); + } + + /// Log a warning message with structured fields + pub fn warn(&self, message: &str, correlation_id: Option<&str>) { + warn!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + "{}", + message + ); + } + + /// Log an error message with structured fields + pub fn error(&self, message: &str, error: Option<&dyn std::error::Error>, correlation_id: Option<&str>) { + error!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + error = error.map(|e| e.to_string()).as_deref(), + "{}", + message + ); + } + + /// Log a debug message with structured fields + pub fn debug(&self, message: &str, correlation_id: Option<&str>) { + debug!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + "{}", + message + ); + } + + /// Log camera-related events + pub fn camera_event(&self, event: &str, camera_id: &str, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + camera_id = camera_id, + camera_event = event, + "Camera event: {}", + event + ); + } + + /// Log detection-related events + pub fn detection_event(&self, detection_type: &str, confidence: f64, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + detection_type = detection_type, + confidence = confidence, + "Detection event: {} (confidence: {:.2})", + detection_type, + confidence + ); + } + + /// Log storage-related events + pub fn storage_event(&self, operation: &str, file_path: &str, file_size: Option, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + storage_operation = operation, + file_path = file_path, + file_size = file_size, + "Storage event: {}", + operation + ); + } + + /// Log communication-related events + pub fn communication_event(&self, operation: &str, endpoint: &str, status_code: Option, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + communication_operation = operation, + endpoint = endpoint, + status_code = status_code, + "Communication event: {}", + operation + ); + } + + /// Log hardware-related events + pub fn hardware_event(&self, component: &str, event: &str, temperature: Option, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + hardware_component = component, + hardware_event = event, + temperature = temperature, + "Hardware event: {} - {}", + component, + event + ); + } + + /// Log configuration-related events + pub fn config_event(&self, operation: &str, config_key: &str, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + config_operation = operation, + config_key = config_key, + "Configuration event: {}", + operation + ); + } + + /// Log startup events + pub fn startup_event(&self, component: &str, version: &str, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + startup_component = component, + version = version, + "Component started: {} v{}", + component, + version + ); + } + + /// Log shutdown events + pub fn shutdown_event(&self, component: &str, reason: &str, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + shutdown_component = component, + shutdown_reason = reason, + "Component shutdown: {} - {}", + component, + reason + ); + } + + /// Log performance metrics + pub fn performance_event(&self, operation: &str, duration_ms: u64, correlation_id: Option<&str>) { + info!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + performance_operation = operation, + duration_ms = duration_ms, + "Performance: {} completed in {}ms", + operation, + duration_ms + ); + } + + /// Log security-related events + pub fn security_event(&self, event: &str, severity: &str, correlation_id: Option<&str>) { + warn!( + service_name = %self.service_name, + device_id = %self.device_id, + correlation_id = correlation_id, + security_event = event, + severity = severity, + "Security event: {} (severity: {})", + event, + severity + ); + } +} + +/// Utility functions for log file management +pub struct LogFileManager { + log_directory: PathBuf, +} + +impl LogFileManager { + pub fn new(log_directory: PathBuf) -> Self { + Self { log_directory } + } + + /// Get all log files in the directory + pub async fn get_log_files(&self) -> Result> { + let mut log_files = Vec::new(); + let mut entries = fs::read_dir(&self.log_directory).await?; + + while let Some(entry) = entries.next_entry().await? { + let path = entry.path(); + if path.is_file() { + if let Some(extension) = path.extension() { + if extension == "log" { + log_files.push(path); + } + } + } + } + + // Sort by modification time (oldest first) + log_files.sort_by_key(|path| { + std::fs::metadata(path) + .and_then(|m| m.modified()) + .unwrap_or(std::time::SystemTime::UNIX_EPOCH) + }); + + Ok(log_files) + } + + /// Get log files that are ready for upload (older than current hour) + pub async fn get_uploadable_log_files(&self) -> Result> { + let all_files = self.get_log_files().await?; + let mut uploadable_files = Vec::new(); + + let current_time = std::time::SystemTime::now(); + let one_hour_ago = current_time - std::time::Duration::from_secs(3600); + + for file_path in all_files { + // Skip the current active log file (usually the most recently modified) + if let Ok(metadata) = std::fs::metadata(&file_path) { + if let Ok(modified_time) = metadata.modified() { + // Only upload files that are older than 1 hour + if modified_time < one_hour_ago { + uploadable_files.push(file_path); + } + } + } + } + + Ok(uploadable_files) + } + + /// Compress a log file using gzip + pub async fn compress_log_file(&self, file_path: &PathBuf) -> Result { + use flate2::{write::GzEncoder, Compression}; + use std::io::Write; + + let file_content = fs::read(file_path).await?; + let compressed_path = file_path.with_extension("log.gz"); + + let compressed_data = tokio::task::spawn_blocking(move || -> Result> { + let mut encoder = GzEncoder::new(Vec::new(), Compression::default()); + encoder.write_all(&file_content)?; + Ok(encoder.finish()?) + }).await??; + + fs::write(&compressed_path, compressed_data).await?; + Ok(compressed_path) + } + + /// Remove a log file + pub async fn remove_log_file(&self, file_path: &PathBuf) -> Result<()> { + fs::remove_file(file_path).await?; + Ok(()) + } + + /// Get total size of all log files + pub async fn get_total_log_size(&self) -> Result { + let log_files = self.get_log_files().await?; + let mut total_size = 0; + + for file_path in log_files { + if let Ok(metadata) = std::fs::metadata(&file_path) { + total_size += metadata.len(); + } + } + + Ok(total_size) + } + + /// Clean up old log files if total size exceeds limit + pub async fn cleanup_old_logs(&self, max_total_size: u64) -> Result<()> { + let total_size = self.get_total_log_size().await?; + + if total_size <= max_total_size { + return Ok(()); + } + + let log_files = self.get_log_files().await?; + let mut current_size = total_size; + + // Remove oldest files until we're under the limit + for file_path in log_files { + if current_size <= max_total_size { + break; + } + + if let Ok(metadata) = std::fs::metadata(&file_path) { + let file_size = metadata.len(); + self.remove_log_file(&file_path).await?; + current_size -= file_size; + + debug!( + "Removed old log file: {} (size: {} bytes)", + file_path.display(), + file_size + ); + } + } + + Ok(()) + } +} + +/// Generate a correlation ID for request tracing +pub fn generate_correlation_id() -> String { + Uuid::new_v4().to_string() +} \ No newline at end of file diff --git a/meteor-edge-client/src/main.rs b/meteor-edge-client/src/main.rs index 297e3c8..e0acb1f 100644 --- a/meteor-edge-client/src/main.rs +++ b/meteor-edge-client/src/main.rs @@ -11,11 +11,15 @@ mod detection; mod storage; mod communication; mod integration_test; +mod logging; +mod log_uploader; use hardware::get_hardware_id; use config::{Config, ConfigManager}; use api::ApiClient; use app::Application; +use logging::{init_logging, LoggingConfig, StructuredLogger, generate_correlation_id}; +use log_uploader::create_log_uploader; #[derive(Parser)] #[command(name = "meteor-edge-client")] @@ -97,8 +101,8 @@ async fn register_device(jwt_token: String, api_url: String) -> Result<()> { Ok(config) if config.registered => { println!("✅ Device is already registered!"); println!(" Hardware ID: {}", config.hardware_id); - if let (Some(device_id), Some(user_id)) = (&config.device_id, &config.user_profile_id) { - println!(" Device ID: {}", device_id); + if let Some(user_id) = &config.user_profile_id { + println!(" Device ID: {}", config.device_id); println!(" User Profile ID: {}", user_id); } if let Some(registered_at) = &config.registered_at { @@ -143,7 +147,7 @@ async fn register_device(jwt_token: String, api_url: String) -> Result<()> { config_manager.save_config(&config)?; println!("🎉 Device registration completed successfully!"); - println!(" Device ID: {}", config.device_id.as_ref().unwrap()); + println!(" Device ID: {}", config.device_id); println!(" Config saved to: {:?}", config_manager.get_config_path()); Ok(()) @@ -173,9 +177,7 @@ async fn show_status() -> Result<()> { Ok(config) => { if config.registered { println!("✅ Registration Status: REGISTERED"); - if let Some(device_id) = &config.device_id { - println!(" Device ID: {}", device_id); - } + println!(" Device ID: {}", config.device_id); if let Some(user_id) = &config.user_profile_id { println!(" User Profile ID: {}", user_id); } @@ -213,17 +215,96 @@ async fn check_health(api_url: String) -> Result<()> { /// Run the main event-driven application async fn run_application() -> Result<()> { + // Load configuration first + let config_manager = ConfigManager::new(); + let config = if config_manager.config_exists() { + config_manager.load_config()? + } else { + eprintln!("❌ Device not registered. Use 'register ' command first."); + std::process::exit(1); + }; + + if !config.registered { + eprintln!("❌ Device not registered. Use 'register ' command first."); + std::process::exit(1); + } + + // Initialize structured logging + let logging_config = LoggingConfig { + service_name: "meteor-edge-client".to_string(), + device_id: config.device_id.clone(), + ..LoggingConfig::default() + }; + + init_logging(logging_config.clone()).await?; + + let logger = StructuredLogger::new( + logging_config.service_name.clone(), + logging_config.device_id.clone(), + ); + + let correlation_id = generate_correlation_id(); + + logger.startup_event( + "meteor-edge-client", + env!("CARGO_PKG_VERSION"), + Some(&correlation_id) + ); + println!("🎯 Initializing Event-Driven Meteor Edge Client..."); + // Start log uploader in background + let log_uploader = create_log_uploader(&config, logger.clone(), logging_config.log_directory.clone()); + let uploader_handle = tokio::spawn(async move { + if let Err(e) = log_uploader.start_upload_task().await { + eprintln!("Log uploader error: {}", e); + } + }); + + logger.info("Log uploader started successfully", Some(&correlation_id)); + // Create the application with a reasonable event bus capacity let mut app = Application::new(1000); + logger.info(&format!( + "Application initialized - Event Bus Capacity: 1000, Initial Subscribers: {}", + app.subscriber_count() + ), Some(&correlation_id)); + println!("📊 Application Statistics:"); println!(" Event Bus Capacity: 1000"); println!(" Initial Subscribers: {}", app.subscriber_count()); // Run the application - app.run().await?; + let app_handle = tokio::spawn(async move { + app.run().await + }); + + // Wait for either the application or log uploader to complete + tokio::select! { + result = app_handle => { + match result { + Ok(Ok(())) => { + logger.shutdown_event("meteor-edge-client", "normal", Some(&correlation_id)); + println!("✅ Application completed successfully"); + } + Ok(Err(e)) => { + logger.error("Application failed", Some(&*e), Some(&correlation_id)); + eprintln!("❌ Application failed: {}", e); + return Err(e); + } + Err(e) => { + logger.error("Application task panicked", Some(&e), Some(&correlation_id)); + eprintln!("❌ Application task panicked: {}", e); + return Err(e.into()); + } + } + } + _ = uploader_handle => { + logger.warn("Log uploader task completed unexpectedly", Some(&correlation_id)); + println!("⚠️ Log uploader completed unexpectedly"); + } + } Ok(()) } diff --git a/meteor-web-backend/package.json b/meteor-web-backend/package.json index 89afa69..4444af2 100644 --- a/meteor-web-backend/package.json +++ b/meteor-web-backend/package.json @@ -24,6 +24,7 @@ "migrate:create": "node-pg-migrate create" }, "dependencies": { + "@aws-sdk/client-cloudwatch": "^3.859.0", "@aws-sdk/client-s3": "^3.856.0", "@aws-sdk/client-sqs": "^3.856.0", "@nestjs/common": "^11.0.1", @@ -32,6 +33,7 @@ "@nestjs/passport": "^11.0.5", "@nestjs/platform-express": "^11.1.5", "@nestjs/schedule": "^6.0.0", + "@nestjs/terminus": "^11.0.0", "@nestjs/typeorm": "^11.0.0", "@types/bcrypt": "^6.0.0", "@types/passport-jwt": "^4.0.1", @@ -43,11 +45,15 @@ "class-validator": "^0.14.2", "dotenv": "^17.2.1", "multer": "^2.0.2", + "nestjs-pino": "^4.4.0", "node-pg-migrate": "^8.0.3", "passport": "^0.7.0", "passport-jwt": "^4.0.1", "passport-local": "^1.0.0", "pg": "^8.16.3", + "pino": "^9.7.0", + "pino-http": "^10.5.0", + "prom-client": "^15.1.3", "reflect-metadata": "^0.2.2", "rxjs": "^7.8.1", "stripe": "^18.4.0", diff --git a/meteor-web-backend/src/app.module.ts b/meteor-web-backend/src/app.module.ts index ed713fe..50ec666 100644 --- a/meteor-web-backend/src/app.module.ts +++ b/meteor-web-backend/src/app.module.ts @@ -1,18 +1,26 @@ import * as dotenv from 'dotenv'; -import { Module } from '@nestjs/common'; +import { Module, NestModule, MiddlewareConsumer } from '@nestjs/common'; import { TypeOrmModule } from '@nestjs/typeorm'; import { ScheduleModule } from '@nestjs/schedule'; +import { LoggerModule } from 'nestjs-pino'; import { AppController } from './app.controller'; import { AppService } from './app.service'; import { AuthModule } from './auth/auth.module'; import { DevicesModule } from './devices/devices.module'; import { EventsModule } from './events/events.module'; import { PaymentsModule } from './payments/payments.module'; +import { LogsModule } from './logs/logs.module'; +import { MetricsModule } from './metrics/metrics.module'; import { UserProfile } from './entities/user-profile.entity'; import { UserIdentity } from './entities/user-identity.entity'; import { Device } from './entities/device.entity'; import { InventoryDevice } from './entities/inventory-device.entity'; import { RawEvent } from './entities/raw-event.entity'; +import { ValidatedEvent } from './entities/validated-event.entity'; +import { CorrelationMiddleware } from './logging/correlation.middleware'; +import { MetricsMiddleware } from './metrics/metrics.middleware'; +import { StructuredLogger } from './logging/logger.service'; +import { pinoConfig } from './logging/logging.config'; // Ensure dotenv is loaded before anything else dotenv.config(); @@ -24,16 +32,17 @@ console.log('Current working directory:', process.cwd()); @Module({ imports: [ + LoggerModule.forRoot(pinoConfig), ScheduleModule.forRoot(), TypeOrmModule.forRoot({ type: 'postgres', url: process.env.DATABASE_URL || 'postgresql://user:password@localhost:5432/meteor_dev', - entities: [UserProfile, UserIdentity, Device, InventoryDevice, RawEvent], + entities: [UserProfile, UserIdentity, Device, InventoryDevice, RawEvent, ValidatedEvent], synchronize: false, // Use migrations instead - logging: ['error', 'warn', 'info', 'log'], - logger: 'advanced-console', + logging: ['error', 'warn'], + logger: 'simple-console', // Simplified to avoid conflicts with pino retryAttempts: 3, retryDelay: 3000, }), @@ -41,8 +50,16 @@ console.log('Current working directory:', process.cwd()); DevicesModule, EventsModule, PaymentsModule, + LogsModule, + MetricsModule, ], controllers: [AppController], - providers: [AppService], + providers: [AppService, StructuredLogger], }) -export class AppModule {} +export class AppModule implements NestModule { + configure(consumer: MiddlewareConsumer) { + consumer + .apply(CorrelationMiddleware, MetricsMiddleware) + .forRoutes('*'); // Apply to all routes + } +} diff --git a/meteor-web-backend/src/auth/auth.controller.ts b/meteor-web-backend/src/auth/auth.controller.ts index 1fa5fe3..e6b2685 100644 --- a/meteor-web-backend/src/auth/auth.controller.ts +++ b/meteor-web-backend/src/auth/auth.controller.ts @@ -13,21 +13,39 @@ import { AuthService } from './auth.service'; import { RegisterEmailDto } from './dto/register-email.dto'; import { LoginEmailDto } from './dto/login-email.dto'; import { JwtAuthGuard } from './guards/jwt-auth.guard'; +import { MetricsService } from '../metrics/metrics.service'; @Controller('api/v1/auth') export class AuthController { - constructor(private readonly authService: AuthService) {} + constructor( + private readonly authService: AuthService, + private readonly metricsService: MetricsService, + ) {} @Post('register-email') @HttpCode(HttpStatus.CREATED) async registerWithEmail(@Body(ValidationPipe) registerDto: RegisterEmailDto) { - return await this.authService.registerWithEmail(registerDto); + try { + const result = await this.authService.registerWithEmail(registerDto); + this.metricsService.recordAuthOperation('register', true, 'email'); + return result; + } catch (error) { + this.metricsService.recordAuthOperation('register', false, 'email'); + throw error; + } } @Post('login-email') @HttpCode(HttpStatus.OK) async loginWithEmail(@Body(ValidationPipe) loginDto: LoginEmailDto) { - return await this.authService.loginWithEmail(loginDto); + try { + const result = await this.authService.loginWithEmail(loginDto); + this.metricsService.recordAuthOperation('login', true, 'email'); + return result; + } catch (error) { + this.metricsService.recordAuthOperation('login', false, 'email'); + throw error; + } } @Get('profile') diff --git a/meteor-web-backend/src/logging/correlation.middleware.ts b/meteor-web-backend/src/logging/correlation.middleware.ts new file mode 100644 index 0000000..2031823 --- /dev/null +++ b/meteor-web-backend/src/logging/correlation.middleware.ts @@ -0,0 +1,27 @@ +import { Injectable, NestMiddleware } from '@nestjs/common'; +import { Request, Response, NextFunction } from 'express'; +import { v4 as uuidv4 } from 'uuid'; + +export interface RequestWithCorrelation extends Request { + correlationId: string; +} + +@Injectable() +export class CorrelationMiddleware implements NestMiddleware { + use(req: RequestWithCorrelation, res: Response, next: NextFunction): void { + // Check if correlation ID already exists in headers (from upstream services) + const existingCorrelationId = req.headers['x-correlation-id'] as string; + + // Generate new correlation ID if none exists + const correlationId = existingCorrelationId || uuidv4(); + + // Attach correlation ID to request object + req.correlationId = correlationId; + + // Add correlation ID to response headers for client visibility + res.setHeader('x-correlation-id', correlationId); + + // Continue with the request + next(); + } +} \ No newline at end of file diff --git a/meteor-web-backend/src/logging/logger.service.ts b/meteor-web-backend/src/logging/logger.service.ts new file mode 100644 index 0000000..33bc87b --- /dev/null +++ b/meteor-web-backend/src/logging/logger.service.ts @@ -0,0 +1,123 @@ +import { Injectable, Scope } from '@nestjs/common'; +import { PinoLogger, InjectPinoLogger } from 'nestjs-pino'; + +export interface LogEntry { + timestamp?: string; + level: string; + service_name: string; + correlation_id?: string | null; + message: string; + [key: string]: any; +} + +@Injectable({ scope: Scope.TRANSIENT }) +export class StructuredLogger { + constructor( + @InjectPinoLogger() private readonly logger: PinoLogger, + ) {} + + private createLogEntry( + level: string, + message: string, + meta: Record = {}, + correlationId?: string, + ): LogEntry { + return { + timestamp: new Date().toISOString(), + level, + service_name: 'meteor-web-backend', + correlation_id: correlationId || null, + message, + ...meta, + }; + } + + info(message: string, meta: Record = {}, correlationId?: string): void { + const logEntry = this.createLogEntry('info', message, meta, correlationId); + this.logger.info(logEntry); + } + + warn(message: string, meta: Record = {}, correlationId?: string): void { + const logEntry = this.createLogEntry('warn', message, meta, correlationId); + this.logger.warn(logEntry); + } + + error(message: string, error?: Error, meta: Record = {}, correlationId?: string): void { + const errorMeta = error + ? { + error: { + name: error.name, + message: error.message, + stack: process.env.NODE_ENV === 'development' ? error.stack : undefined, + }, + ...meta, + } + : meta; + + const logEntry = this.createLogEntry('error', message, errorMeta, correlationId); + this.logger.error(logEntry); + } + + debug(message: string, meta: Record = {}, correlationId?: string): void { + const logEntry = this.createLogEntry('debug', message, meta, correlationId); + this.logger.debug(logEntry); + } + + // Business logic specific log methods + + userAction(action: string, userId: string, details: Record = {}, correlationId?: string): void { + this.info(`User action: ${action}`, { + user_id: userId, + action, + ...details, + }, correlationId); + } + + deviceAction(action: string, deviceId: string, details: Record = {}, correlationId?: string): void { + this.info(`Device action: ${action}`, { + device_id: deviceId, + action, + ...details, + }, correlationId); + } + + eventProcessing(eventId: string, stage: string, details: Record = {}, correlationId?: string): void { + this.info(`Event processing: ${stage}`, { + event_id: eventId, + processing_stage: stage, + ...details, + }, correlationId); + } + + apiRequest(method: string, path: string, statusCode: number, duration: number, correlationId?: string): void { + this.info('API request completed', { + http_method: method, + http_path: path, + http_status_code: statusCode, + response_time_ms: duration, + }, correlationId); + } + + databaseQuery(query: string, duration: number, correlationId?: string): void { + this.debug('Database query executed', { + query_type: query, + query_duration_ms: duration, + }, correlationId); + } + + // Security-related logging + authEvent(event: string, userId?: string, details: Record = {}, correlationId?: string): void { + this.info(`Authentication event: ${event}`, { + auth_event: event, + user_id: userId, + ...details, + }, correlationId); + } + + securityAlert(alert: string, details: Record = {}, correlationId?: string): void { + this.warn(`Security alert: ${alert}`, { + security_alert: alert, + ...details, + }, correlationId); + } +} \ No newline at end of file diff --git a/meteor-web-backend/src/logging/logging.config.ts b/meteor-web-backend/src/logging/logging.config.ts new file mode 100644 index 0000000..2286ca1 --- /dev/null +++ b/meteor-web-backend/src/logging/logging.config.ts @@ -0,0 +1,76 @@ +import { Params } from 'nestjs-pino'; + +export const pinoConfig: Params = { + pinoHttp: { + level: process.env.LOG_LEVEL || 'info', + transport: + process.env.NODE_ENV === 'development' + ? { + target: 'pino-pretty', + options: { + colorize: true, + singleLine: true, + translateTime: 'SYS:standard', + }, + } + : undefined, + formatters: { + log: (object: any) => { + return { + timestamp: new Date().toISOString(), + level: object.level, + service_name: 'meteor-web-backend', + correlation_id: object.req?.correlationId || null, + message: object.msg || object.message, + ...object, + }; + }, + }, + customLogLevel: function (req, res, err) { + if (res.statusCode >= 400 && res.statusCode < 500) { + return 'warn'; + } else if (res.statusCode >= 500 || err) { + return 'error'; + } + return 'info'; + }, + customSuccessMessage: function (req, res) { + if (res.statusCode === 404) { + return 'resource not found'; + } + return `${req.method} ${req.url}`; + }, + customErrorMessage: function (req, res, err) { + return `${req.method} ${req.url} - ${err.message}`; + }, + autoLogging: { + ignore: (req) => { + // Skip logging for health check endpoints + return req.url === '/health' || req.url === '/'; + }, + }, + serializers: { + req: (req) => ({ + method: req.method, + url: req.url, + headers: { + 'user-agent': req.headers['user-agent'], + 'content-type': req.headers['content-type'], + authorization: req.headers.authorization ? '[REDACTED]' : undefined, + }, + correlationId: req.correlationId, + }), + res: (res) => ({ + statusCode: res.statusCode, + headers: { + 'content-type': res.headers['content-type'], + }, + }), + err: (err) => ({ + type: err.constructor.name, + message: err.message, + stack: process.env.NODE_ENV === 'development' ? err.stack : undefined, + }), + }, + }, +}; \ No newline at end of file diff --git a/meteor-web-backend/src/main.ts b/meteor-web-backend/src/main.ts index eb4c9b3..32c4db4 100644 --- a/meteor-web-backend/src/main.ts +++ b/meteor-web-backend/src/main.ts @@ -1,6 +1,7 @@ import * as dotenv from 'dotenv'; import { NestFactory } from '@nestjs/core'; import { ValidationPipe } from '@nestjs/common'; +import { Logger } from 'nestjs-pino'; import { AppModule } from './app.module'; import { json } from 'express'; @@ -9,10 +10,19 @@ dotenv.config(); async function bootstrap() { try { - console.log('=== Starting Meteor Backend ==='); - console.log('Loading .env from:', process.cwd()); - - const app = await NestFactory.create(AppModule); + const app = await NestFactory.create(AppModule, { bufferLogs: true }); + + // Use pino logger for the entire application + app.useLogger(app.get(Logger)); + + const logger = app.get(Logger); + + logger.log({ + message: 'Starting Meteor Backend', + service_name: 'meteor-web-backend', + env: process.env.NODE_ENV, + cwd: process.cwd(), + }); // Configure raw body parsing for webhook endpoints app.use( @@ -40,18 +50,45 @@ async function bootstrap() { const port = process.env.PORT ?? 3000; await app.listen(port); - console.log(`🚀 Application is running on: http://localhost:${port}`); + + logger.log({ + message: 'Application started successfully', + service_name: 'meteor-web-backend', + port: port, + url: `http://localhost:${port}`, + }); } catch (error) { - console.error('❌ Failed to start application:', error); + // Fallback to console if logger is not available + const errorLogger = console; + + errorLogger.error(JSON.stringify({ + timestamp: new Date().toISOString(), + level: 'error', + service_name: 'meteor-web-backend', + message: 'Failed to start application', + error: { + name: error.name, + message: error.message, + stack: error.stack, + }, + })); + if ( error.message.includes('database') || error.message.includes('connection') ) { - console.error('🔍 Database connection error detected. Please check:'); - console.error('1. Database server is running'); - console.error('2. DATABASE_URL in .env is correct'); - console.error('3. Database credentials are valid'); - console.error('4. Network connectivity to database'); + errorLogger.error(JSON.stringify({ + timestamp: new Date().toISOString(), + level: 'error', + service_name: 'meteor-web-backend', + message: 'Database connection error detected', + troubleshooting: [ + 'Database server is running', + 'DATABASE_URL in .env is correct', + 'Database credentials are valid', + 'Network connectivity to database', + ], + })); } process.exit(1); } diff --git a/meteor-web-backend/src/metrics/metrics.controller.ts b/meteor-web-backend/src/metrics/metrics.controller.ts new file mode 100644 index 0000000..72e0fe6 --- /dev/null +++ b/meteor-web-backend/src/metrics/metrics.controller.ts @@ -0,0 +1,13 @@ +import { Controller, Get, Header } from '@nestjs/common'; +import { MetricsService } from './metrics.service'; + +@Controller('metrics') +export class MetricsController { + constructor(private readonly metricsService: MetricsService) {} + + @Get() + @Header('Content-Type', 'text/plain') + async getMetrics(): Promise { + return this.metricsService.getPrometheusMetrics(); + } +} \ No newline at end of file diff --git a/meteor-web-backend/src/metrics/metrics.middleware.ts b/meteor-web-backend/src/metrics/metrics.middleware.ts new file mode 100644 index 0000000..b394d8f --- /dev/null +++ b/meteor-web-backend/src/metrics/metrics.middleware.ts @@ -0,0 +1,82 @@ +import { Injectable, NestMiddleware } from '@nestjs/common'; +import { Request, Response, NextFunction } from 'express'; +import { MetricsService } from './metrics.service'; + +@Injectable() +export class MetricsMiddleware implements NestMiddleware { + constructor(private readonly metricsService: MetricsService) {} + + use(req: Request, res: Response, next: NextFunction): void { + const startTime = Date.now(); + + // Increment active connections + this.metricsService.incrementActiveConnections(); + + // Hook into response finish event + res.on('finish', () => { + const duration = Date.now() - startTime; + const route = this.extractRoute(req); + const endpoint = this.extractEndpoint(req); + + // Record metrics + this.metricsService.recordHttpRequest( + req.method, + route, + res.statusCode, + duration, + endpoint, + ); + + // Decrement active connections + this.metricsService.decrementActiveConnections(); + }); + + next(); + } + + /** + * Extract a normalized route pattern from the request + */ + private extractRoute(req: Request): string { + // Try to get the route from Express route info + if (req.route?.path) { + return req.route.path; + } + + // Fallback to path normalization + const path = req.path || req.url; + + // Normalize common patterns + const normalizedPath = path + // Replace UUIDs with :id + .replace(/\/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, '/:id') + // Replace numeric IDs with :id + .replace(/\/\d+/g, '/:id') + // Replace other potential ID patterns + .replace(/\/[a-zA-Z0-9_-]{20,}/g, '/:id'); + + return normalizedPath; + } + + /** + * Extract endpoint name for better categorization + */ + private extractEndpoint(req: Request): string { + const path = req.path || req.url; + + // Extract the main endpoint category + const pathParts = path.split('/').filter(part => part.length > 0); + + if (pathParts.length === 0) { + return 'root'; + } + + // For API paths like /api/v1/users, return 'users' + if (pathParts[0] === 'api' && pathParts.length > 2) { + return pathParts[2] || 'unknown'; + } + + // For other paths, return the first meaningful part + return pathParts[0] || 'unknown'; + } +} \ No newline at end of file diff --git a/meteor-web-backend/src/metrics/metrics.module.ts b/meteor-web-backend/src/metrics/metrics.module.ts new file mode 100644 index 0000000..77eb702 --- /dev/null +++ b/meteor-web-backend/src/metrics/metrics.module.ts @@ -0,0 +1,10 @@ +import { Module } from '@nestjs/common'; +import { MetricsService } from './metrics.service'; +import { MetricsController } from './metrics.controller'; + +@Module({ + providers: [MetricsService], + controllers: [MetricsController], + exports: [MetricsService], +}) +export class MetricsModule {} \ No newline at end of file diff --git a/meteor-web-backend/src/metrics/metrics.service.ts b/meteor-web-backend/src/metrics/metrics.service.ts new file mode 100644 index 0000000..4adbfa2 --- /dev/null +++ b/meteor-web-backend/src/metrics/metrics.service.ts @@ -0,0 +1,285 @@ +import { Injectable, Logger } from '@nestjs/common'; +import { CloudWatchClient, PutMetricDataCommand, StandardUnit } from '@aws-sdk/client-cloudwatch'; +import { register, Counter, Histogram, Gauge } from 'prom-client'; + +@Injectable() +export class MetricsService { + private readonly logger = new Logger(MetricsService.name); + private readonly cloudWatch: CloudWatchClient; + + // Prometheus metrics + private readonly httpRequestsTotal: Counter; + private readonly httpRequestDuration: Histogram; + private readonly httpActiveConnections: Gauge; + + constructor() { + this.cloudWatch = new CloudWatchClient({ + region: process.env.AWS_REGION || 'us-east-1', + }); + + // Initialize HTTP request counter + this.httpRequestsTotal = new Counter({ + name: 'http_requests_total', + help: 'Total number of HTTP requests', + labelNames: ['method', 'route', 'status_code', 'endpoint'], + registers: [register], + }); + + // Initialize HTTP request duration histogram + this.httpRequestDuration = new Histogram({ + name: 'http_request_duration_seconds', + help: 'Duration of HTTP requests in seconds', + labelNames: ['method', 'route', 'status_code', 'endpoint'], + buckets: [0.01, 0.05, 0.1, 0.5, 1, 2.5, 5, 10], + registers: [register], + }); + + // Initialize active connections gauge + this.httpActiveConnections = new Gauge({ + name: 'http_active_connections', + help: 'Number of active HTTP connections', + registers: [register], + }); + } + + /** + * Record HTTP request metrics + */ + recordHttpRequest( + method: string, + route: string, + statusCode: number, + duration: number, + endpoint?: string, + ): void { + const labels = { + method: method.toUpperCase(), + route, + status_code: statusCode.toString(), + endpoint: endpoint || route, + }; + + // Update Prometheus metrics + this.httpRequestsTotal.inc(labels); + this.httpRequestDuration.observe(labels, duration / 1000); // Convert ms to seconds + + // Send to CloudWatch asynchronously + this.sendHttpMetricsToCloudWatch(method, route, statusCode, duration, endpoint) + .catch(error => { + this.logger.error('Failed to send HTTP metrics to CloudWatch', error); + }); + } + + /** + * Increment active connections + */ + incrementActiveConnections(): void { + this.httpActiveConnections.inc(); + } + + /** + * Decrement active connections + */ + decrementActiveConnections(): void { + this.httpActiveConnections.dec(); + } + + /** + * Record custom business metric + */ + recordCustomMetric( + metricName: string, + value: number, + unit: StandardUnit = StandardUnit.Count, + dimensions?: Record, + ): void { + this.sendCustomMetricToCloudWatch(metricName, value, unit, dimensions) + .catch(error => { + this.logger.error(`Failed to send custom metric ${metricName} to CloudWatch`, error); + }); + } + + /** + * Send HTTP metrics to CloudWatch + */ + private async sendHttpMetricsToCloudWatch( + method: string, + route: string, + statusCode: number, + duration: number, + endpoint?: string, + ): Promise { + const timestamp = new Date(); + const namespace = 'MeteorApp/WebBackend'; + + const dimensions = [ + { Name: 'Method', Value: method.toUpperCase() }, + { Name: 'Route', Value: route }, + { Name: 'StatusCode', Value: statusCode.toString() }, + ]; + + if (endpoint) { + dimensions.push({ Name: 'Endpoint', Value: endpoint }); + } + + const metricData = [ + // Request count metric + { + MetricName: 'RequestCount', + Value: 1, + Unit: StandardUnit.Count, + Timestamp: timestamp, + Dimensions: dimensions, + }, + // Request duration metric + { + MetricName: 'RequestDuration', + Value: duration, + Unit: StandardUnit.Milliseconds, + Timestamp: timestamp, + Dimensions: dimensions, + }, + ]; + + // Add error rate metric for non-2xx responses + if (statusCode >= 400) { + metricData.push({ + MetricName: 'ErrorCount', + Value: 1, + Unit: StandardUnit.Count, + Timestamp: timestamp, + Dimensions: dimensions, + }); + } + + const command = new PutMetricDataCommand({ + Namespace: namespace, + MetricData: metricData, + }); + + await this.cloudWatch.send(command); + } + + /** + * Send custom metric to CloudWatch + */ + private async sendCustomMetricToCloudWatch( + metricName: string, + value: number, + unit: StandardUnit, + dimensions?: Record, + ): Promise { + const timestamp = new Date(); + const namespace = 'MeteorApp/WebBackend'; + + const dimensionArray = dimensions + ? Object.entries(dimensions).map(([key, value]) => ({ + Name: key, + Value: value, + })) + : []; + + const command = new PutMetricDataCommand({ + Namespace: namespace, + MetricData: [ + { + MetricName: metricName, + Value: value, + Unit: unit, + Timestamp: timestamp, + Dimensions: dimensionArray, + }, + ], + }); + + await this.cloudWatch.send(command); + } + + /** + * Get Prometheus metrics for /metrics endpoint + */ + async getPrometheusMetrics(): Promise { + return register.metrics(); + } + + /** + * Record database operation metrics + */ + recordDatabaseOperation( + operation: string, + table: string, + duration: number, + success: boolean, + ): void { + this.recordCustomMetric('DatabaseOperationDuration', duration, StandardUnit.Milliseconds, { + Operation: operation, + Table: table, + Success: success.toString(), + }); + + this.recordCustomMetric('DatabaseOperationCount', 1, StandardUnit.Count, { + Operation: operation, + Table: table, + Success: success.toString(), + }); + } + + /** + * Record authentication metrics + */ + recordAuthOperation(operation: string, success: boolean, provider?: string): void { + this.recordCustomMetric('AuthOperationCount', 1, StandardUnit.Count, { + Operation: operation, + Success: success.toString(), + Provider: provider || 'local', + }); + } + + /** + * Record payment metrics + */ + recordPaymentOperation( + operation: string, + amount: number, + currency: string, + success: boolean, + provider: string, + ): void { + this.recordCustomMetric('PaymentOperationCount', 1, StandardUnit.Count, { + Operation: operation, + Success: success.toString(), + Provider: provider, + Currency: currency, + }); + + if (success) { + this.recordCustomMetric('PaymentAmount', amount, StandardUnit.None, { + Operation: operation, + Provider: provider, + Currency: currency, + }); + } + } + + /** + * Record event processing metrics + */ + recordEventProcessing( + eventType: string, + processingTime: number, + success: boolean, + source?: string, + ): void { + this.recordCustomMetric('EventProcessingDuration', processingTime, StandardUnit.Milliseconds, { + EventType: eventType, + Success: success.toString(), + Source: source || 'unknown', + }); + + this.recordCustomMetric('EventProcessingCount', 1, StandardUnit.Count, { + EventType: eventType, + Success: success.toString(), + Source: source || 'unknown', + }); + } +} \ No newline at end of file diff --git a/package-lock.json b/package-lock.json index 8e3a57e..7d2f996 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8016,6 +8016,7 @@ "version": "0.0.1", "license": "UNLICENSED", "dependencies": { + "@aws-sdk/client-cloudwatch": "^3.859.0", "@aws-sdk/client-s3": "^3.856.0", "@aws-sdk/client-sqs": "^3.856.0", "@nestjs/common": "^11.0.1", @@ -8024,6 +8025,7 @@ "@nestjs/passport": "^11.0.5", "@nestjs/platform-express": "^11.1.5", "@nestjs/schedule": "^6.0.0", + "@nestjs/terminus": "^11.0.0", "@nestjs/typeorm": "^11.0.0", "@types/bcrypt": "^6.0.0", "@types/passport-jwt": "^4.0.1", @@ -8035,11 +8037,15 @@ "class-validator": "^0.14.2", "dotenv": "^17.2.1", "multer": "^2.0.2", + "nestjs-pino": "^4.4.0", "node-pg-migrate": "^8.0.3", "passport": "^0.7.0", "passport-jwt": "^4.0.1", "passport-local": "^1.0.0", "pg": "^8.16.3", + "pino": "^9.7.0", + "pino-http": "^10.5.0", + "prom-client": "^15.1.3", "reflect-metadata": "^0.2.2", "rxjs": "^7.8.1", "stripe": "^18.4.0", @@ -8280,111 +8286,6 @@ "node": ">=14.0.0" } }, - "meteor-web-backend/node_modules/@aws-crypto/sha256-browser": { - "version": "5.2.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-crypto/sha256-js": "^5.2.0", - "@aws-crypto/supports-web-crypto": "^5.2.0", - "@aws-crypto/util": "^5.2.0", - "@aws-sdk/types": "^3.222.0", - "@aws-sdk/util-locate-window": "^3.0.0", - "@smithy/util-utf8": "^2.0.0", - "tslib": "^2.6.2" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/sha256-browser/node_modules/@smithy/is-array-buffer": { - "version": "2.2.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=14.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/sha256-browser/node_modules/@smithy/util-buffer-from": { - "version": "2.2.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/is-array-buffer": "^2.2.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=14.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/sha256-browser/node_modules/@smithy/util-utf8": { - "version": "2.3.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/util-buffer-from": "^2.2.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=14.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/sha256-js": { - "version": "5.2.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-crypto/util": "^5.2.0", - "@aws-sdk/types": "^3.222.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=16.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/supports-web-crypto": { - "version": "5.2.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/util": { - "version": "5.2.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-sdk/types": "^3.222.0", - "@smithy/util-utf8": "^2.0.0", - "tslib": "^2.6.2" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/util/node_modules/@smithy/is-array-buffer": { - "version": "2.2.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=14.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/util/node_modules/@smithy/util-buffer-from": { - "version": "2.2.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/is-array-buffer": "^2.2.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=14.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-crypto/util/node_modules/@smithy/util-utf8": { - "version": "2.3.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/util-buffer-from": "^2.2.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=14.0.0" - } - }, "meteor-web-backend/node_modules/@aws-sdk/client-s3": { "version": "3.856.0", "license": "Apache-2.0", @@ -8762,19 +8663,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@aws-sdk/middleware-host-header": { - "version": "3.840.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-sdk/types": "3.840.0", - "@smithy/protocol-http": "^5.1.2", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@aws-sdk/middleware-location-constraint": { "version": "3.840.0", "license": "Apache-2.0", @@ -8787,31 +8675,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@aws-sdk/middleware-logger": { - "version": "3.840.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-sdk/types": "3.840.0", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-sdk/middleware-recursion-detection": { - "version": "3.840.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-sdk/types": "3.840.0", - "@smithy/protocol-http": "^5.1.2", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@aws-sdk/middleware-sdk-s3": { "version": "3.856.0", "license": "Apache-2.0", @@ -8925,21 +8788,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@aws-sdk/region-config-resolver": { - "version": "3.840.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-sdk/types": "3.840.0", - "@smithy/node-config-provider": "^4.1.3", - "@smithy/types": "^4.3.1", - "@smithy/util-config-provider": "^4.0.0", - "@smithy/util-middleware": "^4.0.4", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@aws-sdk/signature-v4-multi-region": { "version": "3.856.0", "license": "Apache-2.0", @@ -8971,17 +8819,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@aws-sdk/types": { - "version": "3.840.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@aws-sdk/util-arn-parser": { "version": "3.804.0", "license": "Apache-2.0", @@ -8992,40 +8829,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@aws-sdk/util-endpoints": { - "version": "3.848.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-sdk/types": "3.840.0", - "@smithy/types": "^4.3.1", - "@smithy/url-parser": "^4.0.4", - "@smithy/util-endpoints": "^3.0.6", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-sdk/util-locate-window": { - "version": "3.804.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@aws-sdk/util-user-agent-browser": { - "version": "3.840.0", - "license": "Apache-2.0", - "dependencies": { - "@aws-sdk/types": "3.840.0", - "@smithy/types": "^4.3.1", - "bowser": "^2.11.0", - "tslib": "^2.6.2" - } - }, "meteor-web-backend/node_modules/@aws-sdk/util-user-agent-node": { "version": "3.856.0", "license": "Apache-2.0", @@ -9048,17 +8851,6 @@ } } }, - "meteor-web-backend/node_modules/@aws-sdk/xml-builder": { - "version": "3.821.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@babel/code-frame": { "version": "7.27.1", "dev": true, @@ -11048,17 +10840,6 @@ "@sinonjs/commons": "^3.0.0" } }, - "meteor-web-backend/node_modules/@smithy/abort-controller": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@smithy/chunked-blob-reader": { "version": "5.0.0", "license": "Apache-2.0", @@ -11080,52 +10861,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@smithy/config-resolver": { - "version": "4.1.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/node-config-provider": "^4.1.3", - "@smithy/types": "^4.3.1", - "@smithy/util-config-provider": "^4.0.0", - "@smithy/util-middleware": "^4.0.4", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/core": { - "version": "3.7.2", - "license": "Apache-2.0", - "dependencies": { - "@smithy/middleware-serde": "^4.0.8", - "@smithy/protocol-http": "^5.1.2", - "@smithy/types": "^4.3.1", - "@smithy/util-base64": "^4.0.0", - "@smithy/util-body-length-browser": "^4.0.0", - "@smithy/util-middleware": "^4.0.4", - "@smithy/util-stream": "^4.2.3", - "@smithy/util-utf8": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/credential-provider-imds": { - "version": "4.0.6", - "license": "Apache-2.0", - "dependencies": { - "@smithy/node-config-provider": "^4.1.3", - "@smithy/property-provider": "^4.0.4", - "@smithy/types": "^4.3.1", - "@smithy/url-parser": "^4.0.4", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@smithy/eventstream-codec": { "version": "4.0.4", "license": "Apache-2.0", @@ -11186,20 +10921,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@smithy/fetch-http-handler": { - "version": "5.1.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/protocol-http": "^5.1.2", - "@smithy/querystring-builder": "^4.0.4", - "@smithy/types": "^4.3.1", - "@smithy/util-base64": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@smithy/hash-blob-browser": { "version": "4.0.4", "license": "Apache-2.0", @@ -11213,19 +10934,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@smithy/hash-node": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "@smithy/util-buffer-from": "^4.0.0", - "@smithy/util-utf8": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@smithy/hash-stream-node": { "version": "4.0.4", "license": "Apache-2.0", @@ -11238,27 +10946,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@smithy/invalid-dependency": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/is-array-buffer": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@smithy/md5-js": { "version": "4.0.4", "license": "Apache-2.0", @@ -11271,413 +10958,6 @@ "node": ">=18.0.0" } }, - "meteor-web-backend/node_modules/@smithy/middleware-content-length": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/protocol-http": "^5.1.2", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/middleware-endpoint": { - "version": "4.1.17", - "license": "Apache-2.0", - "dependencies": { - "@smithy/core": "^3.7.2", - "@smithy/middleware-serde": "^4.0.8", - "@smithy/node-config-provider": "^4.1.3", - "@smithy/shared-ini-file-loader": "^4.0.4", - "@smithy/types": "^4.3.1", - "@smithy/url-parser": "^4.0.4", - "@smithy/util-middleware": "^4.0.4", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/middleware-retry": { - "version": "4.1.18", - "license": "Apache-2.0", - "dependencies": { - "@smithy/node-config-provider": "^4.1.3", - "@smithy/protocol-http": "^5.1.2", - "@smithy/service-error-classification": "^4.0.6", - "@smithy/smithy-client": "^4.4.9", - "@smithy/types": "^4.3.1", - "@smithy/util-middleware": "^4.0.4", - "@smithy/util-retry": "^4.0.6", - "tslib": "^2.6.2", - "uuid": "^9.0.1" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/middleware-retry/node_modules/uuid": { - "version": "9.0.1", - "funding": [ - "https://github.com/sponsors/broofa", - "https://github.com/sponsors/ctavan" - ], - "license": "MIT", - "bin": { - "uuid": "dist/bin/uuid" - } - }, - "meteor-web-backend/node_modules/@smithy/middleware-serde": { - "version": "4.0.8", - "license": "Apache-2.0", - "dependencies": { - "@smithy/protocol-http": "^5.1.2", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/middleware-stack": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/node-config-provider": { - "version": "4.1.3", - "license": "Apache-2.0", - "dependencies": { - "@smithy/property-provider": "^4.0.4", - "@smithy/shared-ini-file-loader": "^4.0.4", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/node-http-handler": { - "version": "4.1.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/abort-controller": "^4.0.4", - "@smithy/protocol-http": "^5.1.2", - "@smithy/querystring-builder": "^4.0.4", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/property-provider": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/protocol-http": { - "version": "5.1.2", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/querystring-builder": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "@smithy/util-uri-escape": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/querystring-parser": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/service-error-classification": { - "version": "4.0.6", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/shared-ini-file-loader": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/signature-v4": { - "version": "5.1.2", - "license": "Apache-2.0", - "dependencies": { - "@smithy/is-array-buffer": "^4.0.0", - "@smithy/protocol-http": "^5.1.2", - "@smithy/types": "^4.3.1", - "@smithy/util-hex-encoding": "^4.0.0", - "@smithy/util-middleware": "^4.0.4", - "@smithy/util-uri-escape": "^4.0.0", - "@smithy/util-utf8": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/smithy-client": { - "version": "4.4.9", - "license": "Apache-2.0", - "dependencies": { - "@smithy/core": "^3.7.2", - "@smithy/middleware-endpoint": "^4.1.17", - "@smithy/middleware-stack": "^4.0.4", - "@smithy/protocol-http": "^5.1.2", - "@smithy/types": "^4.3.1", - "@smithy/util-stream": "^4.2.3", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/types": { - "version": "4.3.1", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/url-parser": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/querystring-parser": "^4.0.4", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-base64": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/util-buffer-from": "^4.0.0", - "@smithy/util-utf8": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-body-length-browser": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-body-length-node": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-buffer-from": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/is-array-buffer": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-config-provider": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-defaults-mode-browser": { - "version": "4.0.25", - "license": "Apache-2.0", - "dependencies": { - "@smithy/property-provider": "^4.0.4", - "@smithy/smithy-client": "^4.4.9", - "@smithy/types": "^4.3.1", - "bowser": "^2.11.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-defaults-mode-node": { - "version": "4.0.25", - "license": "Apache-2.0", - "dependencies": { - "@smithy/config-resolver": "^4.1.4", - "@smithy/credential-provider-imds": "^4.0.6", - "@smithy/node-config-provider": "^4.1.3", - "@smithy/property-provider": "^4.0.4", - "@smithy/smithy-client": "^4.4.9", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-endpoints": { - "version": "3.0.6", - "license": "Apache-2.0", - "dependencies": { - "@smithy/node-config-provider": "^4.1.3", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-hex-encoding": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-middleware": { - "version": "4.0.4", - "license": "Apache-2.0", - "dependencies": { - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-retry": { - "version": "4.0.6", - "license": "Apache-2.0", - "dependencies": { - "@smithy/service-error-classification": "^4.0.6", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-stream": { - "version": "4.2.3", - "license": "Apache-2.0", - "dependencies": { - "@smithy/fetch-http-handler": "^5.1.0", - "@smithy/node-http-handler": "^4.1.0", - "@smithy/types": "^4.3.1", - "@smithy/util-base64": "^4.0.0", - "@smithy/util-buffer-from": "^4.0.0", - "@smithy/util-hex-encoding": "^4.0.0", - "@smithy/util-utf8": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-uri-escape": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-utf8": { - "version": "4.0.0", - "license": "Apache-2.0", - "dependencies": { - "@smithy/util-buffer-from": "^4.0.0", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "meteor-web-backend/node_modules/@smithy/util-waiter": { - "version": "4.0.6", - "license": "Apache-2.0", - "dependencies": { - "@smithy/abort-controller": "^4.0.4", - "@smithy/types": "^4.3.1", - "tslib": "^2.6.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, "meteor-web-backend/node_modules/@sqltools/formatter": { "version": "1.2.5", "license": "MIT" @@ -13217,10 +12497,6 @@ "node": ">=18" } }, - "meteor-web-backend/node_modules/bowser": { - "version": "2.11.0", - "license": "MIT" - }, "meteor-web-backend/node_modules/brace-expansion": { "version": "1.1.12", "dev": true, @@ -14408,22 +13684,6 @@ ], "license": "BSD-3-Clause" }, - "meteor-web-backend/node_modules/fast-xml-parser": { - "version": "5.2.5", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/NaturalIntelligence" - } - ], - "license": "MIT", - "dependencies": { - "strnum": "^2.1.0" - }, - "bin": { - "fxparser": "src/cli/cli.js" - } - }, "meteor-web-backend/node_modules/fastq": { "version": "1.19.1", "dev": true, @@ -17548,13 +16808,6 @@ "node": ">=0.10.0" } }, - "meteor-web-backend/node_modules/split2": { - "version": "4.2.0", - "license": "ISC", - "engines": { - "node": ">= 10.x" - } - }, "meteor-web-backend/node_modules/sprintf-js": { "version": "1.0.3", "dev": true, @@ -17753,16 +17006,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "meteor-web-backend/node_modules/strnum": { - "version": "2.1.1", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/NaturalIntelligence" - } - ], - "license": "MIT" - }, "meteor-web-backend/node_modules/superagent": { "version": "10.2.3", "dev": true, @@ -19082,6 +18325,631 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/@aws-crypto/sha256-browser": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/sha256-browser/-/sha256-browser-5.2.0.tgz", + "integrity": "sha512-AXfN/lGotSQwu6HNcEsIASo7kWXZ5HYWvfOmSNKDsEqC4OashTp8alTmaz+F7TC2L083SFv5RdB+qU3Vs1kZqw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/sha256-js": "^5.2.0", + "@aws-crypto/supports-web-crypto": "^5.2.0", + "@aws-crypto/util": "^5.2.0", + "@aws-sdk/types": "^3.222.0", + "@aws-sdk/util-locate-window": "^3.0.0", + "@smithy/util-utf8": "^2.0.0", + "tslib": "^2.6.2" + } + }, + "node_modules/@aws-crypto/sha256-browser/node_modules/@smithy/is-array-buffer": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@smithy/is-array-buffer/-/is-array-buffer-2.2.0.tgz", + "integrity": "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@aws-crypto/sha256-browser/node_modules/@smithy/util-buffer-from": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@smithy/util-buffer-from/-/util-buffer-from-2.2.0.tgz", + "integrity": "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/is-array-buffer": "^2.2.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@aws-crypto/sha256-browser/node_modules/@smithy/util-utf8": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/@smithy/util-utf8/-/util-utf8-2.3.0.tgz", + "integrity": "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/util-buffer-from": "^2.2.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@aws-crypto/sha256-js": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/sha256-js/-/sha256-js-5.2.0.tgz", + "integrity": "sha512-FFQQyu7edu4ufvIZ+OadFpHHOt+eSTBaYaki44c+akjg7qZg9oOQeLlk77F6tSYqjDAFClrHJk9tMf0HdVyOvA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/util": "^5.2.0", + "@aws-sdk/types": "^3.222.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=16.0.0" + } + }, + "node_modules/@aws-crypto/supports-web-crypto": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/supports-web-crypto/-/supports-web-crypto-5.2.0.tgz", + "integrity": "sha512-iAvUotm021kM33eCdNfwIN//F77/IADDSs58i+MDaOqFrVjZo9bAal0NK7HurRuWLLpF1iLX7gbWrjHjeo+YFg==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + } + }, + "node_modules/@aws-crypto/util": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/@aws-crypto/util/-/util-5.2.0.tgz", + "integrity": "sha512-4RkU9EsI6ZpBve5fseQlGNUWKMa1RLPQ1dnjnQoe07ldfIzcsGb5hC5W0Dm7u423KWzawlrpbjXBrXCEv9zazQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.222.0", + "@smithy/util-utf8": "^2.0.0", + "tslib": "^2.6.2" + } + }, + "node_modules/@aws-crypto/util/node_modules/@smithy/is-array-buffer": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@smithy/is-array-buffer/-/is-array-buffer-2.2.0.tgz", + "integrity": "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@aws-crypto/util/node_modules/@smithy/util-buffer-from": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@smithy/util-buffer-from/-/util-buffer-from-2.2.0.tgz", + "integrity": "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/is-array-buffer": "^2.2.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@aws-crypto/util/node_modules/@smithy/util-utf8": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/@smithy/util-utf8/-/util-utf8-2.3.0.tgz", + "integrity": "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/util-buffer-from": "^2.2.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/@aws-sdk/client-cloudwatch": { + "version": "3.859.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/client-cloudwatch/-/client-cloudwatch-3.859.0.tgz", + "integrity": "sha512-tnUP6qRKG1vEXTkZr1u+T+vqyrSF+pSveK3maaERGvsgABrFEAR6KU0yb93A1sSmYsfHd/O1CVzkQB6zecc44g==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/sha256-browser": "5.2.0", + "@aws-crypto/sha256-js": "5.2.0", + "@aws-sdk/core": "3.858.0", + "@aws-sdk/credential-provider-node": "3.859.0", + "@aws-sdk/middleware-host-header": "3.840.0", + "@aws-sdk/middleware-logger": "3.840.0", + "@aws-sdk/middleware-recursion-detection": "3.840.0", + "@aws-sdk/middleware-user-agent": "3.858.0", + "@aws-sdk/region-config-resolver": "3.840.0", + "@aws-sdk/types": "3.840.0", + "@aws-sdk/util-endpoints": "3.848.0", + "@aws-sdk/util-user-agent-browser": "3.840.0", + "@aws-sdk/util-user-agent-node": "3.858.0", + "@smithy/config-resolver": "^4.1.4", + "@smithy/core": "^3.7.2", + "@smithy/fetch-http-handler": "^5.1.0", + "@smithy/hash-node": "^4.0.4", + "@smithy/invalid-dependency": "^4.0.4", + "@smithy/middleware-compression": "^4.1.15", + "@smithy/middleware-content-length": "^4.0.4", + "@smithy/middleware-endpoint": "^4.1.17", + "@smithy/middleware-retry": "^4.1.18", + "@smithy/middleware-serde": "^4.0.8", + "@smithy/middleware-stack": "^4.0.4", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/node-http-handler": "^4.1.0", + "@smithy/protocol-http": "^5.1.2", + "@smithy/smithy-client": "^4.4.9", + "@smithy/types": "^4.3.1", + "@smithy/url-parser": "^4.0.4", + "@smithy/util-base64": "^4.0.0", + "@smithy/util-body-length-browser": "^4.0.0", + "@smithy/util-body-length-node": "^4.0.0", + "@smithy/util-defaults-mode-browser": "^4.0.25", + "@smithy/util-defaults-mode-node": "^4.0.25", + "@smithy/util-endpoints": "^3.0.6", + "@smithy/util-middleware": "^4.0.4", + "@smithy/util-retry": "^4.0.6", + "@smithy/util-utf8": "^4.0.0", + "@smithy/util-waiter": "^4.0.6", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/client-sso": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/client-sso/-/client-sso-3.858.0.tgz", + "integrity": "sha512-iXuZQs4KH6a3Pwnt0uORalzAZ5EXRPr3lBYAsdNwkP8OYyoUz5/TE3BLyw7ceEh0rj4QKGNnNALYo1cDm0EV8w==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/sha256-browser": "5.2.0", + "@aws-crypto/sha256-js": "5.2.0", + "@aws-sdk/core": "3.858.0", + "@aws-sdk/middleware-host-header": "3.840.0", + "@aws-sdk/middleware-logger": "3.840.0", + "@aws-sdk/middleware-recursion-detection": "3.840.0", + "@aws-sdk/middleware-user-agent": "3.858.0", + "@aws-sdk/region-config-resolver": "3.840.0", + "@aws-sdk/types": "3.840.0", + "@aws-sdk/util-endpoints": "3.848.0", + "@aws-sdk/util-user-agent-browser": "3.840.0", + "@aws-sdk/util-user-agent-node": "3.858.0", + "@smithy/config-resolver": "^4.1.4", + "@smithy/core": "^3.7.2", + "@smithy/fetch-http-handler": "^5.1.0", + "@smithy/hash-node": "^4.0.4", + "@smithy/invalid-dependency": "^4.0.4", + "@smithy/middleware-content-length": "^4.0.4", + "@smithy/middleware-endpoint": "^4.1.17", + "@smithy/middleware-retry": "^4.1.18", + "@smithy/middleware-serde": "^4.0.8", + "@smithy/middleware-stack": "^4.0.4", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/node-http-handler": "^4.1.0", + "@smithy/protocol-http": "^5.1.2", + "@smithy/smithy-client": "^4.4.9", + "@smithy/types": "^4.3.1", + "@smithy/url-parser": "^4.0.4", + "@smithy/util-base64": "^4.0.0", + "@smithy/util-body-length-browser": "^4.0.0", + "@smithy/util-body-length-node": "^4.0.0", + "@smithy/util-defaults-mode-browser": "^4.0.25", + "@smithy/util-defaults-mode-node": "^4.0.25", + "@smithy/util-endpoints": "^3.0.6", + "@smithy/util-middleware": "^4.0.4", + "@smithy/util-retry": "^4.0.6", + "@smithy/util-utf8": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/core": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.858.0.tgz", + "integrity": "sha512-iWm4QLAS+/XMlnecIU1Y33qbBr1Ju+pmWam3xVCPlY4CSptKpVY+2hXOnmg9SbHAX9C005fWhrIn51oDd00c9A==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "3.840.0", + "@aws-sdk/xml-builder": "3.821.0", + "@smithy/core": "^3.7.2", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/property-provider": "^4.0.4", + "@smithy/protocol-http": "^5.1.2", + "@smithy/signature-v4": "^5.1.2", + "@smithy/smithy-client": "^4.4.9", + "@smithy/types": "^4.3.1", + "@smithy/util-base64": "^4.0.0", + "@smithy/util-body-length-browser": "^4.0.0", + "@smithy/util-middleware": "^4.0.4", + "@smithy/util-utf8": "^4.0.0", + "fast-xml-parser": "5.2.5", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-env": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.858.0.tgz", + "integrity": "sha512-kZsGyh2BoSRguzlcGtzdLhw/l/n3KYAC+/l/H0SlsOq3RLHF6tO/cRdsLnwoix2bObChHUp03cex63o1gzdx/Q==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@smithy/property-provider": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-http": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.858.0.tgz", + "integrity": "sha512-GDnfYl3+NPJQ7WQQYOXEA489B212NinpcIDD7rpsB6IWUPo8yDjT5NceK4uUkIR3MFpNCGt9zd/z6NNLdB2fuQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@smithy/fetch-http-handler": "^5.1.0", + "@smithy/node-http-handler": "^4.1.0", + "@smithy/property-provider": "^4.0.4", + "@smithy/protocol-http": "^5.1.2", + "@smithy/smithy-client": "^4.4.9", + "@smithy/types": "^4.3.1", + "@smithy/util-stream": "^4.2.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-ini": { + "version": "3.859.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.859.0.tgz", + "integrity": "sha512-KsccE1T88ZDNhsABnqbQj014n5JMDilAroUErFbGqu5/B3sXqUsYmG54C/BjvGTRUFfzyttK9lB9P9h6ddQ8Cw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "3.858.0", + "@aws-sdk/credential-provider-env": "3.858.0", + "@aws-sdk/credential-provider-http": "3.858.0", + "@aws-sdk/credential-provider-process": "3.858.0", + "@aws-sdk/credential-provider-sso": "3.859.0", + "@aws-sdk/credential-provider-web-identity": "3.858.0", + "@aws-sdk/nested-clients": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@smithy/credential-provider-imds": "^4.0.6", + "@smithy/property-provider": "^4.0.4", + "@smithy/shared-ini-file-loader": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-node": { + "version": "3.859.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.859.0.tgz", + "integrity": "sha512-ZRDB2xU5aSyTR/jDcli30tlycu6RFvQngkZhBs9Zoh2BiYXrfh2MMuoYuZk+7uD6D53Q2RIEldDHR9A/TPlRuA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/credential-provider-env": "3.858.0", + "@aws-sdk/credential-provider-http": "3.858.0", + "@aws-sdk/credential-provider-ini": "3.859.0", + "@aws-sdk/credential-provider-process": "3.858.0", + "@aws-sdk/credential-provider-sso": "3.859.0", + "@aws-sdk/credential-provider-web-identity": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@smithy/credential-provider-imds": "^4.0.6", + "@smithy/property-provider": "^4.0.4", + "@smithy/shared-ini-file-loader": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-process": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.858.0.tgz", + "integrity": "sha512-l5LJWZJMRaZ+LhDjtupFUKEC5hAjgvCRrOvV5T60NCUBOy0Ozxa7Sgx3x+EOwiruuoh3Cn9O+RlbQlJX6IfZIw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@smithy/property-provider": "^4.0.4", + "@smithy/shared-ini-file-loader": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-sso": { + "version": "3.859.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.859.0.tgz", + "integrity": "sha512-BwAqmWIivhox5YlFRjManFF8GoTvEySPk6vsJNxDsmGsabY+OQovYxFIYxRCYiHzH7SFjd4Lcd+riJOiXNsvRw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/client-sso": "3.858.0", + "@aws-sdk/core": "3.858.0", + "@aws-sdk/token-providers": "3.859.0", + "@aws-sdk/types": "3.840.0", + "@smithy/property-provider": "^4.0.4", + "@smithy/shared-ini-file-loader": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-web-identity": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.858.0.tgz", + "integrity": "sha512-8iULWsH83iZDdUuiDsRb83M0NqIlXjlDbJUIddVsIrfWp4NmanKw77SV6yOZ66nuJjPsn9j7RDb9bfEPCy5SWA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "3.858.0", + "@aws-sdk/nested-clients": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@smithy/property-provider": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/middleware-host-header": { + "version": "3.840.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-host-header/-/middleware-host-header-3.840.0.tgz", + "integrity": "sha512-ub+hXJAbAje94+Ya6c6eL7sYujoE8D4Bumu1NUI8TXjUhVVn0HzVWQjpRLshdLsUp1AW7XyeJaxyajRaJQ8+Xg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "3.840.0", + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/middleware-logger": { + "version": "3.840.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-logger/-/middleware-logger-3.840.0.tgz", + "integrity": "sha512-lSV8FvjpdllpGaRspywss4CtXV8M7NNNH+2/j86vMH+YCOZ6fu2T/TyFd/tHwZ92vDfHctWkRbQxg0bagqwovA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "3.840.0", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/middleware-recursion-detection": { + "version": "3.840.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-recursion-detection/-/middleware-recursion-detection-3.840.0.tgz", + "integrity": "sha512-Gu7lGDyfddyhIkj1Z1JtrY5NHb5+x/CRiB87GjaSrKxkDaydtX2CU977JIABtt69l9wLbcGDIQ+W0uJ5xPof7g==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "3.840.0", + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/middleware-user-agent": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-user-agent/-/middleware-user-agent-3.858.0.tgz", + "integrity": "sha512-pC3FT/sRZ6n5NyXiTVu9dpf1D9j3YbJz3XmeOOwJqO/Mib2PZyIQktvNMPgwaC5KMVB1zWqS5bmCwxpMOnq0UQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@aws-sdk/util-endpoints": "3.848.0", + "@smithy/core": "^3.7.2", + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/nested-clients": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.858.0.tgz", + "integrity": "sha512-ChdIj80T2whoWbovmO7o8ICmhEB2S9q4Jes9MBnKAPm69PexcJAK2dQC8yI4/iUP8b3+BHZoUPrYLWjBxIProQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-crypto/sha256-browser": "5.2.0", + "@aws-crypto/sha256-js": "5.2.0", + "@aws-sdk/core": "3.858.0", + "@aws-sdk/middleware-host-header": "3.840.0", + "@aws-sdk/middleware-logger": "3.840.0", + "@aws-sdk/middleware-recursion-detection": "3.840.0", + "@aws-sdk/middleware-user-agent": "3.858.0", + "@aws-sdk/region-config-resolver": "3.840.0", + "@aws-sdk/types": "3.840.0", + "@aws-sdk/util-endpoints": "3.848.0", + "@aws-sdk/util-user-agent-browser": "3.840.0", + "@aws-sdk/util-user-agent-node": "3.858.0", + "@smithy/config-resolver": "^4.1.4", + "@smithy/core": "^3.7.2", + "@smithy/fetch-http-handler": "^5.1.0", + "@smithy/hash-node": "^4.0.4", + "@smithy/invalid-dependency": "^4.0.4", + "@smithy/middleware-content-length": "^4.0.4", + "@smithy/middleware-endpoint": "^4.1.17", + "@smithy/middleware-retry": "^4.1.18", + "@smithy/middleware-serde": "^4.0.8", + "@smithy/middleware-stack": "^4.0.4", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/node-http-handler": "^4.1.0", + "@smithy/protocol-http": "^5.1.2", + "@smithy/smithy-client": "^4.4.9", + "@smithy/types": "^4.3.1", + "@smithy/url-parser": "^4.0.4", + "@smithy/util-base64": "^4.0.0", + "@smithy/util-body-length-browser": "^4.0.0", + "@smithy/util-body-length-node": "^4.0.0", + "@smithy/util-defaults-mode-browser": "^4.0.25", + "@smithy/util-defaults-mode-node": "^4.0.25", + "@smithy/util-endpoints": "^3.0.6", + "@smithy/util-middleware": "^4.0.4", + "@smithy/util-retry": "^4.0.6", + "@smithy/util-utf8": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/region-config-resolver": { + "version": "3.840.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/region-config-resolver/-/region-config-resolver-3.840.0.tgz", + "integrity": "sha512-Qjnxd/yDv9KpIMWr90ZDPtRj0v75AqGC92Lm9+oHXZ8p1MjG5JE2CW0HL8JRgK9iKzgKBL7pPQRXI8FkvEVfrA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "3.840.0", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/types": "^4.3.1", + "@smithy/util-config-provider": "^4.0.0", + "@smithy/util-middleware": "^4.0.4", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/token-providers": { + "version": "3.859.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.859.0.tgz", + "integrity": "sha512-6P2wlvm9KBWOvRNn0Pt8RntnXg8fzOb5kEShvWsOsAocZeqKNaYbihum5/Onq1ZPoVtkdb++8eWDocDnM4k85Q==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "3.858.0", + "@aws-sdk/nested-clients": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@smithy/property-provider": "^4.0.4", + "@smithy/shared-ini-file-loader": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/types": { + "version": "3.840.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.840.0.tgz", + "integrity": "sha512-xliuHaUFZxEx1NSXeLLZ9Dyu6+EJVQKEoD+yM+zqUo3YDZ7medKJWY6fIOKiPX/N7XbLdBYwajb15Q7IL8KkeA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/util-endpoints": { + "version": "3.848.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/util-endpoints/-/util-endpoints-3.848.0.tgz", + "integrity": "sha512-fY/NuFFCq/78liHvRyFKr+aqq1aA/uuVSANjzr5Ym8c+9Z3HRPE9OrExAHoMrZ6zC8tHerQwlsXYYH5XZ7H+ww==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "3.840.0", + "@smithy/types": "^4.3.1", + "@smithy/url-parser": "^4.0.4", + "@smithy/util-endpoints": "^3.0.6", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/util-locate-window": { + "version": "3.804.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/util-locate-window/-/util-locate-window-3.804.0.tgz", + "integrity": "sha512-zVoRfpmBVPodYlnMjgVjfGoEZagyRF5IPn3Uo6ZvOZp24chnW/FRstH7ESDHDDRga4z3V+ElUQHKpFDXWyBW5A==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@aws-sdk/util-user-agent-browser": { + "version": "3.840.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/util-user-agent-browser/-/util-user-agent-browser-3.840.0.tgz", + "integrity": "sha512-JdyZM3EhhL4PqwFpttZu1afDpPJCCc3eyZOLi+srpX11LsGj6sThf47TYQN75HT1CarZ7cCdQHGzP2uy3/xHfQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "3.840.0", + "@smithy/types": "^4.3.1", + "bowser": "^2.11.0", + "tslib": "^2.6.2" + } + }, + "node_modules/@aws-sdk/util-user-agent-node": { + "version": "3.858.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/util-user-agent-node/-/util-user-agent-node-3.858.0.tgz", + "integrity": "sha512-T1m05QlN8hFpx5/5duMjS8uFSK5e6EXP45HQRkZULVkL3DK+jMaxsnh3KLl5LjUoHn/19M4HM0wNUBhYp4Y2Yw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/middleware-user-agent": "3.858.0", + "@aws-sdk/types": "3.840.0", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + }, + "peerDependencies": { + "aws-crt": ">=1.0.0" + }, + "peerDependenciesMeta": { + "aws-crt": { + "optional": true + } + } + }, + "node_modules/@aws-sdk/xml-builder": { + "version": "3.821.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.821.0.tgz", + "integrity": "sha512-DIIotRnefVL6DiaHtO6/21DhJ4JZnnIwdNbpwiAhdt/AVbttcE4yw925gsjur0OGv5BTYXQXU3YnANBYnZjuQA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, "node_modules/@babel/runtime": { "version": "7.28.2", "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.28.2.tgz", @@ -19186,6 +19054,76 @@ "@nestjs/core": "^10.0.0 || ^11.0.0" } }, + "node_modules/@nestjs/terminus": { + "version": "11.0.0", + "resolved": "https://registry.npmjs.org/@nestjs/terminus/-/terminus-11.0.0.tgz", + "integrity": "sha512-c55LOo9YGovmQHtFUMa/vDaxGZ2cglMTZejqgHREaApt/GArTfgYYGwhRXPLq8ZwiQQlLuYB+79e9iA8mlDSLA==", + "license": "MIT", + "dependencies": { + "boxen": "5.1.2", + "check-disk-space": "3.4.0" + }, + "peerDependencies": { + "@grpc/grpc-js": "*", + "@grpc/proto-loader": "*", + "@mikro-orm/core": "*", + "@mikro-orm/nestjs": "*", + "@nestjs/axios": "^2.0.0 || ^3.0.0 || ^4.0.0", + "@nestjs/common": "^10.0.0 || ^11.0.0", + "@nestjs/core": "^10.0.0 || ^11.0.0", + "@nestjs/microservices": "^10.0.0 || ^11.0.0", + "@nestjs/mongoose": "^11.0.0", + "@nestjs/sequelize": "^10.0.0 || ^11.0.0", + "@nestjs/typeorm": "^10.0.0 || ^11.0.0", + "@prisma/client": "*", + "mongoose": "*", + "reflect-metadata": "0.1.x || 0.2.x", + "rxjs": "7.x", + "sequelize": "*", + "typeorm": "*" + }, + "peerDependenciesMeta": { + "@grpc/grpc-js": { + "optional": true + }, + "@grpc/proto-loader": { + "optional": true + }, + "@mikro-orm/core": { + "optional": true + }, + "@mikro-orm/nestjs": { + "optional": true + }, + "@nestjs/axios": { + "optional": true + }, + "@nestjs/microservices": { + "optional": true + }, + "@nestjs/mongoose": { + "optional": true + }, + "@nestjs/sequelize": { + "optional": true + }, + "@nestjs/typeorm": { + "optional": true + }, + "@prisma/client": { + "optional": true + }, + "mongoose": { + "optional": true + }, + "sequelize": { + "optional": true + }, + "typeorm": { + "optional": true + } + } + }, "node_modules/@nuxt/opencollective": { "version": "0.4.1", "resolved": "https://registry.npmjs.org/@nuxt/opencollective/-/opencollective-0.4.1.tgz", @@ -19202,6 +19140,623 @@ "npm": ">=5.10.0" } }, + "node_modules/@opentelemetry/api": { + "version": "1.9.0", + "resolved": "https://registry.npmjs.org/@opentelemetry/api/-/api-1.9.0.tgz", + "integrity": "sha512-3giAOQvZiH5F9bMlMiv8+GSPMeqg0dbaeo58/0SlA9sxSqZhnUtxzX9/2FzyhS9sWQf5S0GJE0AKBrFqjpeYcg==", + "license": "Apache-2.0", + "engines": { + "node": ">=8.0.0" + } + }, + "node_modules/@smithy/abort-controller": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/abort-controller/-/abort-controller-4.0.4.tgz", + "integrity": "sha512-gJnEjZMvigPDQWHrW3oPrFhQtkrgqBkyjj3pCIdF3A5M6vsZODG93KNlfJprv6bp4245bdT32fsHK4kkH3KYDA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/config-resolver": { + "version": "4.1.4", + "resolved": "https://registry.npmjs.org/@smithy/config-resolver/-/config-resolver-4.1.4.tgz", + "integrity": "sha512-prmU+rDddxHOH0oNcwemL+SwnzcG65sBF2yXRO7aeXIn/xTlq2pX7JLVbkBnVLowHLg4/OL4+jBmv9hVrVGS+w==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/node-config-provider": "^4.1.3", + "@smithy/types": "^4.3.1", + "@smithy/util-config-provider": "^4.0.0", + "@smithy/util-middleware": "^4.0.4", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/core": { + "version": "3.7.2", + "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.7.2.tgz", + "integrity": "sha512-JoLw59sT5Bm8SAjFCYZyuCGxK8y3vovmoVbZWLDPTH5XpPEIwpFd9m90jjVMwoypDuB/SdVgje5Y4T7w50lJaw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/middleware-serde": "^4.0.8", + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "@smithy/util-base64": "^4.0.0", + "@smithy/util-body-length-browser": "^4.0.0", + "@smithy/util-middleware": "^4.0.4", + "@smithy/util-stream": "^4.2.3", + "@smithy/util-utf8": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/credential-provider-imds": { + "version": "4.0.6", + "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.0.6.tgz", + "integrity": "sha512-hKMWcANhUiNbCJouYkZ9V3+/Qf9pteR1dnwgdyzR09R4ODEYx8BbUysHwRSyex4rZ9zapddZhLFTnT4ZijR4pw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/node-config-provider": "^4.1.3", + "@smithy/property-provider": "^4.0.4", + "@smithy/types": "^4.3.1", + "@smithy/url-parser": "^4.0.4", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/fetch-http-handler": { + "version": "5.1.0", + "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.1.0.tgz", + "integrity": "sha512-mADw7MS0bYe2OGKkHYMaqarOXuDwRbO6ArD91XhHcl2ynjGCFF+hvqf0LyQcYxkA1zaWjefSkU7Ne9mqgApSgQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/protocol-http": "^5.1.2", + "@smithy/querystring-builder": "^4.0.4", + "@smithy/types": "^4.3.1", + "@smithy/util-base64": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/hash-node": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/hash-node/-/hash-node-4.0.4.tgz", + "integrity": "sha512-qnbTPUhCVnCgBp4z4BUJUhOEkVwxiEi1cyFM+Zj6o+aY8OFGxUQleKWq8ltgp3dujuhXojIvJWdoqpm6dVO3lQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "@smithy/util-buffer-from": "^4.0.0", + "@smithy/util-utf8": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/invalid-dependency": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/invalid-dependency/-/invalid-dependency-4.0.4.tgz", + "integrity": "sha512-bNYMi7WKTJHu0gn26wg8OscncTt1t2b8KcsZxvOv56XA6cyXtOAAAaNP7+m45xfppXfOatXF3Sb1MNsLUgVLTw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/is-array-buffer": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/is-array-buffer/-/is-array-buffer-4.0.0.tgz", + "integrity": "sha512-saYhF8ZZNoJDTvJBEWgeBccCg+yvp1CX+ed12yORU3NilJScfc6gfch2oVb4QgxZrGUx3/ZJlb+c/dJbyupxlw==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/middleware-compression": { + "version": "4.1.15", + "resolved": "https://registry.npmjs.org/@smithy/middleware-compression/-/middleware-compression-4.1.15.tgz", + "integrity": "sha512-cYBwWOOMQFpQwokCWChnKITmbTGrn8diYaHmX0eosRLBSyUJFvwaFN+pf4rVfsyBsTnr8UpUNfvjw1fDhZgOGA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.7.2", + "@smithy/is-array-buffer": "^4.0.0", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "@smithy/util-config-provider": "^4.0.0", + "@smithy/util-middleware": "^4.0.4", + "@smithy/util-utf8": "^4.0.0", + "fflate": "0.8.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/middleware-compression/node_modules/fflate": { + "version": "0.8.1", + "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.1.tgz", + "integrity": "sha512-/exOvEuc+/iaUm105QIiOt4LpBdMTWsXxqR0HDF35vx3fmaKzw7354gTilCh5rkzEt8WYyG//ku3h3nRmd7CHQ==", + "license": "MIT" + }, + "node_modules/@smithy/middleware-content-length": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/middleware-content-length/-/middleware-content-length-4.0.4.tgz", + "integrity": "sha512-F7gDyfI2BB1Kc+4M6rpuOLne5LOcEknH1n6UQB69qv+HucXBR1rkzXBnQTB2q46sFy1PM/zuSJOB532yc8bg3w==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/middleware-endpoint": { + "version": "4.1.17", + "resolved": "https://registry.npmjs.org/@smithy/middleware-endpoint/-/middleware-endpoint-4.1.17.tgz", + "integrity": "sha512-S3hSGLKmHG1m35p/MObQCBCdRsrpbPU8B129BVzRqRfDvQqPMQ14iO4LyRw+7LNizYc605COYAcjqgawqi+6jA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.7.2", + "@smithy/middleware-serde": "^4.0.8", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/shared-ini-file-loader": "^4.0.4", + "@smithy/types": "^4.3.1", + "@smithy/url-parser": "^4.0.4", + "@smithy/util-middleware": "^4.0.4", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/middleware-retry": { + "version": "4.1.18", + "resolved": "https://registry.npmjs.org/@smithy/middleware-retry/-/middleware-retry-4.1.18.tgz", + "integrity": "sha512-bYLZ4DkoxSsPxpdmeapvAKy7rM5+25gR7PGxq2iMiecmbrRGBHj9s75N74Ylg+aBiw9i5jIowC/cLU2NR0qH8w==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/node-config-provider": "^4.1.3", + "@smithy/protocol-http": "^5.1.2", + "@smithy/service-error-classification": "^4.0.6", + "@smithy/smithy-client": "^4.4.9", + "@smithy/types": "^4.3.1", + "@smithy/util-middleware": "^4.0.4", + "@smithy/util-retry": "^4.0.6", + "tslib": "^2.6.2", + "uuid": "^9.0.1" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/middleware-serde": { + "version": "4.0.8", + "resolved": "https://registry.npmjs.org/@smithy/middleware-serde/-/middleware-serde-4.0.8.tgz", + "integrity": "sha512-iSSl7HJoJaGyMIoNn2B7czghOVwJ9nD7TMvLhMWeSB5vt0TnEYyRRqPJu/TqW76WScaNvYYB8nRoiBHR9S1Ddw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/middleware-stack": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/middleware-stack/-/middleware-stack-4.0.4.tgz", + "integrity": "sha512-kagK5ggDrBUCCzI93ft6DjteNSfY8Ulr83UtySog/h09lTIOAJ/xUSObutanlPT0nhoHAkpmW9V5K8oPyLh+QA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/node-config-provider": { + "version": "4.1.3", + "resolved": "https://registry.npmjs.org/@smithy/node-config-provider/-/node-config-provider-4.1.3.tgz", + "integrity": "sha512-HGHQr2s59qaU1lrVH6MbLlmOBxadtzTsoO4c+bF5asdgVik3I8o7JIOzoeqWc5MjVa+vD36/LWE0iXKpNqooRw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/property-provider": "^4.0.4", + "@smithy/shared-ini-file-loader": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/node-http-handler": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.1.0.tgz", + "integrity": "sha512-vqfSiHz2v8b3TTTrdXi03vNz1KLYYS3bhHCDv36FYDqxT7jvTll1mMnCrkD+gOvgwybuunh/2VmvOMqwBegxEg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/abort-controller": "^4.0.4", + "@smithy/protocol-http": "^5.1.2", + "@smithy/querystring-builder": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/property-provider": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/property-provider/-/property-provider-4.0.4.tgz", + "integrity": "sha512-qHJ2sSgu4FqF4U/5UUp4DhXNmdTrgmoAai6oQiM+c5RZ/sbDwJ12qxB1M6FnP+Tn/ggkPZf9ccn4jqKSINaquw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/protocol-http": { + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/@smithy/protocol-http/-/protocol-http-5.1.2.tgz", + "integrity": "sha512-rOG5cNLBXovxIrICSBm95dLqzfvxjEmuZx4KK3hWwPFHGdW3lxY0fZNXfv2zebfRO7sJZ5pKJYHScsqopeIWtQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/querystring-builder": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/querystring-builder/-/querystring-builder-4.0.4.tgz", + "integrity": "sha512-SwREZcDnEYoh9tLNgMbpop+UTGq44Hl9tdj3rf+yeLcfH7+J8OXEBaMc2kDxtyRHu8BhSg9ADEx0gFHvpJgU8w==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "@smithy/util-uri-escape": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/querystring-parser": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/querystring-parser/-/querystring-parser-4.0.4.tgz", + "integrity": "sha512-6yZf53i/qB8gRHH/l2ZwUG5xgkPgQF15/KxH0DdXMDHjesA9MeZje/853ifkSY0x4m5S+dfDZ+c4x439PF0M2w==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/service-error-classification": { + "version": "4.0.6", + "resolved": "https://registry.npmjs.org/@smithy/service-error-classification/-/service-error-classification-4.0.6.tgz", + "integrity": "sha512-RRoTDL//7xi4tn5FrN2NzH17jbgmnKidUqd4KvquT0954/i6CXXkh1884jBiunq24g9cGtPBEXlU40W6EpNOOg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/shared-ini-file-loader": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/shared-ini-file-loader/-/shared-ini-file-loader-4.0.4.tgz", + "integrity": "sha512-63X0260LoFBjrHifPDs+nM9tV0VMkOTl4JRMYNuKh/f5PauSjowTfvF3LogfkWdcPoxsA9UjqEOgjeYIbhb7Nw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/signature-v4": { + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.1.2.tgz", + "integrity": "sha512-d3+U/VpX7a60seHziWnVZOHuEgJlclufjkS6zhXvxcJgkJq4UWdH5eOBLzHRMx6gXjsdT9h6lfpmLzbrdupHgQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/is-array-buffer": "^4.0.0", + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "@smithy/util-hex-encoding": "^4.0.0", + "@smithy/util-middleware": "^4.0.4", + "@smithy/util-uri-escape": "^4.0.0", + "@smithy/util-utf8": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/smithy-client": { + "version": "4.4.9", + "resolved": "https://registry.npmjs.org/@smithy/smithy-client/-/smithy-client-4.4.9.tgz", + "integrity": "sha512-mbMg8mIUAWwMmb74LoYiArP04zWElPzDoA1jVOp3or0cjlDMgoS6WTC3QXK0Vxoc9I4zdrX0tq6qsOmaIoTWEQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.7.2", + "@smithy/middleware-endpoint": "^4.1.17", + "@smithy/middleware-stack": "^4.0.4", + "@smithy/protocol-http": "^5.1.2", + "@smithy/types": "^4.3.1", + "@smithy/util-stream": "^4.2.3", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/types": { + "version": "4.3.1", + "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.3.1.tgz", + "integrity": "sha512-UqKOQBL2x6+HWl3P+3QqFD4ncKq0I8Nuz9QItGv5WuKuMHuuwlhvqcZCoXGfc+P1QmfJE7VieykoYYmrOoFJxA==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/url-parser": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/url-parser/-/url-parser-4.0.4.tgz", + "integrity": "sha512-eMkc144MuN7B0TDA4U2fKs+BqczVbk3W+qIvcoCY6D1JY3hnAdCuhCZODC+GAeaxj0p6Jroz4+XMUn3PCxQQeQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/querystring-parser": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-base64": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/util-base64/-/util-base64-4.0.0.tgz", + "integrity": "sha512-CvHfCmO2mchox9kjrtzoHkWHxjHZzaFojLc8quxXY7WAAMAg43nuxwv95tATVgQFNDwd4M9S1qFzj40Ul41Kmg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/util-buffer-from": "^4.0.0", + "@smithy/util-utf8": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-body-length-browser": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/util-body-length-browser/-/util-body-length-browser-4.0.0.tgz", + "integrity": "sha512-sNi3DL0/k64/LO3A256M+m3CDdG6V7WKWHdAiBBMUN8S3hK3aMPhwnPik2A/a2ONN+9doY9UxaLfgqsIRg69QA==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-body-length-node": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/util-body-length-node/-/util-body-length-node-4.0.0.tgz", + "integrity": "sha512-q0iDP3VsZzqJyje8xJWEJCNIu3lktUGVoSy1KB0UWym2CL1siV3artm+u1DFYTLejpsrdGyCSWBdGNjJzfDPjg==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-buffer-from": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/util-buffer-from/-/util-buffer-from-4.0.0.tgz", + "integrity": "sha512-9TOQ7781sZvddgO8nxueKi3+yGvkY35kotA0Y6BWRajAv8jjmigQ1sBwz0UX47pQMYXJPahSKEKYFgt+rXdcug==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/is-array-buffer": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-config-provider": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/util-config-provider/-/util-config-provider-4.0.0.tgz", + "integrity": "sha512-L1RBVzLyfE8OXH+1hsJ8p+acNUSirQnWQ6/EgpchV88G6zGBTDPdXiiExei6Z1wR2RxYvxY/XLw6AMNCCt8H3w==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-defaults-mode-browser": { + "version": "4.0.25", + "resolved": "https://registry.npmjs.org/@smithy/util-defaults-mode-browser/-/util-defaults-mode-browser-4.0.25.tgz", + "integrity": "sha512-pxEWsxIsOPLfKNXvpgFHBGFC3pKYKUFhrud1kyooO9CJai6aaKDHfT10Mi5iiipPXN/JhKAu3qX9o75+X85OdQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/property-provider": "^4.0.4", + "@smithy/smithy-client": "^4.4.9", + "@smithy/types": "^4.3.1", + "bowser": "^2.11.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-defaults-mode-node": { + "version": "4.0.25", + "resolved": "https://registry.npmjs.org/@smithy/util-defaults-mode-node/-/util-defaults-mode-node-4.0.25.tgz", + "integrity": "sha512-+w4n4hKFayeCyELZLfsSQG5mCC3TwSkmRHv4+el5CzFU8ToQpYGhpV7mrRzqlwKkntlPilT1HJy1TVeEvEjWOQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/config-resolver": "^4.1.4", + "@smithy/credential-provider-imds": "^4.0.6", + "@smithy/node-config-provider": "^4.1.3", + "@smithy/property-provider": "^4.0.4", + "@smithy/smithy-client": "^4.4.9", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-endpoints": { + "version": "3.0.6", + "resolved": "https://registry.npmjs.org/@smithy/util-endpoints/-/util-endpoints-3.0.6.tgz", + "integrity": "sha512-YARl3tFL3WgPuLzljRUnrS2ngLiUtkwhQtj8PAL13XZSyUiNLQxwG3fBBq3QXFqGFUXepIN73pINp3y8c2nBmA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/node-config-provider": "^4.1.3", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-hex-encoding": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/util-hex-encoding/-/util-hex-encoding-4.0.0.tgz", + "integrity": "sha512-Yk5mLhHtfIgW2W2WQZWSg5kuMZCVbvhFmC7rV4IO2QqnZdbEFPmQnCcGMAX2z/8Qj3B9hYYNjZOhWym+RwhePw==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-middleware": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/@smithy/util-middleware/-/util-middleware-4.0.4.tgz", + "integrity": "sha512-9MLKmkBmf4PRb0ONJikCbCwORACcil6gUWojwARCClT7RmLzF04hUR4WdRprIXal7XVyrddadYNfp2eF3nrvtQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-retry": { + "version": "4.0.6", + "resolved": "https://registry.npmjs.org/@smithy/util-retry/-/util-retry-4.0.6.tgz", + "integrity": "sha512-+YekoF2CaSMv6zKrA6iI/N9yva3Gzn4L6n35Luydweu5MMPYpiGZlWqehPHDHyNbnyaYlz/WJyYAZnC+loBDZg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/service-error-classification": "^4.0.6", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-stream": { + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/@smithy/util-stream/-/util-stream-4.2.3.tgz", + "integrity": "sha512-cQn412DWHHFNKrQfbHY8vSFI3nTROY1aIKji9N0tpp8gUABRilr7wdf8fqBbSlXresobM+tQFNk6I+0LXK/YZg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/fetch-http-handler": "^5.1.0", + "@smithy/node-http-handler": "^4.1.0", + "@smithy/types": "^4.3.1", + "@smithy/util-base64": "^4.0.0", + "@smithy/util-buffer-from": "^4.0.0", + "@smithy/util-hex-encoding": "^4.0.0", + "@smithy/util-utf8": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-uri-escape": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/util-uri-escape/-/util-uri-escape-4.0.0.tgz", + "integrity": "sha512-77yfbCbQMtgtTylO9itEAdpPXSog3ZxMe09AEhm0dU0NLTalV70ghDZFR+Nfi1C60jnJoh/Re4090/DuZh2Omg==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-utf8": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@smithy/util-utf8/-/util-utf8-4.0.0.tgz", + "integrity": "sha512-b+zebfKCfRdgNJDknHCob3O7FpeYQN6ZG6YLExMcasDHsCXlsXCEuiPZeLnJLpwa5dvPetGlnGCiMHuLwGvFow==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/util-buffer-from": "^4.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/util-waiter": { + "version": "4.0.6", + "resolved": "https://registry.npmjs.org/@smithy/util-waiter/-/util-waiter-4.0.6.tgz", + "integrity": "sha512-slcr1wdRbX7NFphXZOxtxRNA7hXAAtJAXJDE/wdoMAos27SIquVCKiSqfB6/28YzQ8FCsB5NKkhdM5gMADbqxg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/abort-controller": "^4.0.4", + "@smithy/types": "^4.3.1", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, "node_modules/@tokenizer/inflate": { "version": "0.2.7", "resolved": "https://registry.npmjs.org/@tokenizer/inflate/-/inflate-0.2.7.tgz", @@ -19232,6 +19787,15 @@ "integrity": "sha512-R/BdP7OxEMc44l2Ex5lSXHoIXTB2JLNa3y2QISIbr58U/YcsffyQrYW//hZSdrfxrjRZj3GcUoxMPGdO8gSYuw==", "license": "MIT" }, + "node_modules/ansi-align": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/ansi-align/-/ansi-align-3.0.1.tgz", + "integrity": "sha512-IOfwwBF5iczOjp/WeY4YxyjqAFMQoZufdQWDd19SEExbVLNXqvpzSJ/M7Za4/sCPmQ0+GRquoA7bGcINcxew6w==", + "license": "ISC", + "dependencies": { + "string-width": "^4.1.0" + } + }, "node_modules/ansi-regex": { "version": "5.0.1", "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", @@ -19256,6 +19820,49 @@ "url": "https://github.com/chalk/ansi-styles?sponsor=1" } }, + "node_modules/atomic-sleep": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/atomic-sleep/-/atomic-sleep-1.0.0.tgz", + "integrity": "sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ==", + "license": "MIT", + "engines": { + "node": ">=8.0.0" + } + }, + "node_modules/bintrees": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/bintrees/-/bintrees-1.0.2.tgz", + "integrity": "sha512-VOMgTMwjAaUG580SXn3LacVgjurrbMme7ZZNYGSSV7mmtY6QQRh0Eg3pwIcntQ77DErK1L0NxkbetjcoXzVwKw==", + "license": "MIT" + }, + "node_modules/bowser": { + "version": "2.11.0", + "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.11.0.tgz", + "integrity": "sha512-AlcaJBi/pqqJBIQ8U9Mcpc9i8Aqxn88Skv5d+xBX006BY5u8N3mGLHa5Lgppa7L/HfwgwLgZ6NYs+Ag6uUmJRA==", + "license": "MIT" + }, + "node_modules/boxen": { + "version": "5.1.2", + "resolved": "https://registry.npmjs.org/boxen/-/boxen-5.1.2.tgz", + "integrity": "sha512-9gYgQKXx+1nP8mP7CzFyaUARhg7D3n1dF/FnErWmu9l6JvGpNUN278h0aSb+QjoiKSWG+iZ3uHrcqk0qrY9RQQ==", + "license": "MIT", + "dependencies": { + "ansi-align": "^3.0.0", + "camelcase": "^6.2.0", + "chalk": "^4.1.0", + "cli-boxes": "^2.2.1", + "string-width": "^4.2.2", + "type-fest": "^0.20.2", + "widest-line": "^3.1.0", + "wrap-ansi": "^7.0.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/call-bind-apply-helpers": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", @@ -19285,11 +19892,22 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/camelcase": { + "version": "6.3.0", + "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-6.3.0.tgz", + "integrity": "sha512-Gmy6FhYlCY7uOElZUSbxo2UCDH8owEk996gkbrpsgGtrJLM3J7jGxl9Ic7Qwwj4ivOE5AWZWRMecDdF7hqGjFA==", + "license": "MIT", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/chalk": { "version": "4.1.2", "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz", "integrity": "sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==", - "dev": true, "license": "MIT", "dependencies": { "ansi-styles": "^4.1.0", @@ -19306,7 +19924,6 @@ "version": "7.2.0", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz", "integrity": "sha512-qpCAvRl9stuOHveKsn7HncJRvv501qIacKzQlO/+Lwxc9+0q2wLyv4Dfvt80/DPn2pqOBsJdDiogXGR9+OvwRw==", - "dev": true, "license": "MIT", "dependencies": { "has-flag": "^4.0.0" @@ -19315,6 +19932,27 @@ "node": ">=8" } }, + "node_modules/check-disk-space": { + "version": "3.4.0", + "resolved": "https://registry.npmjs.org/check-disk-space/-/check-disk-space-3.4.0.tgz", + "integrity": "sha512-drVkSqfwA+TvuEhFipiR1OC9boEGZL5RrWvVsOthdcvQNXyCCuKkEiTOTXZ7qxSf/GLwq4GvzfrQD/Wz325hgw==", + "license": "MIT", + "engines": { + "node": ">=16" + } + }, + "node_modules/cli-boxes": { + "version": "2.2.1", + "resolved": "https://registry.npmjs.org/cli-boxes/-/cli-boxes-2.2.1.tgz", + "integrity": "sha512-y4coMcylgSCdVinjiDBuR8PCC2bLjyGTwEmPb9NHR/QaNU6EUOXcTY/s6VjGMD6ENSEaeQYHCY0GNGS5jfMwPw==", + "license": "MIT", + "engines": { + "node": ">=6" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/cliui": { "version": "8.0.1", "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", @@ -19490,12 +20128,39 @@ "node": ">=6" } }, + "node_modules/fast-redact": { + "version": "3.5.0", + "resolved": "https://registry.npmjs.org/fast-redact/-/fast-redact-3.5.0.tgz", + "integrity": "sha512-dwsoQlS7h9hMeYUq1W++23NDcBLV4KqONnITDV9DjfS3q1SgDGVrBdvvTLUotWtPSD7asWDV9/CmsZPy8Hf70A==", + "license": "MIT", + "engines": { + "node": ">=6" + } + }, "node_modules/fast-safe-stringify": { "version": "2.1.1", "resolved": "https://registry.npmjs.org/fast-safe-stringify/-/fast-safe-stringify-2.1.1.tgz", "integrity": "sha512-W+KJc2dmILlPplD/H4K9l9LcAHAfPtP6BY84uVLXQ6Evcz9Lcg33Y2z1IVblT6xdY54PXYVHEv+0Wpq8Io6zkA==", "license": "MIT" }, + "node_modules/fast-xml-parser": { + "version": "5.2.5", + "resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-5.2.5.tgz", + "integrity": "sha512-pfX9uG9Ki0yekDHx2SiuRIyFdyAr1kMIMitPvb0YBo8SUfKvia7w7FIyd/l6av85pFYRhZscS75MwMnbvY+hcQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "strnum": "^2.1.0" + }, + "bin": { + "fxparser": "src/cli/cli.js" + } + }, "node_modules/fflate": { "version": "0.8.2", "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.2.tgz", @@ -19591,7 +20256,6 @@ "version": "4.0.0", "resolved": "https://registry.npmjs.org/has-flag/-/has-flag-4.0.0.tgz", "integrity": "sha512-EykJT/Q1KjTWctppgIAgfSO0tKVuZUjhgMr17kqTumMl6Afv3EISleU7qZUzoXDFTAHTDC4NOoG/ZxU3EvlMPQ==", - "dev": true, "license": "MIT", "engines": { "node": ">=8" @@ -19717,6 +20381,22 @@ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "license": "MIT" }, + "node_modules/nestjs-pino": { + "version": "4.4.0", + "resolved": "https://registry.npmjs.org/nestjs-pino/-/nestjs-pino-4.4.0.tgz", + "integrity": "sha512-+GMNlcNWDRrMtlQftfcxN+5pV2C25A4wsYIY7cfRJTMW4b8IFKYReDrG1lUp5LGql9fXemmnVJ2Ww10iIkCZPQ==", + "hasInstallScript": true, + "license": "MIT", + "engines": { + "node": ">= 14" + }, + "peerDependencies": { + "@nestjs/common": "^8.0.0 || ^9.0.0 || ^10.0.0 || ^11.0.0", + "pino": "^7.5.0 || ^8.0.0 || ^9.0.0", + "pino-http": "^6.4.0 || ^7.0.0 || ^8.0.0 || ^9.0.0 || ^10.0.0", + "rxjs": "^7.1.0" + } + }, "node_modules/object-inspect": { "version": "1.13.4", "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", @@ -19729,6 +20409,15 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/on-exit-leak-free": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/on-exit-leak-free/-/on-exit-leak-free-2.1.2.tgz", + "integrity": "sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA==", + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, "node_modules/path-to-regexp": { "version": "8.2.0", "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.2.0.tgz", @@ -19738,6 +20427,84 @@ "node": ">=16" } }, + "node_modules/pino": { + "version": "9.7.0", + "resolved": "https://registry.npmjs.org/pino/-/pino-9.7.0.tgz", + "integrity": "sha512-vnMCM6xZTb1WDmLvtG2lE/2p+t9hDEIvTWJsu6FejkE62vB7gDhvzrpFR4Cw2to+9JNQxVnkAKVPA1KPB98vWg==", + "license": "MIT", + "dependencies": { + "atomic-sleep": "^1.0.0", + "fast-redact": "^3.1.1", + "on-exit-leak-free": "^2.1.0", + "pino-abstract-transport": "^2.0.0", + "pino-std-serializers": "^7.0.0", + "process-warning": "^5.0.0", + "quick-format-unescaped": "^4.0.3", + "real-require": "^0.2.0", + "safe-stable-stringify": "^2.3.1", + "sonic-boom": "^4.0.1", + "thread-stream": "^3.0.0" + }, + "bin": { + "pino": "bin.js" + } + }, + "node_modules/pino-abstract-transport": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/pino-abstract-transport/-/pino-abstract-transport-2.0.0.tgz", + "integrity": "sha512-F63x5tizV6WCh4R6RHyi2Ml+M70DNRXt/+HANowMflpgGFMAym/VKm6G7ZOQRjqN7XbGxK1Lg9t6ZrtzOaivMw==", + "license": "MIT", + "dependencies": { + "split2": "^4.0.0" + } + }, + "node_modules/pino-http": { + "version": "10.5.0", + "resolved": "https://registry.npmjs.org/pino-http/-/pino-http-10.5.0.tgz", + "integrity": "sha512-hD91XjgaKkSsdn8P7LaebrNzhGTdB086W3pyPihX0EzGPjq5uBJBXo4N5guqNaK6mUjg9aubMF7wDViYek9dRA==", + "license": "MIT", + "dependencies": { + "get-caller-file": "^2.0.5", + "pino": "^9.0.0", + "pino-std-serializers": "^7.0.0", + "process-warning": "^5.0.0" + } + }, + "node_modules/pino-std-serializers": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/pino-std-serializers/-/pino-std-serializers-7.0.0.tgz", + "integrity": "sha512-e906FRY0+tV27iq4juKzSYPbUj2do2X2JX4EzSca1631EB2QJQUqGbDuERal7LCtOpxl6x3+nvo9NPZcmjkiFA==", + "license": "MIT" + }, + "node_modules/process-warning": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/process-warning/-/process-warning-5.0.0.tgz", + "integrity": "sha512-a39t9ApHNx2L4+HBnQKqxxHNs1r7KF+Intd8Q/g1bUh6q0WIp9voPXJ/x0j+ZL45KF1pJd9+q2jLIRMfvEshkA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "license": "MIT" + }, + "node_modules/prom-client": { + "version": "15.1.3", + "resolved": "https://registry.npmjs.org/prom-client/-/prom-client-15.1.3.tgz", + "integrity": "sha512-6ZiOBfCywsD4k1BN9IX0uZhF+tJkV8q8llP64G5Hajs4JOeVLPCwpPVcpXy3BwYiUGgyJzsJJQeOIv7+hDSq8g==", + "license": "Apache-2.0", + "dependencies": { + "@opentelemetry/api": "^1.4.0", + "tdigest": "^0.1.1" + }, + "engines": { + "node": "^16 || ^18 || >=20" + } + }, "node_modules/qs": { "version": "6.14.0", "resolved": "https://registry.npmjs.org/qs/-/qs-6.14.0.tgz", @@ -19753,6 +20520,21 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/quick-format-unescaped": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/quick-format-unescaped/-/quick-format-unescaped-4.0.4.tgz", + "integrity": "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==", + "license": "MIT" + }, + "node_modules/real-require": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/real-require/-/real-require-0.2.0.tgz", + "integrity": "sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==", + "license": "MIT", + "engines": { + "node": ">= 12.13.0" + } + }, "node_modules/reflect-metadata": { "version": "0.2.2", "resolved": "https://registry.npmjs.org/reflect-metadata/-/reflect-metadata-0.2.2.tgz", @@ -19777,6 +20559,15 @@ "tslib": "^2.1.0" } }, + "node_modules/safe-stable-stringify": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/safe-stable-stringify/-/safe-stable-stringify-2.5.0.tgz", + "integrity": "sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA==", + "license": "MIT", + "engines": { + "node": ">=10" + } + }, "node_modules/shell-quote": { "version": "1.8.3", "resolved": "https://registry.npmjs.org/shell-quote/-/shell-quote-1.8.3.tgz", @@ -19862,12 +20653,30 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/sonic-boom": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/sonic-boom/-/sonic-boom-4.2.0.tgz", + "integrity": "sha512-INb7TM37/mAcsGmc9hyyI6+QR3rR1zVRu36B0NeGXKnOOLiZOfER5SA+N7X7k3yUYRzLWafduTDvJAfDswwEww==", + "license": "MIT", + "dependencies": { + "atomic-sleep": "^1.0.0" + } + }, "node_modules/spawn-command": { "version": "0.0.2", "resolved": "https://registry.npmjs.org/spawn-command/-/spawn-command-0.0.2.tgz", "integrity": "sha512-zC8zGoGkmc8J9ndvml8Xksr1Amk9qBujgbF0JAIWO7kXr43w0h/0GJNM/Vustixu+YE8N/MTrQ7N31FvHUACxQ==", "dev": true }, + "node_modules/split2": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz", + "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==", + "license": "ISC", + "engines": { + "node": ">= 10.x" + } + }, "node_modules/string-width": { "version": "4.2.3", "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", @@ -19914,6 +20723,18 @@ } } }, + "node_modules/strnum": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/strnum/-/strnum-2.1.1.tgz", + "integrity": "sha512-7ZvoFTiCnGxBtDqJ//Cu6fWtZtc7Y3x+QOirG15wztbdngGSkht27o2pyGWrVy0b4WAy3jbKmnoK6g5VlVNUUw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT" + }, "node_modules/strtok3": { "version": "10.3.4", "resolved": "https://registry.npmjs.org/strtok3/-/strtok3-10.3.4.tgz", @@ -19946,6 +20767,24 @@ "url": "https://github.com/chalk/supports-color?sponsor=1" } }, + "node_modules/tdigest": { + "version": "0.1.2", + "resolved": "https://registry.npmjs.org/tdigest/-/tdigest-0.1.2.tgz", + "integrity": "sha512-+G0LLgjjo9BZX2MfdvPfH+MKLCrxlXSYec5DaPYP1fe6Iyhf0/fSmJ0bFiZ1F8BT6cGXl2LpltQptzjXKWEkKA==", + "license": "MIT", + "dependencies": { + "bintrees": "1.0.2" + } + }, + "node_modules/thread-stream": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/thread-stream/-/thread-stream-3.1.0.tgz", + "integrity": "sha512-OqyPZ9u96VohAyMfJykzmivOrY2wfMSf3C5TtFJVgN+Hm6aj+voFhlK+kZEIv2FBh1X6Xp3DlnCOfEQ3B2J86A==", + "license": "MIT", + "dependencies": { + "real-require": "^0.2.0" + } + }, "node_modules/token-types": { "version": "6.0.4", "resolved": "https://registry.npmjs.org/token-types/-/token-types-6.0.4.tgz", @@ -19979,6 +20818,18 @@ "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", "license": "0BSD" }, + "node_modules/type-fest": { + "version": "0.20.2", + "resolved": "https://registry.npmjs.org/type-fest/-/type-fest-0.20.2.tgz", + "integrity": "sha512-Ne+eE4r0/iWnpAxD852z3A+N0Bt5RN//NjJwRd2VFHEmrywxf5vsZlh4R6lixl6B+wz/8d+maTSAkN1FIkI3LQ==", + "license": "(MIT OR CC0-1.0)", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/uid": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/uid/-/uid-2.0.2.tgz", @@ -20003,6 +20854,31 @@ "url": "https://github.com/sponsors/sindresorhus" } }, + "node_modules/uuid": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/uuid/-/uuid-9.0.1.tgz", + "integrity": "sha512-b+1eJOlsR9K8HJpow9Ok3fiWOWSIcIzXodvv0rQjVoOVNpWMpxf1wZNpt4y9h10odCNrqnYp1OBzRktckBe3sA==", + "funding": [ + "https://github.com/sponsors/broofa", + "https://github.com/sponsors/ctavan" + ], + "license": "MIT", + "bin": { + "uuid": "dist/bin/uuid" + } + }, + "node_modules/widest-line": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/widest-line/-/widest-line-3.1.0.tgz", + "integrity": "sha512-NsmoXalsWVDMGupxZ5R08ka9flZjjiLvHVAWYOKtiKM8ujtZWr9cRffak+uSE48+Ob8ObalXpwyeUiyDD6QFgg==", + "license": "MIT", + "dependencies": { + "string-width": "^4.0.0" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/wrap-ansi": { "version": "7.0.0", "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",