Health & Queue Stats

Monitor the health of your Mend instance and track queue statistics for operational insights.

Health Check

GET /health

Public No authentication required

Check if the service is healthy and all dependencies are operational.

Response

{
  "status": "healthy",
  "timestamp": "2025-10-18T22:00:00Z",
  "system": {
    "memory_free_mb": 2048.5,
    "disk_free_gb": 125.3,
    "disk_used_gb": 74.7,
    "disk_total_gb": 200.0
  },
  "dependencies": {
    "redis": {
      "status": "healthy",
      "latency_ms": 1.23,
      "last_check": "2025-10-18T22:00:00Z"
    },
    "s3": {
      "status": "healthy",
      "latency_ms": 45.67,
      "last_check": "2025-10-18T22:00:00Z"
    }
  }
}

Status Values

healthy

All systems operational

degraded

Some non-critical issues detected

unhealthy

Critical dependencies unavailable

Example

curl http://localhost:8080/health

---

Liveness Check

GET /health/live

Public No authentication required

Simple check to verify the service is running. Used by orchestrators like Kubernetes.

Response

{
  "status": "healthy",
  "timestamp": "2025-10-18T22:00:00Z"
}

Example

curl http://localhost:8080/health/live

Note

This endpoint always returns 200 if the service is running, regardless of dependency health.

---

Readiness Check

GET /health/ready

Public No authentication required

Detailed check to verify the service is ready to accept requests. Validates all dependencies.

Response (Ready)

{
  "status": "healthy",
  "timestamp": "2025-10-18T22:00:00Z",
  "dependencies": {
    "redis": {
      "status": "healthy",
      "latency_ms": 1.23
    },
    "s3": {
      "status": "healthy",
      "latency_ms": 45.67
    }
  }
}

Response (Not Ready)

503 Service Unavailable

{
  "status": "unhealthy",
  "timestamp": "2025-10-18T22:00:00Z",
  "dependencies": {
    "redis": {
      "status": "unhealthy",
      "error": "connection refused",
      "latency_ms": 0
    }
  }
}

Example

curl http://localhost:8080/health/ready

Kubernetes Integration

Use /health/live for liveness probes and /health/ready for readiness probes in your Kubernetes deployment.

---

Queue Statistics

GET /api/v1/queue/stats

Protected Requires API key

Get statistics for all processing queues.

Response

{
  "total_jobs": 150,
  "total_pending": 25,
  "total_processing": 10,
  "total_completed": 110,
  "total_failed": 5,
  "queues": {
    "image_resize": {
      "name": "image_resize",
      "pending": 10,
      "processing": 3,
      "completed": 45,
      "failed": 2
    },
    "image_optimize": {
      "name": "image_optimize",
      "pending": 8,
      "processing": 2,
      "completed": 35,
      "failed": 1
    },
    "video_thumbnail": {
      "name": "video_thumbnail",
      "pending": 5,
      "processing": 3,
      "completed": 20,
      "failed": 1
    },
    "audio_convert": {
      "name": "audio_convert",
      "pending": 2,
      "processing": 2,
      "completed": 10,
      "failed": 1
    }
  }
}

Example

cURL

curl -H "X-API-Key: your_api_key_here" \
  http://localhost:8080/api/v1/queue/stats

JavaScript

const response = await fetch('http://localhost:8080/api/v1/queue/stats', {
  headers: {
    'X-API-Key': 'your_api_key_here'
  }
});
const stats = await response.json();
console.log(`Total jobs: ${stats.total_jobs}`);
console.log(`Pending: ${stats.total_pending}`);

---

Use Cases

Monitoring Dashboards

Display real-time queue statistics and health status

Auto-Scaling

Scale workers based on pending job counts

Alerting

Trigger alerts when queues grow too large or dependencies fail

Load Balancing

Route jobs based on queue depths

---

Monitoring Integration

Prometheus Metrics

GET /metrics

Public No authentication required

Prometheus-compatible metrics endpoint for monitoring.

curl http://localhost:8080/metrics

Example metrics:

# HELP mend_jobs_total Total number of jobs processed
# TYPE mend_jobs_total counter
mend_jobs_total{type="image_resize",status="completed"} 1234

# HELP mend_queue_depth Current queue depth
# TYPE mend_queue_depth gauge
mend_queue_depth{queue="image_resize"} 10

# HELP mend_job_duration_seconds Job processing duration
# TYPE mend_job_duration_seconds histogram
mend_job_duration_seconds_bucket{type="image_resize",le="1"} 450

Learn more in the Metrics Guide.

Grafana Dashboard

Import the provided Grafana dashboard for comprehensive monitoring:

• Queue depths and throughput
• Job success/failure rates
• Processing times and latencies
• System resource usage

---

Health Check Best Practices

Recommendations

1. Monitor regularly: Check health every 30-60 seconds
2. Set up alerts: Notify on degraded or unhealthy status
3. Use readiness probes: Prevent traffic to unhealthy instances
4. Track trends: Monitor queue growth over time
5. Capacity planning: Scale workers based on queue stats

Example Monitoring Script

#!/bin/bash

# Check health and alert if unhealthy
HEALTH=$(curl -s http://localhost:8080/health | jq -r '.status')

if [ "$HEALTH" != "healthy" ]; then
  echo "ALERT: Service is $HEALTH"
  # Send alert (email, Slack, PagerDuty, etc.)
fi

# Check queue depth
PENDING=$(curl -s -H "X-API-Key: $API_KEY" \
  http://localhost:8080/api/v1/queue/stats | jq '.total_pending')

if [ "$PENDING" -gt 100 ]; then
  echo "ALERT: Queue depth is $PENDING (threshold: 100)"
  # Scale up workers
fi

---

Kubernetes Health Probes

Example Kubernetes deployment with health checks:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mend-api
spec:
  template:
    spec:
      containers:
      - name: mend-api
        image: mend:latest
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10

---

Response Codes

Endpoint	Success	Unhealthy
/health	200	200 (with status: unhealthy)
/health/live	200	200 (always)
/health/ready	200	503
/api/v1/queue/stats	200	500
/metrics	200	200 (always)

---

Next Steps

• Set up Metrics & Monitoring
• Learn about Deployment
• Explore Architecture