Real-time Alerts & Emergency Notifications: Critical Communication When It Matters Most

• Server Downtime: Immediate alert when production servers go down
• Database Issues: Notification of connection failures, high CPU usage, or disk space warnings
• API Performance: Alert when response times exceed thresholds or error rates spike
• Security Threats: Instant notification of unusual login attempts or suspicious activity
• Backup Failures: Alert when scheduled backups don't complete successfully

• Typhoon/Flood Warnings: Alert residents to evacuate immediately
• Earthquake Alerts: Instant notification of seismic activity
• Weather Emergencies: Storm, tornado, or extreme weather warnings
• Public Health Alerts: Disease outbreak notifications or vaccine availability
• Community Safety: Missing persons, Amber alerts, or security threats

• Payment Processing Failures: Alert when credit card processing encounters errors
• Inventory Warnings: Notification when stock levels fall below thresholds
• Order Fulfillment Issues: Alert when orders can't be fulfilled or shipped
• Customer Service Escalation: Notify managers of critical customer complaints
• Financial Alerts: Unusual transaction activity or fraud detection

• Lab Result Alerts: Notify doctors of critical or abnormal test results immediately
• Patient Deterioration: Alert medical staff when patient vital signs become critical
• Medication Alerts: Notification of drug interactions or allergies
• Hospital Staff Emergency: Rapid notification during medical emergencies (Code Blue, etc.)
• Equipment Failure: Alert when medical equipment malfunctions or needs maintenance

• Application performance monitoring (APM) tools detect issues
• Sensor networks detect environmental changes
• Security systems identify suspicious activity
• Database monitoring detects performance problems

# Example: Server down alert
if server_status == 'down' && time_down > 30_seconds
  trigger_alert(
    severity: 'critical',
    message: 'Production server is down',
    recipients: ['ops-team@company.com']
  )
end

POST /api/v1/sms_messages
{
  "api_token": "your_api_token",
  "phone_number": "639171234567",
  "message": "ALERT: Production server down (Server-1). Investigating. Reply YES to confirm."
}

• Critical: System down, security breach, data loss risk → Send SMS immediately
• High: Performance degradation, failing services → Send SMS after 5 minutes if unresolved
• Medium: Warnings and non-critical issues → Email only, no SMS

# Escalation example
if alert_triggered
  send_sms_to(primary_on_call)
  wait(5.minutes)
  
  if no_acknowledgment
    send_sms_to(secondary_on_call)
    send_sms_to(manager)
  end
end

# Alert with response options
Message: "ALERT: Database error. Reply: 
1=Investigating, 2=Starting maintenance, 3=Escalate"

User replies: "1"
System acknowledges and logs response

Good: "CRITICAL: API response time >5s. CPU at 95%. Restart job queued."
Bad: "There's an issue"

Good: "SECURITY: 50 failed logins detected from 203.45.67.x in 2 min"
Bad: "Security alert"

Good: "DATA: Backup failed. Last successful: 2 days ago. Action?"
Bad: "Backup error"

1. Monitoring system detects API unavailable (HTTP 502)
2. Immediately sends SMS: "CRITICAL: Payment API down. Customers can't checkout. ETA: investigating"
3. Team responds via SMS: "1=Restarting service"
4. Follow-up SMS after 2 min: "Payment API restored. Monitoring for stability."

1. Security system detects 30 failed logins from unknown IP in 5 minutes
2. Sends SMS: "SECURITY: Bruteforce detected from 210.35.42.x. Account locked. Reply: 1=Investigate, 2=Reset password"
3. Team member replies: "1"
4. Follow-up SMS: "IP blocked. Checking for unauthorized access. Monitor email for detailed report."

1. Vital signs monitor detects critical heart rate (35 bpm)
2. Sends SMS: "CRITICAL: Patient (Room 305) HR=35. O2=88%. Respond: Attending"
3. Doctor responds: "En route"
4. System logs response and notifies nursing station

• Only send SMS for truly critical issues
• Use email/Slack for non-critical notifications
• Implement alert deduplication (don't send same alert multiple times)
• Use escalation instead of spamming multiple messages

• Start with severity: CRITICAL, HIGH, WARNING
• Identify the problem in 1-2 sentences
• Include impact: "5,000 users affected"
• Suggest action: "Check email for details" or "Reply for options"

• CRITICAL alerts: 24/7 (life-threatening, security, revenue impact)
• HIGH alerts: Only during business hours or to on-call staff
• Allow team members to customize alert preferences
• Respect do-not-disturb settings for off-hours

• Include ticket/incident reference number
• Link to dashboard or monitoring system: "See status.company.com"
• Mention related alerts: "See related CPU alert 15 min ago"
• Include metric values that triggered alert

• Log SMS responses automatically
• Track response time (critical metric for incident response)
• Escalate if no acknowledgment within timeframe
• Review alert acknowledgment data for process improvement

• Monthly "fire drill" tests of critical alerts
• Verify all phone numbers are current and working
• Test escalation paths to ensure everyone gets notified
• Practice incident response procedures with team

import requests
import os
from datetime import datetime

def check_server_health(server_id):
    # Check server status
    status = get_server_status(server_id)
    
    if status['cpu'] > 90:
        send_alert(
            f"HIGH: {server_id} CPU at {status['cpu']}%. "
            f"Memory: {status['memory']}%. "
            f"Investigate: status.company.com/server/{server_id}"
        )
    
    if status['available'] == False:
        send_critical_alert(
            f"CRITICAL: {server_id} is DOWN. "
            f"Last response: {status['last_check']}. "
            f"Action needed immediately!"
        )

def send_alert(message):
    """Send non-critical alert"""
    url = 'https://iprogsms.com/api/v1/sms_messages'
    
    data = {
        'api_token': os.getenv('IPROG_SMS_API_TOKEN'),
        'phone_number': os.getenv('ON_CALL_PRIMARY'),  # Primary on-call
        'message': message
    }
    
    response = requests.post(url, json=data)
    return response.json()

def send_critical_alert(message):
    """Send critical alert with escalation"""
    primary = send_alert(message)
    
    # If primary doesn't acknowledge in 2 minutes, escalate
    time.sleep(120)
    
    if not is_acknowledged(primary['message_id']):
        escalate_alert(message)

def escalate_alert(message):
    """Escalate to backup on-call and manager"""
    recipients = [
        os.getenv('ON_CALL_BACKUP'),
        os.getenv('MANAGER_PHONE')
    ]
    
    for phone in recipients:
        send_sms(phone, f"ESCALATION: {message}")

# Usage
check_server_health('server-prod-1')

# app/services/database_alert_service.rb
class DatabaseAlertService
  def self.check_database_health
    db_status = ActiveRecord::Base.connection.execute('SELECT 1').first
    disk_usage = ActiveRecord::Base.connection.execute('SELECT * FROM disk_info').first
    
    if disk_usage['used_percent'] > 85
      send_alert(
        "HIGH: Database disk at #{disk_usage['used_percent']}%. " \
        "Free space: #{disk_usage['free_gb']}GB. " \
        "Cleanup or expand soon."
      )
    end
  rescue StandardError => e
    send_critical_alert(
      "CRITICAL: Database unreachable! Error: #{e.message}. " \
      "Attempting failover. Check logs immediately."
    )
  end
  
  def self.send_alert(message)
    SmsService.send_message(
      phone: ENV['ALERT_PHONE'],
      message: message,
      priority: :high
    )
  end
end

# Schedule with Sidekiq
class DatabaseHealthCheckJob
  include Sidekiq::Job
  
  sidekiq_options retry: 0
  
  def perform
    DatabaseAlertService.check_database_health
  end
end

• Alert Response Time: How long until someone acknowledges the alert
• Mean Time to Detect (MTTD): How quickly issues are identified
• Mean Time to Recovery (MTTR): How quickly issues are resolved after detection
• Alert Accuracy: Percentage of alerts that correspond to actual issues
• Alert Volume: Number of alerts per day (watch for alert fatigue)
• Escalation Rate: Percentage of alerts that required escalation