Complete Webhook Implementation Guide: Build Reliable Event-Driven APIs

Webhooks power real-time integrations for millions of applications. This comprehensive guide covers everything you need to build production-ready webhooks: security, reliability, retry logic, testing, and best practices used by industry leaders like Stripe, GitHub, and Shopify.

What Are Webhooks?

Webhooks are HTTP callbacks that deliver event notifications in real-time. Instead of your application repeatedly asking a service "anything new?" (polling), the service sends you a message when something happens.

Key Concept

Webhooks are "reverse APIs" — instead of you calling an API endpoint, the API calls your endpoint.

How Webhooks Work

1. Registration: You register a webhook URL with a service (e.g., https://yourdomain.com/webhooks/stripe)
2. Event Occurs: Something happens in the service (payment succeeded, PR merged, order created)
3. HTTP POST: The service sends an HTTP POST request to your URL with event data
4. Processing: Your application receives and processes the event
5. Acknowledgment: You respond with HTTP 200 to confirm receipt

// Example webhook payload from Stripe
POST https://yourdomain.com/webhooks/stripe
Content-Type: application/json
Stripe-Signature: t=1678886400,v1=5257a869...

{
  "id": "evt_1MqLN82eZvKYlo2C8JzKzzYI",
  "type": "payment_intent.succeeded",
  "data": {
    "object": {
      "id": "pi_3MqLN82eZvKYlo2C0XHzlJC9",
      "amount": 2000,
      "currency": "usd",
      "status": "succeeded"
    }
  },
  "created": 1678886400
}

When to Use Webhooks vs Polling

✅ Use Webhooks When:

• Events are unpredictable (payments, form submissions)
• Real-time notifications matter (security alerts)
• You want to reduce API calls (cost, rate limits)
• Events are rare (password resets, signups)
• You need push-based architecture

⚠️ Use Polling When:

• You need to query historical data
• Webhook receiver can't be publicly accessible
• Provider doesn't support webhooks
• You need pull-based control (rate limiting)
• Events are frequent and you batch process

Cost Comparison: Webhooks vs Polling

Scenario: Monitoring 1,000 payment transactions/day

❌ Polling Approach (every 30 seconds)

2,880 API calls/day × 1,000 resources = 2.88M API calls/day

✅ Webhook Approach

1,000 events/day = 1,000 webhook deliveries/day

Result: 99.97% reduction in API calls

Webhook Architecture Patterns

Pattern 1: Direct Processing (Simple)

// Direct processing - good for fast operations (<5s)
app.post('/webhooks/stripe', async (req, res) => {
  const event = req.body;
  
  // Verify signature
  const isValid = verifySignature(req);
  if (!isValid) return res.status(401).send('Invalid signature');
  
  // Process event
  await handleStripeEvent(event);
  
  // Acknowledge receipt
  res.status(200).json({ received: true });
});

async function handleStripeEvent(event) {
  switch (event.type) {
    case 'payment_intent.succeeded':
      await updateOrderStatus(event.data.object.id, 'paid');
      await sendConfirmationEmail(event.data.object);
      break;
    // ... other event types
  }
}

// ✅ Pros: Simple, immediate processing
// ❌ Cons: Timeout risk, blocks webhook receiver

Pattern 2: Queue-Based Processing (Production-Ready)

// Queue-based - best for production
import { Queue } from 'bullmq';

const webhookQueue = new Queue('webhooks', {
  connection: { host: 'localhost', port: 6379 }
});

app.post('/webhooks/stripe', async (req, res) => {
  const event = req.body;
  
  // Verify signature
  const isValid = verifySignature(req);
  if (!isValid) return res.status(401).send('Invalid signature');
  
  // Add to queue (fast, non-blocking)
  await webhookQueue.add('stripe-event', event, {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 }
  });
  
  // Acknowledge immediately
  res.status(200).json({ received: true });
});

// Worker processes queue in background
const worker = new Worker('webhooks', async (job) => {
  const event = job.data;
  await handleStripeEvent(event);
}, { connection: { host: 'localhost', port: 6379 } });

// ✅ Pros: Fast acknowledgment, handles failures, scalable
// ❌ Cons: More infrastructure (Redis, workers)

Best Practice

Respond within 5 seconds to avoid timeouts. Use queues for any processing that takes longer than 2-3 seconds.

Pattern 3: Event Sourcing (Enterprise)

// Store all events, process later
app.post('/webhooks/stripe', async (req, res) => {
  const event = req.body;
  
  // Verify signature
  const isValid = verifySignature(req);
  if (!isValid) return res.status(401).send('Invalid signature');
  
  // Store event immediately
  await db.events.insert({
    provider: 'stripe',
    type: event.type,
    payload: event,
    received_at: new Date(),
    processed: false
  });
  
  // Acknowledge
  res.status(200).json({ received: true });
});

// Background processor
setInterval(async () => {
  const unprocessed = await db.events.findUnprocessed();
  for (const event of unprocessed) {
    await processEvent(event);
    await db.events.update(event.id, { processed: true });
  }
}, 5000);

// ✅ Pros: Full audit trail, replayable, debugging-friendly
// ❌ Cons: More storage, eventual consistency

Security: Signature Verification

⚠️ Critical Security Warning

Never trust webhook payloads without signature verification. Anyone can POST data to your webhook endpoint. Always verify the sender's identity.

HMAC Signature Verification (Stripe Pattern)

import crypto from 'crypto';

function verifyStripeSignature(req, secret) {
  const signature = req.headers['stripe-signature'];
  const body = JSON.stringify(req.body);
  
  // Parse signature header
  const elements = signature.split(',');
  const timestamp = elements.find(e => e.startsWith('t='))?.split('=')[1];
  const sig = elements.find(e => e.startsWith('v1='))?.split('=')[1];
  
  // Prevent replay attacks (reject events older than 5 minutes)
  const currentTime = Math.floor(Date.now() / 1000);
  if (currentTime - parseInt(timestamp) > 300) {
    throw new Error('Webhook timestamp too old');
  }
  
  // Compute expected signature
  const signedPayload = `${timestamp}.${body}`;
  const expectedSig = crypto
    .createHmac('sha256', secret)
    .update(signedPayload, 'utf8')
    .digest('hex');
  
  // Constant-time comparison (prevents timing attacks)
  return crypto.timingSafeEqual(
    Buffer.from(sig, 'hex'),
    Buffer.from(expectedSig, 'hex')
  );
}

// Usage in Express
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), (req, res) => {
  try {
    const isValid = verifyStripeSignature(req, process.env.STRIPE_WEBHOOK_SECRET);
    if (!isValid) return res.status(401).send('Invalid signature');
    
    // Process webhook...
    res.status(200).json({ received: true });
  } catch (err) {
    console.error('Signature verification failed:', err);
    res.status(400).send(`Webhook Error: ${err.message}`);
  }
});

Other Security Best Practices

1. HTTPS Only

Never accept webhooks over HTTP. Attackers can intercept and modify payloads.

// Enforce HTTPS in production
if (process.env.NODE_ENV === 'production' && req.protocol !== 'https') {
  return res.status(403).send('HTTPS required');
}

2. IP Allowlisting (Optional Layer)

Some providers publish their webhook sender IPs. Combine with signature verification for defense-in-depth.

// Example: GitHub webhook IPs
const GITHUB_IPS = [
  '192.30.252.0/22', '185.199.108.0/22', '140.82.112.0/20'
];

function isFromGitHub(ip) {
  return GITHUB_IPS.some(range => ipInRange(ip, range));
}

3. Idempotency Keys

Webhooks may be delivered multiple times. Use idempotency keys to prevent duplicate processing.

// Store processed event IDs
const processedEvents = new Set();

async function processEvent(event) {
  // Check if already processed
  if (processedEvents.has(event.id)) {
    console.log(`Event ${event.id} already processed, skipping`);
    return;
  }
  
  // Process event
  await handleEvent(event);
  
  // Mark as processed
  processedEvents.add(event.id);
  // In production, store in database/Redis
}

4. Rate Limiting

Protect against webhook floods (accidental or malicious).

import rateLimit from 'express-rate-limit';

const webhookLimiter = rateLimit({
  windowMs: 60 * 1000, // 1 minute
  max: 100, // 100 requests per minute per IP
  message: 'Too many webhook requests'
});

app.post('/webhooks/*', webhookLimiter);

Reliability: Retry Logic & Idempotency

Provider-Side Retry Logic

Most webhook providers (like Stripe, GitHub, Shopify) automatically retry failed deliveries. Understanding their retry behavior is critical for building reliable receivers.

Typical Retry Pattern (Stripe)

• Immediately after failure
• 5 seconds later
• 30 seconds later
• 2 minutes later
• 5 minutes later
• 10 minutes later
• 30 minutes later
• 1 hour later
• Up to 3 days total retry window

What Triggers a Retry?

❌ Failures (Will Retry)

• HTTP 5xx responses (500, 502, 503, 504)
• Connection timeout (>5-10 seconds)
• DNS resolution failure
• SSL/TLS handshake failure
• Connection refused

✅ Success (No Retry)

• HTTP 200-299 responses
• HTTP 4xx responses (already processed or invalid)
• Empty 200 response

Key Insight

Return 200 for successfully received webhooks, even if processing fails later. Use queues/async processing to decouple receipt from processing. Return 4xx only for permanently invalid requests (bad signature, malformed payload).

Implementing Idempotency

// Production-ready idempotent webhook handler
import { createHash } from 'crypto';

class WebhookProcessor {
  constructor(db) {
    this.db = db;
  }

  async process(event) {
    const eventId = event.id;
    const idempotencyKey = this.generateIdempotencyKey(event);

    // Check if already processed
    const existing = await this.db.webhookEvents.findOne({
      where: { eventId, provider: 'stripe' }
    });

    if (existing) {
      if (existing.status === 'completed') {
        console.log(`Event ${eventId} already processed`);
        return { status: 'duplicate', processed: false };
      }
      // If failed, allow retry
    }

    // Record receipt
    await this.db.webhookEvents.upsert({
      eventId,
      provider: 'stripe',
      type: event.type,
      idempotencyKey,
      payload: event,
      status: 'processing',
      receivedAt: new Date()
    });

    try {
      // Process event
      const result = await this.handleEvent(event);

      // Mark as completed
      await this.db.webhookEvents.update(
        { eventId },
        { status: 'completed', processedAt: new Date(), result }
      );

      return { status: 'success', processed: true };
    } catch (error) {
      // Mark as failed
      await this.db.webhookEvents.update(
        { eventId },
        { status: 'failed', error: error.message }
      );
      throw error;
    }
  }

  generateIdempotencyKey(event) {
    // Hash critical fields to detect duplicate content
    const content = JSON.stringify({
      type: event.type,
      data: event.data,
      created: event.created
    });
    return createHash('sha256').update(content).digest('hex');
  }

  async handleEvent(event) {
    switch (event.type) {
      case 'payment_intent.succeeded':
        return await this.handlePaymentSuccess(event.data.object);
      // ... other handlers
    }
  }
}

Database Schema for Webhook Events

-- PostgreSQL schema
CREATE TABLE webhook_events (
  id SERIAL PRIMARY KEY,
  event_id VARCHAR(255) NOT NULL,
  provider VARCHAR(50) NOT NULL,
  type VARCHAR(100) NOT NULL,
  idempotency_key VARCHAR(64) NOT NULL,
  payload JSONB NOT NULL,
  status VARCHAR(20) NOT NULL, -- processing, completed, failed
  received_at TIMESTAMP NOT NULL DEFAULT NOW(),
  processed_at TIMESTAMP,
  error TEXT,
  result JSONB,
  
  -- Indexes
  UNIQUE(provider, event_id),
  INDEX idx_status (status),
  INDEX idx_type (type),
  INDEX idx_received (received_at)
);

-- Cleanup old events (run daily)
DELETE FROM webhook_events 
WHERE processed_at < NOW() - INTERVAL '90 days';

Complete Implementation (Node.js)

Production-ready Express webhook receiver with all best practices:

// webhook-receiver.js
import express from 'express';
import crypto from 'crypto';
import { Queue, Worker } from 'bullmq';
import { PrismaClient } from '@prisma/client';

const app = express();
const prisma = new PrismaClient();

// Configure queue
const webhookQueue = new Queue('webhooks', {
  connection: { host: process.env.REDIS_HOST, port: 6379 }
});

// Raw body for signature verification
app.use('/webhooks', express.raw({ type: 'application/json' }));

// Rate limiting
import rateLimit from 'express-rate-limit';
const limiter = rateLimit({
  windowMs: 60 * 1000,
  max: 100
});
app.use('/webhooks', limiter);

// Signature verification
function verifySignature(rawBody, signature, secret) {
  const hmac = crypto.createHmac('sha256', secret);
  const digest = hmac.update(rawBody).digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(digest)
  );
}

// Main webhook endpoint
app.post('/webhooks/stripe', async (req, res) => {
  const signature = req.headers['stripe-signature'];
  const rawBody = req.body;

  // Verify signature
  try {
    const isValid = verifySignature(
      rawBody,
      signature,
      process.env.STRIPE_WEBHOOK_SECRET
    );
    if (!isValid) {
      return res.status(401).json({ error: 'Invalid signature' });
    }
  } catch (err) {
    console.error('Signature verification failed:', err);
    return res.status(400).json({ error: 'Signature verification failed' });
  }

  // Parse event
  const event = JSON.parse(rawBody.toString());

  // Check for duplicate
  const existing = await prisma.webhookEvent.findUnique({
    where: { 
      provider_eventId: { provider: 'stripe', eventId: event.id }
    }
  });

  if (existing?.status === 'completed') {
    return res.status(200).json({ 
      received: true, 
      duplicate: true 
    });
  }

  // Store event
  await prisma.webhookEvent.upsert({
    where: { 
      provider_eventId: { provider: 'stripe', eventId: event.id }
    },
    create: {
      provider: 'stripe',
      eventId: event.id,
      type: event.type,
      payload: event,
      status: 'processing',
      receivedAt: new Date()
    },
    update: {
      status: 'processing',
      receivedAt: new Date()
    }
  });

  // Add to queue for async processing
  await webhookQueue.add('stripe-event', event, {
    jobId: event.id, // Prevents duplicate jobs
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 }
  });

  // Acknowledge receipt immediately
  res.status(200).json({ received: true });
});

// Worker to process queue
const worker = new Worker('webhooks', async (job) => {
  const event = job.data;
  
  try {
    await processStripeEvent(event);
    
    // Mark as completed
    await prisma.webhookEvent.update({
      where: { 
        provider_eventId: { provider: 'stripe', eventId: event.id }
      },
      data: {
        status: 'completed',
        processedAt: new Date()
      }
    });
  } catch (error) {
    console.error(`Failed to process event ${event.id}:`, error);
    
    // Mark as failed
    await prisma.webhookEvent.update({
      where: { 
        provider_eventId: { provider: 'stripe', eventId: event.id }
      },
      data: {
        status: 'failed',
        error: error.message
      }
    });
    
    throw error; // Trigger retry
  }
}, { 
  connection: { host: process.env.REDIS_HOST, port: 6379 }
});

async function processStripeEvent(event) {
  switch (event.type) {
    case 'payment_intent.succeeded':
      await handlePaymentSuccess(event.data.object);
      break;
    case 'payment_intent.failed':
      await handlePaymentFailure(event.data.object);
      break;
    case 'customer.subscription.created':
      await handleSubscriptionCreated(event.data.object);
      break;
    case 'customer.subscription.deleted':
      await handleSubscriptionCanceled(event.data.object);
      break;
    default:
      console.log(`Unhandled event type: ${event.type}`);
  }
}

async function handlePaymentSuccess(paymentIntent) {
  // Update order status
  await prisma.order.update({
    where: { paymentIntentId: paymentIntent.id },
    data: { status: 'paid' }
  });
  
  // Send confirmation email
  await sendEmail({
    to: paymentIntent.receipt_email,
    subject: 'Payment Successful',
    template: 'payment-confirmation',
    data: { amount: paymentIntent.amount, currency: paymentIntent.currency }
  });
}

// Start server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Webhook receiver listening on port ${PORT}`);
});

Testing Webhooks Locally

1. Using ngrok (Tunneling)

# Install ngrok
brew install ngrok  # macOS
# or download from https://ngrok.com

# Start your local server
npm start  # Running on http://localhost:3000

# Create tunnel in another terminal
ngrok http 3000

# Output:
# Forwarding  https://abc123.ngrok.io -> http://localhost:3000

# Use the ngrok URL in your webhook provider:
# https://abc123.ngrok.io/webhooks/stripe

⚠️ ngrok Limitations

• Free tier: URL changes every restart
• Paid tier: Fixed subdomain (e.g., yourapp.ngrok.io)
• Not suitable for production (use for development only)

2. Using Webhook.site (Testing)

Webhook.site is a free tool for inspecting webhook payloads without writing code.

1. Visit webhook.site
2. Copy your unique URL (e.g., https://webhook.site/abc123...)
3. Configure that URL in your webhook provider
4. Trigger an event and watch it appear on webhook.site
5. Inspect headers, body, and raw payload

3. Provider Test Events (Recommended)

Most webhook providers offer test event triggers:

Stripe CLI

# Install Stripe CLI
brew install stripe/stripe-cli/stripe

# Login
stripe login

# Forward webhooks to local server
stripe listen --forward-to localhost:3000/webhooks/stripe

# Trigger test event
stripe trigger payment_intent.succeeded

GitHub Webhook Testing

GitHub shows recent webhook deliveries in repository settings:

1. Go to Settings → Webhooks
2. Click on your webhook
3. View "Recent Deliveries"
4. Click "Redeliver" to resend a past event

4. Unit Testing Webhook Handlers

// webhook.test.js
import { describe, it, expect, vi } from 'vitest';
import request from 'supertest';
import app from './webhook-receiver';
import crypto from 'crypto';

describe('Stripe Webhook', () => {
  it('should accept valid webhook', async () => {
    const payload = {
      id: 'evt_test_123',
      type: 'payment_intent.succeeded',
      data: { object: { id: 'pi_123', amount: 2000 } }
    };
    
    const signature = generateSignature(payload);
    
    const response = await request(app)
      .post('/webhooks/stripe')
      .set('Stripe-Signature', signature)
      .send(payload);
    
    expect(response.status).toBe(200);
    expect(response.body.received).toBe(true);
  });

  it('should reject invalid signature', async () => {
    const payload = { id: 'evt_test_123' };
    
    const response = await request(app)
      .post('/webhooks/stripe')
      .set('Stripe-Signature', 'invalid')
      .send(payload);
    
    expect(response.status).toBe(401);
  });

  it('should handle duplicate events', async () => {
    const payload = { id: 'evt_duplicate_123', type: 'payment_intent.succeeded' };
    const signature = generateSignature(payload);
    
    // First delivery
    await request(app)
      .post('/webhooks/stripe')
      .set('Stripe-Signature', signature)
      .send(payload);
    
    // Duplicate delivery
    const response = await request(app)
      .post('/webhooks/stripe')
      .set('Stripe-Signature', signature)
      .send(payload);
    
    expect(response.body.duplicate).toBe(true);
  });
});

function generateSignature(payload) {
  const timestamp = Math.floor(Date.now() / 1000);
  const body = JSON.stringify(payload);
  const signedPayload = `${timestamp}.${body}`;
  const signature = crypto
    .createHmac('sha256', process.env.STRIPE_WEBHOOK_SECRET)
    .update(signedPayload)
    .digest('hex');
  return `t=${timestamp},v1=${signature}`;
}

Monitoring & Debugging

Key Metrics to Track

Delivery Metrics

• Delivery success rate: Target 99.9%+
• Average response time: Target <500ms
• P99 response time: Target <2s
• Failed deliveries: Alert if >1%

Processing Metrics

• Processing success rate: Target 99%+
• Queue depth: Alert if >1000
• Processing latency: Time from receipt to completion
• Duplicate events: Track idempotency hits

Logging Best Practices

// Structured logging for webhooks
import winston from 'winston';

const logger = winston.createLogger({
  format: winston.format.json(),
  transports: [
    new winston.transports.File({ filename: 'webhooks.log' })
  ]
});

app.post('/webhooks/stripe', async (req, res) => {
  const startTime = Date.now();
  const event = JSON.parse(req.body.toString());
  
  // Log receipt
  logger.info('webhook.received', {
    provider: 'stripe',
    eventId: event.id,
    eventType: event.type,
    timestamp: new Date().toISOString()
  });

  try {
    // Process...
    
    // Log success
    logger.info('webhook.processed', {
      provider: 'stripe',
      eventId: event.id,
      eventType: event.type,
      duration: Date.now() - startTime,
      status: 'success'
    });
    
    res.status(200).json({ received: true });
  } catch (error) {
    // Log failure
    logger.error('webhook.failed', {
      provider: 'stripe',
      eventId: event.id,
      eventType: event.type,
      duration: Date.now() - startTime,
      error: error.message,
      stack: error.stack
    });
    
    res.status(500).json({ error: 'Processing failed' });
  }
});

Debugging Failed Webhooks

1. Check Provider Dashboard

Most providers show webhook delivery attempts with response codes and bodies.

• Stripe: Developers → Webhooks → Click webhook → View logs
• GitHub: Settings → Webhooks → Recent Deliveries
• Shopify: Settings → Notifications → Webhooks → View details

2. Replay Failed Events

Don't lose data from failed deliveries. Most providers allow replaying events:

// Stripe: Replay from dashboard or CLI
stripe events resend evt_1MqLN82eZvKYlo2C8JzKzzYI

// GitHub: Click "Redeliver" in webhook settings

// Or build your own replay mechanism
async function replayFailedEvents() {
  const failed = await db.webhookEvents.findMany({
    where: { status: 'failed', receivedAt: { gte: Date.now() - 86400000 } }
  });
  
  for (const event of failed) {
    await processEvent(event.payload);
  }
}

3. Monitor with Datadog / New Relic

Set up APM monitoring for webhook endpoints:

// Example: Datadog APM
import tracer from 'dd-trace';
tracer.init({ service: 'webhook-receiver' });

// Automatically tracks:
// - Response times
// - Error rates
// - Request throughput
// - Database query performance

Alerts to Set Up

Critical Alerts

• Webhook endpoint down: HTTP 5xx rate >1% for 5 minutes
• Processing backlog: Queue depth >1000 for 10 minutes
• Provider unreachable: Failed deliveries from specific provider
• Duplicate storm: >50% duplicate events (might indicate replay attack)

Real-World Provider Examples

Stripe Webhooks

Common Event Types

• payment_intent.succeeded - Payment completed
• payment_intent.failed - Payment failed
• customer.subscription.created - New subscription
• customer.subscription.deleted - Subscription canceled
• invoice.payment_failed - Recurring payment failed
• charge.dispute.created - Customer disputed charge

Check Stripe's status if webhooks stop arriving.

GitHub Webhooks

Common Event Types

• push - Code pushed to repository
• pull_request - PR opened/closed/merged
• issues - Issue created/updated/closed
• release - Release published
• deployment_status - Deployment succeeded/failed

// GitHub webhook handler
import crypto from 'crypto';

function verifyGitHubSignature(req, secret) {
  const signature = req.headers['x-hub-signature-256'];
  const body = JSON.stringify(req.body);
  
  const hmac = crypto.createHmac('sha256', secret);
  const digest = 'sha256=' + hmac.update(body).digest('hex');
  
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(digest)
  );
}

app.post('/webhooks/github', async (req, res) => {
  const isValid = verifyGitHubSignature(req, process.env.GITHUB_WEBHOOK_SECRET);
  if (!isValid) return res.status(401).send('Invalid signature');
  
  const event = req.headers['x-github-event'];
  const payload = req.body;
  
  switch (event) {
    case 'push':
      await handlePush(payload);
      break;
    case 'pull_request':
      await handlePR(payload);
      break;
  }
  
  res.status(200).json({ received: true });
});

Monitor GitHub's status if webhook deliveries fail.

Shopify Webhooks

Common Event Types

• orders/create - New order placed
• orders/updated - Order status changed
• products/create - New product added
• customers/create - New customer registered
• fulfillments/create - Order fulfilled

Check Shopify's status if webhooks stop arriving.

Twilio Webhooks

Twilio uses webhooks for SMS, voice calls, and WhatsApp messages. Unlike most providers, Twilio expects your endpoint to return TwiML (XML) responses for certain events.

// Twilio SMS webhook
app.post('/webhooks/twilio/sms', (req, res) => {
  const from = req.body.From;
  const body = req.body.Body;
  
  console.log(`SMS from ${from}: ${body}`);
  
  // Respond with TwiML
  res.type('text/xml');
  res.send(`<?xml version="1.0" encoding="UTF-8"?>
    <Response>
      <Message>Thanks for your message!</Message>
    </Response>
  `);
});

Monitor Twilio's status if webhooks aren't being delivered.

Common Mistakes to Avoid

❌ 1. Not Verifying Signatures

Problem: Anyone can POST data to your webhook endpoint. Without signature verification, attackers can trigger fake events (fake payments, fake subscriptions).

Solution: Always verify HMAC signatures. Never trust webhook payloads without verification.

❌ 2. Slow Processing Causes Timeouts

Problem: Processing complex logic (sending emails, updating multiple database records) takes 10+ seconds. Provider times out and retries, causing duplicate processing.

Solution: Respond with 200 within 5 seconds. Use queues for heavy processing.

❌ 3. No Idempotency Handling

Problem: Webhooks can be delivered multiple times. Without idempotency checks, you might charge customers twice, send duplicate emails, or create duplicate records.

Solution: Track processed event IDs in your database. Check before processing.

❌ 4. Returning 5xx for Invalid Requests

Problem: Returning 500 for malformed payloads triggers retries. Provider keeps retrying a request that will never succeed.

Solution: Return 4xx (400, 401) for permanently invalid requests. Return 5xx only for temporary failures.

❌ 5. Not Monitoring Failed Deliveries

Problem: Webhook endpoint is down for 2 hours. Provider stops retrying. You never know events were lost.

Solution: Set up alerts for webhook failures. Check provider dashboards regularly. Have a backfill plan.

❌ 6. Exposing Secrets in Logs

Problem: Logging raw webhook payloads exposes sensitive data (customer emails, payment details, API keys).

Solution: Redact sensitive fields before logging. Only log event IDs and types for debugging.

❌ 7. No Testing Strategy

Problem: Webhooks only tested in production. Bug causes payment confirmation emails to fail for 3 days before discovery.

Solution: Use provider CLIs (Stripe CLI, GitHub CLI) for local testing. Write unit tests. Test signature verification.

Production Readiness Checklist

Security ✓

☐Signature verification implemented for all webhook providers
☐HTTPS enforced (reject HTTP requests)
☐Rate limiting configured (100 requests/minute recommended)
☐Timestamp validation to prevent replay attacks
☐Secrets stored in environment variables (not hardcoded)

Reliability ✓

☐Response time <5 seconds (queue-based processing)
☐Idempotency handling (track processed event IDs)
☐Database persistence for all webhook events
☐Retry logic for transient failures
☐Dead letter queue for permanently failed events

Monitoring ✓

☐Structured logging (event ID, type, status, duration)
☐Alerts for failed deliveries (>1% failure rate)
☐Alerts for processing backlog (queue depth >1000)
☐Dashboard showing delivery success rate, response times
☐Provider dashboard checks (Stripe, GitHub, etc.)

Testing ✓

☐Unit tests for signature verification
☐Unit tests for idempotency handling
☐Integration tests with provider test events
☐Load testing (simulate burst of 100+ events)
☐Test duplicate event handling

Documentation ✓

☐Runbook for webhook failures
☐List of webhook URLs registered with providers
☐Secret rotation procedure documented
☐Event replay procedure documented

Conclusion

Webhooks are the backbone of real-time integrations. When implemented correctly — with signature verification, queue-based processing, idempotency handling, and comprehensive monitoring — they enable reliable, cost-effective event-driven architectures.

The patterns in this guide power production systems processing billions of events per day. Start with Pattern 1 (direct processing) for simple use cases, then graduate to Pattern 2 (queue-based) as your scale grows.

Key Takeaways

• Security first: Always verify signatures
• Respond fast: Use queues for heavy processing
• Handle duplicates: Implement idempotency
• Monitor everything: Set up alerts for failures
• Test thoroughly: Use provider CLIs and unit tests

For more on API reliability and monitoring, see our guides on API Dependency Monitoring and API Rate Limiting.

Monitor the status of your webhook providers: Stripe, GitHub, Shopify, Twilio, Auth0, Okta, SendGrid, and more on APIStatusCheck.com.

Last updated: March 9, 2026 | APIStatusCheck.com — Real-time API status monitoring for developers