API Webhooks: Complete Implementation Guide

Webhooks are the backbone of modern event-driven architectures. Instead of constantly polling an API for updates, webhooks allow services to push real-time notifications directly to your application when events occur.

This guide covers everything you need to implement, secure, and monitor production-grade webhook integrations.

What Are Webhooks?

A webhook is a user-defined HTTP callback. When a specific event happens in a source system, it sends an HTTP POST request to a URL you specify, delivering event data in real time.

Common webhook use cases:

• Payment processors (Stripe, PayPal) notifying you of completed transactions
• Git platforms (GitHub, GitLab) triggering CI/CD pipelines on code pushes
• Communication tools (Slack, Discord) receiving chat events
• CRM systems (Salesforce, HubSpot) syncing contact updates
• Monitoring services notifying you of system alerts

Webhooks vs polling:

Approach	Latency	Server Load	Real-time	Complexity
Webhooks	Instant	Low	Yes	Medium-High
Polling	Delayed	High	No	Low-Medium

Webhooks eliminate the need for constant API requests, reducing server load by 95%+ compared to aggressive polling.

Webhook Architecture Patterns

1. Direct Endpoint Pattern

The simplest approach — webhook receiver processes events synchronously:

app.post('/webhooks/stripe', async (req, res) => {
  const event = req.body;
  
  // Process immediately
  await processPayment(event);
  
  res.status(200).send('OK');
});

Pros: Simple, immediate processing
Cons: Slow processing blocks the sender, risk of timeouts

Use when: Processing takes <2 seconds and rarely fails

2. Queue-Based Pattern (Recommended)

Webhook receiver acknowledges immediately, then processes asynchronously:

app.post('/webhooks/stripe', async (req, res) => {
  const event = req.body;
  
  // Acknowledge receipt immediately
  res.status(200).send('OK');
  
  // Queue for async processing
  await queue.add('webhook-processing', event);
});

Pros: Fast response time, better reliability, can retry failed jobs
Cons: Added infrastructure complexity

Use when: Processing takes >2 seconds or involves external API calls

3. Fan-Out Pattern

One webhook triggers multiple downstream actions:

app.post('/webhooks/github', async (req, res) => {
  const event = req.body;
  
  res.status(200).send('OK');
  
  // Trigger multiple async handlers
  await Promise.all([
    notifySlack(event),
    updateDatabase(event),
    triggerCICD(event),
    logToDataWarehouse(event)
  ]);
});

Use when: Events need to trigger multiple independent workflows

Webhook Security Best Practices

1. Signature Verification (Critical)

Always verify webhook signatures to prevent spoofed requests.

Example: Verifying Stripe webhooks

const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

app.post('/webhooks/stripe', async (req, res) => {
  const sig = req.headers['stripe-signature'];
  let event;

  try {
    event = stripe.webhooks.constructEvent(
      req.body,
      sig,
      process.env.STRIPE_WEBHOOK_SECRET
    );
  } catch (err) {
    console.error('Webhook signature verification failed:', err.message);
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  // Signature verified — safe to process
  await processEvent(event);
  res.status(200).send('OK');
});

Example: Custom HMAC signature

const crypto = require('crypto');

function verifyWebhookSignature(payload, signature, secret) {
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');
  
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expectedSignature)
  );
}

app.post('/webhooks/custom', (req, res) => {
  const signature = req.headers['x-webhook-signature'];
  const rawBody = req.body; // Must be raw string, not parsed JSON
  
  if (!verifyWebhookSignature(rawBody, signature, process.env.WEBHOOK_SECRET)) {
    return res.status(401).send('Invalid signature');
  }
  
  // Process webhook...
  res.status(200).send('OK');
});

Critical: Use crypto.timingSafeEqual to prevent timing attacks.

🔐 Storing webhook secrets securely? 1Password lets you reference secrets in code with op://vault/item/field URIs and inject them at runtime — no plaintext secrets in your .env files or source code.

2. IP Allowlisting

Restrict webhook endpoints to known sender IPs:

const ALLOWED_IPS = [
  '192.168.1.100',
  '10.0.0.0/8' // CIDR notation
];

function isAllowedIP(ip) {
  return ALLOWED_IPS.some(allowed => {
    if (allowed.includes('/')) {
      return isInCIDRRange(ip, allowed);
    }
    return ip === allowed;
  });
}

app.post('/webhooks/internal', (req, res) => {
  const clientIP = req.headers['x-forwarded-for'] || req.connection.remoteAddress;
  
  if (!isAllowedIP(clientIP)) {
    return res.status(403).send('IP not allowed');
  }
  
  // Process webhook...
});

When to use: Internal webhooks or services with published IP ranges

3. HTTPS Only

Never accept webhooks over plain HTTP in production:

app.use((req, res, next) => {
  if (req.headers['x-forwarded-proto'] !== 'https' && process.env.NODE_ENV === 'production') {
    return res.status(403).send('HTTPS required');
  }
  next();
});

4. Idempotency Keys

Process each webhook event exactly once:

const processedEvents = new Set(); // In production, use Redis

app.post('/webhooks/payments', async (req, res) => {
  const eventId = req.body.id;
  
  if (processedEvents.has(eventId)) {
    console.log(`Duplicate event ${eventId} — skipping`);
    return res.status(200).send('OK');
  }
  
  await processPayment(req.body);
  processedEvents.add(eventId);
  
  res.status(200).send('OK');
});

Production implementation with Redis:

const redis = require('redis').createClient();

async function isEventProcessed(eventId) {
  const exists = await redis.get(`webhook:${eventId}`);
  if (exists) return true;
  
  await redis.setex(`webhook:${eventId}`, 86400, '1'); // 24h TTL
  return false;
}

Error Handling and Retry Logic

Sender-Side Retries

When sending webhooks, implement exponential backoff:

async function sendWebhook(url, payload, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const response = await fetch(url, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(payload),
        timeout: 5000
      });
      
      if (response.ok) {
        return { success: true };
      }
      
      // Retry on 5xx errors, not 4xx
      if (response.status >= 500 && attempt < maxRetries - 1) {
        const delay = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
        await new Promise(resolve => setTimeout(resolve, delay));
        continue;
      }
      
      return { success: false, status: response.status };
    } catch (error) {
      if (attempt === maxRetries - 1) {
        return { success: false, error: error.message };
      }
      
      const delay = Math.pow(2, attempt) * 1000;
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

Receiver-Side Error Responses

Return appropriate HTTP status codes:

app.post('/webhooks/events', async (req, res) => {
  try {
    await processEvent(req.body);
    res.status(200).send('OK');
  } catch (error) {
    if (error.code === 'VALIDATION_ERROR') {
      // Don't retry validation errors
      res.status(400).json({ error: error.message });
    } else {
      // Sender should retry server errors
      res.status(500).json({ error: 'Processing failed' });
    }
  }
});

Status code guide:

• 200-299: Success (no retry)
• 400-499: Client error (no retry)
• 500-599: Server error (sender should retry)

Dead Letter Queue Pattern

Store failed webhooks for manual review:

async function processWebhook(event) {
  try {
    await handleEvent(event);
  } catch (error) {
    await deadLetterQueue.add({
      event,
      error: error.message,
      timestamp: new Date(),
      retryCount: event.retryCount || 0
    });
    throw error;
  }
}

Testing Webhooks Locally

1. Using ngrok

Expose your local server to the internet:

# Install ngrok
brew install ngrok

# Start your local server
npm start # Running on localhost:3000

# Expose it
ngrok http 3000

ngrok gives you a public URL like https://abc123.ngrok.io — use this as your webhook endpoint during development.

2. Webhook Testing Tools

RequestBin / Webhook.site:

• Provides instant webhook URLs
• Displays incoming requests with headers and body
• Useful for debugging webhook payloads

Postman:

# Send test webhook
curl -X POST http://localhost:3000/webhooks/test \
  -H "Content-Type: application/json" \
  -d '{"event": "test", "data": {"id": 123}}'

3. Replay Production Webhooks

Log incoming webhooks for replay:

app.post('/webhooks/stripe', async (req, res) => {
  // Log for replay
  await fs.appendFile('webhook-logs.jsonl', JSON.stringify({
    timestamp: new Date(),
    headers: req.headers,
    body: req.body
  }) + '\n');
  
  await processWebhook(req.body);
  res.status(200).send('OK');
});

Replay logged webhooks:

cat webhook-logs.jsonl | while read line; do
  curl -X POST http://localhost:3000/webhooks/stripe \
    -H "Content-Type: application/json" \
    -d "$line"
done

Monitoring Webhook Health

Key Metrics to Track

1. Delivery success rate

   • Target: >99.5%
   • Alert if <95% over 5 minutes

2. Response time

   • Target: <2 seconds
   • Alert if p95 >5 seconds

3. Retry rate

   • Track how often senders retry
   • High retry rate = receiver issues

4. Processing queue depth

   • For async patterns
   • Alert if queue grows unbounded

Monitoring Implementation

const metrics = {
  received: 0,
  processed: 0,
  failed: 0,
  responseTime: []
};

app.post('/webhooks/monitor', async (req, res) => {
  const startTime = Date.now();
  metrics.received++;
  
  try {
    await processWebhook(req.body);
    metrics.processed++;
    res.status(200).send('OK');
  } catch (error) {
    metrics.failed++;
    res.status(500).send('Error');
  } finally {
    metrics.responseTime.push(Date.now() - startTime);
  }
});

// Expose metrics endpoint
app.get('/metrics', (req, res) => {
  const successRate = metrics.processed / metrics.received * 100;
  const avgResponseTime = metrics.responseTime.reduce((a, b) => a + b, 0) / metrics.responseTime.length;
  
  res.json({
    successRate: successRate.toFixed(2) + '%',
    avgResponseTime: avgResponseTime.toFixed(0) + 'ms',
    totalReceived: metrics.received,
    totalFailed: metrics.failed
  });
});

Alerting on Webhook Failures

async function alertOnFailures() {
  const recentFailures = await db.query(`
    SELECT COUNT(*) FROM webhook_logs
    WHERE status = 'failed'
    AND created_at > NOW() - INTERVAL '5 minutes'
  `);
  
  if (recentFailures > 10) {
    await sendAlert({
      service: 'webhooks',
      severity: 'high',
      message: `${recentFailures} webhook failures in last 5 minutes`
    });
  }
}

setInterval(alertOnFailures, 60000); // Check every minute

Common Webhook Pitfalls

1. Blocking on Slow Operations

❌ Bad:

app.post('/webhooks/slow', async (req, res) => {
  await sendEmailToUser(req.body.userId); // 5+ seconds
  res.status(200).send('OK');
});

✅ Good:

app.post('/webhooks/fast', async (req, res) => {
  res.status(200).send('OK');
  await queue.add('send-email', { userId: req.body.userId });
});

2. Not Handling Duplicates

Webhook senders may retry on network blips — always implement idempotency.

3. Ignoring Signature Verification

Never trust webhook payloads without verification. Attackers can forge requests.

4. Logging Sensitive Data

❌ Bad:

console.log('Received webhook:', JSON.stringify(req.body));

✅ Good:

const sanitized = { ...req.body };
delete sanitized.creditCard;
delete sanitized.apiKey;
console.log('Received webhook:', JSON.stringify(sanitized));

5. Synchronous Database Writes

Queue database operations to avoid blocking:

app.post('/webhooks/db', async (req, res) => {
  res.status(200).send('OK');
  
  // Write asynchronously
  await db.insert('events', req.body).catch(err => {
    console.error('DB write failed:', err);
    // Add to retry queue
  });
});

Real-World Examples

Stripe Payment Webhook

app.post('/webhooks/stripe', async (req, res) => {
  const sig = req.headers['stripe-signature'];
  let event;

  try {
    event = stripe.webhooks.constructEvent(req.body, sig, process.env.STRIPE_WEBHOOK_SECRET);
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  switch (event.type) {
    case 'payment_intent.succeeded':
      await handleSuccessfulPayment(event.data.object);
      break;
    case 'payment_intent.payment_failed':
      await handleFailedPayment(event.data.object);
      break;
    case 'customer.subscription.deleted':
      await handleCancellation(event.data.object);
      break;
    default:
      console.log(`Unhandled event type ${event.type}`);
  }

  res.status(200).send('OK');
});

GitHub Push Webhook

app.post('/webhooks/github', async (req, res) => {
  const signature = req.headers['x-hub-signature-256'];
  
  if (!verifyGitHubSignature(req.body, signature)) {
    return res.status(401).send('Invalid signature');
  }

  const event = req.headers['x-github-event'];
  
  if (event === 'push') {
    const { repository, ref, commits } = req.body;
    
    if (ref === 'refs/heads/main') {
      await triggerDeploy(repository.name, commits);
    }
  }

  res.status(200).send('OK');
});

Slack Event Webhook

app.post('/webhooks/slack', async (req, res) => {
  // Slack sends URL verification challenge
  if (req.body.type === 'url_verification') {
    return res.json({ challenge: req.body.challenge });
  }

  // Respond immediately to avoid retries
  res.status(200).send('OK');

  // Process event asynchronously
  const { event } = req.body;
  
  if (event.type === 'message' && !event.bot_id) {
    await handleSlackMessage(event);
  }
});

Webhook URL Design Best Practices

Good patterns:

• /webhooks/stripe — Clear service name
• /webhooks/github/push — Service + event type
• /api/v1/webhooks/payments — Versioned

Avoid:

• /hook — Too generic
• /api/callback — Ambiguous
• /stripe — Conflicts with other endpoints

Security tip: Use unguessable paths for sensitive webhooks:

• /webhooks/stripe/wh_abc123xyz — Adds obscurity layer

Conclusion

Webhooks enable real-time, event-driven integrations that scale efficiently. Key takeaways:

✅ Always verify signatures — Never trust unsigned webhooks
✅ Respond quickly — Acknowledge within 2-5 seconds, process async
✅ Implement idempotency — Handle duplicate deliveries gracefully
✅ Monitor delivery health — Track success rates and latencies
✅ Test thoroughly — Use ngrok and webhook testing tools
✅ Handle failures gracefully — Retry with exponential backoff

When implemented correctly, webhooks reduce API polling by 95%+, delivering instant updates with minimal server load.

Recommended Tools

Monitor webhook delivery endpoints. When your webhook receiver goes down, you miss critical events. Better Stack monitors your webhook endpoints 24/7 and alerts you before missed events cascade into data inconsistencies. {/* affiliate:betterstack */}

Secure webhook signing secrets. Webhook signature verification requires storing shared secrets securely. 1Password manages webhook signing keys across environments with CLI injection — no plaintext secrets in config files. {/* affiliate:1password */}