Connecting Systems That Weren't Designed to Talk

The Fundamental Constraint

Every integration has the same underlying problem: you are coupling your system to something you do not control. The external service can change behaviour, go offline, rate limit you, return unexpected data, or deprecate the endpoint you depend on. Your code runs on your infrastructure, but its correctness depends on infrastructure you cannot see.

This is not a solvable problem in the sense that you can eliminate it. It is a constraint you design around. The goal is not to prevent external services from failing. The goal is to ensure that when they fail (and they will), your system continues to operate in a predictable, recoverable state.

Why Integrations Are Hard

Before diving into patterns and implementation, it is worth being explicit about what makes integrations difficult. These are not edge cases. They are the normal operating conditions you should expect.

The Naive Approach (What Goes Wrong)

Most integration code starts the same way. A developer reads the API documentation, writes a function that makes an HTTP request, parses the response, and moves on. The code works in development. It works in the first few weeks of production. Then it fails.

The naive approach treats the external API like a local function call: synchronous, reliable, and consistent. This assumption is false in every particular.

These are not mistakes made by junior developers. They are the natural result of building features under time pressure without explicit requirements for resilience. The code does what it was asked to do: call the API and return the result. The problem is that "call the API and return the result" is not sufficient specification for production systems.

How We Build Reliable Integrations

Reliable integrations require explicit design decisions about failure modes. Every external call needs answers to specific questions before writing code.

We assume failure

Every API call can fail. We design for it from the start. Before implementing any integration, we answer these questions:

• What happens if this request times out after 5 seconds? After 30 seconds?
• What if the service is down for five minutes? For an hour? For a day?
• What if the response is malformed or missing expected fields?
• What if we get rate-limited?
• What if the request succeeds but we never receive the response?
• What if the same request is submitted twice?

Integrations that assume success work in demos. Integrations that assume failure work in production.

We make operations idempotent

If a request fails and we retry, the same operation must not happen twice. We use unique identifiers (idempotency keys) to prevent duplicate charges, duplicate orders, duplicate records.

Idempotency works by sending a unique key with each operation. If the external system receives the same key twice, it returns the result of the first operation without re-executing. Most payment processors and financial APIs support this. For APIs that do not, we implement idempotency on our side by tracking operations in a database before sending them.

We log everything

When an integration fails, we need to debug it. That means knowing exactly what happened, not reconstructing it from memory or guessing from error messages.

We log:

• Request details: Endpoint, method, headers (sanitised), body (sanitised), timestamp
• Response details: Status code, headers, body, timestamp
• Timing: Total request duration, connection time, time to first byte
• Outcome: Success, failure type, retry count, final disposition
• Context: Correlation ID linking the request to the business operation that triggered it

Structured logs make troubleshooting possible. When something goes wrong at 3am, the logs tell us what happened. We can replay the request, compare it to successful requests, and identify the failure mode without guessing. For more on maintaining complete operational records, see our approach to audit trails.

Sanitisation is critical. Logs must not contain API keys, passwords, credit card numbers, or personal data. We strip these before logging and replace them with masked placeholders.

We isolate external dependencies

We wrap external API calls in abstraction layers. The rest of the codebase interacts with our wrapper, not the external API directly. When the external API changes, we change the wrapper. Every other file in the codebase remains unchanged.

This pattern also allows us to:

• Mock the external API for testing without hitting real endpoints
• Swap providers without touching business logic
• Add logging, retry logic, and circuit breakers in one place
• Version our interface independently of the external API's versioning

Integration Patterns We Use

Different integration scenarios call for different patterns. The choice depends on latency requirements, reliability needs, and the capabilities of the external system.

Pattern selection matrix

Choosing the right pattern depends on your specific requirements. This matrix provides a starting point.

Error Handling Strategies

Different errors require different responses. A network timeout needs retry logic. A 400 Bad Request needs investigation and possibly a code fix. A 429 rate limit needs backoff. Treating all errors the same leads to either excessive retries (wasting resources, hitting rate limits) or insufficient retries (failing operations that would succeed on retry).

Error classification

We classify errors into categories that determine handling behaviour.

Retry implementation

Retry logic has several parameters that need tuning for each integration.

• Maximum retries: How many times to retry before giving up. Typically 3-5 for synchronous operations, higher for queued operations.
• Base delay: Initial wait before first retry. Typically 100ms-1s.
• Backoff multiplier: How much to increase delay between retries. Typically 2x (exponential backoff).
• Maximum delay: Cap on delay to prevent waiting forever. Typically 30s-60s.
• Jitter: Random variation in delay to prevent thundering herd. Add 0-25% random variation.

Circuit breaker pattern

If an external service is failing repeatedly, continuing to call it wastes resources and can cascade into broader system failure. The circuit breaker pattern detects repeated failures and stops calling the failing service for a period, giving it time to recover.

A circuit breaker has three states:

Circuit breakers prevent cascading failures. Without them, a failing external service can bring down your entire application as requests queue up waiting for responses that never come. Thread pools exhaust. Memory fills. Other, healthy integrations start failing because they share resources with the failing one.

Rate Limiting and Throttling

Most external APIs have rate limits. Exceed them and you get 429 errors, temporary bans, or degraded service. Some providers are generous. Others are strict. Either way, your integration needs to operate within the limits.

Understanding rate limits

Rate limits come in several forms:

• Requests per second: Common for real-time APIs. Often 10-100 requests per second.
• Requests per minute/hour: Common for batch APIs. Often 1,000-10,000 per hour.
• Concurrent connections: Limits how many simultaneous requests you can have open.
• Daily quotas: Hard limits that reset at midnight UTC. Common for expensive operations.
• Per-endpoint limits: Different limits for different endpoints. Write operations often have stricter limits than read operations.

Rate limits may apply per API key, per IP address, per user, or per organisation. Read the documentation carefully. Test at scale before production.

Client-side throttling

We implement client-side throttling to stay within limits rather than hitting limits and dealing with 429 errors. This is more efficient and more reliable.

• Token bucket: Track available "tokens" that replenish over time. Each request consumes a token. If no tokens available, wait.
• Sliding window: Track requests in the last N seconds. If at limit, wait until oldest request falls out of window.
• Request queuing: Queue requests and dispatch at a controlled rate. Smooth out bursts.

Handling 429 responses

When you do hit a rate limit:

• Parse the Retry-After header if present. It tells you exactly how long to wait.
• If no header, use exponential backoff starting at 1 second.
• Log rate limit events for monitoring. Frequent rate limits indicate a problem.
• Consider queuing affected requests rather than retrying immediately.

Webhook Implementation

Receiving webhooks from external services requires care. Webhooks invert the normal request/response model: the external service calls you. This means you are running code triggered by external events, on external timing, with external data.

Webhook security

Anyone can send an HTTP request to a public URL. Webhook endpoints must verify that requests actually come from the expected source.

Webhook processing pattern

Webhook endpoints should acknowledge quickly and process later. If your processing takes 30 seconds, the webhook provider times out and retries, creating duplicate events.

Handling duplicate webhooks

Webhook providers retry failed deliveries. Your endpoint might receive the same event multiple times. Processing must be idempotent.

• Store the event ID (most providers include one) before processing
• Check if you have seen this event ID before
• If seen, return 200 without reprocessing
• Use database transactions or atomic operations to prevent race conditions

Some providers send the same event from multiple servers simultaneously for reliability. You may receive duplicates within milliseconds of each other. Simple "check then insert" logic has race conditions. Use database constraints or atomic upserts.

Authentication and Security

API credentials are keys to external systems. Compromise them and attackers can charge credit cards, send emails as you, access customer data, or delete resources. Security is not optional.

Credential management

API credentials never belong in code or version control. Ever. Not even in private repositories.

• Environment variables: The baseline. Credentials injected at runtime, not stored in code.
• Secret management systems: AWS Secrets Manager, HashiCorp Vault, Azure Key Vault. Centralised, audited, access-controlled storage for secrets.
• Rotation support: Credentials should be rotatable without code changes or deployments. When a credential is compromised, you need to rotate immediately.

OAuth implementation

For user-authorised integrations (connecting to a user's account on another service), we implement OAuth 2.0 properly.

• Secure token storage: Access tokens and refresh tokens stored encrypted. Never logged. Never exposed to frontend code.
• Token refresh: Refresh tokens before they expire, not when they fail. Schedule refresh at 80% of token lifetime.
• Revocation handling: Users can revoke access at any time. Detect revoked tokens (usually 401 errors) and prompt for reauthorisation.
• State parameter: Use cryptographically random state in OAuth flows to prevent CSRF attacks.
• PKCE: For public clients (mobile apps, SPAs), use Proof Key for Code Exchange to prevent authorisation code interception.

Least privilege

Request only the permissions the integration actually needs. Broad permissions create security risk (more damage if compromised) and make users hesitant to authorise. If you only need to read contacts, do not request write access. If you only need email, do not request calendar.

Review permissions periodically. As integrations evolve, unused scopes accumulate. Remove them.

Transport security

• TLS only: All API calls over HTTPS. No exceptions. No fallback to HTTP.
• Certificate validation: Verify the server's certificate chain. Do not disable certificate validation for convenience.
• TLS version: Minimum TLS 1.2. Prefer TLS 1.3 where supported.
• Certificate pinning: For high-security integrations, pin the expected certificate or public key.

Monitoring and Alerting

An integration that is not monitored is an integration you learn about when users complain. By then, the damage is done: failed orders, missed emails, broken sync. Monitoring catches problems before they cascade.

What to monitor

Alerting thresholds

Alerts should be actionable. Too sensitive and you get alert fatigue. Too insensitive and you miss real problems.

Health checks

Implement periodic health checks that verify the integration is working end to end. Not just "can we reach the API" but "can we complete a representative operation".

• For payment integrations: verify credentials are valid, test mode available
• For email integrations: send a test email to a monitored inbox
• For data sync: verify last sync time is recent, record counts are plausible

Health checks run on a schedule (every 1-5 minutes) and alert if they fail. They catch problems like expired credentials, changed API endpoints, and silent failures that do not generate errors.

Versioning and Backward Compatibility

External APIs change. New fields appear. Old fields disappear. Endpoints get deprecated. Response formats evolve. Your integration must handle change without breaking.

Defensive parsing

Do not assume the response contains exactly what the documentation says. Parse defensively.

• Use optional fields: If a field might not be present, handle its absence gracefully.
• Ignore unknown fields: New fields in responses should not cause parsing errors.
• Validate types: A field documented as integer might arrive as string. Handle it.
• Handle null vs missing: These are different conditions. A null value is not the same as a missing key.

API versioning strategies

When external APIs version their endpoints, you need a strategy for migration.

Testing against API changes

Record real API responses and use them in tests. When you suspect the API has changed, compare fresh responses against recordings. Automated tests that call real APIs (in sandbox mode) catch changes that documentation does not mention.

Common Failure Modes

Integration failures follow patterns. Knowing the patterns helps you build defences before you encounter the failure in production.

The timeout that succeeded

You send a request. It times out after 30 seconds. You assume it failed and retry. But the original request actually succeeded. Now you have duplicate records, duplicate charges, or inconsistent state.

Defence: Idempotency keys on all mutating operations. Verify state before retrying. For critical operations (payments), query for existing record before creating new one.

The silent data change

The external API starts returning a field in a different format. No error, no warning. Your code parses it into garbage and stores it. You discover the problem a week later when someone notices corrupt data.

Defence: Schema validation on responses. Log warnings for unexpected formats. Monitoring for data quality (null rates, format consistency).

The cascading failure

An external API slows down. Your application's threads block waiting for responses. Thread pool exhausts. Other, unrelated requests start failing. The entire application becomes unresponsive.

Defence: Aggressive timeouts. Separate connection pools per integration. Circuit breakers. Bulkheads (isolate external calls from critical paths).

The credential expiry

OAuth tokens expire. API keys get rotated by the provider. A team member leaves and their personal API key stops working. Integration fails with 401 errors until someone manually fixes it.

Defence: Use service accounts, not personal credentials. Automated token refresh. Monitoring for authentication failures. Documented credential rotation procedures.

The rate limit death spiral

You hit a rate limit. Your retry logic retries immediately. You hit the limit again. You retry more aggressively. The provider extends your cooldown period. You have now created an outage that lasts hours instead of seconds.

Defence: Respect Retry-After headers. Implement exponential backoff with maximum. Queue requests during rate limiting rather than retrying aggressively.

The webhook flood

Something triggers a large number of webhooks in a short period. Your webhook handler processes them synchronously. Database connections exhaust. Background job queue fills. Other operations start failing.

Defence: Acknowledge webhooks immediately, process asynchronously. Rate limit your own processing. Use separate queues or workers for webhook processing.

Testing Integration Code

Integration code is notoriously difficult to test. The external dependency makes tests slow, flaky, and dependent on external state. But untested integration code is a liability. We use multiple testing strategies in combination.

Unit tests with mocks

Mock the HTTP client to return predetermined responses. Test your parsing logic, error handling, and business logic without network calls. Fast, deterministic, run in CI.

The limitation: mocks only return what you tell them to. They do not catch cases where the real API behaves differently than you expected.

Contract tests

Record real API responses and replay them in tests. Periodically refresh recordings to catch API changes. Tools like VCR, Betamax, or Polly automate this.

The limitation: recordings become stale. Real APIs return dynamic data (timestamps, IDs) that must be handled.

Sandbox testing

Most API providers offer sandbox or test environments. Use them for integration testing that exercises real network calls. Test both success paths and error paths (many sandboxes support triggering specific error conditions).

The limitation: sandboxes may not perfectly mirror production behaviour. Rate limits may be different. Some edge cases may not be reproducible.

Chaos testing

Intentionally introduce failures in a controlled environment. Inject network latency. Return error responses. Drop connections. Verify that fallback behaviour works as designed.

• Add 5 second delay to responses. Does your timeout trigger?
• Return 500 errors randomly. Does retry logic engage?
• Close connections mid-response. Do you handle partial data?
• Return 429 for all requests. Does backoff work?

What You Get

Integrations built with these patterns behave predictably in production. When external services fail (and they will), your system degrades gracefully rather than catastrophically.

• ✓
  Work reliably Failure handling, retry logic, and idempotency built in from the start. Transient failures resolve automatically.
• ✓
  Fail gracefully Circuit breakers and bulkheads prevent cascading failures. One bad integration does not bring down the system.
• ✓
  Are debuggable Comprehensive structured logging tells you exactly what happened. Correlation IDs link requests to business operations.
• ✓
  Handle change Abstraction layers isolate you from external API changes. Defensive parsing handles unexpected responses.
• ✓
  Are secure Credentials managed properly, permissions minimised, webhooks verified, transport encrypted.
• ✓
  Are observable Monitoring and alerting catch problems before users report them. Health checks verify end-to-end functionality.

Data flows between your systems reliably. You do not get woken up when an external service has a bad night. When problems do occur, you have the logs, metrics, and tools to diagnose and resolve them quickly.

Further Reading

• Laravel HTTP Client - Official documentation for Laravel's Guzzle wrapper with retry and timeout configuration.
• Stripe Idempotency Keys - Reference implementation of idempotency from one of the best-designed APIs in the industry.
• Circuit Breaker Pattern - Martin Fowler's canonical explanation of the circuit breaker pattern.