SalesEQ - Elevated errors across multiple services in IAD1 (Washington, D.C., USA) – Incident details

Cloudflare → Cloudflare Sites and Services → Network experiencing degraded performance

Elevated errors across multiple services in IAD1 (Washington, D.C., USA)

Resolved
Degraded performance
Started 3 months agoLasted about 5 hours

Affected

Vercel → Functions

Operational from 1:18 AM to 1:18 AM, Degraded performance from 1:18 AM to 4:54 AM, Operational from 4:54 AM to 6:39 AM

Updates
  • Resolved
    UTC
    Resolved

    We've deployed mitigations and all services are operating normally. Traffic may continue to reroute to nearby regions for the next few hours, but we will restore traffic gradually after verifying system availability in the IAD1 region.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are monitoring the results. Backlogs for Workflows and Queues will be processed within the next few hours.

  • Update
    UTC
    Update

    New workflow runs and queue messages are being processed now. Processing the backlog of existing workflow runs and queued messages is still being investigated.

  • Update
    UTC
    Update

    We are continuing to work on a fix for this issue.

  • Update
    UTC
    Update

    New messages to Vercel Workflows and Vercel Queues are being queued, but message processing is paused. Queued messages will be processed when service is restored.

  • Update
    UTC
    Update

    Traffic to the IAD1 region has been re-routed to nearby regions. Functions configured in IAD1 will be invoked in a different region if failover regions are configured, but if no other regions are configured, the function will still be invoked in IAD1.

  • Identified
    UTC
    Identified

    Some Vercel functions that run in the IAD1 region are experiencing elevated invocation failures. We are investigating the issue and will share more information as it becomes available.