Webhooks

Webhooks let you push real-time event notifications from Breeze to any external HTTP endpoint. When a device enrolls, an alert fires, a script completes, or any other tracked event occurs, Breeze sends a signed JSON payload to each webhook configured for that event type. Webhooks are scoped to an organization and support custom headers, HMAC-SHA256 signature verification, automatic retries with exponential backoff, and a full delivery history with per-delivery status tracking.

---

Creating a Webhook

1. Choose the events you want to subscribe to (see Event Types below).

2. Send a POST request with the webhook name, destination URL, signing secret, and event list:

   POST /webhooks
   Content-Type: application/json
   
   {
     "name": "PagerDuty Alerts",
     "url": "https://events.pagerduty.com/integration/abc123/enqueue",
     "secret": "whsec_your_signing_secret",
     "events": ["alert.triggered", "alert.escalated", "device.offline"],
     "headers": [
       { "key": "X-Custom-Source", "value": "breeze-rmm" }
     ]
   }

3. The API validates the URL (must be HTTPS, must not resolve to private/loopback address space), encrypts the secret, and returns the new webhook:

   {
     "id": "d290f1ee-6c54-4b01-90e6-d701748f0851",
     "orgId": "11111111-1111-1111-1111-111111111111",
     "name": "PagerDuty Alerts",
     "url": "https://events.pagerduty.com/integration/abc123/enqueue",
     "events": ["alert.triggered", "alert.escalated", "device.offline"],
     "headers": [{ "key": "X-Custom-Source", "value": "breeze-rmm" }],
     "status": "active",
     "hasSecret": true,
     "createdAt": "2026-02-18T12:00:00.000Z",
     "updatedAt": "2026-02-18T12:00:00.000Z",
     "lastDeliveryAt": null
   }

4. Send a test delivery to verify connectivity (see Testing a Webhook).

Note

Both the secret and the url are encrypted at rest using AES-256-GCM. The secret is never returned in API responses — only the hasSecret boolean indicates whether one is configured. The url is returned by GET responses, but any credentials embedded in it (HTTP basic-auth userinfo and signed query tokens) are stripped before the value is sent to the client.

Organization Scoping

Webhooks follow the standard multi-tenant access model:

Auth Scope	Behavior
Organization	orgId is set automatically from the authenticated context.
Partner	orgId must be provided in the request body. The API verifies the partner has access to that organization.
System	orgId must be provided in the request body.

---

Event Types

Webhooks subscribe to events emitted by the Breeze event bus. Pass one or more event type strings in the events array when creating or updating a webhook.

Device Events

Event	Description
device.enrolled	A new agent has completed enrollment
device.online	A device has come online (heartbeat received)
device.offline	A device has stopped sending heartbeats
device.updated	Device metadata or configuration has changed
device.decommissioned	A device has been decommissioned

Alert Events

Event	Description
alert.triggered	A monitoring rule has fired a new alert
alert.acknowledged	An alert has been acknowledged by a user
alert.resolved	An alert condition has cleared
alert.escalated	An alert has been escalated to a higher tier

Script Events

Event	Description
script.started	Script execution has begun on a device
script.completed	Script execution finished successfully
script.failed	Script execution failed

Automation Events

Event	Description
automation.started	An automation workflow has begun
automation.completed	An automation workflow completed successfully
automation.failed	An automation workflow failed

Policy Events

Event	Description
policy.evaluated	A configuration policy has been evaluated against a device
policy.violation	A device is non-compliant with a policy
policy.compliant	A device has returned to compliance

Patch Events

Event	Description
patch.available	New patches are available for a device
patch.approved	A patch has been approved for installation
patch.installed	A patch was installed successfully
patch.failed	A patch installation failed
patch.rollback	A patch has been rolled back

Other Events

Event	Description
security.score_changed	A device’s security posture score has changed
remote.session.started	A remote access session has begun
remote.session.ended	A remote access session has ended
user.login	A user logged in
user.logout	A user logged out
session.login	A device user session started
session.logout	A device user session ended

---

Delivery Mechanism

When an event is emitted on the Breeze event bus, the webhook delivery system routes it to all active webhooks in the same organization that are subscribed to that event type. Each delivery follows this sequence:

1. The event bus fires the event. The initializeWebhookDelivery subscriber matches the event type against all configured webhooks for the organization.

2. A delivery record is created in the webhook_deliveries table with status pending and attempts set to 0.

3. The delivery job is queued onto the Redis list breeze:webhooks:delivery.

4. The WebhookDeliveryWorker picks up the job, validates the destination URL (including DNS resolution to block SSRF), and sends an HTTP POST with the signed payload.

5. If the response status is 2xx, the delivery is marked delivered. Otherwise, the retry logic takes over.

Payload Format

Every webhook delivery sends a JSON POST body with the following structure:

{
  "id": "event-uuid",
  "type": "device.offline",
  "timestamp": "2026-02-18T12:34:56.789Z",
  "orgId": "11111111-1111-1111-1111-111111111111",
  "data": {
    "deviceId": "abc123",
    "hostname": "WORKSTATION-01",
    "lastSeen": "2026-02-18T12:30:00.000Z"
  }
}

Field	Description
id	Unique event ID (UUID)
type	The event type string (e.g., alert.triggered)
timestamp	ISO 8601 timestamp of when the event was created
orgId	The organization the event belongs to
data	Event-specific payload; contents vary by event type

Request Headers

Every delivery includes these headers:

Header	Description
Content-Type	application/json
User-Agent	Breeze-Webhooks/1.0
X-Breeze-Delivery-Id	Unique ID for this delivery attempt
X-Breeze-Event-Id	The event ID from the payload
X-Breeze-Event-Type	The event type string
X-Breeze-Timestamp	Unix timestamp (milliseconds) of the delivery attempt
X-Breeze-Signature	HMAC-SHA256 signature (only if a secret is configured)
X-Breeze-Signature-Timestamp	Timestamp used in the signature computation

Any custom headers configured on the webhook are merged into the request after the standard headers.

---

Retry Logic

Failed deliveries are retried automatically using exponential backoff. The default retry policy is:

Parameter	Default Value
Maximum retries	5
Initial delay	1,000 ms (1 second)
Backoff multiplier	2x
Maximum delay	300,000 ms (5 minutes)
Delivery timeout	30,000 ms (30 seconds)

The delay between retries follows the formula:

delay = min(initialDelay * (multiplier ^ attempt), maxDelay)

This produces the following schedule for the default policy:

Attempt	Delay
1st retry	1 second
2nd retry	2 seconds
3rd retry	4 seconds
4th retry	8 seconds
5th retry	16 seconds

Custom Retry Policy

You can override the default retry policy per webhook by setting the retryPolicy field in the database. The policy object accepts:

{
  "maxRetries": 10,
  "initialDelayMs": 2000,
  "backoffMultiplier": 3,
  "maxDelayMs": 600000
}

Dead Letter Queue

After all retries are exhausted, the delivery job is moved to a dead letter queue (Redis key breeze:webhooks:dlq). Jobs in the DLQ can be inspected and retried through the worker API.

Caution

Redirects are rejected (redirect: "error"). If your endpoint returns a 3xx redirect, the delivery will be treated as a failure and retried.

---

Signature Verification

When a webhook has a signing secret, every delivery includes an HMAC-SHA256 signature in the X-Breeze-Signature header. Use this to verify that the payload was sent by Breeze and has not been tampered with.

How the Signature Is Computed

The signature input is the concatenation of the timestamp and the raw JSON payload body, separated by a dot:

signature_input = timestamp + "." + payload_body
signature = HMAC-SHA256(secret, signature_input)

The X-Breeze-Signature header value is prefixed with sha256=:

X-Breeze-Signature: sha256=a1b2c3d4e5f6...

The X-Breeze-Signature-Timestamp header contains the same timestamp value used in the signature computation.

Verification Examples

Node.js

import { createHmac, timingSafeEqual } from 'crypto';

function verifyWebhookSignature(payload, secret, signatureHeader, timestampHeader) {
  const expectedSig = signatureHeader.replace('sha256=', '');
  const signatureInput = `${timestampHeader}.${payload}`;
  const computedSig = createHmac('sha256', secret)
    .update(signatureInput)
    .digest('hex');

  return timingSafeEqual(
    Buffer.from(expectedSig, 'hex'),
    Buffer.from(computedSig, 'hex')
  );
}

Python

import hmac
import hashlib

def verify_webhook_signature(payload: bytes, secret: str, signature_header: str, timestamp_header: str) -> bool:
    expected_sig = signature_header.removeprefix("sha256=")
    signature_input = f"{timestamp_header}.{payload.decode('utf-8')}"
    computed_sig = hmac.new(
        secret.encode('utf-8'),
        signature_input.encode('utf-8'),
        hashlib.sha256
    ).hexdigest()

    return hmac.compare_digest(expected_sig, computed_sig)

Tip

Always use a constant-time comparison function (such as timingSafeEqual in Node.js or hmac.compare_digest in Python) to prevent timing attacks.

---

Delivery History

Every webhook maintains a paginated delivery history. Each delivery record captures the event type, payload, response details, and retry state.

Listing Deliveries

GET /webhooks/:id/deliveries?page=1&limit=50&status=failed

All query parameters are optional. Results are ordered by most recent first.

Parameter	Type	Description
page	integer	Page number (default: 1)
limit	integer	Results per page (default: 50, max: 100)
status	string	Filter by delivery status: pending, delivered, failed, retrying

Response:

{
  "data": [
    {
      "id": "delivery-uuid",
      "webhookId": "webhook-uuid",
      "orgId": "org-uuid",
      "status": "delivered",
      "event": "alert.triggered",
      "eventId": "event-uuid",
      "payload": { "alertId": "...", "severity": "critical" },
      "responseStatus": 200,
      "responseBody": "ok",
      "attempt": 1,
      "nextAttemptAt": null,
      "createdAt": "2026-02-18T12:34:56.000Z",
      "deliveredAt": "2026-02-18T12:34:57.123Z"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 50,
    "total": 142
  }
}

Delivery Statistics

When you fetch a single webhook via GET /webhooks/:id, the response includes aggregated delivery statistics:

{
  "id": "webhook-uuid",
  "name": "PagerDuty Alerts",
  "status": "active",
  "deliveryStats": {
    "total": 1284,
    "pending": 2,
    "delivered": 1250,
    "failed": 30,
    "retrying": 2,
    "lastDeliveredAt": "2026-02-18T12:34:57.123Z"
  }
}

Retrying a Failed Delivery

You can manually retry any delivery that has a failed status. The retry creates a new delivery record with the same event type and payload, and queues it for immediate processing.

POST /webhooks/:id/retry/:deliveryId

Response (202 Accepted):

{
  "message": "Delivery retry queued",
  "delivery": {
    "id": "new-delivery-uuid",
    "status": "pending",
    "attempt": 0
  }
}

Note

Only deliveries with status failed can be retried. Attempting to retry a delivery with any other status returns a 400 error.

---

Testing a Webhook

Send a test delivery to verify that your endpoint is reachable and correctly processing payloads:

POST /webhooks/:id/test
Content-Type: application/json

{
  "event": "webhook.test",
  "payload": {
    "message": "Hello from Breeze RMM"
  }
}

Both event and payload are optional. If omitted, the test uses event type webhook.test and a default payload containing a message, timestamp, and webhook ID.

Response (202 Accepted):

{
  "message": "Test delivery queued",
  "delivery": {
    "id": "delivery-uuid",
    "webhookId": "webhook-uuid",
    "status": "pending",
    "event": "webhook.test",
    "attempt": 0
  }
}

The test delivery is queued through the same delivery pipeline as real events, so it exercises the full path including HMAC signing, URL validation, and retry logic.

---

Updating a Webhook

Update any combination of name, URL, secret, events, headers, or status:

PATCH /webhooks/:id
Content-Type: application/json

{
  "events": ["device.enrolled", "device.offline", "alert.triggered"],
  "status": "paused"
}

Status Value	Description
active	Webhook is enabled and will receive deliveries
paused	Webhook is temporarily disabled; no deliveries are sent
failed	Webhook has been marked as failed (can be set manually or by the system)

Note

When you update the url field, the new URL goes through the same SSRF safety validation (HTTPS required, no private/loopback IPs, DNS resolution check) as during creation.

---

Deleting a Webhook

DELETE /webhooks/:id

Caution

Deleting a webhook also removes all associated delivery records from the database. This action cannot be undone.

---

URL Safety Validation

Breeze validates webhook URLs at both creation time and delivery time to prevent SSRF (Server-Side Request Forgery) attacks.

At creation and update time, the API performs synchronous validation including DNS resolution:

• URL must use HTTPS
• Hostname must not be localhost, *.localhost, or *.local
• IP addresses must not be private, loopback, or link-local (covers RFC 1918, RFC 6598, RFC 5737, and IPv6 equivalents)
• DNS resolution is performed to catch hostnames that resolve to blocked address ranges

At delivery time, the worker re-validates the URL before every delivery attempt. If the URL has become unsafe (e.g., DNS now resolves to a private IP), the delivery fails immediately without making an HTTP request.

---

API Reference

Method	Path	Description
GET	/webhooks	List webhooks for the organization (paginated, filterable by status and orgId)
POST	/webhooks	Create a new webhook
GET	/webhooks/:id	Get webhook details including delivery statistics
PATCH	/webhooks/:id	Update webhook name, URL, secret, events, headers, or status
DELETE	/webhooks/:id	Delete a webhook and all its delivery records
GET	/webhooks/:id/deliveries	List delivery history (paginated, filterable by status)
POST	/webhooks/:id/test	Send a test delivery
POST	/webhooks/:id/retry/:deliveryId	Retry a failed delivery

All webhook routes require authentication via the authMiddleware and are scoped to organization, partner, or system roles.

---

Database Schema

webhooks

Column	Type	Description
id	UUID	Primary key
org_id	UUID	Organization this webhook belongs to
name	varchar(255)	Display name
url	text	Destination URL (must be HTTPS); encrypted at rest using AES. GET responses strip any credentials embedded in the URL (HTTP basic-auth userinfo and signed query tokens) before returning it.
secret	text	Encrypted HMAC signing secret (AES-256-GCM)
events	text[]	Array of subscribed event type strings
headers	JSONB	Custom headers sent with each delivery
status	enum	active, disabled, or error
retry_policy	JSONB	Optional custom retry policy override
success_count	integer	Cumulative successful deliveries
failure_count	integer	Cumulative failed deliveries
last_delivery_at	timestamp	Timestamp of the most recent delivery attempt
last_success_at	timestamp	Timestamp of the most recent successful delivery
created_by	UUID	User who created the webhook
created_at	timestamp	Creation timestamp
updated_at	timestamp	Last modification timestamp

webhook_deliveries

Column	Type	Description
id	UUID	Primary key
webhook_id	UUID	Foreign key to webhooks
event_type	varchar(100)	The event type that triggered this delivery
event_id	varchar(100)	The unique event ID
payload	JSONB	The full event payload sent to the endpoint
status	enum	pending, delivered, failed, or retrying
attempts	integer	Number of delivery attempts made
next_retry_at	timestamp	Scheduled time for the next retry (if retrying)
response_status	integer	HTTP response status code from the endpoint
response_body	text	Response body (truncated to 1,000 characters)
response_time_ms	integer	Round-trip time in milliseconds
error_message	text	Error description if the delivery failed
created_at	timestamp	When the delivery record was created
delivered_at	timestamp	When the delivery was successfully completed

---

Troubleshooting

“Invalid webhook URL” error when creating a webhook. The URL must use HTTPS and must not resolve to a private, loopback, or link-local IP address. Hostnames like localhost, *.localhost, and *.local are also blocked. If the hostname cannot be resolved via DNS, creation will fail.

Webhook is not receiving events. Check the webhook status – if it is paused or failed, deliveries are not processed. Verify that the events array includes the event types you expect. Use POST /webhooks/:id/test to confirm the endpoint is reachable.

Deliveries are stuck in retrying status. The worker retries failed deliveries using exponential backoff. Check the nextAttemptAt field on the delivery to see when the next attempt is scheduled. If retries have been exhausted (default: 5 attempts), the delivery moves to failed and the job enters the dead letter queue.

Signature verification fails on my endpoint. Ensure you are computing the HMAC over the correct input: the raw X-Breeze-Signature-Timestamp header value, followed by a dot (.), followed by the raw request body. Use the X-Breeze-Signature-Timestamp header (not X-Breeze-Timestamp) as the timestamp component. The signature uses SHA-256 and is hex-encoded.

“Unsafe webhook URL” error during delivery. The worker re-validates the URL before each delivery attempt. If your DNS records have changed and the hostname now resolves to a private IP range, deliveries will fail. Update the webhook URL to point to a publicly routable address.

“Only failed deliveries can be retried” error. The manual retry endpoint (POST /webhooks/:id/retry/:deliveryId) only accepts deliveries with status failed. Deliveries that are pending, retrying, or already delivered cannot be retried through this endpoint.

“Organization context required” error. Webhook routes require an organization scope. If you are authenticated with a partner or system scope, pass the orgId query parameter (for list) or include it in the request body (for create) to specify which organization’s webhooks you are managing.

Response body is truncated. Webhook delivery response bodies are truncated to 1,000 characters to prevent excessive storage consumption. If you need the full response for debugging, check your endpoint’s own logs.