Upgrade Guide

Docker Compose Upgrade

1. Update the version in .env

   # Set to the version you want (e.g. 0.11.3)
   BREEZE_VERSION=0.11.3

2. Pull and restart

   docker compose pull
   docker compose up -d

   This pulls the new API, web, and binaries images and restarts all services. Database migrations run automatically on API startup when AUTO_MIGRATE=true (the default).

   Expected downtime: Typically 5–15 seconds while containers restart. If the new version includes database migrations, the API container will take additional time to apply them before accepting traffic – usually under 60 seconds for routine schema changes, but potentially several minutes for large data migrations. Agents continue running independently during this window and reconnect automatically.

3. Verify

   # Check API health
   curl -s https://breeze.yourdomain.com/health | jq .version
   
   # Check agent version (on the device)
   breeze-agent version

Tip

The binaries-init container uses pull_policy: always, so docker compose up -d automatically pulls the latest binaries image and refreshes the volume — no manual steps needed.

Version-specific notes

v0.95.0 — authentication & tenancy hardening

This release resets the authentication surface. Plan for a short low-traffic window and expect the following the first time the new API boots:

• Everyone is signed out once. Existing web and mobile sessions are invalidated and everyone signs in again — there is no silent-refresh recovery and no kill switch. Confirm you and your admins have a working multi-factor device or recovery codes before upgrading, since signing in is the only way back in. Agents, the Helper, the portal, and installers are unaffected.
• Connected AI / automation apps must re-authorize. Legacy OAuth refresh tokens are removed; any MCP/OAuth client reconnects and re-consents once.
• Microsoft 365 ticket mailboxes need re-consent. Non-disabled mailboxes drop to a re-auth state; a Partner Admin re-approves each one. Mail arriving during the gap is picked up after re-consent, not lost.
• New required database credential. Production now runs request handlers as an unprivileged, RLS-enforced role and will not start unless one of DATABASE_URL_APP, BREEZE_APP_DB_PASSWORD, or POSTGRES_PASSWORD is set, and it refuses to run as a SUPERUSER/BYPASSRLS role. See Environment Variables → Database. This check runs even when AUTO_MIGRATE=false.
• Behind Cloudflare? Set TRUST_CF_CONNECTING_IP=true (see Environment Variables).
• Migrations. 26 idempotent migrations auto-apply on boot. Most are metadata-only, but one backfills the refresh_token_families table — on a large, long-running deployment, check that table’s size first and expect a slightly longer boot while it applies.

Caution

Any value you set in .env only takes effect if it is also mapped in the api service’s environment: block of your compose file. The shipped compose files map the database-credential variables; if you maintain your own compose file, add them there too.

Agent Updates

Agents update themselves automatically. On each heartbeat (~60s), the API checks if a newer version is available and signals the agent to upgrade. The agent:

1. Downloads the new binary and verifies its SHA-256 checksum
2. Backs up the current binary
3. Replaces the executable and restarts the service
4. Rolls back automatically if the new binary fails to start

During an update, the device status changes to Updating in the dashboard so you can distinguish active updates from offline agents. If the update takes longer than 5 minutes, the device reverts to Offline. After a successful restart, the agent immediately reports its new version (bypassing the normal 0–60 second heartbeat jitter).

Linux Binary-In-Use Handling

On Linux, the updater handles the case where the agent binary is still in use (common with manual installs that don’t use a package manager). The updater removes the old binary before writing the new one to avoid ETXTBSY (“text file busy”) errors. If a file lock error still occurs, it is treated as transient and retried on the next heartbeat.

Watchdog Recovery

The watchdog service automatically restarts the agent after a failed update or crash. If an update leaves the agent in a bad state, the watchdog detects the failure and restarts the previous (backed-up) binary.

The binary source depends on your BINARY_SOURCE setting:

Mode	Agent downloads from
github (default)	GitHub Releases CDN
local	Your API server (or S3 if configured)

See Binary Distribution for details.

Disabling Auto-Update on Specific Devices

Some devices — production database servers, devices in a change-controlled window — should not self-update on the normal heartbeat cadence. Toggle auto-update per device without editing config on the machine:

# Disable auto-update for one device
curl -X POST "https://breeze.example.com/api/v1/devices/<device-id>/auto-update" \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

# Re-enable later
curl -X POST "https://breeze.example.com/api/v1/devices/<device-id>/auto-update" \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

The agent persists this setting across restarts. This is also the way to recover agents that were left with auto-update disabled by a manual or recovery update — see the set_auto_update command.

Controlled agent fleet rollout

By default (AGENT_AUTO_PROMOTE=true), publishing or syncing a new agent release immediately makes it the fleet’s upgrade target — every auto-updating device moves to it on its next heartbeat. If you’d rather stage rollouts (canary first, then the fleet), set AGENT_AUTO_PROMOTE=false to decouple registering a release from promoting it.

1. Turn on controlled rollout. Set AGENT_AUTO_PROMOTE=false in .env and map it in the api service environment: block of your compose file, then restart the API. (Compose only passes through variables you list there.)

2. Publish or sync the release. New binaries register as downloadable but are not promoted — the fleet stays on the current version. Existing devices are unaffected.

3. Canary it. Update a few non-production devices to the new version and verify (you can target an individual device’s update directly).

4. Promote to the fleet. When you’re satisfied, promote the version. This is a platform-admin action and requires MFA:

   curl -X POST "https://breeze.example.com/api/v1/agent-versions/promote" \
     -H "Authorization: Bearer <platform-admin-api-key>" \
     -H "Content-Type: application/json" \
     -d '{"version": "0.84.0"}'

   Auto-updating devices then move to the promoted version on their next heartbeat. To promote a single component (e.g. just the helper or watchdog) while holding the rest, add "component": "helper" to the body.

Tip

Leave AGENT_AUTO_PROMOTE unset (or true) to keep the default behavior where a published release rolls out immediately. The promote action is audit-logged either way.

Pinning Agent and Watchdog Versions

Instead of always tracking the latest promoted release, you can pin the fleet — or a single customer — to a specific version. Settings → Partner Defaults and Settings → Org Defaults each expose an Update version targets card with independent Agent target and Watchdog target selectors. Each selector offers every registered version plus Latest promoted (track the globally promoted release, the default behavior).

Pin precedence is inherit-with-override — deliberately different from the lock model used by the update policy and maintenance window below:

• A partner pin is the inherited default for all of that partner’s organizations.
• Any org can override the partner pin with its own value — including explicitly re-selecting Latest promoted to move that org back to the promoted release while the rest of the fleet stays pinned.
• Agent and watchdog pins resolve independently.

This is what lets a partner pilot a new version on one customer: pin the fleet to the known-good version at the partner level, then override a single org to the candidate version (or to Latest promoted).

Caution

Pins fail closed. If the resolved pin has no registered build for a device’s platform and architecture, the update — including watchdog-recovery pushes — is withheld: the device simply stays on its current version and retries on the next heartbeat. This is surfaced only in server logs and error tracking; there is no dashboard badge for a frozen device. Before pinning broadly, verify the pinned version has builds registered for every platform/architecture in your fleet.

Pins gate alongside the update policy and maintenance window, not instead of them — a pinned update still waits for the maintenance window and respects the update policy.

Agent Update Scheduling

In addition to the per-device auto-update toggle and the partner-level controlled rollout, Breeze supports a structured maintenance window for agent updates. When a maintenance window is configured, agents only accept update signals from the server during the allowed window — outside of it, update prompts are deferred until the next eligible window.

Setting a maintenance window

Agent update maintenance windows are set in Settings → Org Defaults (and optionally in Settings → Partner Defaults to apply fleet-wide). The window configuration includes:

Field	Description
Days	Which days of the week updates are permitted (e.g., Tuesday and Thursday)
Start time	Window start time in UTC
End time	Window end time in UTC

Note

All times are specified and stored in UTC. Communicate the UTC equivalent to your team to avoid scheduling confusion across time zones.

The update policy and maintenance window follow the same partner-locks precedence that governs other policy settings: a field set at the Partner level wins and locks — child organizations cannot override it. The org value applies only where the partner has left that field unset, and each field is merged independently. The update gate that agents check on heartbeat honors this merged result.

Note

The maintenance window gates when the update signal is acted on, not when the binary is downloaded. Agents can pre-download an available update during any window but will only apply it (restart the service) during the allowed window.

When no window is configured

If no maintenance window is set, agents update on their normal heartbeat cadence (~60 seconds after a new version is promoted). Configure a window when you need updates to avoid business hours or to concentrate restarts into a known change window.

Migration Failure Recovery

If the API container fails to start after an upgrade due to a migration error, follow these steps:

1. Check the container logs to identify the failing migration.

   docker compose logs api --tail 100

   Look for lines starting with [auto-migrate]. Common failure messages include:

   • Migration checksum mismatch – a previously applied migration file was modified. This should never happen in normal operation.
   • SQL errors during a specific migration file (e.g., constraint violations, missing columns).

2. Do not manually edit migration files. Migrations are tracked by checksum. Modifying a previously applied migration will cause all future startups to fail.

3. Fix and restart. The most common causes and resolutions:

   • Incompatible extension (e.g., TimescaleDB not installed): Install the required extension and restart.
   • Permissions issue: Ensure the database user has sufficient privileges.
   • Data conflict: Manually resolve the conflicting data in the database, then restart the API container.

   docker compose restart api

4. If unrecoverable, restore your database from a pre-upgrade backup and re-attempt the upgrade after resolving the underlying issue.

Caution

Migrations run inside a transaction. If a migration fails partway through, the transaction is rolled back and no partial changes are applied. The API will not start until all migrations succeed.

Zero-Downtime Upgrades

For production deployments where any downtime is unacceptable, you can perform a blue-green upgrade using two parallel Compose stacks:

1. Deploy the new version alongside the old one. Use a separate Compose project name or directory with the new BREEZE_VERSION and point it at the same database and Redis.

   COMPOSE_PROJECT_NAME=breeze-new BREEZE_VERSION=0.12.0 \
     docker compose -f docker-compose.yml -f docker-compose.override.yml up -d

2. Wait for migrations and health check. The new API container applies any pending migrations on startup. Once the health endpoint returns the expected version, the new stack is ready.

   curl -s http://localhost:NEW_PORT/health | jq .version

3. Switch traffic. Update your reverse proxy (Caddy, nginx, Cloudflare tunnel) to route traffic to the new stack.

4. Tear down the old stack. Once traffic is fully migrated, stop the old containers.

Rolling Back

1. Revert the version

   # In .env
   BREEZE_VERSION=0.11.2

2. Pull and restart

   docker compose pull
   docker compose up -d

Danger

Database migrations are forward-only. If the newer version ran a migration, reverting the API version may cause schema mismatches. Restore from a database backup if needed.

Checking Versions

# API version
curl -s https://breeze.yourdomain.com/health | jq .version

# Agent version (on the device)
breeze-agent version

# Docker image versions
docker compose images

Pre-Release Testing

You can test a release candidate before making it available to users:

1. Tag a prerelease

   git tag v0.11.3-rc.1 && git push --tags

   CI builds all images but does not tag them as latest.

2. Deploy the RC locally

   # In .env
   BREEZE_VERSION=0.11.3-rc.1
   
   docker compose pull
   docker compose up -d

3. Sync agents to the RC using the sync-github endpoint. This fetches the release assets from GitHub and registers them in the Breeze database so agents know a new version is available.

   curl -X POST "https://your-domain/api/v1/agent-versions/sync-github?version=v0.11.3-rc.1" \
     -H "Authorization: Bearer <api-key>"

   The response includes the version that was synced and the list of platform/architecture targets that were registered:

   {
     "version": "0.11.3-rc.1",
     "synced": ["agent:linux/amd64", "agent:macos/arm64", "agent:windows/amd64", "helper:windows/amd64"]
   }

   Agents will pick up the new binary on their next heartbeat (~60 seconds). Omit the ?version= parameter to sync the latest stable release instead of a specific version. This endpoint requires system-level (admin) authentication.

4. Test, then promote

   git tag v0.11.3 && git push --tags

   This pushes latest tags — all users get the update.