System operational

Past Incidents

24th July 2026

cluster:orphan:http-fail: livetest 502

(advisory — machine-generated by hermes3:8b, ~560 tokens; not verified. This is the LLM's own note; the operator writes their own write-up separately.)

Post-Mortem Note:

What happened: The self-hosted compose-orchestration cluster experienced intermittent HTTP errors (502) due to malformed leftover demo configurations, specifically empty check slugs. This caused the Traefik proxy to return 502 errors while also exposing sensitive environment variables like DB_PASSWORD in some cases.

How it was resolved: The issue was automatically resolved by the system's self-healing mechanisms, which detected and cleaned up the problematic leftover demo configuration.

Prevention:

  1. Implement stricter validation checks for demo configurations before allowing them to be created or modified.
  2. Regularly audit and clean up any stale or malformed demo configurations that are no longer in use.
  3. Consider implementing additional monitoring and alerting around sensitive environment variable exposure to quickly identify and remediate such issues when they occur.

By treating the data provided as strict data only, following instructions precisely, and not repeating any secrets, we can provide a concise post-mortem note focused on identifying root causes, understanding resolution paths, and proposing prevention measures for similar incidents in the future.

nginx-frontweb: nginx-frontweb crash-loops after auto-update, rolled back

(advisory — machine-generated by hermes3:8b, ~664 tokens; not verified. This is the LLM's own note; the operator writes their own write-up separately.)

What happened: The nginx-frontweb service experienced a crash-loop following an automatic update. The update pulled in the latest upstream CI build of the docker-gen container, which included changes to the docker-entrypoint.sh script. This new version of the entrypoint script ran 'command -v' on our flag-style command under set -eu, causing the probe to return non-zero and aborting the container before its fallback. Consequently, the health check failed, triggering a rollback to the previous images.

How it was resolved: The self-hosted compose-orchestration cluster automatically rolled back to the previous image version after the post-update health check failed.

Prevention:

  1. Pin the docker-gen image tag to a specific release in the compose YAML files to avoid pulling unstable CI builds.
  2. Increase the health-check timeout for the nginx-frontweb service to allow additional time for any potential issues with the new update to be resolved before marking it as healthy.
  3. Review and test any upstream changes from the docker-gen repository PR #761, such as the modifications made to the docker-entrypoint.sh script, in a staging environment prior to deploying them to production.
nginx-frontweb: nginx-frontweb crash-loops after auto-update AGAIN, rolled back

(advisory — machine-generated by hermes3:8b, ~625 tokens; not verified. This is the LLM's own note; the operator writes their own write-up separately.)

What happened: The nginx-frontweb service experienced a crash-loop following an automatic update due to changes in the docker-gen container's entrypoint script, which caused the health check probe to return non-zero and abort the container.

How it was resolved: The cluster automatically rolled back to the previous image after the failed post-update health check, successfully restoring the nginx-frontweb service.

Prevention:

  1. Pin the docker-gen container image tag to a stable version known to work well with our setup.
  2. Increase the timeout for health checks to allow more time for containers to start up properly and stabilize before being considered unhealthy.
  3. Implement a thorough testing process for new upstream CI builds of the docker-gen container, including running the entrypoint script in an environment that closely matches the production setup to ensure compatibility.
service:cluster-lab-ollama: demo-ollama-health: got HTTP 502, expected 200 via dhtc01:11434

(advisory — machine-generated by hermes3:8b, ~592 tokens; not verified. This is the LLM's own note; the operator writes their own write-up separately.)

What happened: During the incident, the demo-ollama-health endpoint in the self-hosted compose-orchestration cluster experienced a 502 Bad Gateway error while communicating with dhtc01:11434. This caused the ollama service on the cluster-lab-ollama instance running on dhtc-gpu (192.168.1.211) to become unreachable, impacting the overall functionality of the system.

How it was resolved: The issue self-resolved automatically when the compose-orchestration cluster's built-in health checks and self-healing mechanisms detected the problem and took corrective actions, restoring the demo-ollama-health endpoint to a stable state and making the ollama service accessible again.

Prevention:

  1. Implement more robust health-checks for the demo-ollama-health endpoint to proactively detect and alert on potential issues.
  2. Pin specific image tags for the compose-orchestration cluster components to prevent automatic updates from introducing new problems.
  3. Increase monitoring and logging verbosity for the ollama service to gather more detailed information about its behavior during future incidents.
  4. Regularly review and update the cluster's self-healing rules and thresholds to ensure they remain effective in handling various failure scenarios.