Skip to main content

SC Application Development & Delivery Pipeline

The single source of truth for how Symphony Core builds, tests, ships, and operates its internal applications — from a code change on a laptop to a running container in production, plus the infrastructure footprint and cost that sit underneath it.

This standard is prescriptive: it defines the target process every internal-app repo should follow. Where a repo diverges from it today, the deviation is tracked in §13 Governance & fix-forward register rather than silently tolerated.

Relationship to other standards (who owns what)​

This document owns the end-to-end pipeline. Deep, topic-specific rules live in their own standards; this doc references them rather than restating them.

ConcernAuthoritative doc
Architecture principles, language/coding standards, security posturesc-software-dev-guide.md
The go/no-go production gate (blocking checklist, system tiers)production-gate-checklist.md
Change-request lifecycle, commit/branch conventions, requirement numberingsc-change-management-standard.md
Health-endpoint contract (/health, /health/ready)health-check-standards.md
Deploy mechanics for the current stack (exact commands)sc-infrastructure/docs/migration/hetzner-deployment-runbook.md
Live service inventory (ports, containers, status)sc-infrastructure/docs/reference/cross-repo-service-registry.md

Note on sc-software-dev-guide.md: its earlier "Cloud Infrastructure (GCP) / Cloud Run" pipeline described an aspirational target that was never adopted. Production runs on Hetzner + Docker Compose (see §2). That doc's deployment/CI-CD sections now defer to this one.

1. Principles​

  1. One deploy orchestrator. Application repos produce images; sc-infrastructure deploys them. There is exactly one production stack definition. Never run an app's own docker-compose or npm start in production.
  2. Everything that changes production is in git — app code, compose/overlays, ops scripts. Runtime config changes are codified as versioned scripts, not clicked in a UI (see §9).
  3. Single active primary. Exactly one host serves production at a time. Handoffs are deliberate and ordered (see §10).
  4. Secrets never touch git. 1Password is the source of truth; envs are rendered on the host with op inject (see §5).
  5. Cost is a first-class metric. The infrastructure footprint is small and deliberate; it is reviewed on a cadence (see §11).
  6. Reversible by default. Every production change has a known rollback (image pin, hostname flip, docker start).

2. System context​

Symphony Core's internal software is a set of containerized services, each in its own repo, orchestrated by a single infrastructure repo.

  • Application repos own their source, Dockerfile, tests, and CI. They do not own how they run in production.
  • sc-infrastructure is the singleton orchestrator: docker-compose.yml + environment overlays reference each app via build: { context: ../<app-repo> }, names everything symphonycore_*, and is always invoked with COMPOSE_PROJECT_NAME=symphonycore.
  • Deploy topology (current): production = Hetzner sc-prod; RAPTOR = warm standby / rollback; BEAST = cold archive. See §11 for specs and cost.

3. Environments​

Three environments, promoted left to right. A change is proven in each before the next.

EnvironmentWherePurposeData & credentials
DevelopmentLocal (each engineer)Write + unit-test a changeLocal DB/Redis; USE_MOCK_ADAPTERS=true — never real GHL/Slack/Drive
StagingRAPTOR (to be isolated)Integration-test against the real stack shape before prodSandbox/mock creds only + throwaway DB; must NOT touch production GHL token, Slack, Drive, or paid APIs
ProductionHetzner sc-prodServes clientsReal secrets (1Password), real data

Promotion rule: code merges to trunk → CI green → image built → deployed to staging → verified → deployed to prod. Runtime-config changes follow the same left-to-right order via their codified script (§9).

Current-state gap: the staging environment is not yet isolated. RAPTOR is a stopped rollback, not a live sandbox. Until it is stood up with mock/sandbox creds and ADR-006 drill mode (NO_EXTERNAL_WRITES), "staging" testing happens locally with mock adapters. Tracked in §13.

4. Repository & branching model​

  • Trunk-based development. One long-lived trunk per repo; short-lived feature/*, fix/*, docs/* branches off it; small PRs merged back quickly.
  • Trunk name: main. New repos use main.
  • Conventional Commits (feat:, fix:, docs:, chore: …) — enables changelog and version automation. Commit/branch conventions detail: sc-change-management-standard.md.
  • Pull requests are mandatory for trunk: at least one review, CI green, the repo's pre-merge checklist satisfied. Use a .github/pull_request_template.md (GitHub auto-loads it from that path — not templates/).
  • CODEOWNERS on each repo routes reviews.

Current-state gaps: sc-infrastructure is on main; symphony-flow, website-monitoring-system, and ghl-data-sync are still on master — standardize to main. PR templates in symphony-flow/WMS live in templates/ (not auto-loaded); move to .github/. sc-infrastructure has no PR template. Tracked in §13.

5. Coding, testing & secrets standards​

Deep language rules live in sc-software-dev-guide.md; the delivery-relevant baseline:

Stacks

  • Node 20 / TypeScript (symphony-flow, ghl-data-sync): eslint + prettier + tsc --noEmit typecheck.
  • Python 3.13 / FastAPI (website-monitoring-system): ruff + black (line 100) + mypy strict.

Testing

  • Split unit / integration / e2e. Integration/e2e spin up real postgres + redis service containers with USE_MOCK_ADAPTERS=true.
  • Coverage gate: ≥ 70% baseline for every app repo, enforced in CI. Repos already above (WMS 80%) keep their higher bar.
  • Every service exposes the health contract from health-check-standards.md.

Secrets (uniform across all repos)

  • Committed .env.1password manifest of op://… references; gitignored .env.
  • Render with op inject -i .env.1password -o .env (headless on hosts via a 1Password service-account token).
  • One 1Password item per repo in vault sc-Infrastructure-DR (<repo>-env). Real secret values never appear in git or in chat/logs.

Current-state gaps: coverage gates are inconsistent (WMS 80% enforced · symphony-flow 50/60 · ghl-data-sync 0/none · sc-infrastructure n/a). pre-commit is listed as a dev dep in WMS but no .pre-commit-config.yaml is wired in any repo — adopt pre-commit (format + secret-scan) repo-wide. Tracked in §13.

6. Continuous integration (CI)​

Every app repo runs CI in GitHub Actions on push/PR to its trunk. Required jobs:

JobNode/TS reposPython repos
Lint + typecheckeslint, prettier --check, tsc --noEmitruff, black --check, mypy
Unit testsjestpytest
Integration/e2ejest with postgres + redis service containerspytest + postgres service
Coverage gatejest thresholds (≥70%)--cov-fail-under (≥70%)
Schema safety (Python)—create_all + drift audit against real Postgres (catches dialect-only bugs SQLite tests miss)

sc-infrastructure CI validates instead of tests: docker compose config, env-template secret check, super-linter (hadolint/yamllint/shellcheck).

Resolved 2026-08-11: ghl-data-sync now has CI gating build (tsc) + lint (eslint); symphony-flow's CI trigger now includes master so it actually fires. Still open: ghl-data-sync's jest suite is not hermetic (hardcoded LAN Postgres + seeded prod rows) so it is not yet gated — see §13 #12. Both closures tracked in §13.

7. Build & container image standard​

  • Dockerfile: multi-stage where it helps, a slim base (node:20-alpine, python:3.13-slim), runs as non-root, and defines a HEALTHCHECK.
  • Image name: symphonycore/<service>:<tag> (e.g. symphonycore/symphony-flow-api).
  • Provenance: pass GIT_SHA and BUILD_TIME build args and surface them on the health endpoint + OCI labels, so a running container is traceable to a commit.
  • Where images are built — on the deploy host. sc-infrastructure's compose builds each service from its ../<app-repo> context during docker compose up -d --build. There is no container registry today.
    • Why: one host, low change rate, and building on-box avoids registry auth/credential plumbing (which has bitten us on locked-down hosts). It is a deliberate simplification.
    • When to adopt a registry (GHCR): a second production host, a real staging host that should run the identical image, or CI-built images become the source of truth. At that point CI builds and pushes symphonycore/<svc>:<git-sha> to GHCR and the host pulls.

Current-state gaps: ghl-data-sync images are unprefixed (ghl-rest-api, ghl-token-service) — rename to symphonycore/*. Only symphony-flow threads GIT_SHA/BUILD_TIME — standardize across repos. Tracked in §13.

8. Release & versioning​

  • Semantic versioning (MAJOR.MINOR.PATCH) per repo.
  • CHANGELOG.md in Keep-a-Changelog format with an [Unreleased] section; roll it into a dated version block on release and tag the commit (vX.Y.Z).
  • Image tags: :latest is used today for the single-host build-on-box model. When a registry is adopted (§7), pin deploys to an immutable :<git-sha> tag; :latest becomes a convenience alias only.
  • The deploy unit is sc-infrastructure's compose, not an individual app version — a production release is "the set of images currently built + the compose/overlay commit."

Current-state gap: only sc-infrastructure practices real semver + dated tags; app repos sit at static 0.1.0/1.0.0 with [Unreleased]-only changelogs, and WMS uses a sprint-based (non-semver) changelog. Align on semver + tags. Tracked in §13.

9. Deploying to production (Hetzner)​

Two change types, two flows.

9a. Code change (app or infra)​

  1. Merge to trunk; CI green.
  2. On the deploy host (ssh sc-hetzner), pull the latest app + infra repos.
  3. Rebuild + roll the service: 
    COMPOSE_PROJECT_NAME=symphonycore docker compose --env-file .env \
      -f docker-compose.yml -f docker/compose/_prod/docker-compose.prod.yml \
      up -d --build <service>
    The _prod overlay applies production hardening (Traefik 127.0.0.1-only, no published app ports, resource limits, cloudflared tunnel).
  4. Verify health + the affected route; watch logs.

Greenfield (fresh host) additionally runs scripts/setup/create-volumes.sh, the guarded Postgres init (rejects weak/example DB passwords), data restore, and the post-restore runtime state (schedules + registry seed). Full mechanics: sc-infrastructure/docs/migration/hetzner-deployment-runbook.md.

Never assume a code change is live without confirming the container was rebuilt — docker compose ps + image build time.

9b. Runtime-configuration change (config-as-code)​

Some production behavior lives in runtime state, not repo code — e.g. Uptime Kuma monitor parameters, or symphony-flow's Redis-resident schedules. Do not hand-edit these in a UI. Instead:

  1. Codify the change as an idempotent, env-overridable script in sc-infrastructure/scripts/setup/ (pattern: ensure-symphony-flow-schedules.sh, tune-kuma-monitor-retries.sh). Commit it.
  2. Run it against staging (CONTAINER=<staging_container> …), verify.
  3. Run it against prod. The script is the reviewable, repeatable artifact.

This keeps runtime config reproducible and auditable, and gives it the same staging→prod path as code.

Rollback​

  • Code: redeploy the prior image / revert the compose commit.
  • Ingress: flip the affected hostname CNAME back to the standby's tunnel.
  • Whole-host: start the standby's stopped stack (docker start) and re-point hostnames (see §10).

10. Operations​

Monitoring (defense in depth):

  • Gatus (status.symphonycorelabs.com) — aggregates internal service health + a few external checks; alerts to Slack #sc-infrastructure + email.
  • Uptime Kuma (wms.…) — client-website monitoring (the WMS product) + meta-monitoring.
  • UptimeRobot (external, free) — an independent watchdog on flow/status/wms + intake, hosted off all owned infrastructure. This is the layer that catches a whole host going dark (a monitor that lives on the failed host can't alert on its own death).

Backups: scripts/backup/daily-backup.sh at 03:00 UTC — pg_dump -Fc of every keep-set DB + tars of the Uptime-Kuma/Gatus volumes → rclone to Google Drive (sc-infrastructure-backup/hetzner-daily/), with local + remote retention. Hetzner whole-box auto-backups are enabled as a second layer. Restores are rehearsed as part of DR (sc-infrastructure/docs/guides/disaster-recovery-plan.md).

Single-primary discipline: exactly one host is active. The ADR-006 deployment-role guard (../device-deployments) enforces this on Windows hosts, but it is Windows-only, so RAPTOR↔Hetzner single-primary is procedural:

  • On handoff, stop the old primary's active containers token-service first (it's the single GHL OAuth refresher — two of them thrash the same token).
  • On a standby host, clear the restart policy — docker update --restart=no — or a reboot silently re-activates it into a dual-primary (a stopped restart: always container comes back on daemon start).

Incident response & DR: sc-infrastructure/docs/guides/ (operational-runbook, disaster-recovery-plan, troubleshooting). Every production incident gets a blameless write-up.

11. Infrastructure footprint & cost​

Current state as of 2026-08-11. Figures reconcile with sc-infrastructure/docs/migration/hetzner-cutover-execution.md.

Compute​

HostRoleSpecRunsCost/mo
sc-prod (Hetzner)Production primaryCPX32 — x86, 4 vCPU / 8 GB / 160 GB + 4 GB swap, Ubuntu 24.04, FalkensteinThe 13 production containers below~$51 all-in (incl. auto-backups + IPv4)
RAPTORWarm standby / rollbackLocal Windows, 32 GB, on TailscaleStack stopped-but-intact (--restart=no); postgres/redis/traefik/cloudflared kept for rollback + the * wildcard tunnel$0 (owned HW)
BEASTCold archiveLocal PCZero service load; DR-pull target$0 (owned HW)

sc-prod: public 138.201.191.170, Tailscale 100.110.87.50, SSH sc-hetzner (user sc). Only :22 is public (UFW; Traefik binds 127.0.0.1, ingress is Cloudflare-Tunnel only). Vertical scale path: in-place resize CPX32 → CPX42 (16 GB) if memory pressure appears — no rebuild.

Production services (13 containers on sc-prod)​

postgres · redis · traefik · cloudflared · token_service · ghl_rest_api · symphony_flow_api · symphony_flow_worker · symphony_flow_scheduled_worker · wms_api · uptime_kuma · uptime_kuma_themed · gatus.

Deferred / offline (not on sc-prod): symphony-platform (screenshot/evidence/report/ intelligence + MinIO), Community Plane, sc-finance-sync/Firefly, website-testing-automation, pgAdmin, sc-glossary-app.

Supporting services​

ServiceFunctionCost/mo
Cloudflare (Tunnel + Access + DNS)Sole ingress; Access-gates status/wms€0 (free tier)
Google Drive (shared drive)Off-host backup target (rclone)€0 (existing Workspace)
TailscalePrivate host↔host link (DR-pull, ops)€0 (existing)
1PasswordSecret source of truth (op inject)Existing (team)
Hetzner auto-backupsWhole-box snapshotsincluded in the ~$51
UptimeRobotExternal watchdog€0 (free)
DataForSEO · Twilio · GHL · StripeRank data · SMS/voice · CRM · paymentsTBD — fill in

Cost in context​

Line itemMonthly
Hetzner sc-prod (all-in)~$51
Cloudflare / Drive / Tailscale / standby hosts€0 (free/owned)
Total production infrastructure~$51
— for comparison — Claude/AI usage (1 user, measured)~$1,127 API-equivalent (flat Max plan)

Production infrastructure is ~4.5% of the AI compute line — infrastructure is not the cost driver, so decisions favor reliability and simplicity over shaving infra dollars.

Cost governance​

  • Quarterly footprint-&-cost review: reconcile this section against the live Hetzner invoice + the service registry; confirm deferred services are still deferred; fill the TBD third-party costs; decide any resize.
  • Budget alert: set a Hetzner spend threshold with email notification.
  • Scaling decisions are recorded here (the CPX32→CPX42 path) so the footprint stays legible.

12. Adding a new internal application​

  1. Scaffold from the software-dev-project-template (CI, lint/format, PR template, branch rules baked in — see symphony-core-documents/11-engineering/templates/).
  2. Dockerfile per §7; health endpoint per health-check-standards.md; .env.1password manifest + 1Password item in sc-Infrastructure-DR per §5.
  3. Add the service to sc-infrastructure/docker-compose.yml (build context ../<repo>, symphonycore_ name, symphonycore_network) and Traefik labels for its hostname; add it to the _prod overlay if it needs a public route.
  4. Register it in sc-infrastructure/docs/reference/cross-repo-service-registry.md.
  5. Add Gatus/Kuma checks; pass the production-gate-checklist.md before it serves clients.

13. Governance & fix-forward register​

Owner: Systems Team. Review cadence: quarterly (or on any major topology change, e.g. a new host or a registry adoption). Each in-scope app repo links this doc from its CLAUDE.md as the delivery SoT (replacing the per-repo ai-assisted-agile-process.md boilerplate over time).

Deviations from this standard that exist today, to be closed forward:

#GapReposPriorityStatus
1Trunk is master, not mainsymphony-flow, website-monitoring-system, ghl-data-syncMedOpen
2No CI workflow despite a full test suiteghl-data-syncHighResolved 2026-08-11 — added .github/workflows/ci.yml gating build (tsc) + lint (eslint), both green. Tests deferred (see #12).
3CI triggers on main/develop but repo is master → CI never runssymphony-flowHighResolved 2026-08-11 — added master to push/PR triggers (kept main/develop for forward-compat).
4Coverage gate inconsistent (0 / 50 / 80) → set ≥70% baselineghl-data-sync (0), symphony-flow (50/60)MedOpen — ghl-data-sync blocked by #12
5Image names unprefixedghl-data-sync (ghl-rest-api, ghl-token-service)LowOpen
6GIT_SHA/BUILD_TIME provenance only on one repoall except symphony-flowLowOpen
7pre-commit referenced but never wiredallMedOpen
8PR template not in .github/ (not auto-loaded) / missingsymphony-flow, WMS (templates/); sc-infrastructure (none)LowOpen
9Versioning: static 0.x/[Unreleased] only, no tagssymphony-flow, ghl-data-sync, WMS (sprint-based)LowOpen
10Staging environment not yet isolated (mock/sandbox + drill mode)RAPTOR / platformMedOpen
11Per-repo ai-assisted-agile-process.md boilerplate → replace with a link here~10 reposLowOpen
12Test suite not hermetic → not CI-gateable: suites connect to a hardcoded private-LAN Postgres (192.168.68.74) and assert on specific seeded prod rows (e.g. tests/business-rules/evaluate-client-location.test.ts); others block on live network. Needs a test DB + fixtures/seeding + mocked GHL network before npm test can gate CI or a coverage baseline (#4) applies.ghl-data-syncMedOpen — tracked as ghl-data-sync#55 (full evidence + approach there); discovered 2026-08-11 while closing #2
138 pre-existing unit-test failures now visible once CI fires (was masked while CI never ran): payment-confirmed-fanout (skeleton load count), list_locations validateParams/execute assertions, slack-interaction-handler (2 timeouts).symphony-flowMedResolved 2026-08-11 — triaged: all 8 were stale tests / a test-isolation gap, not code bugs (verified vs source + the real ghl-data-sync API contract). Tests aligned; full unit suite 1045/1045 green. No source changed.
14integration-tests CI job hung (>25 min, no timeout-minutes guard) → risk of consuming GitHub's 6h default.symphony-flowMedResolved 2026-08-11 — root cause: integration/e2e/coverage suites hold Postgres/Redis (BullMQ) handles open, so jest never exits. Added --forceExit on those suites + per-job timeout-minutes (lint 10 / unit 15 / integration 20 / e2e 15 / coverage 25). Verified: integration now completes in ~4 min instead of hanging. Root-cause afterAll teardown is a follow-up.
15client-onboarding integration tests pass inputs missing the now-required company_slug field → Missing required field: company_slug. Config/test drift.symphony-flowMedPartially resolved 2026-08-11 — added company_slug to the 22 proceed-path inputs (validation-reject inputs left as-is). This clears the company_slug layer but the suite is still red on deeper drift → see #17. (payment-confirmed-fanout/client-profile-populate had no company_slug issue — verified.)
16website-qa integration tests hit a real Postgres but the CI integration-tests job never provisioned the schema, so tables were missing → 9 failures (relation "website_test_runs" does not exist).symphony-flowMedResolved 2026-08-11 (CI-verified) — added a Provision database schema + run migrations step to the integration + coverage jobs: schema.sql (base tables — workflow_runs/steps that migrations reference but don't create) then db:migrate (feature tables). website-qa suites now green.
17client-onboarding integration suite (22 of ~23 residual integration failures) fails at the validate_not_duplicate step. Root cause (investigated 2026-08-13): the step's drive_folder_not_exists / clickup_folder_not_exists checks (added in executor 04b61a2) hard-throw when they can't resolve a real folder/space id. Those ids come from tier config (config/tiers/*.yaml) whose values are env templates ({{ env.DRIVE_CLIENTS_FOLDER_ID }}, {{ env.CLICKUP_CLIENT_SPACE_ID }}) that are unset in the mock test env; the drive check also needs a google_workspace adapter the suite doesn't register. Before 04b61a2 the step didn't exist, so the mock tests passed. (Adding parent_folder_id to the YAML rule does NOT fix it — the resolved value is still an unset env var.) A separate downstream sitemap_audit.run custom handler (4d6b756) is likely also unregistered in this suite. Decision needed: make the checks tolerant of unresolvable config (skip vs. throw) — a production behavior choice — OR provide fake env + a google_workspace mock in the test (test-only). Plus a handful of unrelated failures in payment-confirmed-fanout/client-profile-populate/others (1–3 each) not yet individually triaged.symphony-flowMedResolved 2026-08-19 (CI-verified) — owner decision: option C (test-only fix; executor stays strict — no production behavior change), decision of record in symphony-flow/docs/sprints/2026-08-issue-52-integration-green-sprint.md. The drive_folder_not_exists rule got its ${DRIVE_CLIENTS_FOLDER_ID} env wiring (the config the decision presumes; client_drive_setup.yaml pattern), the suite supplies fake env + google_workspace/clickup mocks + a sitemap_audit.run stub (the suspected unregistered handler was real), and gained duplicate-detected rejection coverage. Residuals triaged: payment-confirmed-fanout (stale Slack action shape) and client-profile-populate (stale hardcoded ClickUp folder id) fixed in-sprint; the other suites were already green. Coverage thresholds rebaselined to first-green actuals as ratchet floors. Integration + Coverage jobs green on master (run 32229461059). Close-out on symphony-flow#52; symphony-flow 824b073, 1fba0d2.

Changelog​

VersionDateChange
1.22026-08-19Register #17 resolved: symphony-flow integration suite greened via option C (test-only fix, executor stays strict) per symphony-flow#52; Integration + Coverage CI jobs green on master for the first time (coverage thresholds rebaselined to first-green actuals as ratchet floors).
1.12026-08-11Closed the two High-priority CI gaps in the fix-forward register: added build+lint CI to ghl-data-sync (#2) and fixed symphony-flow's CI trigger to include master (#3); greened symphony-flow's lint gate (132 pre-existing errors fixed, behavior-preserving). Recorded newly-discovered gaps: #12 (ghl-data-sync test suite non-hermetic) and #13 (symphony-flow 8 pre-existing unit-test failures now visible).
1.02026-08-11Initial standard. Establishes the Hetzner + Docker-Compose delivery pipeline as canonical; supersedes the GCP/Cloud-Run pipeline in sc-software-dev-guide.md; records the current infrastructure footprint & cost and the cross-repo fix-forward register.