Pearl Platform — Multi-Environment Capability

01

Architecture — The 2-VM Model

A single production VM for stable deployments and a single shared test VM that hosts all feature-branch environments simultaneously.

🚀

Production VM (Dedicated)

Trigger: Manual dispatch only (workflow_dispatch)
Approval: Required (GitHub Environment gate)
IIS: Single Default Web Site, all apps
Database: Production databases
Destroy: Never — permanent

🧪

Shared Test VM (Multi-Env)

Trigger: Push to feature/*, env/*
Approval: Not required (auto-deploy)
IIS: One site per branch environment
Database: Shared test databases
Destroy: Auto on branch delete or daily orphan scan

Branch → Environment mapping

master → Production VM (dedicated, approval-gated, auto-creates VM on first merge)
test → Infrastructure owner for shared test VM (create/destroy via Terraform)
feature/auth-refactor → IIS site "auth-refactor" on shared VM (cannot touch VM itself)
feature/billing-fix → IIS site "billing-fix" on shared VM (cannot touch VM itself)
fix/report-export → IIS site "report-export" on shared VM (cannot touch VM itself)

Infrastructure Ownership — Who controls what

test branch — ONLY branch that can create/destroy the shared test VM (Terraform)
master branch — ONLY branch that can create the production VM (auto-bootstrap, no destroy)
feature/* branches — Can ONLY deploy/remove IIS sites. Cannot touch VMs, Terraform, or other environments.

Feature branches ride ON infrastructure — they don't manage it.

02

How It Works — Triggers & Developer Flow

No new VMs are created. The shared test VM is always running. A test "environment" is just an IIS site + folder on disk — created automatically when you push.

When does a test environment get created?

❌ Creating a branch does NOT trigger anything.
❌ No new VM is provisioned.
✅ Trigger: PUSHING CODE to a feature/* or env/* branch.

When you git push origin feature/my-thing:
1. GitHub detects a push to a branch matching feature/**
2. deploy-test.yaml fires automatically
3. Code is built, then an Azure Run Command creates an IIS site on the existing shared test VM
4. Subsequent pushes to the same branch update the existing environment (not duplicate)

The "environment" is just: a folder (F:\envs\my-thing\), an IIS website, and an app pool. That's it.

Event	Branch Pattern	Workflow	Result
Push code	`feature/`, `env/`	`deploy-test.yaml`	Build → Deploy IIS site on shared test VM
Push again	Same branch	`deploy-test.yaml`	Build → Update existing site (idempotent)
Manual dispatch	`master`	`deploy-production.yaml`	Ensure infra → Build → Approval → Deploy to production VM
Push code	`test`	No deploy workflow	Safe working branch — infrastructure owner for test VM
Manual dispatch	`test`	`create-environment.yaml`	Terraform creates/ensures shared test VM
Delete branch	`feature/`, `env/`	`cleanup-test-env.yaml`	Remove IIS site + files (NOT the VM itself)
Daily 3am	—	`cleanup-test-env.yaml`	Remove orphan sites (branch no longer exists)

What feature branches CANNOT do

❌ Cannot create or destroy the shared test VM
❌ Cannot affect other branches' IIS sites
❌ Cannot modify infrastructure (Terraform)
✅ Can only deploy/update/remove their OWN IIS site on the existing VM

Developer Workflow Steps

1

Push to a feature branch. git push origin feature/auth-refactor — this is the ONLY trigger. GitHub Actions builds and deploys automatically.

2

Access your test environment. Browse to http://auth-refactor.pearl.test/pearl-azure — your branch is running in isolation from other branches.

3

Push more commits. Each push automatically updates your environment — same workflow fires, site is updated in place.

4

Create a PR and get approval. Reviewers can access your test environment to verify changes visually.

5

Merge to master. Production is deployed via manual dispatch when ready. Deleting the feature branch triggers automatic cleanup.

03

Production VM — Auto-Bootstrap & Protected

The production VM is managed via manual dispatch of deploy-production.yaml. The workflow includes an ensure-infrastructure job (Terraform, idempotent — first time creates the VM, subsequent times skips). Deployment requires manual approval.

Aspect	Production Configuration
Deploy trigger	Manual dispatch only (`workflow_dispatch`)
Infrastructure	Auto-created by `ensure-infrastructure` job (Terraform idempotent — first time creates, subsequent times skips)
Approval	GitHub Environment protection rule — manual approval required before deploy
IIS Structure	Default Web Site with standard app structure at `F:\apps\`
Worker Services	Full set: QueueProcessor, SystemChecker, AISpooler, Totem
Database	Production databases (PearlData, PearlQueues, PearlUsers, PearlBilling)
Deploy method	Azure Run Commands — same proven pattern as current deploy

Self-bootstrapping production

First dispatch: Terraform creates the production VM (~10-15 min extra) → build → approval → deploy
Every subsequent dispatch: Terraform sees "no changes" (~30s) → build → approval → deploy

No manual infrastructure provisioning. Just dispatch deploy-production.yaml and the ensure-infrastructure job handles it.

Safety

Production cannot be deployed to without manual dispatch and approval. There is no destroy workflow for production. Only manual dispatch triggers the production pipeline — feature/fix/env branches go to the test VM automatically.

04

Shared Test VM — Multiple Environments

One VM hosts all test environments simultaneously. Each feature branch gets its own IIS site, app pool, and file directory — but they share the same VM resources and test databases.

📂

File Isolation

Each environment deploys to its own directory:
F:\envs\auth-refactor\pearl-azure\
F:\envs\billing-fix\pearl-azure\
F:\envs\report-export\pearl-azure\

🌐

IIS Isolation

Each environment gets its own IIS site + app pool:
Site: pearl-auth-refactor
Pool: PearlAppPool-auth-refactor
Host: auth-refactor.pearl.test

⚙️

Worker Services (Shared)

One set of worker services processes from the shared test database. Workers don't need per-branch isolation since the database is shared anyway.

Capacity

The shared VM can comfortably host 5-10 simultaneous environments. Each environment only consumes disk space for its deployed files (~200-400MB per env). IIS app pool isolation prevents one environment's crash from affecting others.

05

IIS Multi-Site Structure

Each test environment is a separate IIS website with host-header binding. The host header routes incoming requests to the correct environment without port conflicts.

IIS Site Name	Host Header	Physical Path	App Pool
`pearl-auth-refactor`	`auth-refactor.pearl.test`	`F:\envs\auth-refactor\`	`PearlAppPool-auth-refactor`
`pearl-billing-fix`	`billing-fix.pearl.test`	`F:\envs\billing-fix\`	`PearlAppPool-billing-fix`
`pearl-report-export`	`report-export.pearl.test`	`F:\envs\report-export\`	`PearlAppPool-report-export`

Why host-header binding

All environments listen on port 80. IIS uses the Host header to route to the correct site. No port conflicts. Developers just need the correct hostname in their browser (or hosts file).

06

Database Strategy — Shared

Simple and fast: one database set for production, one shared set for all test environments.

Database Set	Used By	Connection Strings
Production databases	Production VM only	Production web.config → PearlData, PearlQueues, etc.
Test databases (shared)	ALL test environments	All test web.configs → PearlData_test, PearlQueues_test, etc.

Trade-offs

Pro: Zero spin-up delay for databases. No per-env restore needed. Simple to manage.
Con: Test environments can see each other's data changes. One developer's queue job might be picked up by another's test. This is acceptable for feature testing.

07

CI/CD Workflow Design

Three new workflows handle the full lifecycle: test deploy, production deploy, and cleanup.

🧪

deploy-test.yaml

Trigger: Push to feature/*, env/*
Action: Build → Stage to blob → Run Command → Invoke-RemoteArtifactDeployment.ps1 → Deploy-TestEnvironment.ps1
Approval: None (auto-deploy)
Pattern: Same as deploy.yaml (SAS URLs, blob staging, output blob parsing)

🚀

deploy-production.yaml

Trigger: Manual dispatch only (workflow_dispatch)
Action: Ensure infra → Build → Approval → Stage to blob → Run Command → Deploy to web + worker VMs
Approval: Required (GitHub Environment gate)
Pattern: Full pipeline with optional infrastructure refresh

🧹

cleanup-test-env.yaml

Trigger: Branch delete + daily schedule
Action: Run Command → Cleanup-TestEnvironment.ps1 (direct, no artifact download)
Approval: None (automatic)
Speed: ~30 seconds

08

Access And Routing

How developers reach each test environment in their browser.

1

Request enters Azure Firewall. Developer browses to http://auth-refactor.pearl.test/pearl-azure

2

Firewall DNAT forwards to test VM. Port 80 traffic → shared test VM private IP (10.2.1.10)

3

IIS routes by Host header. The Host: auth-refactor.pearl.test header matches the correct IIS site binding.

4

Correct environment serves the response. IIS site pearl-auth-refactor serves content from F:\envs\auth-refactor\

Developer Setup

Step 1: Allowlist Your IP

The firewall blocks all traffic by default. Add your IP to firewall_allowed_source_ips in
src/terraform/environments/test/terraform.tfvars, then run terraform apply.

See _internal_docs/Managing-External-Web-Access.md for full instructions.

Step 2: Get the Firewall Public IP

terraform output firewall_public_ip
or: az network public-ip show -n pip-pearl-test-firewall -g rg-pearl-test-hub --query ipAddress -o tsv

Note: The IP changes after each environment destroy/rebuild cycle.

Step 3: Configure DNS (choose one)

Option A — Hosts file (per environment):

Windows	`C:\Windows\System32\drivers\etc\hosts` (edit as Admin)
macOS / Linux	`/etc/hosts` (edit with sudo)

Add entries:
<firewall-ip> auth-refactor.pearl.test
<firewall-ip> billing-fix.pearl.test

Option B — Wildcard DNS (recommended, one-time):
*.pearl.test → <firewall-ip>
Configure via dnsmasq, corporate DNS, or local DNS resolver. Covers all environments automatically.

Step 4: Access Your Environment

http://{env-name}.pearl.test/pearl-azure/login.aspx

Helper Script

List all active environments

Run locally (requires az CLI):
./scripts/show-test-environments.sh

Shows: firewall IP, all deployed environments, ready-to-use URLs, and hosts file entries.

Troubleshooting

Can't reach the site (timeout)?

1. Check your IP is allowlisted: curl -s ifconfig.me
2. Verify firewall IP is correct (may have changed after rebuild)
3. Confirm DNS resolves: ping {env-name}.pearl.test should show the firewall IP
4. Verify VM is running in Azure Portal

Wrong env or 404?

• Ensure a deploy-test workflow succeeded for your branch
• Branch must use feature/, fix/, or env/ prefix
• Host header must match IIS binding exactly

09

Lifecycle Flow Diagrams

Visual representation of the full create → deploy → destroy lifecycle.

End-to-End Flow

flowchart TD subgraph REPO["Repository"] MASTER[master branch] FEAT1[feature/auth-refactor] FEAT2[feature/billing-fix] end subgraph CICD["GitHub Actions"] MASTER -->|manual dispatch| PROD_WF[deploy-production.yaml] FEAT1 -->|push| TEST_WF[deploy-test.yaml] FEAT2 -->|push| TEST_WF FEAT1 -->|branch deleted| CLEAN[cleanup-test-env.yaml] end subgraph PROD["Production VM - Dedicated"] PROD_WF -->|approval required| PROD_RC[Azure Run Command] PROD_RC --> PROD_IIS["IIS Default Web Site
F:\\apps\\pearl-azure"] PROD_IIS --> PROD_DB[("Production Databases")] end subgraph TEST["Shared Test VM - Multi-Environment"] TEST_WF -->|auto-deploy| TEST_RC[Azure Run Command] TEST_RC --> ENV1["IIS: pearl-auth-refactor
Host: auth-refactor.pearl.test
F:\\envs\\auth-refactor\\"] TEST_RC --> ENV2["IIS: pearl-billing-fix
Host: billing-fix.pearl.test
F:\\envs\\billing-fix\\"] ENV1 --> TEST_DB[("Shared Test Databases")] ENV2 --> TEST_DB CLEAN -->|remove| ENV1 end

CI/CD Sequence

sequenceDiagram participant Dev as Developer participant GH as GitHub Actions participant AZ as Azure Run Commands participant VM as Shared Test VM Note over Dev,VM: Create Test Environment Dev->>GH: Push to feature/auth-refactor GH->>GH: Build on windows-2022 GH->>AZ: Run Command on test VM AZ->>VM: Deploy-TestEnvironment.ps1 -EnvName auth-refactor VM->>VM: Create F:\envs\auth-refactor\ VM->>VM: Create IIS site + app pool VM->>VM: Extract artifacts + inject config VM-->>GH: Validation results JSON GH->>Dev: Environment ready at auth-refactor.pearl.test Note over Dev,VM: Deploy to Production Dev->>GH: Merge PR to master GH->>Dev: Approval request Dev->>GH: Approve GH->>AZ: Run Command on production VM AZ->>VM: Deploy-Production.ps1 GH->>Dev: Production updated Note over Dev,VM: Cleanup Dev->>GH: Delete feature/auth-refactor GH->>AZ: Run Command on test VM AZ->>VM: Cleanup-TestEnvironment.ps1 -EnvName auth-refactor VM->>VM: Remove IIS site + delete files GH->>Dev: Environment cleaned up

10

Implementation Roadmap

Total estimated effort: 5-7 working days for full capability.

Phase	Effort	Deliverable
1. Deploy scripts	1-2 days	`Deploy-TestEnvironment.ps1` + `Cleanup-TestEnvironment.ps1`
2. Production workflow	1 day	`deploy-production.yaml` with approval gate
3. Test workflow	1-2 days	`deploy-test.yaml` triggered on feature branches
4. Cleanup workflow	0.5-1 day	`cleanup-test-env.yaml` on branch delete + schedule
5. Access setup	0.5 day	DNS/hosts configuration + documentation
6. Demonstrate	1 day	2 feature branches running simultaneously, one merged

Cost summary

Production VM (new): ~£110-220/month
Shared test VM: £0 additional (already exists)
Test environments: £0 (same VM, just disk space)
Total new cost: ~£110-220/month (production VM only)

Safety guardrails

• Production requires manual approval — cannot auto-deploy
• Production has no destroy workflow
• Test environments cleaned up on branch delete + daily orphan scan at 03:00 UTC
• Maximum ~10 concurrent test environments (soft limit)
• Disk space monitoring alerts at 80% usage

Built on existing proven patterns

This multi-environment capability does NOT reinvent deployment. It reuses:
• Invoke-RemoteArtifactDeployment.ps1 — Same Run Command entry point (downloads scripts + artifacts from blob)
• DeploymentHelpers.psm1 — Reuses Sync-DirectoryContents, Grant-IisReadAccess, Assert-DirectoryMatchesSource
• Apply-EnvironmentWebConfig.ps1 — Same connection string injection (environment-metadata.json + Key Vault)
• deploy.yaml pattern — Same SAS URL generation, blob staging, VM readiness polling, output blob parsing
• build.yaml — Same build process (windows-2022, .zop deps, MSBuild + aspnet_compiler)

Only the IIS topology changes: test environments use separate IIS Websites with host-header bindings instead of Applications under Default Web Site.

This multi-environment capability builds on the existing proven CI/CD system (Azure Run Commands, GitHub-hosted runners, structured validation). The implementation extends — not replaces — the current working deployment pattern.