Pearl Platform — Deployment & Operations Manual

01

Architecture Overview

How all the pieces fit together.

Key Concepts

Environment Metadata: JSON artifact produced by create-environment that describes all infrastructure. All deploy workflows consume it.

Azure Run Command v2: Remote PowerShell execution on VMs without SSH/RDP. Scripts are uploaded to blob storage and executed via Azure APIs.

Host-Header Routing: IIS routes requests to the correct test environment based on the Host HTTP header. No port conflicts.

02

Workflow Inventory

All GitHub Actions workflows and when to use them.

Workflow	Trigger	Purpose
`create-environment.yaml`	Manual	Provision/destroy infrastructure via Terraform
`deploy-production.yaml`	Manual	Build master + deploy to production VMs
`deploy-test.yaml`	Push to feature/fix/env branches	Auto-build + deploy isolated test environment
`cleanup-test-env.yaml`	Branch delete / Daily / Manual	Remove stale test environments
`build.yaml`	Manual	Standalone build (no deploy)
`deploy.yaml`	Manual	Deploy specific version or rollback
`windows-build.yml`	Manual	Legacy validation build

03

Create Environment

Provisioning infrastructure for the first time.

⚠️ Prerequisites Required

Before running create-environment on a new subscription, you must complete all manual setup steps (OIDC, Key Vault, storage accounts, blobs, SQLMI, GitHub variables). See the New Subscription Setup Guide for the complete checklist.

1

Actions → Create Environment → Run workflow. Select target_environment: test, mode: plan

2

Review the Terraform plan. Check what resources will be created (VMs, VNets, firewall, storage, etc.)

3

Run again with mode: apply. Terraform provisions all infrastructure (~15 minutes)

4

Metadata published. environment-metadata.json artifact is uploaded for deploy workflows to consume

What gets created

Hub VNet (Firewall + Bastion) • Spoke VNet (Web, Worker, Build subnets) • NAT Gateway • NSGs • Azure Firewall • Bastion Host • Web VM (D4s_v5) • Worker VM (D2s_v5) • Key Vault • Storage Account • VNet Peering to SQLMI

04

Destroy Environment

Tearing down infrastructure completely.

1

Run with mode: destroy-plan. Preview what will be removed

2

Run with mode: destroy. All VMs, networking, storage, and secrets are destroyed

⚠️ Warning

Destroy removes ALL resources. All test environments are lost. The firewall public IP changes on rebuild. You'll need to run apply again and re-push feature branches to recreate environments.

05

Build Process

How source code becomes deployable artifacts.

flowchart LR A[Checkout Code] --> B[Download Vendor DLLs
Telerik + Ajax] B --> C[MSBuild Web Tier
ASP.NET 4.x] C --> D[Build Worker Tier] D --> E[Package ZIPs
+ Release Manifest] E --> F[Upload Artifacts]

Vendor Dependencies

Proprietary DLLs stored encrypted in Azure Blob Storage. The build downloads, verifies SHA256, decrypts with Key Vault password, and passes to MSBuild.

Bundle	Key Vault Secret	Contents
Telerik	`TelerikDLL-PassKey`	Telerik.Web.UI.dll
Ajax	`AjaxDLL-PassKey`	AjaxControlToolkit.dll

Version Naming

Context	Format	Example
Production	`prod-{run}.{attempt}-{sha7}`	prod-15.1-abc1234
Test	`test-{run}.{attempt}-{sha7}`	test-42.1-def5678
Standalone	`gha-{run}.{attempt}-{sha7}`	gha-8.1-abc1234

06

Deploy Test Environment

Automatic deployment on every feature branch push.

Trigger

Push to any feature/* or env/* branch.

1

Derive env name. feature/auth-refactor → auth-refactor

2

Build on Windows runner. Full MSBuild from the branch code

3

Stage to Azure Storage. Upload ZIP + scripts, generate SAS URLs

4

Deploy via Run Command. Extracts files, creates IIS site, binds host header

5

Post URL on PR. Comments environment URL on associated pull request

What happens on the VM

Creates directory F:\envs\{env}\pearl-azure\ • Extracts ZIP • Creates IIS app pool PearlAppPool-{env} • Creates website pearl-{env} with binding *:80:{env}.pearl.test • Applies web.config • Health check

Branch → Environment Mapping

Branch	Environment Name
`feature/auth-refactor`	auth-refactor
`fix/login-bug`	login-bug
`env/staging`	staging
`feature/very-long-name-here`	very-long-name-here (max 20 chars)

Build failure = no deploy

The deploy job depends on the build job (needs: [build]). If build fails, deploy is automatically skipped.

07

Deploy Production

Manual deployment to production VMs.

Trigger

Manual dispatch only: Actions → Deploy Production → Run workflow

1

Ensure infrastructure. Terraform plan + apply (idempotent). Skip with skip_infrastructure: true

2

Build from master. Full MSBuild producing web + worker + manifest

3

Deploy web to web VM. Via Azure Run Command v2

4

Deploy worker to worker VM. Via Azure Run Command v2

5

Record history. Updates deployment-history.json in blob storage

Key differences from test

• Deploys BOTH web + worker (test deploys web only)
• Runs Terraform infrastructure check first
• Records deployment history for rollback support
• No auto-trigger — manual dispatch only
• Concurrency: only one production deploy at a time (no cancellation)

Inputs

Input	Default	Description
`skip_infrastructure`	false	Skip Terraform check (assume VM exists)
`build_run_id`	""	Deploy from specific build run; empty = fresh build

08

Rollback

Reverting to a previous known-good deployment.

1

Actions → Deploy Release → Run workflow. Set mode: rollback

2

Workflow loads rollback reference. Identifies previous successful deployment from history

3

Validates prior artifacts exist. Confirms ZIPs are still in blob storage

4

Deploys previous version. Both web and worker VMs receive the prior release

5

Validates rollback. Health checks confirm the rolled-back version is running

09

Accessing Test Environments

How to reach deployed test environments in your browser.

1

Allowlist your IP. Add to firewall_allowed_source_ips in terraform.tfvars, run terraform apply

2

Get firewall IP. terraform output firewall_public_ip or az network public-ip show -n pip-pearl-test-firewall -g rg-pearl-test-hub --query ipAddress -o tsv

3

Configure DNS. Add hosts file entry: <firewall-ip> {env}.pearl.test
Or wildcard: *.pearl.test → firewall-ip (one-time setup)

4

Browse. http://{env}.pearl.test/pearl-azure/login.aspx

Helper Script

Run ./scripts/show-test-environments.sh to see all active environments, their URLs, and the hosts file entries you need.

Hosts file locations

Windows: C:\Windows\System32\drivers\etc\hosts (edit as Admin)
macOS / Linux: /etc/hosts (edit with sudo)

10

Cleanup & Maintenance

How test environments are removed when no longer needed.

Automatic (Branch Delete)

When a branch is deleted (e.g., after PR merge), the cleanup workflow triggers automatically and removes the IIS site, app pool, and files.

Scheduled (Daily at 03:00 UTC)

Lists all pearl-* IIS sites on the VM, compares against active branches via GitHub API, and removes any orphaned environments.

Manual

Actions → Cleanup Test Environment → Run workflow → enter the environment name (e.g., auth-refactor)

Safety

• Never touches sites without pearl- prefix
• Never touches Default Web Site
• Idempotent: returns "not-found" if env already removed
• Skips gracefully if VM is stopped

11

Manual Operations

Common operational tasks and how to perform them.

Build without deploying

Actions → Build Deliverables → Run workflow. Select branch and environment.

Deploy a specific version

Actions → Deploy Release → mode: deploy. Provide artifact_version or build_run_id.

Check current state

# List all test environments
./scripts/show-test-environments.sh

# Get firewall IP
az network public-ip show -n pip-pearl-test-firewall -g rg-pearl-test-hub --query ipAddress -o tsv

# Check VM status
az vm get-instance-view --name vm-pearl-test-web --resource-group rg-pearl-test-spoke \
  --query "instanceView.statuses[?starts_with(code,'PowerState/')].displayStatus" -o tsv

12

Troubleshooting

Common issues and their solutions.

Build Failures

Symptom	Fix
"Missing required configuration values"	Add missing variable in GitHub Settings → Environments
"SHA256 mismatch"	Update `CI_BUNDLE_SHA256` variable after re-uploading bundle
MSBuild errors	Fix the code on the branch

Deployment Failures

Symptom	Fix
"Test VM not found"	Run create-environment with mode=apply
Run Command timeout	RDP to VM via Bastion, check IIS Manager
"No create-environment run found"	Run create-environment first (generates metadata artifact)

Access Issues

Symptom	Fix
Connection timeout	Check IP allowlist + DNS resolves to correct IP
404 / wrong env	Verify host header matches; check deploy-test succeeded
Firewall IP changed	After rebuild: `terraform output firewall_public_ip`

13

Secrets & Configuration

Required GitHub Actions variables and secrets.

Variables (Repository Level)

Variable	Purpose
`AZURE_CLIENT_ID`	SP / MI client ID for OIDC
`AZURE_TENANT_ID`	Azure AD tenant
`AZURE_SUBSCRIPTION_ID`	Target subscription
`CI_BUNDLE_STORAGE_ACCOUNT`	Storage with vendor DLLs
`CI_BUNDLE_KEYVAULT_NAME`	KV with decrypt passwords
`CI_BUNDLE_SHA256_TELERIK`	Integrity hash for Telerik
`CI_BUNDLE_SHA256_AJAX`	Integrity hash for Ajax
`TF_BACKEND_RESOURCE_GROUP_NAME`	Terraform state RG
`TF_BACKEND_STORAGE_ACCOUNT_NAME`	Terraform state storage
`TF_BACKEND_CONTAINER_NAME`	Terraform state container

Secrets

Secret	Purpose
`TF_VAR_SQL_ADMIN_PASSWORD`	SQLMI admin password
`TF_VAR_VM_ADMIN_PASSWORD`	VM admin password

14

Script Reference

All deployment and build scripts.

VM Deployment Scripts

Script	Purpose
`Deploy-TestEnvironment.ps1`	Create/update test IIS env
`Cleanup-TestEnvironment.ps1`	Destroy test IIS env
`Deploy-Production.ps1`	Production deploy (wraps Deploy-WebArtifact)
`Deploy-WebArtifact.ps1`	Core web deploy logic
`Invoke-RemoteArtifactDeployment.ps1`	Run Command entry point (routes to deploy script)
`DeploymentHelpers.psm1`	Shared utility module (28 functions)

CI Build Scripts

Script	Purpose
`Prepare-CiBuildDependencies.ps1`	Download + decrypt vendor DLLs
`Invoke-CiWebBuild.ps1`	MSBuild web tier
`Invoke-CiWorkerBuild.ps1`	Build worker tier
`Package-CiArtifacts.ps1`	ZIP + manifest
`generate_environment_metadata.py`	Terraform → metadata JSON
`show-test-environments.sh`	List active envs + URLs

15

Decision Tree

What workflow to use for each situation.

flowchart TD Q{What do you need?} Q -->|"Set up infra
(first time)"| A[Create Environment
mode: apply] Q -->|"Deploy feature
for testing"| B[Push to
feature/* branch] Q -->|"Deploy to
production"| C[Deploy Production
manual dispatch] Q -->|"Rollback
production"| D[Deploy Release
mode: rollback] Q -->|"Deploy specific
version"| E[Deploy Release
mode: deploy] Q -->|"Clean up
test env"| F[Delete branch
or manual dispatch] Q -->|"Just build
no deploy"| G[Build Deliverables
manual dispatch] Q -->|"Destroy
everything"| H[Create Environment
mode: destroy] Q -->|"See active
environments"| I["./scripts/
show-test-environments.sh"]