Central Build, Bootstrap, Deployment, And Validation Guide
This presentation site is now the practical documentation hub for rebuilding and validating the Pearl test environment. It is aligned to the updated repository runbooks, the tested Terraform code, the observed VM bootstrap behavior, and the real validation flow used against the web, worker, and build virtual machines.
The old presentation was mainly a proposal. The repository documentation is now stronger because it was updated after actual Terraform work, VM extension retries, bootstrap verification, and manual validation against the live test VMs.
The updated repository runbooks are the engineering source of truth. This website is the easiest place to understand those runbooks, follow them in the right order, and navigate to the correct validation path without guesswork.
You do not need to guess which markdown file to open first. This site shows the right route.
The content now reflects the tested Terraform environment, not only the original design proposal.
The notes now match what build, web, and worker bootstrap actually prepare on the VMs.
The runtime validation order is based on what actually worked during manual testing of the environment.
The easiest way to avoid confusion is to start with the path that matches your job today. This page is the traffic controller.
Start with the Central Runbook. It explains what each phase proves, why the order matters, and what each VM is for.
Open guide.html firstUse the Terraform Build Guide for variables, installer storage, apply behavior, bootstrap expectations, and re-runs.
Open terraform-guide.htmlJump to the VM Validation Map below to identify the right detailed guide for web, worker, build, or end-to-end checks.
Runbook first, detailed reference secondUse the Reuse Pattern section to separate what is Pearl-specific from what can be applied to a new project.
See Reuse PatternIf your immediate goal is proving the application environment works, you do not need to wait for the build VM first. Build the artifacts on a trusted Windows machine, deploy web first, then worker, then run the shared database and functional checks. Validate the build VM afterward as the repeatable build host.
This is the recommended sequence for the current repository state. Terraform gives you a strong baseline, but runtime proof still depends on built application content and structured manual checks.
| Stage | What you are proving | Why it matters |
|---|---|---|
| Terraform + Bootstrap | The platform exists: network, storage, SQL MI, VM structure, and baseline guest setup. | This is the foundation, but not full application proof. |
| Web Deployment | The web VM has real built content in the expected F:\apps folders. |
IIS checks mean very little if the folders are still empty placeholders. |
| Web Validation | IIS, app routing, Solr, Memcached, and internal host naming work. | The worker tier depends on the web tier in several places. |
| Worker Validation | The services, Totem, logs, and execution paths work on the worker VM. | This proves the background-processing tier is alive. |
| Database + Functional Checks | The environment is truly operational, not just prepared. | This is where queue jobs, Totem flow, and worker-to-data behavior matter. |
| Build VM Validation | The build VM can become the repeatable build host. | This is a separate proof from runtime readiness. |
Use this when the practical question is simple: which guide do I open next? Each card explains what the guide proves, when you use it, and how it fits into the larger flow.
Use this to confirm IIS, web content deployment, Solr, Memcached, and internal host resolution.
Best used when: web output is already copied to the web VM.
Start with: the Central Runbook web sections.
Best first runtime proofUse this after worker binaries are deployed to test Totem, service startup, log paths, and local runtime behavior.
Best used when: the web tier is already proven far enough to support worker dependencies.
Start with: the Central Runbook worker sections.
After web tier is provenUse this to decide whether the environment is genuinely operational instead of only provisioned and bootstrapped.
Best used when: web and worker basic validations both pass.
Start with: the shared validation section of the Central Runbook.
Real application proofUse this to prove the build VM can build web and worker artifacts and become the repeatable build host.
Best used when: you want to stop depending on another trusted Windows machine.
Start with: the Build VM path in the Central Runbook.
Separate from runtime proofWeb and worker validation tell you whether the application environment works. Build VM validation tells you whether the build host can be trusted for repeatable delivery later. They support each other, but they are not the same proof.
One of the biggest sources of confusion was the gap between the original presentation and the tested implementation. This is the corrected summary of what is now automated or prepared versus what still needs manual or pipeline work.
| Area | Now automated or prepared | Still not fully automated |
|---|---|---|
| Infrastructure | Resource groups, network, Bastion, firewall intent, NAT egress, SQL MI, storage, and VMs. | All future environment variants and every long-term enterprise control around them. |
| Bootstrap | Role-specific VM preparation for the web, worker, and build tiers. | Final application deployment and full config transformation. |
| Web Tier | IIS features, app folders, Java, Solr, Memcached, and IIS app structure. | Copying the real built web content into the prepared folders. |
| Worker Tier | Folders, framework features, NSSM, service placeholders, and internal host aliases. | Real worker binaries, final configs, and steady-state service enablement. |
| Build Tier | Git, NuGet, NSSM, PowerShell modules, Blob-backed build-tool prerequisites, and runner footprint. | A complete artifact promotion and deployment pipeline into the environment. |
The tested environment uses F: for build, web, and worker application data paths. Older D: assumptions are stale.
The current bootstrap path expects required build tooling and installers to be available through the configured Blob storage layout.
Bastion tunneling is part of the intended environment now, but scp still requires SSH to exist on the guest VM.
Bootstrap creates places for apps to live. It does not magically provide the final built application payloads.
The CI/CD system is fully working. Three GitHub Actions workflows handle the complete lifecycle from infrastructure provisioning through build, deployment, and validation. Deployment uses Azure VM Run Commands (v2) for remote execution without inbound network requirements.
create-environment.yaml — Terraform plan/apply/destroy with environment metadata publication. Supports full destroy/rebuild cycles.
build.yaml — Builds web and worker on GitHub-hosted windows-2022 runners. Downloads encrypted vendor bundle from Blob, validates SHA256, decrypts with Key Vault password, NuGet restore, MSBuild, and packages versioned artifacts.
deploy.yaml — Deploys to web/worker VMs via Azure VM Run Commands (v2). Web deployed first → validated → worker deployed → validated. Includes SQL connectivity retry logic (3 attempts, 5s delay) for post-rebuild stability.
Invoke-RemoteValidation.ps1 — Runs per-VM checks (HTTP health, IIS status, Windows services, SQL TCP connectivity). Produces structured JSON results parsed by the workflow for pass/fail gating.
After a full environment destroy/rebuild, the Azure Firewall public IP changes (new static allocation). Use terraform output firewall_public_ip to get the new IP for external web access. SQL connectivity may briefly timeout during VNet peering route propagation — the worker validation retry logic handles this automatically.
This site should not only help with Pearl. It should also teach the repeatable shape of building a serious test environment for another legacy or multi-tier system.
Always distinguish what Terraform and bootstrap prepare from what the application deployment still must deliver.
Do not test end-to-end behavior before proving the web tier, worker tier, and data targeting are correct.
Every environment needs plain-English runbooks, not only code, scripts, and comments.
The manual checks tell you what future smoke tests, deployment gates, and rollback checks need to exist.
The old proposal subfolder still had two useful things worth preserving: the cost-control logic and the reminder that a serious environment has separate platform, data, and operational cost drivers. Those points are now carried forward here so the website remains the main planning surface.
| Cost area | Current baseline | Operational meaning | Main control lever |
|---|---|---|---|
| Networking | GBP 845.87 / month | Firewall and Bastion dominate the fixed access cost of the estate. | Rightsize architecture only if the environment can safely tolerate reduced access and inspection controls. |
| SQL Managed Instance | GBP 660.15 / month | The main data tier is not a simple stop-start service like a VM. | Use reservations, licence benefits, or resizing. Do not assume a pause button exists. |
| Three virtual machines | GBP 517.78 / month | These are the easiest resources to reduce when the estate is idle. | Stop and deallocate VMs when not in use, while remembering storage and retained services still bill. |
| Monitoring and management | GBP 248.13 / month combined | Log Analytics, backup, update management, and retained recovery points keep charging quietly. | Tune retention, diagnostics, and backup scope instead of assuming they stop with the VMs. |
VM compute can be stopped and deallocated. That reduces compute cost, but disks, backups, and several control-plane services still continue to bill.
SQL Managed Instance, Azure Firewall, and Bastion are the classic trap. They are major recurring costs and are not managed like a simple paused VM.
For long-lived estates, look first at reservations, Dev/Test pricing, Azure Hybrid Benefit, and rightsizing. Those matter more than wishful “pause everything” assumptions.
The old proposal-era presentation material is now treated as archived reference. The live website pages above are the current operator-facing source.
Archived — no longer maintainedTreat these four pages as the core website navigation set. Together they cover orientation, the full operator flow, the infrastructure-specific build and bootstrap reference, and the fully operational GitHub CI/CD automation.
Use this for orientation, quick decisions, and deciding what to open next.
You are hereUse it for the fully operational GitHub Actions workflows, Azure Run Commands deployment, validation gates, and environment configuration.
Use this for the operator path: build order, deploy order, validation order, and plain-English explanations.
Use this for variables, apply flow, bootstrap expectations, drift handling, and tested infrastructure troubleshooting.
Use this for the mirrored long-form commands, paths, services, logs, access patterns, and deep validation checks from the internal docs.
Use this for the scalability plan: architecture options, implementation roadmap, database strategy, and cost projections for multiple environments.
Complete reference: all workflows, build process, environment lifecycle, test environment access, rollback procedures, secrets & troubleshooting.
Everything that must be configured manually before running create-environment on a new Azure subscription: OIDC, Key Vault, storage, blobs, SQLMI, GitHub variables.
Start here, move into the Central Runbook when you need a step-by-step process, and open the Terraform Build Guide only when you are changing infrastructure or diagnosing bootstrap and apply behavior.
If you are about to use this environment again, this is the shortest safe action list.
1. Review the Central Runbook.
2. If infrastructure changes are needed, use the Terraform Build Guide.
3. Build web and worker artifacts on a trusted Windows machine if the build VM is not yet your proven host.
4. Validate the web VM.
5. Validate the worker VM.
6. Run the shared database and functional checks.
7. Validate the build VM as the repeatable build host.
8. Convert repeated manual steps into automation after they are proven.