Detailed Reference Library — Mirrored From The Tested Internal Runbooks

Pearl Detailed Reference

The Long-Form Website Manual

This page is the deep operational mirror of the internal documentation. Use it when you need the actual paths, service names, bootstrap log locations, access guidance, command examples, and validation proof points without jumping back into the markdown files.

Audience Operators, engineers, reviewers Purpose Execute and verify the environment from the website alone Scope Web, worker, build, database, access, status, scripts

Open Central Runbook Open Terraform Guide Open GitHub CI/CD Guide Jump To VM Checks

Scroll to explore

How To Use This Page

The website now has three layers. The home page is orientation. The runbook is the flow. This page is the deep operational reference.

Use this page during execution

Keep it open while you connect to VMs, inspect logs, verify paths, and run manual checks.

Access routes
Expected folders and hostnames
Bootstrap log locations
Command examples

What it mirrors

The content below mirrors the current tested internal docs rather than the older proposal material.

VM access guide
Web, worker, build validation guides
Database and functional validation guide
Status-check and bootstrap guidance

What it is not

This page is not a replacement for judgment. It is the detailed evidence map for proving the environment works.

Provisioned does not equal deployed
Deployed does not equal operational
Build-host proof is separate from runtime proof

The practical rule

Use the Central Runbook to decide what comes next. Use this Detailed Reference page to execute the check in detail.

Environment And Platform Overview

This is the condensed architecture context from the older presentation and codebase deep-dive, rewritten so it supports the current tested environment instead of competing with it.

0 Databases Main SQL MI estate

0 Tables+ Across the current data model

0 Internal Endpoints Service-heavy legacy platform

0 Core Components Web, utility, and worker apps

flowchart TB subgraph Users[Internet and Admin Access] Operators[Operators] Clients[Clients] Admins[Admins via Bastion] end subgraph AzureTest[Azure Test Environment] subgraph WebTier[Web Tier] WebVM[vm-pearl-test-web\nIIS\npearl-azure\npearl-webservices\nutility apps\nSolr\nMemcached] end subgraph WorkerTier[Worker Tier] WorkerVM[vm-pearl-test-worker\nQueue Processor\nSystem Checker\nAI Spooler\nTotem] end subgraph BuildTier[Build Tier] BuildVM[vm-pearl-test-build\nGit\nNuGet\nBuild Tools\nrunner footprint\nartifact staging] end subgraph DataTier[Data Tier] SQLMI[Azure SQL Managed Instance\n17 databases] end end Operators --> WebVM Clients --> WebVM Admins --> WebVM Admins --> WorkerVM Admins --> BuildVM WebVM --> SQLMI WorkerVM --> SQLMI BuildVM --> WebVM BuildVM --> WorkerVM

Repository-To-Tier Mapping

This is the practical map between the repository and the test-environment tiers. It helps answer which source folder feeds which VM and which runtime responsibility.

Repository path	Runtime tier	Role in the environment
`src/pearl-azure`	Web VM	Main ASP.NET Web Forms application for operators, admins, and client portal use.
`src/pearl-webservices-azure`	Web VM	Internal service-heavy web application used by background flows and utility endpoints.
`src/utility-server/*`	Web VM	Payments, reporting, and Xero-related utility applications.
`src/queue-processor-azure`	Worker VM	Queue-claim and HTTP job execution worker.
`src/system-checker`	Worker VM	Health-monitoring and status-recording worker.
`src/ai-spooler`	Worker VM	AI QC fetch and processing worker.
`src/totem-2-cloud-nosql`	Worker VM	Long-poll notification service for browser event delivery.
`scripts/windows/Build-WebTier.ps1`	Build VM	Checked-in CLI build flow for the web-tier publish output.
`src/terraform/environments/test`	Provisioning	Test-environment Terraform composition and module wiring.

Access And Private Connectivity Reference

The test VMs use private IPs. You should assume direct public RDP does not exist. The current preferred access pattern is Azure Bastion.

Access method	Use it when	Important note
Azure Bastion in the portal	You need immediate browser-based access with the least setup.	This is the easiest immediate path for all three VMs.
Bastion native client / tunnel	You want a better Mac-native workflow using a local client.	The current Terraform enables native-client and tunneling features.
Point-to-Site VPN	You want your Mac to behave like it is inside the Azure VNet.	This is useful for longer-term admin work but is heavier than Bastion.
SCP / SSH	You specifically need SSH-style transfer or shell access.	Bastion tunneling does not create SSH on the guest. The VM still needs an SSH listener.

az extension add --name bastion
az extension update --name bastion

az account set --subscription f777ae09-2af4-4616-b497-19e2d1f05482

az network bastion tunnel \
  --name bas-pearl-test \
  --resource-group rg-pearl-test-hub \
  --target-resource-id /subscriptions/f777ae09-2af4-4616-b497-19e2d1f05482/resourceGroups/rg-pearl-test-spoke/providers/Microsoft.Compute/virtualMachines/vm-pearl-test-web \
  --resource-port 3389 \
  --port 50001

Simple access rule

If you only need to get into a VM quickly, use Azure Bastion in the portal. If you want a better local-client experience from macOS, use Bastion tunneling. Only treat SSH or SCP as valid if the guest actually has SSH configured.

Bootstrap Script, Storage, And Log Map

One of the easiest ways to lose time is not knowing whether a failure belongs to Terraform, the VM extension, or the guest bootstrap script. This map tells you where the evidence lives.

Repository bootstrap scripts

src/terraform/bootstrap/windows/common.ps1
src/terraform/bootstrap/windows/bootstrap-web.ps1
src/terraform/bootstrap/windows/bootstrap-worker.ps1
src/terraform/bootstrap/windows/bootstrap-build.ps1

Expected storage containers

backup
restore
scripts
artifacts

Main guest evidence locations

C:\Bootstrap\Logs
C:\Bootstrap\State
C:\Bootstrap\Packages
F:\vs-package-cache on build VM

VM role	Expected primary log	Expected primary state marker
Build	`C:\Bootstrap\Logs\bootstrap-build.log`	`C:\Bootstrap\State\build-<bootstrapVersion>`
Worker	`C:\Bootstrap\Logs\bootstrap-worker.log`	`C:\Bootstrap\State\worker-<bootstrapVersion>`
Web	`C:\Bootstrap\Logs`	`C:\Bootstrap\State`

Web VM Deep Reference

Use this section after the web payload is deployed to the web VM (either manually or via the deploy.yaml GitHub Actions workflow using Azure Run Commands). The purpose is to prove the interactive application tier is actually alive, not just pre-created in IIS.

Expected application paths

F:\apps\pearl-azure
F:\apps\pearl-webservices
F:\apps\utility-server\payments
F:\apps\utility-server\reporting
F:\apps\utility-server\xero

Expected IIS applications

/pearl-azure
/pearl-webservices
/utility-payments
/utility-reporting
/utility-xero

Expected host aliases

memcached.private.pearl
pearlinternal.private.pearl
pearl3.private.pearl
pearl4.private.pearl
solrlb.private.pearl

$paths = @(
  'F:\apps\pearl-azure',
  'F:\apps\pearl-webservices',
  'F:\apps\utility-server\payments',
  'F:\apps\utility-server\reporting',
  'F:\apps\utility-server\xero',
  'C:\solr',
  'C:\memcached'
)

$paths | ForEach-Object {
  [pscustomobject]@{ Path = $_; Exists = Test-Path $_ }
} | Format-Table -AutoSize

$repoRoot = 'C:\path\to\message-direct'
$publishRoot = Join-Path $repoRoot 'publish'
$aspnetCompiler = 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_compiler.exe'

& $aspnetCompiler -v /pearl-azure -p (Join-Path $repoRoot 'src\pearl-azure') -f -u (Join-Path $publishRoot 'pearl-azure')
& $aspnetCompiler -v /pearl-webservices -p (Join-Path $repoRoot 'src\pearl-webservices-azure') -f -u (Join-Path $publishRoot 'pearl-webservices')
& $aspnetCompiler -v /utility-payments -p (Join-Path $repoRoot 'src\utility-server\payments') -f -u (Join-Path $publishRoot 'utility-payments')
& $aspnetCompiler -v /utility-reporting -p (Join-Path $repoRoot 'src\utility-server\reporting') -f -u (Join-Path $publishRoot 'utility-reporting')
& $aspnetCompiler -v /utility-xero -p (Join-Path $repoRoot 'src\utility-server\xero') -f -u (Join-Path $publishRoot 'utility-xero')

Critical web build warning

pearl-azure is an ASP.NET Web Site project, not a normal project-file web app. That means the CLI precompile step depends on a correctly seeded source bin folder.

Telerik.Web.UI.dll
AjaxControlToolkit.dll
Newtonsoft.Json.dll
AWSSDK.Core.dll and AWSSDK.S3.dll
Otp.NET.dll
Potentially GCheckout*.dll

Deployment destination after build

Copy the publish output, not the staging source tree, into the web VM application folders.

publish\pearl-azure to F:\apps\pearl-azure
publish\pearl-webservices to F:\apps\pearl-webservices
publish\utility-payments to F:\apps\utility-server\payments
publish\utility-reporting to F:\apps\utility-server\reporting
publish\utility-xero to F:\apps\utility-server\xero

Exact dependency-handling rule

Do not start aspnet_compiler.exe until the source src\pearl-azure\bin folder is deliberately seeded and validated. If that folder is incomplete, the precompile failure later in the flow is only a delayed symptom.

$repoRoot = 'C:\path\to\message-direct'
$packagesRoot = Join-Path $repoRoot 'packages'
$pearlBin = Join-Path $repoRoot 'src\pearl-azure\bin'
$pearlPackagesConfig = Join-Path $repoRoot 'src\pearl-azure\packages.config'
$nuget = 'C:\Program Files (x86)\NuGet\nuget.exe'

New-Item -ItemType Directory -Path $packagesRoot -Force | Out-Null
New-Item -ItemType Directory -Path $pearlBin -Force | Out-Null

& $nuget restore $pearlPackagesConfig -PackagesDirectory $packagesRoot

$dllPatterns = @(
  'Microsoft.Web.SessionState.SqlInMemory.dll',
  'AWSSDK.Core.dll',
  'AWSSDK.S3.dll',
  'Newtonsoft.Json.dll',
  'Otp.NET.dll'
)

foreach ($pattern in $dllPatterns) {
  $match = Get-ChildItem $packagesRoot -Recurse -Filter $pattern -ErrorAction SilentlyContinue |
    Sort-Object FullName |
    Select-Object -First 1

  if ($match) {
    Copy-Item $match.FullName -Destination $pearlBin -Force
    Write-Host "Copied $($match.Name)"
  }
}

$repoRoot = 'C:\path\to\message-direct'
$pearlBin = Join-Path $repoRoot 'src\pearl-azure\bin'

robocopy '\\server\share\pearl-azure\bin' $pearlBin Telerik.Web.UI.dll AjaxControlToolkit.dll GCheckout*.dll /R:2 /W:2

Get-ChildItem $pearlBin |
  Where-Object { $_.Name -match '^(AjaxControlToolkit|Telerik\.Web\.UI|Newtonsoft\.Json|AWSSDK\.Core|AWSSDK\.S3|Otp(\.NET|Net)|GCheckout|Microsoft\.Web\.SessionState\.SqlInMemory).*\.dll$' } |
  Select-Object Name, Length, LastWriteTime |
  Sort-Object Name |
  Format-Table -AutoSize

$repoRoot = 'C:\path\to\message-direct'
$pearlBin = Join-Path $repoRoot 'src\pearl-azure\bin'

$requiredDllChecks = @(
  'Telerik.Web.UI*.dll',
  'AjaxControlToolkit*.dll',
  'Newtonsoft.Json*.dll',
  'AWSSDK.Core*.dll',
  'AWSSDK.S3*.dll',
  'Otp.NET*.dll',
  'GCheckout*.dll',
  'Microsoft.Web.SessionState.SqlInMemory*.dll'
)

$missing = @()
foreach ($pattern in $requiredDllChecks) {
  if (-not (Get-ChildItem $pearlBin -Filter $pattern -ErrorAction SilentlyContinue)) {
    $missing += $pattern
  }
}

if ($missing.Count -gt 0) {
  throw ('pearl-azure bin is incomplete. Missing DLL families: ' + ($missing -join ', '))
}

$ErrorActionPreference = 'Stop'

$repoRoot = $PWD.Path
$packagesRoot = Join-Path $repoRoot 'packages'
$nuget = 'C:\Program Files (x86)\NuGet\nuget.exe'

& $nuget restore (Join-Path $repoRoot 'src\utility-server\payments\packages.config') -PackagesDirectory $packagesRoot
& $nuget restore (Join-Path $repoRoot 'src\utility-server\reporting\packages.config') -PackagesDirectory $packagesRoot
& $nuget restore (Join-Path $repoRoot 'src\utility-server\xero\packages.config') -PackagesDirectory $packagesRoot

powershell -ExecutionPolicy Bypass -File .\scripts\windows\Build-WebTier.ps1 `
  -ManualDependencySource 'C:\approved\pearl-azure-vendor-bin' `
  -CleanPublish `
  -AllowMissingGCheckout

What the build script stages

The checked-in build script may precompile from a staging tree under src\publish\_precompile-source\<site> instead of directly from the checked-out source.

Treat it as build-only working state
Do not commit staged rewrites back into app source
Do not copy _precompile-source to the VM
Deploy only the published folders under src\publish\<site>

Current build-only compatibility behavior

The staged copy is where the build flow applies temporary compiler relaxations and limited source rewrites for legacy Web Site projects.

pearl-webservices uses relaxed VB compiler options during staged precompile
utility-reporting uses staged code-behind compatibility fixes
The goal is successful clean-room precompile without changing repository runtime code

Known-good manifest comparison workflow

If policy allows you to inspect a working deployment but not copy DLLs from it, use the dependency manifest workflow below. This gives you exact names, versions, and tokens before you precompile.

$binPath = 'C:\path\to\known-good\pearl-azure\bin'
$outputPath = 'C:\temp\pearl-azure-bin-manifest.csv'

$rows = foreach ($file in Get-ChildItem $binPath -Filter *.dll | Sort-Object Name) {
    $assemblyName = $null
    $publicKeyToken = ''

    try {
        $assemblyName = [System.Reflection.AssemblyName]::GetAssemblyName($file.FullName)
        $tokenBytes = $assemblyName.GetPublicKeyToken()
        if ($tokenBytes) {
            $publicKeyToken = -join ($tokenBytes | ForEach-Object { $_.ToString('x2') })
        }
    }
    catch {
    }

    [pscustomobject]@{
        Name = $file.Name
        AssemblyVersion = if ($assemblyName) { $assemblyName.Version.ToString() } else { '' }
        FileVersion = $file.VersionInfo.FileVersion
        ProductVersion = $file.VersionInfo.ProductVersion
        PublicKeyToken = $publicKeyToken
        Length = $file.Length
        LastWriteTime = $file.LastWriteTime
    }
}

$rows | Export-Csv -Path $outputPath -NoTypeInformation
$rows | Format-Table -AutoSize
Write-Host "Saved manifest to $outputPath"

$binPath = 'C:\path\to\known-good\pearl-azure\bin'

Get-ChildItem $binPath -Filter *.dll |
  Where-Object { $_.Name -match '^(AjaxControlToolkit|Telerik\.Web\.UI|Newtonsoft\.Json|AWSSDK\.Core|AWSSDK\.S3|Otp(\.NET|Net)|GCheckout|Microsoft\.Web\.SessionState\.SqlInMemory).*\.dll$' } |
  ForEach-Object {
    $asm = $null
    $pkt = ''
    try {
      $asm = [System.Reflection.AssemblyName]::GetAssemblyName($_.FullName)
      $bytes = $asm.GetPublicKeyToken()
      if ($bytes) {
        $pkt = -join ($bytes | ForEach-Object { $_.ToString('x2') })
      }
    } catch {}

    [pscustomobject]@{
      Name = $_.Name
      AssemblyVersion = if ($asm) { $asm.Version.ToString() } else { '' }
      FileVersion = $_.VersionInfo.FileVersion
      ProductVersion = $_.VersionInfo.ProductVersion
      PublicKeyToken = $pkt
    }
  } |
  Format-Table -AutoSize

$reference = Import-Csv 'C:\temp\pearl-azure-bin-manifest.csv'
$candidate = Get-ChildItem 'C:\path\to\message-direct\src\pearl-azure\bin' -Filter *.dll |
  ForEach-Object {
    $asm = $null
    try { $asm = [System.Reflection.AssemblyName]::GetAssemblyName($_.FullName) } catch {}
    [pscustomobject]@{
      Name = $_.Name
      AssemblyVersion = if ($asm) { $asm.Version.ToString() } else { '' }
      FileVersion = $_.VersionInfo.FileVersion
    }
  }

Compare-Object $reference $candidate -Property Name, AssemblyVersion, FileVersion

Check	What a pass means	What a fail usually means
Folders contain real content	The web tier was actually deployed.	Only bootstrap placeholders exist.
IIS apps exist and bind correctly	The web tier is shaped correctly for testing.	IIS structure or host-binding drift exists.
Solr and Memcached respond	The main support services are ready for realistic app behavior.	Bootstrap did not finish or services are not running.
Connection strings point to SQL MI	The web tier is aimed at the test data platform.	Old environment settings survived deployment.

Worker VM Deep Reference

The worker VM proves the background-processing layer. Only treat it as working when the real deployed binaries are present and the services can run without immediate failure.

Expected application folders

F:\apps\queue-processor
F:\apps\system-checker
F:\apps\ai-spooler
F:\apps\totem
F:\QueueProcessor\Logs

Expected Windows services

PearlQueueProcessor
PearlSystemChecker
PearlAISpooler
PearlTotem

Expected deployed binaries

QueueProcessor.exe
SystemChecker.exe
AISpooler.exe
Totem2.exe

$paths = @(
  'F:\apps\queue-processor',
  'F:\apps\system-checker',
  'F:\apps\ai-spooler',
  'F:\apps\totem',
  'F:\QueueProcessor\Logs',
  'C:\totemscripts',
  'C:\totem',
  'C:\Bootstrap\Logs',
  'C:\Bootstrap\State'
)

$paths | ForEach-Object {
  [pscustomobject]@{ Path = $_; Exists = Test-Path $_ }
} | Format-Table -AutoSize

Get-Service PearlQueueProcessor, PearlSystemChecker, PearlAISpooler, PearlTotem |
  Select-Object Name, Status, StartType | Format-Table -AutoSize

Get-CimInstance Win32_Service |
  Where-Object { $_.Name -in @('PearlQueueProcessor','PearlSystemChecker','PearlAISpooler','PearlTotem') } |
  Select-Object Name,State,StartMode,PathName | Format-List

Invoke-WebRequest 'http://localhost/ping' -UseBasicParsing
Invoke-WebRequest 'http://localhost/stats' -UseBasicParsing

$sessionId = 'opsx-manual-' + [guid]::NewGuid().ToString('N')
$notifyKey = 'opsx-test-key'
Invoke-WebRequest "http://localhost/register?sessionid=$sessionId¬ifyon=%7Btoast:$notifyKey%7D" -UseBasicParsing
Invoke-WebRequest "http://localhost/notify?key=$notifyKey" -UseBasicParsing
Invoke-WebRequest "http://localhost/poll?sessionid=$sessionId" -UseBasicParsing

$queueProbe = 'F:\QueueProcessor\Logs\manual-validation-probe.txt'
Set-Content -Path $queueProbe -Value ('validated ' + (Get-Date))
Get-Item $queueProbe | Select-Object FullName,Length,LastWriteTime

Get-Content 'C:\Bootstrap\Logs\bootstrap-worker.log' -Tail 120

Get-Content C:\Windows\System32\drivers\etc\hosts |
  Select-String 'memcached.private.pearl|pearlinternal.private.pearl|pearl3.private.pearl|pearl4.private.pearl|solrlb.private.pearl'

Quick worker rule

If only the folders and placeholder services exist, the worker VM is deployed but not proven. To call it working, the services must start, Totem must respond, logs must be writable, and at least one synthetic functional action must succeed.

Build VM Deep Reference

The build VM is the repeatable build-host proof. It should not be treated as ready just because the VM exists. The toolchain, folders, logs, package cache, and repo build smoke tests all matter.

Expected working folders

F:\build
F:\build\repo
F:\build\artifacts
F:\build\scripts
F:\restore-tools

Expected tool locations

C:\Program Files\Git\cmd\git.exe
C:\Tools\nuget\nuget.exe
C:\Tools\nssm\nssm.exe
C:\BuildTools\MSBuild\Current\Bin\MSBuild.exe
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_compiler.exe

Expected build evidence

C:\Bootstrap\Logs\bootstrap-build.log
C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe
F:\vs-package-cache
Runner footprint under C:\actions-runner if configured

$paths = @(
  'F:\build',
  'F:\build\repo',
  'F:\build\artifacts',
  'F:\build\scripts',
  'F:\restore-tools',
  'C:\BuildTools',
  'C:\Bootstrap\Logs'
)

$paths | ForEach-Object {
  [pscustomobject]@{ Path = $_; Exists = Test-Path $_ }
} | Format-Table -AutoSize

& 'C:\Program Files\Git\cmd\git.exe' --version
& 'C:\Tools\nuget\nuget.exe' help | Select-Object -First 5
Get-Content 'C:\Bootstrap\Logs\bootstrap-build.log' -Tail 120

$vswhere = 'C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe'
& $vswhere -latest -products * -format json

Get-ChildItem 'C:\BuildTools\MSBuild\Current\Bin' -Filter MSBuild.exe -ErrorAction SilentlyContinue |
  Select-Object FullName,Length,LastWriteTime

Test-Path 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_compiler.exe'

Get-ChildItem 'C:\actions-runner' | Select-Object Name,Length,LastWriteTime | Format-Table -AutoSize
Test-Path 'C:\actions-runner\.runner'

$repoRoot = 'F:\build\repo\message-direct'
$timestamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$logPath = Join-Path $repoRoot ("src\publish\build-webtier-$timestamp.log")

Set-Location $repoRoot

powershell -ExecutionPolicy Bypass -File .\scripts\windows\Build-WebTier.ps1 `
  -ManualDependencySource 'C:\approved\pearl-azure-vendor-bin' `
  -CleanPublish `
  -AllowMissingGCheckout *>&1 | Tee-Object -FilePath $logPath

$msbuild = 'C:\BuildTools\MSBuild\Current\Bin\MSBuild.exe'
$outRoot = 'F:\build\artifacts\worker-smoke'
New-Item -ItemType Directory -Path $outRoot -Force | Out-Null

& $msbuild (Join-Path $repoRoot 'src\queue-processor-azure\MessageQueueProcessor.sln') /t:Build /p:Configuration=Release /p:OutDir="$outRoot\QueueProcessor\\"
& $msbuild (Join-Path $repoRoot 'src\system-checker\SystemChecker.sln') /t:Build /p:Configuration=Release /p:OutDir="$outRoot\SystemChecker\\"
& $msbuild (Join-Path $repoRoot 'src\ai-spooler\AISpooler\AISpooler.sln') /t:Build /p:Configuration=Release /p:OutDir="$outRoot\AISpooler\\"
& $msbuild (Join-Path $repoRoot 'src\totem-2-cloud-nosql\Totem2.sln') /t:Build /p:Configuration=Release /p:OutDir="$outRoot\Totem2\\"

Build-host decision rule

If the toolchain exists but the repo still cannot build on the VM, the build VM is bootstrapped but not yet ready for repeatable use.

Database And Functional Validation Reference

This is the real operational proof. It tells you whether the environment works as an application environment instead of only a set of provisioned Windows machines. The automated CI/CD pipeline (deploy.yaml) includes SQL connectivity checks with retry logic: 3 attempts, 15-second TCP timeout, 5-second delay between retries. This accommodates VNet peering propagation delays (15–60s) that occur after environment recreation.

Validation goal	What to prove	Main warning
Connection string targeting	Web and worker config point to the intended test SQL Managed Instance.	Placeholder passwords or old host families are immediate fail conditions.
Basic SQL read/write	Web and worker tiers can open connections and perform low-impact test operations.	Use synthetic markers and roll back writes when possible.
Session state reachability	The ASP.NET session-state catalog is reachable from the deployed web configuration.	If session state points elsewhere, the web tier is not aligned.
Functional flows	Queue jobs, Totem flows, System Checker activity, and AI Spooler reachability all behave as expected.	Legacy hard-coded dependencies are still known risk areas.

[xml]$config = Get-Content 'F:\apps\pearl-azure\web.config'
$config.configuration.connectionStrings.add |
  Select-Object name, connectionString |
  Format-List

$sessionState = $config.configuration.'system.web'.sessionState
$sessionState

Test-NetConnection 'pearlsqlmi.639b83d4d534.database.windows.net' -Port 1433

[xml]$config = Get-Content 'F:\apps\pearl-azure\web.config'
$queuesConnection = ($config.configuration.connectionStrings.add | Where-Object { $_.name -eq 'Queues' }).connectionString
$aspnetConnection = ($config.configuration.connectionStrings.add | Where-Object { $_.name -eq 'ASPNET' }).connectionString

$queuesConnection
$aspnetConnection

$connectionString = $queuesConnection
$sql = 'SELECT TOP 5 ID, Source, Processed, DateTimeAdded FROM Process_JobQueue ORDER BY ID DESC'

$conn = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$conn.Open()
$reader = $cmd.ExecuteReader()
$table = New-Object System.Data.DataTable
$table.Load($reader)
$conn.Close()
$table | Format-Table -AutoSize

$connectionString = $queuesConnection

$conn = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$conn.Open()
$tran = $conn.BeginTransaction()

try {
  $cmd = $conn.CreateCommand()
  $cmd.Transaction = $tran
  $cmd.CommandText = @"
INSERT INTO Process_Diagnostics(SQLDateTime, server, LocalDateTime, GuessedSQLDateTime)
VALUES (GETDATE(), @server, @local, @guessed)
"@
  $cmd.Parameters.Add('@server',[System.Data.SqlDbType]::VarChar,50).Value = 'OPSX-MANUAL-VALIDATION-WEB'
  $cmd.Parameters.Add('@local',[System.Data.SqlDbType]::VarChar,100).Value = (Get-Date).ToString('o')
  $cmd.Parameters.Add('@guessed',[System.Data.SqlDbType]::VarChar,100).Value = (Get-Date).ToString('o')
  $rows = $cmd.ExecuteNonQuery()
  Write-Host "Rows inserted inside transaction: $rows"
  $tran.Rollback()
  Write-Host 'Rollback completed.'
}
catch {
  try { $tran.Rollback() } catch {}
  throw
}
finally {
  $conn.Close()
}

$connectionString = $aspnetConnection
$sql = 'SELECT TOP 20 name FROM sys.tables ORDER BY name'

$conn = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$conn.Open()
$reader = $cmd.ExecuteReader()
$table = New-Object System.Data.DataTable
$table.Load($reader)
$conn.Close()
$table | Format-Table -AutoSize

$connectionString = 'Server=tcp:YOUR-SQLMI.database.windows.net,1433;Persist Security Info=False;User ID=YOURUSER;Password=YOURPASSWORD;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Initial Catalog=PearlQueues'

Test-NetConnection 'pearlsqlmi.639b83d4d534.database.windows.net' -Port 1433

$sql = 'SELECT TOP 5 ID, Source, Processed, Server, DateTimeAdded FROM Process_JobQueue ORDER BY ID DESC'

$conn = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$conn.Open()
$reader = $cmd.ExecuteReader()
$table = New-Object System.Data.DataTable
$table.Load($reader)
$conn.Close()
$table | Format-Table -AutoSize

$connectionString = 'Server=tcp:YOUR-SQLMI.database.windows.net,1433;Persist Security Info=False;User ID=YOURUSER;Password=YOURPASSWORD;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Initial Catalog=PearlQueues'

$conn = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$conn.Open()
$tran = $conn.BeginTransaction()

try {
  $cmd = $conn.CreateCommand()
  $cmd.Transaction = $tran
  $cmd.CommandText = @"
INSERT INTO Process_Diagnostics(SQLDateTime, server, LocalDateTime, GuessedSQLDateTime)
VALUES (GETDATE(), @server, @local, @guessed)
"@
  $cmd.Parameters.Add('@server',[System.Data.SqlDbType]::VarChar,50).Value = 'OPSX-MANUAL-VALIDATION-WORKER'
  $cmd.Parameters.Add('@local',[System.Data.SqlDbType]::VarChar,100).Value = (Get-Date).ToString('o')
  $cmd.Parameters.Add('@guessed',[System.Data.SqlDbType]::VarChar,100).Value = (Get-Date).ToString('o')
  $rows = $cmd.ExecuteNonQuery()
  Write-Host "Rows inserted inside transaction: $rows"
  $tran.Rollback()
  Write-Host 'Rollback completed.'
}
catch {
  try { $tran.Rollback() } catch {}
  throw
}
finally {
  $conn.Close()
}

$connectionString = 'Server=tcp:YOUR-SQLMI.database.windows.net,1433;Persist Security Info=False;User ID=YOURUSER;Password=YOURPASSWORD;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Initial Catalog=PearlQueues'
$targetUrl = 'http://pearlinternal.private.pearl/pearl-azure/exposed/returnsystemstatus.aspx'

$sql = @"
INSERT INTO Process_JobQueue(DateTimeAdded, DateTimeToRun, Source, ResourceCode, URL, Processed)
VALUES (GETDATE(), GETDATE(), 'OPSX-MANUAL-VALIDATION', 'OPSXTEST', @url, 0);
SELECT SCOPE_IDENTITY() AS NewId;
"@

$conn = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$null = $cmd.Parameters.Add('@url',[System.Data.SqlDbType]::VarChar,1000)
$cmd.Parameters['@url'].Value = $targetUrl
$conn.Open()
$jobId = $cmd.ExecuteScalar()
$conn.Close()

Write-Host "Created validation queue job ID $jobId"

$jobId = 12345
$connectionString = 'Server=tcp:YOUR-SQLMI.database.windows.net,1433;Persist Security Info=False;User ID=YOURUSER;Password=YOURPASSWORD;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Initial Catalog=PearlQueues'

for ($i = 0; $i -lt 24; $i++) {
  $sql = "SELECT ID, Source, Processed, Server, DateTimeOfBatch, DateTimeCompleted, Duration, Attempts FROM Process_JobQueue WHERE ID = $jobId"
  $conn = New-Object System.Data.SqlClient.SqlConnection($connectionString)
  $cmd = $conn.CreateCommand()
  $cmd.CommandText = $sql
  $conn.Open()
  $reader = $cmd.ExecuteReader()
  $table = New-Object System.Data.DataTable
  $table.Load($reader)
  $conn.Close()
  $table | Format-Table -AutoSize
  if ($table.Rows.Count -gt 0 -and $table.Rows[0].Processed -ne 0 -and $table.Rows[0].Processed -ne 1) { break }
  Start-Sleep -Seconds 5
}

-- Optional direct SQL cleanup in SSMS or sqlcmd after the queue validation
SELECT ID, Source, Processed, Server, DateTimeOfBatch, DateTimeCompleted, Duration, Attempts
FROM Process_JobQueue
WHERE ID = <your job id>;

DELETE FROM Process_JobQueue
WHERE ID = <your job id>;

$sessionId = 'opsx-manual-' + [guid]::NewGuid().ToString('N')
$notifyKey = 'opsx-functional-key'

$register = Invoke-WebRequest "http://localhost/register?sessionid=$sessionId¬ifyon=%7Btoast:$notifyKey%7D" -UseBasicParsing
$notify = Invoke-WebRequest "http://localhost/notify?key=$notifyKey" -UseBasicParsing
$poll = Invoke-WebRequest "http://localhost/poll?sessionid=$sessionId" -UseBasicParsing

$register.StatusCode
$notify.StatusCode
$notify.Content
$poll.StatusCode
$poll.Content

$connectionString = 'Server=YOUR-CHECKING-DB;Initial Catalog=Checking;User ID=YOURUSER;Password=YOURPASSWORD'
$sql = 'SELECT TOP 20 ID, Name, Active, DateTime, DateTimeLastRun, LastResult, Status FROM CheckJobs ORDER BY ID'

$conn = New-Object System.Data.SqlClient.SqlConnection($connectionString)
$cmd = $conn.CreateCommand()
$cmd.CommandText = $sql
$conn.Open()
$reader = $cmd.ExecuteReader()
$table = New-Object System.Data.DataTable
$table.Load($reader)
$conn.Close()
$table | Format-Table -AutoSize

Start-Service 'PearlSystemChecker'
Get-Service 'PearlSystemChecker'

Invoke-WebRequest 'http://10.0.0.12/aiqc/getmessagestoprocess.aspx' -UseBasicParsing -TimeoutSec 20

Test-NetConnection 10.0.0.12 -Port 80

Start-Service 'PearlAISpooler'
Get-Service 'PearlAISpooler'

Get-WinEvent -LogName Application -MaxEvents 80 |
  Where-Object { $_.ProviderName -match 'Application Error|.NET Runtime|NSSM' -or $_.Message -match 'AISpooler' } |
  Select-Object TimeCreated,ProviderName,Id,LevelDisplayName,Message |
  Format-List

How to score the queue validation

Processed = 10 means the worker completed the synthetic job successfully
Processed = 2 means the job failed
Server should identify the worker instance that claimed the job
DateTimeCompleted should be populated on terminal completion

Current migration risk to record honestly

If System Checker still targets 10.0.1.36, the environment is not fully migrated
If AI Spooler only works because it still reaches legacy endpoints, count that as an amber result, not green
If session state points anywhere except the intended SQL MI family, fail the validation immediately

Copy-paste SQL only for SSMS users

This subsection is for operators who prefer SSMS or sqlcmd instead of PowerShell. It assumes you already connected to the correct SQL MI target identified from the deployed web configuration.

-- Read proof against PearlQueues
        SELECT TOP 5 ID, Source, Processed, Server, DateTimeAdded
        FROM Process_JobQueue
        ORDER BY ID DESC;

        -- Low-impact write proof with rollback
        BEGIN TRANSACTION;

        INSERT INTO Process_Diagnostics(SQLDateTime, server, LocalDateTime, GuessedSQLDateTime)
        VALUES (GETDATE(), 'OPSX-MANUAL-VALIDATION-SQL', CONVERT(varchar(100), SYSDATETIMEOFFSET(), 127), CONVERT(varchar(100), SYSDATETIMEOFFSET(), 127));

        SELECT TOP 5 *
        FROM Process_Diagnostics
        ORDER BY SQLDateTime DESC;

        ROLLBACK TRANSACTION;

-- Session-state reachability proof against the ASPNET catalog
        SELECT TOP 20 name
        FROM sys.tables
        ORDER BY name;

-- Insert a synthetic queue job and capture the new ID
        DECLARE @TargetUrl varchar(1000) = 'http://pearlinternal.private.pearl/pearl-azure/exposed/returnsystemstatus.aspx';

        INSERT INTO Process_JobQueue(DateTimeAdded, DateTimeToRun, Source, ResourceCode, URL, Processed)
        VALUES (GETDATE(), GETDATE(), 'OPSX-MANUAL-VALIDATION', 'OPSXTEST', @TargetUrl, 0);

        SELECT CAST(SCOPE_IDENTITY() AS int) AS NewId;

-- Poll the validation job after insertion
        SELECT ID, Source, Processed, Server, DateTimeOfBatch, DateTimeCompleted, Duration, Attempts
        FROM Process_JobQueue
        WHERE ID = <your job id>;

        -- Interpretation:
        -- Processed = 10 means success
        -- Processed = 2 means terminal failure

-- Optional cleanup after queue validation
        DELETE FROM Process_JobQueue
        WHERE ID = <your job id>;

-- System Checker inspection if you have an approved Checking database target
        SELECT TOP 20 ID, Name, Active, DateTime, DateTimeLastRun, LastResult, Status
        FROM CheckJobs
        ORDER BY ID;

Current migration risk to remember

Some worker-side code paths still show legacy dependency evidence in source. If deployed behavior still points to old IPs or non-test endpoints, the environment is not fully migrated even if the services start.

Quick Status Check Commands

These are the fast commands to answer whether the environment is stable enough for more work, whether bootstrap is still running, and whether Terraform and Azure still align.

export SUBSCRIPTION_ID="f777ae09-2af4-4616-b497-19e2d1f05482"
export SPOKE_RG="rg-pearl-test-spoke"
export HUB_RG="rg-pearl-test-hub"
export DATA_RG="rg-pearl-test-data"
export BUILD_VM="vm-pearl-test-build"
export WEB_VM="vm-pearl-test-web"
export WORKER_VM="vm-pearl-test-worker"
export FIREWALL_NAME="fw-pearl-test"
export KV_NAME="pearl-test-kv"
export INSTALLER_STORAGE="stglobalartifacts17fef6"
export TEST_STORAGE="stpearltest17fef6"

az account set --subscription "$SUBSCRIPTION_ID"

az vm list -g "$SPOKE_RG" -d \
  --query "[].{vm:name, power:powerState, provisioning:provisioningState, privateIp:privateIps}" \
  -o table

for vm in "$BUILD_VM" "$WEB_VM" "$WORKER_VM"; do
  az vm extension show \
    -g "$SPOKE_RG" \
    --vm-name "$vm" \
    -n "bootstrap-$vm" \
    --query "{name:name, provisioningState:provisioningState, bootstrapVersion:settings.bootstrapVersion}" \
    -o json
done

TF_VAR_subscription_id="$SUBSCRIPTION_ID" \
TF_VAR_sql_admin_password='YOUR_SQL_PASSWORD' \
TF_VAR_vm_admin_password='YOUR_VM_PASSWORD' \
terraform -chdir="/Users/mestella/Documents/NetzonProjects/message-direct/message-direct/src/terraform/environments/test" plan -input=false

Portal checks that still matter

VM Overview for power and provisioning state
Extensions + applications for bootstrap status
Resource group Deployments for failed nested operations
Activity log for extension retries and failures

What healthy usually looks like

VMs show Running
Bootstrap extensions show Succeeded or a clearly active update state
Terraform plan does not show unexpected drift
Guest logs do not end with unresolved fatal bootstrap errors

Troubleshooting Patterns To Recognize Fast

These are the repeat offenders that make the environment look more broken than it is, or more ready than it really is.

Terraform succeeded but the app is broken

This is usually a deployment or configuration issue, not a resource-creation failure.

Firewall IP changed after rebuild

The Azure Firewall public IP is dynamic and changes after every destroy/rebuild cycle. Retrieve the new IP with terraform output firewall_public_ip and update any external access rules.

Bootstrap folders exist but workloads do not run

That usually means the environment is prepared but the real payloads or runtime configs are still incomplete.

Old D: assumptions keep showing up

The current tested environment uses F: for the main application and build working paths.

SCP assumptions fail unexpectedly

RDP or Bastion tunneling is not the same thing as guest-level SSH support.