Sitra HQ and BCG Runbook
System startup and shutdown
Use this runbook to start or shut down the systems at Sitra HQ or BCG. Follow the steps in order, verify each result, and record the outcome.
Important: If a required check fails, a command returns an unexpected error, or the result is uncertain, stop, capture the evidence and contact SRT. Do not guess recovery steps or declare the site recovered while checks remain unresolved.
Guided workflow
Choose what you need to do. The guide will show one step at a time. Only mark a step complete when you have carried it out and confirmed the expected result.
Select an action and site to begin.
Stop here. Do not continue or guess a recovery step. Capture the error or other evidence and contact SRT. Do not declare the site recovered while checks remain unresolved.
Workflow steps completed. This does not automatically confirm the site is operational. Record the checks and hand back only if all required checks passed.
Before you begin
- Use your authorised PC and open XCP Centre.
- Connect to the correct pool: Sitra HQ — DC01 or BCG— DC02.
- Start up assumes the physical hosts, storage, power and network are available. If any are unavailable, stop and contact SRT.
- Keep XCP Centre running on your PC. Shutting down ADMIN01 should not close this session.
- For a two-site restart, keep separate checks and records for Sitra HQ and BCG.
Part A — Start the system
Choose the site or sites to recover. For one site, leave the other site's running services alone. For both sites, follow the cross-site order shown below.
Step 1 — Open XCP Centre and confirm the pool
- On your authorised PC, open XCP Centre and connect to the pool for the site you are working on.
- Expand the selected pool and its hosts. Confirm the site before starting or stopping any VM.
- If ADMIN01 is off and you need ADMIN access, try ADMIN01 in the other pool. From there, open XCP and connect to the affected pool. If both ADMIN01 VMs are offline, or the other pool's ADMIN01 cannot reach the affected pool, stop and contact SRT. Do not guess an iDRAC address.
| Site | Pool | VM naming |
|---|---|---|
| Sitra HQ | DC01 | VM names end in -DC01 |
| KCG | DC02 | VM names end in -DC02 |
There is one NDC at each site, each with DSN01 and DCN01. AD01 and AD02 are the two domain-controller VMs in each pool. For example, AD01 at Sitra is AD01-DC01.
Step 2 — Check whether the required VMs are running
Check the required VMs in XCP Centre. Leave any VM that is already running alone; start only VMs that are stopped. If all required VMs are running, continue to Step 3.
If VMs need to be started, use this order:
- AD01 — wait for the agreed Windows ready screen.
- AD02 — it must be on a different physical host from AD01; wait for its ready screen.
- ADMIN01 — check that it starts without errors. Keep XCP Centre open on your PC.
- ZM01, XOA and JB01 — wait for JB01 to reach its logon screen.
- REG — wait for its logon screen.
- MATA01, NR01 and MS02 — confirm the data-proxy VM is NR01 in the selected pool.
- GeoVS VMs: DCN01, DSN01, NCN01, NSN01, RCN01 and RSN01. Check host placement below.
Host placement checks
Step 3 — Check AD01, AD02 and REG
Open each VM console. Check AD01 and AD02 for errors. Confirm REG is at its logon screen. If these checks pass, continue to application startup. For a two-site restart, complete these checks in both pools before starting the application stages.
If both ADMIN01 VMs are offline, or the other pool's ADMIN01 cannot reach the affected pool, stop and contact SRT for the confirmed recovery route.
Step 4 — Start applications
Use the VM consoles in the selected Sitra or KCG pool. Sign in using approved credentials. Run one command at a time and wait for the agreed healthy result. If an application is already running, check it; do not restart it just to repeat the sequence.
A. Start MATA01
Run on MATA01-DC01 or MATA01-DC02, depending on the selected pool:
sudo su cd /opt/mata docker plugin enable loki:latest ./up
MATA uses /opt/mata. A Loki “already enabled” message is acceptable. For other errors, stop, capture the error and contact SRT.
B. Start the data proxy on NR01
Select NR01 in the correct pool and run:
sudo su cd /opt/docker ./up
Wait for the data proxy to be healthy before continuing.
C. Start GeoVS applications in order
| Stage | Application order | Check |
|---|---|---|
| NDC | DSN01 → DCN01 | Service node healthy, then main GeoVS server healthy. |
| NCC | NSN01 → NCN01 | Service node healthy, then NCC GeoVS server healthy. |
| RCC | RSN01 → RCN01 | Service node healthy, then RCC GeoVS server healthy. |
On each GeoVS application VM, run:
sudo su cd /opt/docker docker plugin enable loki:latest ./up
One site restarting: complete NDC → NCC → RCC in the selected pool. Leave the other site's running services alone.
Both sites restarting: complete MATA and NR startup at both sites. Then start DSN01 → DCN01 at Sitra and KCG. Check both NDCs are healthy before starting NCC at either site. Complete NCC at both sites, then RCC at both sites.
Complete each application and its healthy check before starting the next. If a command fails or the result is unclear, stop and contact SRT.
Step 5 — Check the system after start up
Use the agreed observation period. Do not clear restart counters to make a check pass. Distinguish historic restarts from new restarts after recovery. This guide does not set a fixed AIS message-rate or target-count threshold; compare activity with what is expected for the site.
Check the relevant system views and records, including MATA, AIS, GeoVS targets and replication, and PRTG alarms where applicable.
All required checks pass: record the results and hand back.
Any failed, missing or uncertain check: capture the evidence and contact SRT before declaring the site recovered.
Step 6 — Record results and hand back
Save dated screenshots of the MATA overview, AIS, GeoVS targets and replication, plus any PRTG alarm. Record the site, restart time, actions taken and check results in Zuper or the agreed incident record. Keep a separate result for Sitra and KCG.
Tell the site contact the system is ready only when the required checks pass. Do not mark an unresolved recovery as complete.
Part B — Shut down the system
Shutdown is for one site at a time. Confirm the selected pool and agree the shutdown with the site contact before starting.
Step 1 — Record where the VMs are running
Shutdown flow: connect to the Sitra or BCG pool → record VM locations → stop the six GeoVS VMs → stop MATA01 and REG → stop AD02, AD01 and ADMIN01 → follow the agreed final equipment shutdown sequence.
Work on the selected pool only. Do not shut down the other pool by mistake.
Agree the shutdown with the site contact. Confirm high availability handling and the final equipment shutdown order before starting. Expand the selected pool and its hosts and record the current host and status of every VM. Keep this record for restart.
| Group | VMs to record | Host / status notes |
|---|---|---|
| Domain controllers | AD01, AD02 | ________________ |
| Management | ADMIN01 | ________________ |
| Services | ZM01, XOA, JB01 | ________________ |
| Registry | REG | ________________ |
| MATA and data proxy | MATA01, NR01, MS02 | ________________ |
| GeoVS | DCN01, DSN01, NCN01, NSN01, RCN01, RSN01 | ________________ |
Step 2 — Stop the six GeoVS VMs
Stop the VMs in this order: RCN01 → NCN01 → DCN01 → RSN01 → NSN01 → DSN01.
On each VM, stop the application first:
sudo su cd /opt/docker ./down
Wait until the application has fully stopped, allowing up to five minutes. If it does not stop or shows an error, contact SRT. Once it has stopped, run:
shutdown now
Check the VM is stopped in XCP before moving to the next one.
Stop and contact SRT: if a VM restarts unexpectedly. Do not keep shutting it down.
Step 3 — Stop MATA01, then REG
On MATA01 for the selected site, stop the application:
sudo su cd /opt/mata/ ./down
Wait for the application to stop fully, allowing up to five minutes. Then run the command below and confirm the VM is stopped:
shutdown now
On REG for the selected site, run:
sudo shutdown now
Step 4 — Shut down the Windows VMs
Use the normal Windows shutdown action in this order: AD02 → AD01 → ADMIN01. Confirm each VM is stopped in XCP before continuing.
If your XCP session is running on an ADMIN01 VM, reconnect from your authorised PC before shutting that ADMIN01 down.
Step 5 — Follow the approved final shutdown sequence
For remaining VMs, physical hosts, SAN, switches and UPS, follow SRT's approved shutdown sequence for the site.
Do not remove power until the full site shutdown sequence is complete.