Site reliability engineering practice / Calgary
Site reliability engineering for Calgary teams.
Distributed services need an operating model for the moments when a site loses connectivity or telemetry arrives late. Separate missing evidence from confirmed recovery, define the service boundary and make ownership clear between field-connected systems and central operations.
What this practice covers
What SRE covers
A distributed service can behave differently at a remote site and at its central platform. Define the user outcome at each boundary, account for delayed evidence and agree who decides when intervention is needed. Verify recovery using the service state as well as the signals arriving from it.
Reliability when connectivity and telemetry are delayed
Connectivity loss can leave central operators uncertain about a remote service. The operating model should distinguish a known failure, an expected disconnected state and a gap in available evidence.
Hypothetical example: a remote application continues recording work while its connection is unavailable. When the link returns, old telemetry arrives alongside new events. A fresh arrival time alone does not prove the service is currently healthy.
Recovery evidence needs the event time, expected local behavior and reconciliation state. It should show whether work was preserved and synchronized, not simply whether the connection returned.
Responsibility boundaries identify who can act locally, who can change the central platform and who confirms the complete recovery. Any intervention affecting physical operations requires the appropriate domain owner and agreed procedures.
- Evidence freshness
- Distinguish event time, arrival time and missing signals when assessing service state.
- Recovery ownership
- Name the local and central owners, decision rights and escalation path.
- Reconciliation checks
- Verify preserved work, synchronization and the restored user outcome before closing an incident.
What that gives your team
- A clear distinction between missing signals and known failure
- Recovery decisions with named local and central owners
- Evidence that the restored service has reconciled its work
Service indicators and objectives
Define a service-level indicator (SLI) for a user journey, then set a service-level objective (SLO) over an agreed period. Use the objective to make reliability decisions explicit.
Error budgets and change decisions
An error budget is the unreliability allowed by an SLO over its measurement period. Use budget consumption alongside incident evidence to decide when to prioritize reliability work or proceed with change.
Dependency and failure analysis
Examine how timeouts, queues, retries and upstream failures affect the complete service journey. Identify where isolation, graceful degradation or recovery checks are needed.
Operational toil and automation
Identify repetitive operational work that is manual, reactive and offers little lasting value. Consider automation or system changes where they reduce that work without hiding risk.
Incident readiness and learning
Connect usable alerts, runbooks, escalation paths and decision owners. Turn incident findings into tracked engineering work and verify whether the changes address the failure pattern.
Production change and verification
Define the expected behavior, review signals during change and agree when to stop or roll back. Verify recovery and record remaining risks before closing the work.
How it works
How SRE work moves
Define the disconnected state, choose an observable improvement and verify recovery across local and central boundaries.
Map the service and evidence gaps.
Identify local behavior, central dependencies and expected connectivity. Define which observations are sufficient to assess the service and which leave its state unknown.
Set the recovery boundary.
Agree on intervention authority, escalation and checks for the next improvement. Account for delayed telemetry and any domain-specific operating procedures.
Verify reconciliation and ownership.
Confirm the user outcome, preserved work and synchronized state. Record remaining uncertainty and update the handoff between local and central owners.
Engagement models
Choose how we work together
Choose consulting for a bounded decision about service boundaries, recovery evidence or operating ownership. Other models address engineering contribution, implementation or scoped ongoing operation. Define local authority and central responsibilities explicitly; this page does not imply field staffing or continuous coverage.
Assessment, architecture review and technical direction for a defined reliability decision.
ExploreA model for engineering contribution inside a client-directed team, with responsibilities and availability agreed separately.
ExploreA model for scoped production response with agreed systems, activation paths and responsibility boundaries.
ExploreA model for ownership of a bounded engineering workstream, with decision rights and handback conditions defined.
ExploreA model for a defined technical outcome with implementation scope, acceptance checks and operating handoff.
ExploreA model for ongoing maintenance, troubleshooting and improvement of named systems under agreed terms.
ExploreQuestions answered
SRE questions, answered
Practical questions about service objectives, reliability work and choosing the right working model.
How should delayed telemetry affect a reliability decision?
Check when an event occurred as well as when it arrived. Treat missing or stale evidence as uncertainty until the operating model provides another reliable observation; a new arrival does not necessarily describe the current service state.
What evidence should close a distributed-service incident?
Confirm the agreed user outcome, local and central state, and any preserved work that needs reconciliation. Record which owners performed those checks and identify any remaining uncertainty before closing the incident.
Does an SLO guarantee uptime?
An SLO is a target used to evaluate service behavior and guide engineering decisions. It does not itself create a contractual service-level agreement or guarantee an outcome.
Does SRE require replacing the monitoring stack?
No. Start with the signals already available and check whether they describe the user journey and its dependencies. Add or change instrumentation when the existing evidence leaves a decision unclear.
Does this page include continuous on-call coverage?
No. Coverage windows, activation, included systems, access, escalation responsibilities and response expectations require a separate scoped agreement.
Next step
Make distributed recovery responsibilities explicit
Describe the local service, its central dependencies and what happens when connectivity or evidence is missing. Include the recovery checks and decision owners already in place.