Site reliability engineering practice / Vancouver
Site reliability engineering for Vancouver teams.
Connected services can fail between components that each appear healthy. Make queues, retries and partner boundaries part of the reliability model. Use objectives and recovery evidence to understand whether a complete user journey works when a dependency slows down or becomes unavailable.
What this practice covers
What SRE covers
Reliability across integrations depends on more than endpoint availability. Describe how work moves through services, where it can wait and what users see when a dependency fails. Use that model to choose indicators, isolate failures and verify completed recovery.
Reliability across asynchronous and partner boundaries
An asynchronous workflow needs a definition of completion and an acceptable delay. Queue age, failed work and end-to-end completion can reveal problems that a component health check misses.
Hypothetical example: an application accepts a request while a partner endpoint is unavailable. The request remains queued. Reliability measurement should distinguish accepted work from completed work and make the waiting state visible.
Retries, timeouts and isolation need deliberate boundaries. Repeating failed work without limits can increase load or repeat side effects; isolating a dependency can keep an unrelated journey usable.
Recovery includes checking the backlog and the final user outcome. A restored endpoint is only part of the evidence when work can remain delayed, duplicated or incomplete.
- Completion objective
- Measure the complete workflow and define which delays count against the objective.
- Failure boundary
- Identify where retries stop, work is isolated and ownership passes between teams.
- Recovery evidence
- Confirm that delayed work is reconciled and the intended user outcome is restored.
What that gives your team
- A clear distinction between accepted and completed work
- Explicit ownership across integration boundaries
- Recovery checks that include the backlog
Service indicators and objectives
Define a service-level indicator (SLI) for a user journey, then set a service-level objective (SLO) over an agreed period. Use the objective to make reliability decisions explicit.
Error budgets and change decisions
An error budget is the unreliability allowed by an SLO over its measurement period. Use budget consumption alongside incident evidence to decide when to prioritize reliability work or proceed with change.
Dependency and failure analysis
Examine how timeouts, queues, retries and upstream failures affect the complete service journey. Identify where isolation, graceful degradation or recovery checks are needed.
Operational toil and automation
Identify repetitive operational work that is manual, reactive and offers little lasting value. Consider automation or system changes where they reduce that work without hiding risk.
Incident readiness and learning
Connect usable alerts, runbooks, escalation paths and decision owners. Turn incident findings into tracked engineering work and verify whether the changes address the failure pattern.
Production change and verification
Define the expected behavior, review signals during change and agree when to stop or roll back. Verify recovery and record remaining risks before closing the work.
How it works
How SRE work moves
Trace a connected workflow, choose the failure boundary to improve and verify the outcome beyond component recovery.
Trace work across dependencies.
Map synchronous calls, queues and partner handoffs. Define completion, acceptable delay and the owner responsible at each boundary.
Bound the failure behavior.
Review timeouts, retries and isolation against incident evidence. Choose a change and define the signals that show whether it contains the failure.
Verify end-to-end recovery.
Check the backlog, repeated side effects and final user outcome. Update runbooks and assign follow-up across the affected owners.
Engagement models
Choose how we work together
Use consulting for a bounded assessment of architecture or dependency decisions. Other working models can address engineering contribution, a defined implementation or scoped operational responsibility. Make partner interfaces and client decision authority explicit in every model.
Assessment, architecture review and technical direction for a defined reliability decision.
ExploreA model for engineering contribution inside a client-directed team, with responsibilities and availability agreed separately.
ExploreA model for scoped production response with agreed systems, activation paths and responsibility boundaries.
ExploreA model for ownership of a bounded engineering workstream, with decision rights and handback conditions defined.
ExploreA model for a defined technical outcome with implementation scope, acceptance checks and operating handoff.
ExploreA model for ongoing maintenance, troubleshooting and improvement of named systems under agreed terms.
ExploreQuestions answered
SRE questions, answered
Practical questions about service objectives, reliability work and choosing the right working model.
How should an asynchronous workflow be measured?
Define the event that starts the journey, the evidence of completion and the acceptable delay. Measure incomplete or late work against that definition rather than treating an accepted request as a completed outcome.
What does failure isolation mean for connected services?
It means limiting how a dependency failure affects other work. Timeouts, bounded retries and separated processing can be part of that design; the right boundary depends on side effects, user impact and recovery needs.
Does an SLO guarantee uptime?
An SLO is a target used to evaluate service behavior and guide engineering decisions. It does not itself create a contractual service-level agreement or guarantee an outcome.
Does SRE require replacing the monitoring stack?
No. Start with the signals already available and check whether they describe the user journey and its dependencies. Add or change instrumentation when the existing evidence leaves a decision unclear.
Does this page include continuous on-call coverage?
No. Coverage windows, activation, included systems, access, escalation responsibilities and response expectations require a separate scoped agreement.
Next step
Make a dependency boundary easier to operate
Describe the workflow, the systems it crosses and what users experience when a dependency fails. Include any queue, retry or recovery evidence your team already has.