Chris Ayers

Senior Site Reliability Engineer
Azure CXP AzRel
Microsoft

BlueSky: @chris-ayers.com
LinkedIn: - chris-l-ayers
Blog: https://chris-ayers.com/
GitHub: Codebytes
Mastodon: @Chrisayers@hachyderm.io
Twitter: @Chris_L_Ayers

Agenda Items

Reliability

Why Reliability Matters

A system is considered "reliable" if it can consistently serve users under normal or abnormal conditions.

• Reliability in distributed systems involves:
  • Consistent performance despite failures.
  • Graceful degradation when certain components become unavailable.
  • Rapid recovery within acceptable time limits.

Understanding Reliability and Resiliency

• Failures are inevitable in distributed systems.
• Workloads must detect, withstand, and recover from failures within acceptable timeframes.
• Ensuring availability for users to access workloads as promised.
• Failures have an impact on revenue, reputation, and customer trust.

FAILURE IS ALWAYS AN OPTION

Financial Impact of Downtime

• Revenue Loss: Downtime can cost businesses over $1 million per hour, especially for e-commerce and online services.
• Increased Expenses: Includes emergency maintenance, staff overtime, and potential penalties for SLA breaches.
• Legal Liabilities: Potential lawsuits from customers or clients affected by downtime.
• Insurance Challenges: Downtime can affect insurance coverage and premiums.
• Operational Costs: Additional costs for restoring systems, data recovery, and preventive measures to avoid future downtime.

Reliability Levels

https://uptime.is/five-nines

Service Level Objectives

Understanding RPOs and RTOs

• Recovery Time Objectives (RTOs): The maximum acceptable downtime before services must be restored.
• Recovery Point Objectives (RPOs): The amount of data that can be lost during a disruption.

Azure Cloud Architecture

Implementing Reliability

Azure-Customer Shared Responsibility Model

Azure Well-Architected Framework

• Provides best practices and guidance for building high-quality Azure solutions.
• Ensures workloads are reliable, secure, efficient, and cost-effective.

Microsoft Azure Well-Architected Framework Pillars

Design Principles for Reliable Workloads

Design for Business Requirements

• Gather business requirements focusing on the workload's intended utility.
• Cover user experience, data, workflows, and unique constraints or sensitivities.
• Clearly state expectations and confirm goals are feasible and documented.

Design for Business Requirements

Design for Resilience

• The workload must continue to operate (fully or partially) despite failures.
• Expect platform outages, resource shortages, and other faults.
• Build resiliency to ensure fault tolerance and graceful degradation.

Design for Resilience

Design for Recovery

• The workload must recover gracefully from failures of any magnitude.
• Even highly resilient systems need disaster preparedness.
• On the data layer, plan to repair or restore state if corruption occurs.

Design for Recovery

Design for Operations

• Shift left in operations to anticipate failures early.
• Test failures frequently in development.
• Ensure shared visibility across teams for dependency status.
• Use diagnostics and alerts from observable systems for rapid incident management.

Design for Operations

Design for Operations

Keep It Simple

• Avoid overengineering architecture, code, and ops.
• Simplicity reduces inefficiencies and misconfigurations.
• Maintain a balanced approach to avoid single points of failure.

Trade-Offs

Trade-Offs in Applying the Framework

• Align trade-offs with business priorities
• Use scenario planning to assess impacts
• Continuously iterate and monitor performance

Trade-Off: Reliability vs. Cost Optimization

Trade-Off: Reliability vs. Security

Trade-Off: Reliability vs. Operational Excellence

Trade-Off: Reliability vs. Performance Efficiency

Failure Mode Analysis (FMA)

We've explored design principles and tradeoffs. Next, let's discuss a crucial activity: proactively identifying and mitigating possible failure modes.

Identifying and Mitigating Failures

Proactive Failure Identification

• Recognize potential weaknesses before they cause outages.
• Tools like checklists, incident post-mortems, and dependency mapping can help.

Mitigating Strategies

• Architect for graceful degradation, fallback strategies, and clear failover mechanisms.
• Enhance observability to detect issues quickly.

Prioritizing Risks and Impacts

Risk Prioritization

• Identify high-impact risks and allocate resources efficiently.
• Distinguish between low-probability/high-impact events and daily minor failures.

Impact Mitigation

• Implement strategies to minimize consequences (e.g., immediate failover, data replication).
• Document potential risks to guide quick resolution.

Single Points of Failure (SPOFs)

• An SPOF is a system part that, if it fails, disrupts the entire workload.
• Identifying SPOFs is critical for improving reliability and availability.

Mitigating SPOFs

• Redundancy: Duplicate components or services.
• Failover Mechanisms: Seamlessly switch to backups.
• Load Balancing: Distribute traffic across multiple instances.

Failure Examples

Failure Examples

Azure Regions and Availability Zones

• Azure has 60+ regions worldwide, each in a specific geography with defined data residency boundaries.
• Regions are groups of datacenters connected by low-latency, high-capacity networks.
• Some regions are sovereign clouds with limited feature availability (e.g., Azure Government).
• Region Map

Azure Region Pairs and Nonpaired Regions

Region Pairs

• Some Azure regions are paired for built-in geo-redundancy (e.g., East US <=> West US).
• Certain services (like geo-redundant storage) rely on these pairs to replicate data.
• Region pairs help with sequential updates, prioritized recovery, and physical isolation.
• However, you still must design your own disaster recovery; pairing alone doesn't guarantee automatic failover.

Nonpaired Regions

• Many newer regions don't have a paired counterpart.
• Often, they rely on availability zones for redundancy.
• Azure services can still replicate data across any region combination.
• Choose any set of regions to meet your business, latency, and compliance needs.

Active-Active vs. Active-Passive

Azure Availability Zones

Azure Availability Zones Infrastructure

• AZs are physically separate datacenters within an Azure region.
• Each zone has its own power, cooling, and networking.
• Low-latency (<2ms) connections link them, reducing local outage impact.
• Not all regions have AZs; check availability.
• Region Support

Types of Availability Zone Support

• Zone-Redundant
• Zonal
• Zone-Resilient
• Non-Zonal (Regional)
• Services Support

Zonal Resources

• Pinned to a specific availability zone.
• Combine multiple zonal deployments for high reliability.

Zonal Resources

Customer Responsibilities:

• Deploy and manage resources in each availability zone.
• Configure and manage data replication.
• Use a load balancer to distribute requests.
• Choose active-passive or active-active models.
• Handle failover when an AZ is unavailable.

Zone-Redundant Resources

• Spread automatically across multiple availability zones.
• Microsoft manages request distribution and data replication.
• Automatic failover if a zone goes down.

Zone-Resilient vs. Non-Zonal (Regional)

Availability Zones and Azure Updates

Update Deployment

• Microsoft deploys updates to a single AZ at a time.
• Minimizes impact on active workloads.
• Running across multiple zones allows continuous service during updates.

Data Replication: Storage Options

Example Scenarios

Line-of-Business Application

• Requirements: High reliability, minimal downtime, strong performance, cost efficiency.
• Approach: Zone-redundant deployment with regional backups.

Internal Application

• Requirements: Cost sensitivity, acceptable downtime risk.
• Approach: Locally redundant deployment plus cross-region backups (or zone-redundant).

Example Scenarios

Legacy Application Migration

• Requirements: High performance, resiliency.
• Approach: Zone-redundant or zonal pinned deployments, possibly with passive failover.

Healthcare Application

• Requirements: Data residency, regulatory compliance.
• Approach: Multi-zone, multi-region with strict compliance considerations.

Example Scenarios

Banking System

• Requirements: Mission-critical, extremely high reliability.
• Approach: Multi-zone and multi-region deployment (Active-Active).

Software as a Service (SaaS)

• Requirements: Geographically distributed users, data residency constraints.
• Approach: Multi-zone, multi-region or zone-redundant single-region with global traffic distribution.

Enhancing Resiliency with Fault Isolation

• Fault isolation prevents cascading failures and maintains availability.
• Azure Availability Zones distribute workloads across multiple data centers, reducing outage risks in a single location.

Deploying Across Zones for Higher Availability

• Deploying workloads across multiple AZs strengthens availability.
• Provides built-in fault isolation to handle datacenter-specific failures.

Architecting for Reliability

• Beyond AZs and multi-region approaches, landing zones, reference architectures, and best practices further ensure resilient workloads in Azure.

Load Testing and Chaos Engineering

Load Testing

Ensuring Performance and Scalability

• Evaluates system behavior under peak conditions.
• Identifies performance bottlenecks.
• Guides infrastructure and capacity planning.
• Ensures reliable performance under heavy workloads.

Load Testing Strategies

Tools for Load Testing

• Popular choices: JMeter, LoadRunner, Locust, Gatling.
• Azure Load Testing leverages JMeter and Locust.
• Choose a tool based on your needs and budget.

Chaos Engineering

After validating performance, we can further strengthen resilience by deliberately introducing controlled failures.

• Chaos Engineering tests how systems behave under unexpected conditions.
• Identifies weak points in architecture and processes.
• Fosters a culture of experimentation and continuous improvement.

Principles of Chaos Engineering

• Simulate real-world failures to validate resilience.
• Conduct controlled experiments, preferably in production with safety measures.
• Ensure minimal user impact through strict guardrails.

Benefits of Chaos Engineering

• Uncovers unforeseen issues before they affect customers.
• Encourages proactive system improvements and learning.
• Strengthens organizational readiness for incident response.

Tools for Chaos Engineering

• Azure Chaos Studio for Azure workloads.
• Other tools:
  • Gremlin
  • Chaos Monkey
  • LitmusChaos

Integrating Testing into Operations

• Combine load testing and chaos engineering into continuous practices.

• Validate system resilience comprehensively and frequently.

Reference Architectures & Resources

• Utilize established Azure reference architectures for reliability.

• Leverage available documentation and best practices for consistency and effectiveness.

Azure Landing Zones

Secure Structure

• Azure Landing Zones create a secure, organized foundation for Azure environments.
• Enforce identity, network, and governance policies at scale.

Reliable Implementation

• A standardized environment supports consistent deployments.
• Simplifies resource management and reduces misconfigurations.

Azure Architecture Center

• Centralized repository of best practices, reference architectures, and design patterns for Azure solutions.
• Provides comprehensive guidance on building reliable, scalable, and secure cloud solutions.
• Offers scenario-specific architectures, recommended practices, and templates to accelerate your workload design.

Explore Azure Architecture Center

Well-Architected Workloads

• Align workloads with business outcomes using the Azure Well-Architected Framework
• Balancing functional requirements and nonfunctional trade-offs
• Integrate design fundamentals, trade-offs, and operational best practices
• Well-Architected Workloads

Mission-Critical Workloads

Learn about designing mission-critical applications on Azure for high availability, reliability, and performance:

Mission-Critical Workloads

Enterprise Web App Patterns

Guidance for web apps on Azure, offering prescriptive architecture, code, and configuration aligned with the Well-Architected Framework:

Enterprise Web App Patterns

Azure Verified Modules

• Pre-built, tested, and Azure-verified modules that accelerate development.
• Ensures compliance with best practices and standards.

Azure Verified Modules

APRL: Azure Platform Resiliency Library

• Curated catalog of resiliency recommendations, plus Azure Resource Graph queries for identifying non-compliant resources.
• Guidance by resource type, specialized workloads, WAF best practices, and automation scripts.

APRL: Azure Platform Resiliency Library

Azure Review Checklists

• Use these to ensure your architecture aligns with best practices.
• Covers diverse Azure services to catch issues before they become critical.

Azure Review Checklists

Conclusion

• Design Principles: Focus on resilience, recovery, and simplicity for robust workloads.
• Tradeoffs: Every decision impacts cost, security, operational excellence, and performance.
• Proactive Reliability: Utilize Failure Mode Analysis to anticipate and mitigate failures.
• Continuous Improvement: Leverage Availability Zones, multi-region deployments, chaos engineering, and load testing to enhance reliability.

Next Steps

• Review your architecture with Azure Review Checklists.
• Validate designs against Microsoft's established Reference Architectures.
• Implement regular chaos engineering and load testing practices.
• Stay informed on reliability best practices and continuously refine your approach.

Resources