Open-source News

How to use a circuit breaker pattern for site reliability engineering Robert Kimani Thu, 06/23/2022 - 03:00 Register or Login to like Register or Login to like

Distributed systems have unique requirements for the site reliability engineer (SRE). One of the best ways to maintain site reliability is to enforce a certain set of best practices. These act as guidelines for configuring infrastructure and policies.

What is a circuit breaker pattern?

A circuit breaker pattern saves your service from halting or crashing when another service is not responding.

Image by:

(Robert Kimani, CC BY-SA 40)

For instance, in the example image, the business logic microservices talk to the User Auth service, Account service, and Gateway service.

As you can see, the "engine" of this business application is the business logic microservice. There's a pitfall, though. Suppose the account service breaks. Maybe it runs out of memory, or maybe a back end database goes down. When this happens, the calling microservice starts to have back pressure until it eventually breaks. This results in poor performance of your web services, which causes other components to break. A circuit breaker pattern must be implemented to rescue your distributed systems from this issue.

Implementing a circuit breaker pattern

When a remote service fails, the ideal scenario is for it to fail fast. Instead of bringing down the entire application, run your application with reduced functionality when it loses contact with something it relies upon. For example, keep your application online, but sacrifice the availability of its account service. Making a design like a circuit breaker helps avoids cascading failure.

Here's the recipe for a circuit breaker design:

1. Track the number of failures encountered while calling a remote service.

2. Fail (open the circuit) when a pre-defined count or timeout is reached.

3. Wait for a pre-defined time again and retry connecting to the remote service.

4. Upon successful connection, close the circuit (meaning you can re-establish connectivity to the remote service.) If the service keeps failing, however, restart the counter.

Instead of hanging on to a poorly performing remote service, you fail fast so you can move on with other operations of your application.

More DevOps resources What is DevOps? The ultimate DevOps hiring guide DevOps monitoring tools guide A guide to implementing DevSecOps Download the DevOps glossary eBook: Ansible for DevOps Latest DevOps articles Open source circuit breaker pattern

There are very specific components provided by open source to enable circuit breaker logic in your infrastructure. First, use a proxy layer, such as Istio. Istio is a technology independent solution and uses a  "Blackbox" approach (meaning that it sits outside your application).

Alternately, you can use Hystrix, an open source library from Netflix that's widely and successfully used by many application teams. Hystrix gets built into your code. It's generally accepted that Hystrix can provide more functionality and features than Istio, but it must be "hard coded" into your application. Whether that matters to your application depends on how you manage your code base and what features you actually require.

Implementing effective load balancing

Load balancing is the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing as efficient as possible. Load balancing can optimize response time, and avoid unevenly overloading some computer nodes while other computer nodes are left idle.

Image by:

(Robert Kimani, CC BY-SA 40)

Users on the left side talk to a virtual IP (vIP) on a load balancer. The load balancer distributes tasks across a pool of systems. Users are not aware of the back-end system, and only interact with the system through the load balancer.

Load balancing helps to ensure high availability (HA) and good performance. High availability means that failures within the pool of servers can be tolerated, such that your services remain available a high percentage of the time. By adding more servers to your pool, effective performance is increased by horizontally spreading the load (this is called horizontal scaling.)

 3 ways to do load balancing

1. DNS Load Balancing: This is the most straight forward and is free. You do not have to purchase any specialized hardware because it's the basic functionality of DNS protocol, you'd simply return multiple A-records An A-record maps a hostname to an IP address. When the DNS server returns multiple IP addresses for a single hostname, the client will choose one IP address at random, in other words the load balancing is automatically done for you from the client.

2. Dedicated Hardware-Based Load Balancer: This is what is typically used within a data center, it's feature-rich and highly performant.

3. When you have a Software Load Balancer, you do not need to have a dedicated hardware for this type of load balancing, you can simply install the load balancer software on commodity hardware and use it for your load balancing use cases.

Nginx and HAProxy are mostly used as software load balancers and can be very useful in development environments where you can quickly spin up their functionality.

DNS load balancing

A DNS server may return multiple IP addresses when a client looks up a hostname. This is because in a large-scale deployment, a network target isn't just running on one server or even just on one cluster. Generally, the client software chooses an IP address at random from the addresses returned. It's not easy to control which IP a client chooses.

Unless you have custom software built to determine the health of a server, pure DNS load balancing doesn't know whether a server is up or down, so a client could be sent to a server that is down.

Clients also cache DNS entries by design, and it's not always easy to clear them. Many vendors offer the ability to choose which datacenter a client gets routed to based on geographic location. Ultimately, an important way to influence how clients reach your services is through load balancing.

Dedicated load balancers

Load balancing can happen at Layer3 (Network), Layer4 (Transport), or Layer7 (Application) of the OSI (Open Systems Interconnection) model.

• An L3 load balancer operates on the source and destination IP addresses.
• An L4 load balancer works with IP and port numbers (using the TCP protocol).
• An L7 load balancer operates at the final layer, the application layer, by making use of the entire payload. It makes use of the entire application data, including the HTTP URL, cookies, headers, and so on. It's able to provide rich functionality for routing. For instance, you can send a request to a particular service based on the type of data that's coming in. For example, when a video file is requested, an L7 load balancer can send that request to a streaming appliance.

There are 5 common load balancing methods in use today:

• Round Robin: This is the simplest and most common. The targets in a load balancing pool are rotated in a loop.
• Weighted Round Robin: Like a round robin, but with manually-assigned importance for the back-end targets.
• Least Loaded or Least Connections: Requests are routed based on the load of the targets reported by the target back-end.
• Least Loaded with Slow Start or Least Response Time: Targets are chosen based on its load, and requests are gradually increased over time to prevent flooding the least loaded back-end.
• Utilization Limit: Based on queries per second (QPS), or events per second (EPS), reported by the back-end.

There are software-based load balancers. This can be software running on commodity hardware.You can pick up any Linux box and install HAProxy or Nginx on it. Software load balancers tend to evolve quickly, as software often does, and is relatively inexpensive.

You can also use hardware-based, purpose-built load balancer appliances. These can be expensive, but they may offer unique features for specialized markets.

Transitioning to Canary-based Deployment

In my previous article, I explained how Canary-based deployments help the SRE ensure smooth upgrades and transitions. The exact method you use for a Canary-based rollout depends on your requirements and environment.

Generally, you release changes on one server first, just to see how the deployed package fares. What you are looking for at this point is to ensure that there are no catastrophic failures, for example, a server doesn't come up or the server crashes after it comes up.

Once that one server deployment is successful, you want to proceed to install it in up to 1% of the servers. Again this 1% is a generic guideline but it really depends on the number of servers you have and your environment. You can then proceed to release for early adaptors if it's applicable in your environment.

Finally, once canary testing is complete, you'd want to release for all users at that point.

Keep in mind that a failed canary is a serious issue. Analyze and implement robust testing processes so that issues like this can be caught during testing phase and not during canary testing.

Reliability matters

Reliability is key, so utilize a circuit breaker pattern to fail fast in a distributed system. You can use Hystrix libraries or Istio.

Design load balancing with a mix of DNS and dedicated load balancers, especially if your application is web-scale. DNS load balancing may not be fully reliable, primarily because the DNS servers may not know which back-end servers are healthy or even up and running.

Finally, you must use canary releases, but remember that a canary is a not a replacement for testing.

Learn what a circuit breaker pattern is, how to implement it, and how to do load balancing.

Image by:

Opensource.com

What to read next What you need to know about site reliability engineering How SREs can achieve effective incident response A site reliability engineer's guide to change management This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Inscryption Mac and Linux ports now available - Hypertext  htxt.africa

500 Individuals Received Training & Certification Scholarships by The Linux Foundation  It's FOSS News

The post How to Install SUSE Linux Enterprise Server 15 SP4 first appeared on Tecmint: Linux Howtos, Tutorials & Guides .

SUSE Enterprise Linux Server (SLES) is a modern and modular Linux distribution that was developed mainly for servers and mainframes. It focuses on supporting production workloads and is typically used by large organizations to

The post How to Install SUSE Linux Enterprise Server 15 SP4 first appeared on Tecmint: Linux Howtos, Tutorials & Guides.

Adding to the list of features slowly building up that will be destined for the Linux 5.20 cycle, Jens Axboe has queued up the support for async buffered writes with XFS when using IO_uring can deliver some significant performance advantages...

Linux 5.20 To Support Async Buffered Writes For XFS + IO_uring For Big Performance Boost  Phoronix

Linux Foundation works toward improved data-center efficiency  Network World

Linux Foundation works toward improved data-center efficiency  Network World

OpenZFS 2.1.5 was released this afternoon as the newest maintenance release for this open-source ZFS file-system implementation that currently works on Linux and FreeBSD systems...

If you are interested in online and in-person training and certifications in open source software development and key open source software, such as Linux and Kubernetes, see our special discount just for readers of this post. Scroll to the end.

.avia-image-container.av-l4r7y18w-23bd95cd1b4c332d68794621edf0f908 .av-image-caption-overlay-center{ color:#ffffff; } #top .flex_column.av-l4r8i0um-ffd5f33c11d6b8121ad0af9aa8165bbb{ margin-top:0px; margin-bottom:0px; } .responsive #top #wrap_all .flex_column.av-l4r8i0um-ffd5f33c11d6b8121ad0af9aa8165bbb{ margin-top:0px; margin-bottom:0px; }

Tomorrow night, in the skies over Congress Bridge in Austin, Texas, 300 drones will work in concert to provide a lightshow to entertain but also inform about the power of open source software to drive innovation in our world, making an impact in every life, every day.

#top .flex_column.av-l4r8idp6-d2d10d1a72bde363213590ecfac5a1b2{ margin-top:0px; margin-bottom:0px; } .responsive #top #wrap_all .flex_column.av-l4r8idp6-d2d10d1a72bde363213590ecfac5a1b2{ margin-top:0px; margin-bottom:0px; } #top .flex_column.av-ee45y-db8219349ecb83f3fecf044f511bb21a{ margin-top:0px; margin-bottom:5px; } .responsive #top #wrap_all .flex_column.av-ee45y-db8219349ecb83f3fecf044f511bb21a{ margin-top:0px; margin-bottom:5px; }

Backing up a bit, open source software often conjures up inaccurate visions and presumptions that just aren’t true. No need to conjure those up – we all know what they are. The reality is that open source software (OSS) has transformed our world and become the backbone of our digital economy and the foundation of our digital world. 

The reality is that open source software (OSS) has transformed our world and become the backbone of our digital economy and the foundation of our digital world. 

Some quick, fun facts: 

• In vertical software stacks across industries, open source penetration ranges from 20 to 85 percent of the overall software used
• Linux fuels 90%+ of web servers and Internet-connected devices
• The Android mobile operating system is built on the Linux kernel
• Immensely popular libraries and tools to build web applications, such as: AMP, Appium, Dojo, jQuery, Marko, Node.js and so many more are open source
• The world’s top 100 supercomputers run Linux
• 100% of mainframe customers use Linux
• The major cloud-service providers – AWS, Google, and Microsoft – all utilize open-source software to run their services and host open-source solutions delivered through the cloud

Open source software is about organizations coming together to collectively solve common problems so they can separately innovate and differentiate on top of the common baseline. They see they are better off pooling resources to make the baseline better. Sometimes it is called “coopetition.” It generally means that while companies may be in competition with each other in certain areas, they can still cooperate on others.

I borrowed from a well-known tagline from my childhood in the headline – open source does bring good things to life. 

Fueling Drone Innovation 

Drones were introduced to the world through military applications and then toys we could all easily fly (well, my personal track record is abysmal). But the reality is that drones are seeing a variety of commercial applications, such as energy facility inspection for oil, gas, and solar, search and rescue, firefighting, and more, with new uses coming online all of the time. We aren’t at The Jetsons level yet, but they are making our lives easier and safer (and some really cool aerial shots).

Much of that innovation comes from open source coopetition. 

The Linux Foundation hosts the Dronecode Foundation, which fosters open source code and standards critical to the worldwide drone industry. In a recent blog post, the general manager, Ramón Roche, discusses some of the ways open source has created an ecosystem of interoperability,  which leads to users having more choice and flexibility. 

Building the Foundation

Ramón recounts how it all started with the creation of Pixhawk, open standards for drone hardware, with the goal to make drones fly autonomously using computer vision. Working to overcome the lack of computing power and technology in 2008, Lorenz Meier, then a student, set out to build the necessary flight control software and hardware. Realizing the task’s scale, he sought the help of fourteen fellow students, many of whom were more experienced than him, to make it happen. They built Pixhawk and kick started an open source community around various technologies. It, “enabled talented people worldwide to collaborate and create a full-scale solution that was reusable and standardized. By giving their technology a permissive open source license, they opened it to everyone for use and collaboration.”

Benefits of Openness in the Real World

The innovation and technological backbone we see in drones is thanks to open software, hardware, and standards. Dronecode’s blog has interviews with Max Tubman of Freefly Systems talks about how open standards are enabling interoperability of various payloads amongst partners in the Open Ecosystem. Also, Bobby Watts of Watts Innovation explains the power of standardization and how it has streamlined their interoperability with other ecosystem partners like Gremsy and Drone Rescue Systems.

The innovation and technological backbone we see in drones is thanks to open software, hardware, and standards

Check out both interviews here and read about what is next.

The story of open source driving innovation in the drone industry is just one of thousands of examples of how open source is driving global innovation. Whether you know it or not, you use open source software every minute of every hour of every day.

Training promo

Use promo code DRONE25 here to receive up to 25% off of Linux Foundation’s training, taken by millions of students around the world. Expires on June 30, 2022. View the whole catalog, from AI and blockchain to web and application development, we have something for you.  

The post Open Source Brings Good Things to Life appeared first on Linux Foundation.

Zoom Is Shutting Down Its Chromebook App: What This Means for Users  MUO - MakeUseOf

KaOS Linux 2022.06 Released with KDE Plasma 5.25 Desktop and Calamares 3.3 Installer  9to5Linux

RISC-V sets out more open-source CPU specs  The Register

Last year I signed up for SpaceX's Starlink satellite Internet service with hopes of using it to replace the Internet connection used for running Phoronix. After months of using Starlink and carrying out thousands of benchmarks, Starlink in the US midwest / Chicagoland area has proven reliable but the performance can be rather volatile still and it was frustrating at first waiting for some Starlink accessories to ship, but the self-service nature and simplicity of the setup were great.

Cardano Foundation joins The Linux Foundation as a Gold Member  Cardano Feed

WSL2 on Windows Server 2022 hits Windows Update  The Register

Back in March there were Linux kernel patches posted for review to boot Linux on the Nintendo Wii U game console after this Linux porting work has long been done out-of-tree. A new iteration of those Wii U enablement patches for the Linux kernel have now been posted...

Linux Software Market SWOT Analysis by Size, Status and Forecast to 2021-2030 | IBM, Oracle, Novell – Indian Defence News  Indian Defence News

Wine developers have released VKD3D 1.4, the newest version of their Direct3D 12 on Vulkan implementation that is useful with Wine for enjoying newer Windows games on Linux...

Installing MetaTrader 4 on Linux  EarnForex News