Building a scalable and high available system will require some sort of load balancing. For one it gives you the ability to scale out to tackle high loads and it allows you to do deployments without having any downtime:
- Bootstrap new box and wait until ready
- Put new box into load balancer, observe
- Gracefully take old box out of load balancer
- Destroy old box
Every developer who takes devops seriously knows that you have to have a staging environment. Since your staging environment should be as close to the production, we decided to put a load balancer in front of our staging environment, like we do it fro prod. Why you ask? Because our load balancers are (in our case) not transparent: They
- Do TLS termination
- Perform request throttling
- Add http headers (like client’s IP address)
Our system has two HTTP endpoints: Our website (customer acquisition and delivery of our browser app) and our API. As a start-up you have limited resources, especially when you are in Beta phase. So we decided in favor of HAproxy running on an EC2 instance, instead of a managed Elastic load balancer (ELB).
Important things to mention why ELB might be a better pick for you: You can add or remove instances conveniently via the AWS API, so it can be part of your Continuous Deployment pipeline. Plus it is a load balancer with better uptime (multi AZ, managed service, …). However, you might not go for the ELB if you do not want to invest too much money, when you have only one box running behind the ELB 99% of the time. So we decided to go with HAproxy.
We came up with an automation for the inclusion / removal of back-end servers in the HAproxy config.
Since our boxes are managed via chef, we can use the
knife search node feature.
Assuming you also use chef (not chef-solo), you can easily query for other nodes in a recipe:
You see the
NOT exclude_from_loadbalancer thing? That is a part of the search
query that tells chef to not return nodes that have set an attribute called
true. We set this attribute to
per default for all our boxes through a base-recipe, but you can also leave it
undefined. By having this toggle, we can take nodes gracefully out of the load balancer
true in the
on the node, then re-run
chef-client on the load balancer.
And this is the excerpt from the haproxy.cfg.erb template file file:
You either let
chef-client run periodically on the load balancer, or you can
knife ssh "role:loadbalancer" "sudo chef-client".
blog comments powered by Disqus