Automated_Load_testing_with_AKS_8a1ce79726.jpg

Automated load testing with Locust and Kubernetes via AKS

In this article we will walk through of how to set up load testing with Locust and Kubernetes via AKS. The cost should be minimal and 100% automated, covering the deployment, monitoring and reporting.

 

Introduction

 

We are all searching for the perfect tools or frameworks for our work. In the context of load testing, what we look for is something powerful and simple, yet flexible at the same time. Ideally, you would just need edit a few variables in a configuration file, run a simple command line, get the results and be done! 😺

Oh right.. also you might want to keep a low cost, and only pay for what you need when you need it, no more, no less 💸

On the same concept as my previous article with parallel UI test automation, we will use:

  • Locust.io as our load test scripting framework;
  • Kubernetes via Azure Kubernetes Service aka. AKS;
  • PowerShell Core to automate the whole thing with the help of Azure CLI and kubectl.

 


 

Goal

 

What we want to achieve here is a simple way of running a load test script with thousands of users in distributed mode while just running a simple command line.

In the background, this is what will play out:

  • Creation of K8s cluster (defined with the needed number of nodes);
  • Deploy the server and then clients that will generate the load;
  • Wait for the load test to finish;
  • Retrieve the results;
  • Dispose the K8s cluster and associated resources

 


 

Just a few words about Locust

 

I must say this is by far the best framework for load testing I have encountered so far 🤯

It benefits from different features / capabilities:

  • The use of Python scripts which makes it simple, yet flexible at the same time;
  • Runs in distributed mode, essential for running a large number of users;
  • Has a wide range of configuration possible via command line / configuration file or environment variables;
  • Allows for customization and extensions on different parts of its architecture (logging, event hooks, clients etc.)
  • Though we will stick with the headless mode in this article, Locust also has a nice web-based UI.

I invite you to have look at their website and their documentation for more info 📘

 


 

Requirements

 

In order to experiment with this setup, the only thing that is required is an Azure subscription in order to use AKS.

If Azure is not an option, then you’ll have to adapt the PowerShell script AKSService.ps1 to communicate with another service (GKE, EKS…)

 

Setup

 

Our setup is pretty simple and only consist of four distinct sections:

  • Configuration files, one per environment (including local);
  • A set of PowerShell scripts to automate the flow from the cluster creation, to its disposal, including the execution of the load test;
  • Our Python load test scripts;
  • K8s template deployment files.

 


 

Flow

 

Here’s an overview of the flow automated by the PowerShell script:

Flow and architecture overview

  1. Cluster creation via AKS
  2. Creation of the resources needed for Locust server and clients, including a persistent volume for storing the reports;
  3. Creation of the server and mounting the necessary resources;
  4. Creation of the clients and mounting the necessary resources;
  5. Running the load test script previously mounted;
  6. Wait until the load test is done and export the results stored in the persistent volume;
  7. Dispose the cluster and all associated resources.

 


 

Configuration file

 

Here’s how one of our configuration files look like:

{
    "cluster":
    {
        "resourceGroup": "n/a",
        "name": "your-cluster-name",
        "nodeCount": 1,
        "vmSize": "n/a",
        "wait": true,
        "isLocal": true
    },
    "template": 
    {
        "image": "your-docker-image",
        "replicas": 5,
        "download-resources": true
    },
    "server": 
    {
        "master": true,
        "headless": true,
        "host": "https://your-base-url.com",
        "expect-workers": 5,
        "users": 20,
        "spawn-rate": 1,
        "run-time": "1m",
        "print-stats": true,
        "csv": "/reports/load-test",
        "html": "/reports/load-test.html"
    },
    "client": 
    {
        "worker": true,
        "headless": true,
        "host": "https://your-base-url.com"
    },
    "script": "../locust-scripts/simple/locustfile.py"

}

It contains five different sections:

  • The cluster parameters: resource group, name, VM size, number of nodes… Additionally, if you run locally, you can set isLocal to true.
  • The template parameters to be replaced in our K8s templates;
  • The Locust server parameters. These will be used to create a locust.conf file only available to the server. It is where you specify how many users will be simulated, how long, the spawn rate, and the reports that will be generated etc.. You can find the complete list of parameters available in the Locust documentation.
  • The Locust clients parameters. Not much needed there. Only specifying headless and the base URL is enough.
  • The path to our Locust script.

 

We have a few yaml templates needed in order to make things work properly. This includes:

  • A PV/PVC for storing the reports at the end of the load test
  • A pod template for the Locust server
  • A deployment template for the Locust clients
  • Two services to allow communication between the server and clients.

Two things of note there:

  • Some parameters are set directly from the configuration file, such as {replicas}, {image} or {download-resources}. Can be useful to define extra environment variables to be used directly as part of our locust load test scripts.

 


 

Deployment files

 

We have a few yaml templates needed in order to make things work properly. This includes:

  • A PV/PVC for storing the reports at the end of the load test
  • A pod template for the Locust server
  • A deployment template for the Locust clients
  • Two services to allow communication between the server and clients.

Two things of note there:

  • Some parameters are set directly from the configuration file, such as {replicas}, {image} or {download-resources}. Can be useful to define extra environment variables to be used directly as part of our locust load test scripts.
apiVersion: "apps/v1"
kind: Deployment
metadata:
  name: locust-client-deployment
spec:
  replicas: {replicas}
  ...
    spec:
      containers:
      - image: {image}
        ...
        env:
          ...
          - name: LOCUST_DOWNLOAD_RESOURCES
            value: "{download-resources}"
        ...

NB: LOCUST_MASTER_NODE_HOST should have the same value as the ip defined in our K8s service so that the clients can reach out to the server ⚠️

...
env:
  - name: LOCUST_MASTER_NODE_HOST
    value: "10.96.0.2"
  ...
apiVersion: v1
kind: Service
metadata:
  name: locust-server-service-connect
spec:
  ports:
    - port: 5557
      protocol: TCP
      targetPort: 5557
  clusterIP: 10.96.0.2
  ...

 


 

PowerShell Scripts

 

The PowerShell script is pretty similar to the one in my previous article, and it is maybe not so relevant to show some code in here. What is important, is that they are at the core of the automation and control the whole execution flow.

 

Docker image

 

The official locust docker image is mounted as read-only, and runs a non-root user for security purpose. This poses a problem since it prevents us from writing, not only in the container itself, but also to the persistent volume we mounted to our container.

Since our load test is quite ephemeral, we don’t really need to achieve a high security level to be fair. We can then use our own Docker image running a root user instead and adding any extra 3rd party library needed for our load test scripts 💡

FROM locustio/locust

RUN pip3 install beautifulsoup4

USER root
ENTRYPOINT ["locust"]

ENV PYTHONUNBUFFERED=1

On top of that, you can add any 3rd party libraries you may need for your Python script, like in this example with beautifulsoup4 for example. Feel free to use mine in Docker hub

 


 

Results

 

Currently, the server will generate 5 different files by default. This includes csv reports with exceptions, failures, and stats per page, along with an html report showing the trend over time 💹

reports
│   load-test_exceptions.csv
│   load-test_failures.csv
│   load-test_stats_history.csv
│   load-test_stats.csv
│   loadt-test.html

Charts report.png

Locust provides also some good customization when it comes to reporting, which can be useful if one needs to log different data or output them in a different format.

 


 

Conclusion

 

To this day, Locust is probably one of the best load testing framework to use in combination of Kubernetes. It has all the qualities for that, and I don’t think there is much arguing about that 😎

Earlier articles from other sources already explained how to deploy Locust in a Kubernetes environment. What I wanted to achieve here is a simple way to automate the whole flow and keep the complexity as low as possible instead.

The real added value of this article is takes place on other levels:

  • Scripting all the way up from the cluster creation to its disposal while keeping simplicity with a single command line and configuration file.
  • Avoiding using cloud-based load testing tools and all the costs associated to them.
  • Experimenting with the possibilities offered by PowerShell, Azure CLI and Kubernetes.

 

Code

 

All the code is available in my GitHub repo for those who wish to experiment and orient their strategy toward load testing in CI environment.

 

Going further

 

From there, it should not be so difficult to integrate this into your CI/CD pipeline. There are a couple of easy alternatives:

  1. Parse the reports generated, and pass/fail accordingly.
  2. Use Locust capabilities for controlling the process exit code and adapt it to your own context.

See Locust documentation for more info 🔧

 

References

 

https://github.com/deliveryhero/helm-charts/tree/master/stable/locust https://medium.com/locust-io-experiments/locust-io-experiments-running-in-kubernetes-95447571a550