Today we are going to do something different. Today is load testing day! Today is also the day when we will be borrowing something from the
enemy Java World: Apache JMeter. What is JMeter? Let me shamelessly paste here the definition from their website:
The Apache JMeter™ application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.
But by the way, what is load testing? To keep it simple, it’s a specific domain of testing aimed at measuring how a given system behaves when being used by a relatively large number of simultaneous users. What large number means depends entirely of the goals you want to achieve. For instance, it might be written somewhere in the specifications of your project that “the application must handle up to 100 concurrent users with a maximum response time of 1s and an average response time of 200ms”. Load testing, and tools like JMeter, can help you prove that your application just does that. And if it doesn’t, then you might want to either review your specs or get to work on that performance!
My objective for today’s post is twofold:
- Get more familiar with load testing in general. This is something I’ve never done outside of purely academic circles (read, at school) and I would like to introduce it to my day-to-day job. I think it’s bad not to test such things but let’s be honest, performance concerns are often pushed to the far end of the backlog until, well surprise surprise, things start to get ugly. Who would have thought that several users would use your app at the same time, right?
- Get started with JMeter. I requested it at work in the hope to finally be able to use it in my current project, so I figured I might take the beast for a ride at home first. My Java colleagues said good things about it and it is free, so why not?
That being said, let’s load test!
Installing and running JMeter
First things first, let’s grab JMeter from their official website:
JMeter greats you with the following screen and an empty test plan:
- Create 100 users that will constantly call our PizzaManager REST-like API for 10 seconds. The API consists of 5 methods:
- Get a pizza by name
- Get all pizzas (names only)
- Get all pizzas (full details)
- Add a new pizza to the system
- Delete a pizza by name
- Measure how our system behaves, and more particularly the following elements:
- Response time: the time elapsed from the moment a request is sent to the moment its response has been received. The lower the better.
- Throughput: the number of requests the system is able to process per second. The higher the better.
Let’s add the first JMeter component to our test plan: a Thread Group. This component will be responsible for creating the fake users who will test your application. Each user will be represented by a separate, independent thread, thus the name of the component.
(click on the .gif to enlarge)
The Number of Threads (users) parameter is explicit. Setting it to 100 will create 100 new users who will call our API in a concurrent fashion. I set the Loop Count parameter to Forever so the users keep calling the API in loop. We could have defined a predefined value instead, for instance 3. In that case each user would have made exactly 3 calls to the API. Finally, by enabling the scheduler, we can define the Duration (seconds) parameter. Here I set it to 10 so the users keep calling the PizzaManager API for 10 seconds.
It’s also worth mentioning the Ramp-up parameter. You can use it if you want to progressively reach the number of users over a given period of time. The graph below would represent the progressive creation of 100 users over a ramp-up period of 5 seconds:
Calling the API
In order to access our PizzaManager REST-like API, we need some kind of component to make create and send HTTP requests. Obviously JMeter is well equipped to do so. There are 2 components that we will be looking at: HTTP Request Defaults and HTTP Request.
the HTTP Request Defaults component is available under the Config Element category:
We can add our second component type, HTTP Request. It is available under the Sampler category.
It looks very similar to the previous component, but here we can define the exact path to our the methods we want to call as well as the appropriate HTTP verbs. Here’s the test plan hierarchy after I’ve added all 5 methods that we want to test:
The last category of components of interest to us today is what JMeter calls listeners. Those components measure criteria like response times and throughput and display the results to the users. The full list of listeners is available here. We will be using the listener called Summary Report here. The screenshot below shows how to add it to our test plan:
Running the test plan
As you can see on the top right corner, the test runs for 10 seconds and create 100 users, as we defined in the Thread Group. Let’s go over the results of the execution (I excluded some other columns for clarity purposes):
In the 10 seconds the test ran, over 8100 HTTP requests were sent to the PizzaManager (roughly 1600 requests per method), resulting in an overall throughput of 804 operations per seconds. The average response time was around 120ms with peaks between 350 and 400ms. If we were to compare those results with our fake requirements from above (“the application must handle up to 100 concurrent users with a maximum response time of 1s and an average response time of 200ms”), we could say that our application runs fast enough to satisfy the specs.
Of course, this is just an introduction and we might want to run more tests with a different amount of users and various durations to make sure that everything works as expected.
Small bonus – functional testing
Aside from load testing, you can also do functional testing with JMeter. In order to do so, you can add the View Results Tree listener to your test plan. This will record all HTTP requests made along with their responses. You can use it to make sure the API works as expected from a business point of view. If the screenshot below, we can see that one of the DeletePizza requests failed with a 500 HTTP status code (Internal Server Error).
let private pizzas = new Dictionary<string, Pizza>() let deleteCI (name:string) = let pizza = pizzas.Keys |> Seq.tryFind (fun p -> p.ToLower() = name.ToLower()) match pizza with | None -> Failure(sprintf "Failed to delete '%s'. The pizza could not be found!" name) | Some p -> pizzas.Remove(p) |> ignore Success(sprintf "'%s' was successfully deleted from the database." name)
This is because the pizza was found at the beginning of the function was being removed in the meantime by another user, before the following line was called:
pizzas.Remove(p) |> ignore
The View Results Tree component also allows use to visualize the JSON data sent as a response to a request:
This is also a useful feature for checking if your API returns the expected results.
That’s it for today’s post. I hope you found this introduction to JMeter useful. It seems to be a very powerful tool and I am looking forward to using it more regularly in the future! Please let me know if you have more experience using JMeter or other similar Software. What features do you use? How often do you use it? Do you use it in some automated way? I’d be very interested in your feedback.