Load-testing the PizzaManager REST-like API with JMeter

Today we are going to do something different. Today is load testing day! Today is also the day when we will be borrowing something from the enemy Java World: Apache JMeter. What is JMeter? Let me shamelessly paste here the definition from their website:

The Apache JMeter™ application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

But by the way, what is load testing? To keep it simple, it’s a specific domain of testing aimed at measuring how a given system behaves when being used by a relatively large number of simultaneous users. What large number means depends entirely of the goals you want to achieve. For instance, it might be written somewhere in the specifications of your project that “the application must handle up to 100 concurrent users with a maximum response time of 1s and an average response time of 200ms”. Load testing, and tools like JMeter, can help you prove that your application just does that. And if it doesn’t, then you might want to either review your specs or get to work on that performance!

My objective for today’s post is twofold:

  • Get more familiar with load testing in general. This is something I’ve never done outside of purely academic circles (read, at school) and I would like to introduce it to my day-to-day job. I think it’s bad not to test such things but let’s be honest, performance concerns are often pushed to the far end of the backlog until, well surprise surprise, things start to get ugly. Who would have thought that several users would use your app at the same time, right?

  • Get started with JMeter. I requested it at work in the hope to finally be able to use it in my current project, so I figured I might take the beast for a ride at home first. My Java colleagues said good things about it and it is free, so why not?

That being said, let’s load test!

Installing and running JMeter

First things first, let’s grab JMeter from their official website:

You can unzip it somewhere on your local machine and run the GUI client by using the ApacheJMeter.jar file from the bin folder, as shown below:

Setting up a new test plan

JMeter greats you with the following screen and an empty test plan:

Let’s create a test plan that does the following:

  • Create 100 users that will constantly call our PizzaManager REST-like API for 10 seconds. The API consists of 5 methods:
    • Get a pizza by name
    • Get all pizzas (names only)
    • Get all pizzas (full details)
    • Add a new pizza to the system
    • Delete a pizza by name
  • Measure how our system behaves, and more particularly the following elements:
    • Response time: the time elapsed from the moment a request is sent to the moment its response has been received. The lower the better.
    • Throughput: the number of requests the system is able to process per second. The higher the better.

Creating users

Let’s add the first JMeter component to our test plan: a Thread Group. This component will be responsible for creating the fake users who will test your application. Each user will be represented by a separate, independent thread, thus the name of the component.

(click on the .gif to enlarge)

We now need to configure the Thread group according to our requirements above:

The Number of Threads (users) parameter is explicit. Setting it to 100 will create 100 new users who will call our API in a concurrent fashion. I set the Loop Count parameter to Forever so the users keep calling the API in loop. We could have defined a predefined value instead, for instance 3. In that case each user would have made exactly 3 calls to the API. Finally, by enabling the scheduler, we can define the Duration (seconds) parameter. Here I set it to 10 so the users keep calling the PizzaManager API for 10 seconds.

It’s also worth mentioning the Ramp-up parameter. You can use it if you want to progressively reach the number of users over a given period of time. The graph below would represent the progressive creation of 100 users over a ramp-up period of 5 seconds:

By setting the parameter to 0, I declare that I want all my users to be created from the very start of the test.

Calling the API

In order to access our PizzaManager REST-like API, we need some kind of component to make create and send HTTP requests. Obviously JMeter is well equipped to do so. There are 2 components that we will be looking at: HTTP Request Defaults and HTTP Request.

the HTTP Request Defaults component is available under the Config Element category:

It allows to configure default HTTP settings for all the HTTP requests added to the current Thread Group. Here’s how ours looks like:

We just defined the protocol, server name and port number we want to use to access our API (http://localhost:9000/).

We can add our second component type, HTTP Request. It is available under the Sampler category.

It looks very similar to the previous component, but here we can define the exact path to our the methods we want to call as well as the appropriate HTTP verbs. Here’s the test plan hierarchy after I’ve added all 5 methods that we want to test:

And here are the details for each element, in the same order as above:

The POST method is a bit particular because it also features a JSON body, as shown below:

And that’s it for the configuration of our calls. Time to go over the last step: measuring!

Measuring performance

The last category of components of interest to us today is what JMeter calls listeners. Those components measure criteria like response times and throughput and display the results to the users. The full list of listeners is available here. We will be using the listener called Summary Report here. The screenshot below shows how to add it to our test plan:

Running the test plan

It’s time to run our test plan! In order to do so, just click on the play button in the ribbon:Here it is in action!
(click on the .gif to enlarge)

As you can see on the top right corner, the test runs for 10 seconds and create 100 users, as we defined in the Thread Group. Let’s go over the results of the execution (I excluded some other columns for clarity purposes):

In the 10 seconds the test ran, over 8100 HTTP requests were sent to the PizzaManager (roughly 1600 requests per method), resulting in an overall throughput of 804 operations per seconds. The average response time was around 120ms with peaks between 350 and 400ms. If we were to compare those results with our fake requirements from above (“the application must handle up to 100 concurrent users with a maximum response time of 1s and an average response time of 200ms”), we could say that our application runs fast enough to satisfy the specs.

Of course, this is just an introduction and we might want to run more tests with a different amount of users and various durations to make sure that everything works as expected.

Small bonus – functional testing

Aside from load testing, you can also do functional testing with JMeter. In order to do so, you can add the View Results Tree listener to your test plan. This will record all HTTP requests made along with their responses. You can use it to make sure the API works as expected from a business point of view. If the screenshot below, we can see that one of the DeletePizza requests failed with a 500 HTTP status code (Internal Server Error).

After investigation, it occurred that the following code was causing an error:

let private pizzas = new Dictionary<string, Pizza>()

let deleteCI (name:string) =
    let pizza = pizzas.Keys |> Seq.tryFind (fun p -> p.ToLower() = name.ToLower())
    match pizza with
    | None -> Failure(sprintf "Failed to delete '%s'. The pizza could not be found!" name)
    | Some p -> 
        pizzas.Remove(p) |> ignore
        Success(sprintf "'%s' was successfully deleted from the database." name)

This is because the pizza was found at the beginning of the function was being removed in the meantime by another user, before the following line was called:

pizzas.Remove(p) |> ignore

The View Results Tree component also allows use to visualize the JSON data sent as a response to a request:

This is also a useful feature for checking if your API returns the expected results.

That’s it for today’s post. I hope you found this introduction to JMeter useful. It seems to be a very powerful tool and I am looking forward to using it more regularly in the future! Please let me know if you have more experience using JMeter or other similar Software. What features do you use? How often do you use it? Do you use it in some automated way? I’d be very interested in your feedback.


Leave a Reply

Your email address will not be published. Required fields are marked *