Java

Road to Continuous Deployment Part 1 – Performance Testing

The next set of Articles are documenting a presentation I’m working on to demonstrate Continuous Delivery, and how in a world where CD is desired, it’s becoming increasingly important for the appropriate level of testing to happen. I’ll be breaking them down to tackle one topic at a time, finally building up to a full demo on how I feel CD can work.

Firstly, I’m going to start with a problem CI/CD can address – Performance Testing. Not so much on how to performance test or how CI/CD can build systems that don’t require it, but how a continuous delivery pipeline can quickly alert developers to potential problems that could otherwise remain undetected and take ages to debug once in production.

Nothing new or groovy here, but a problem most developers (and definitely sql developers) are familiar with: the N+1 problem. Java developers using Hibernate will be familiar with the @OneToMany annotation, but less so on the @Fetch annotation, and even less so on the implications of changing the FetchMode. I aim to demonstrate how something as simple as changing a Hibernate @OneToMany Fetch strategy can drastically affect the performance of a system

There are several good reasons why you may want to do this:
* This is a reasonable thing to do: Maybe you want to lazy load children, no point eagerly loading all details with a join
* Might not always be a bad thing to do (perhaps most of the time only a few children are accessed, which negates the overall performance impace) but testing should be done to assess the potential impact on performance

Side-bar: The demo project for this article was originally ported from the spring boot data rest example within their samples, however the @Fetch annotation appears to be ignored, which makes it difficult to demonstrate.
This article gives a good direction on what I expected to happen and what the problem likely is: http://www.solidsyntax.be/2013/10/17/fetching-collections-hibernate/
I suspect the spring boot configuration doesn’t use the Criteria API behind the scenes, which means the @Fetch annotation will be ignored.

The application is a simple school class registration system, with the domain modelled around classes and students. One GET resource is available which returns all classes and students as children nodes. Below is a sample of the returned json:

  {
    "id": 3,
    "className": "Oakleaf Grammar School - Class 3",
    "students": [
      {
        "id": 970,
        "firstName": "Marie",
        "lastName": "Robinson"
      },
      {
        "id": 2277,
        "firstName": "Steve",
        "lastName": "Parker"
      },
      {
        "id": 4303,
        "firstName": "Lillian",
        "lastName": "Carpenter"
      },
      {
        "id": 9109,
        "firstName": "Samuel",
        "lastName": "Woods"
      }
    ]
  }

So at this point, I have a simple application whose performance can be altered by simply changing the @Fetch annotation, but how can we test the effect of this?

This is what Gatling is designed for. It is a performance and load testing library written in Scala, which has a really nice DSL that can be used to express scenarios for testing load on systems

This is code required to define a scenario for testing our system:

import io.gatling.core.Predef._
import io.gatling.http.Predef._

class SchoolClassLoadTest extends Simulation {

val httpConf = http
.baseURL("http://localhost:8080") // Here is the root for all relative URLs

val scn = scenario("Scenario Name").repeat(10, "n") {

exec(http("request_1") // Using http protocol
.get("/v1/class") // Get http method with the relative url
.check(status.is(200))) // Check response status code is 200 (OK)
  }
}

Not much is it?

Side-bar: Gatling also comes with is a recorder UI. This allows you to record all interactions with applications over HTTP and save them as scenarios. Gatling achieves this by acting as a proxy, which you can use like any other web proxy. You can route web browser or application traffic via the Gatling Recorder Proxy, and it will record all interactions made through it and save them as a scenario similar to the one above, which can then be tweaked later.

Once you have your scenario defined, you can configure your simulations to meet whatever load and duration parameters you like. For example in our app I’m going to run the test simulating 10 concurrent each making 10 requests:

setUp(scn.inject(atOnceUsers(10)).protocols(httpConf))

This is just a simple example of what you can do with Gatling: you can specify ramp up and down of users to test how an app scales; pause between steps to simulate human interaction; and much more. For more about what you can do with Gatling check out their QuickStart Page

Back to our app. I’m now going to hit this simulation against our app whilst it’s using the FetchMode.JOIN config:
N+1WithJoin

N.B. I’ve warmed up the DB cache before running both the before and after benchmarks by running the simulation once before I’ve recorded the results.

Above is the baseline for our app – you can see the mean response time is 221ms, and the max is 378ms. The 95 percentile is 318ms. Now look what happens when I simply change the @Fetch strategy from JOIN to SELECT:
N+1WithSelect

The average response time has increased from 221ms to 3.4 seconds, with the 95th percentage rocketing up to 8.3 seconds! Something as simple as an annoatation change can have such a dramatic impact on performance. The worrying thing is this is so easy to do: it could be done by someone unfamiliar with Hibernate and database performance tuning; by someone frantically looking to find some performance gain and changing a number of things arbitrarily; or what I consider worse – someone who believes that the benefits of doing this outweigh the performance knock, but fail to gather the evidence to back it up!

Now that we’ve got the tools required to test performance, in my next article I’ll look into how this can be proactively monitored between releases, stay tuned…

Using Spotify Docker Plugin on Windows

So recently I have been looking into using Docker to automate parts of my testing and deployment, Using my lightweight CEP component (https://github.com/foyst/smalldata-cep) that uses Siddhi under the covers as a basis to develop my skills in Continuous Integration / Deployment.

Currently using Windows 10 on my personal laptop and never one to give up a challenge, I finally got Docker Toolbox running on Windows with version 1.8.2, with boot2docker and Docker Quickstart Terminal.

My next goal was to configure Docker builds directly to my boot2docker VM using Spotify’s Docker Maven Plugin (https://github.com/spotify/docker-maven-plugin) as a step towards Continuous Integration / Deployment. The idea being whenever I run the deploy phase of my Maven build, a Docker image of my software would be built, ready to run immediately afterwards.

This is the maven configuration I used within my “runner” module’s pom.xml:

<plugins>
    <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
    </plugin>
    <plugin>
        <groupId>com.spotify</groupId>
        <artifactId>docker-maven-plugin</artifactId>
        <version>0.3.5</version>
        <configuration>
            <baseImage>nimmis/java:oracle-8-jdk</baseImage>
            <imageName>foyst/smalldata-cep</imageName>
            <exposes>
                <expose>8080</expose>
            </exposes>
            <entryPoint>["java", "-jar", "/app/${project.build.finalName}.jar"]</entryPoint>
            <!-- copy the service's jar file from target into the root directory of the image -->
            <resources>
                <resource>
                    <targetPath>/app/</targetPath>
                    <directory>${project.build.directory}</directory>
                    <include>${project.build.finalName}.jar</include>
                </resource>
            </resources>
        </configuration>
    </plugin>
</plugins>

I was trying to run it within IntelliJ on Windows, and was getting the following error:

Docker - Socket Write Error

Reading the Spotify Docker readme on GitHub states that it uses the DOCKER_HOST environment variable to determine the location of the Docker Daemon, and if this isn’t set it uses localhost:2375. So I tried setting this to the IP of the VM.

20151108 Docker 7

Unsuccessful again with this error:

Docker Error 2

This hadn’t worked either. Next thing to do was ssh onto the boot2docker VM and do a “netstat -apn” command, to see what ports Docker was listening on

Docker Machine Netstat

Docker was actually listening on 2376, not 2375! So after changing this I got back to the Socket write error from previous attempts.

A quick Google for this brought a bug up for the plugin (https://github.com/spotify/docker-maven-plugin/issues/51), mentioning other environment variables that potentially needed setting up, but at this point I didn’t know if they’d fix my specific problem, or even if they did what to set them to.

So next thing I tried to do was go back to Docker basics and build an image of my Spring Boot uberjar using a standard Dockerfile (completely outside of Maven):

FROM nimmis/java:oracle-8-jdk

WORKDIR /app
ADD smalldata-cep-runner-1.0-SNAPSHOT.jar /app/smalldata-cep-runner.jar

EXPOSE 8080
CMD ["java", "-jar", "/app/smalldata-cep-runner.jar"]

Then from within the Docker Quickstart Terminal I was able to run this command within the folder that had my Dockerfile in it: “docker build -t foyst/smalldata-cep .”

This successfully built my image in my boot2docker VM, which I could then run with “docker run –rm -p 8080:8080 foyst/smalldata-cep”

Ok, so using Vanilla Docker build I could successfully create an image, happy days! So from there I moved on to running “mvn docker:build” from PowerShell…

Still an error. Then it occurred to me to try “mvn docker:build” from the Docker Quickstart Terminal:

20151108 Docker 5

Success!! So what’s the difference? Eventually it occurred to me that the Docker Quickstart Terminal must somehow be configured out of the box to communicate with the boot2docker VM. So I started focusing my search on that

Always review the documentation, this came in handy: http://docs.docker.com/engine/installation/windows/#from-your-shell

This allowed my to actually configure the shell of my choice (Powershell), and knowing how that worked I could then apply the same to my IntelliJ configuration.

20151108 Docker 6

Using Docker in Windows is still quite a PITA, but now the client works natively on Windows and you understand what goes on behind the scenes it’s possible to get it working quite nicely.

Next I will be looking into automated builds and deployments using a combination of GitHub, Jenkins and Docker…