Kubernetes Rolling Updates: implementing a Readiness Probe with Spring Boot Actuator

During a Rolling Update on Kubernetes, if a service has a Readiness Probe defined, Kubernetes will use the results of calling this heathcheck to determine when updated pods are ready to start accepting traffic.

Kubernetes supports two probes to determine the health of a pod:

  • the Readiness Probe: used to determine if a pod is able to accept traffic
  • the Liveliness Probe: used to determine if a pod is appropriately responding, and if not, it will be killed and a new pod restarted

Spring Boot’s Actuator default healthcheck to indicate if a service is up and ready for traffic can be used for a Kubernetes Readiness Probe. To include in an existing Spring Boot service, add the Actuator maven dependency:


This adds the default healthcheck accessible by /actuator/health, and returns a 200 (and json response { “status” : “up”} ) if the service is up and running. 

To configure Kubernetes to call this Actuator healthcheck to determine the health of a pod, add a readinessProbe section to the container spec section for your deployment.yaml:

- name: exampleservice-b
image: kevinhooke/examplespringboot-b:latest
imagePullPolicy: Always
- containerPort: 8080
path: /example-b/v1/actuator/health
port: 8080
initialDelaySeconds: 5
timeoutSeconds: 5

Kubernetes will call this endpoint to check when the pod is deployed and ready for traffic. During a rolling update, as new pods are created with an updated image, you’ll see their status go from 0/1 available to 1/1 available as soon as the Spring Boot service has completed startup and the healthcheck is responding.

The gif below shows deployment of an update image to a pod. Notice how as new pods are created, they move from 0/1 to 1/1 and then when they are ready, the old pods are destroyed:

Caching Spring Boot RESTController responses with Spring Cache and Redis

Spring Boot provides easy integration for caching responses using a number of cache providers. See the section in the docs here. Depending on what type of response you’re trying to cache however, there’s a range of issues you can run into.

If you’re interested in a working configuration then skip to the end, otherwise what follows is the steps I went through to try and get this working as I wanted. If you run into one of these error messages (which are surprisingly descriptive) then maybe you’ll find this post from a search and this will help you get to a solution).

I’m using Redis as a cache provider. To enable this I added the following to my Spring Boot application.properties:


Add this dependency to your pom.xml:


And then add Redis support:



I’m using JPA with h2 for testing, so I’ve also added:


and then for h2:


To enable caching support in your Spring Boot app, add the @EnableCaching annotation:


To enable support for using a RedisTemplate directly (I think Spring Boot Cache when configured to use Redis as your cache provided uses this config too) add this @Bean to your @Configuration class:

public RedisTemplate<String, String> redisTemplate(RedisConnectionFactory cf) {
  RedisTemplate<String, String> redisTemplate = new RedisTemplate<String, String>();
  return redisTemplate;

… and configure the Jedis client library:

public JedisConnectionFactory redisConnectionFactory() {
  JedisConnectionFactory redisConnectionFactory = new JedisConnectionFactory();

  // Defaults
  return redisConnectionFactory;

You also need to configure the RedisCacheManager as the cache provider:

public CacheManager cacheManager(RedisTemplate redisTemplate) {
  RedisCacheManager cacheManager = new RedisCacheManager(redisTemplate);

  // Number of seconds before expiration. Defaults to unlimited (0)
  return cacheManager;

The first requirement for caching your responses is that the response needs to be Serializable, otherwise you’ll see this error in your response from an @RestController:

“status”: 500,
“error”: “Internal Server Error”,
“exception”: “org.springframework.data.redis.serializer.SerializationException”,
“message”: “Cannot serialize; nested exception is org.springframework.core.serializer.support.SerializationFailedException: Failed to serialize object using DefaultSerializer; nested exception is java.lang.IllegalArgumentException: DefaultSerializer requires a Serializable payload but received an object of type [org.springframework.http.ResponseEntity]”

ResponseEntity is the wrapper you typically use for your responses from a @RestController, and it’s not Serializable. From this question here, this suggests using a SpEL expression to tell the cache support to use the body of the response as a the key, and not the ResponseEntity itself (which makes sense):

@Cacheable(cacheNames = "addresses", key="#response?.body")

With this you now get:

Null key returned for cache operation (maybe you are using named params on classes without debug info?) Builder[public org.springframework.http.ResponseEntity kh.springcloud.service1.Service1RestController.getAddresses(java.lang.String) throws org.springframework.web.client.RestClientException] caches=[addresses] | key=’#response?.body’ | keyGenerator=” | cacheManager=” | cacheResolver=” | condition=” | unless=” | sync=’false'”

Thinking what I really need is the id of the address I’m returning as the key, I modified the SpEL to this:


but this is not really what I want either if I’m attempting to return a list of Addresses from GET /addresses?state=CA. What I really want as the key of the results to be cached is the State value that I’m querying on, so trying this approach:


this says request is null.

Looking at the examples in this question, it looks like you can reference the method parameter names directly, so:

@Cacheable(cacheNames = "addresses", key="'state")

Now trying a request we get this:

“message”: “EL1008E: Property or field ‘state’ cannot be found on object of type ‘org.springframework.cache.interceptor.CacheExpressionRootObject’ – maybe not public?”,

Ok, how about key=”{#state}” – this gives same error as before.

“message”: “Expression [{ #state ] @0: EL1044E: Unexpectedly ran out of input”,

I’m really reaching for a solution at this point 🙂 What if we step back and simplify the api we’re trying to cache, and add a findById api and try to cache this by id:

@Cacheable(cacheNames = "addresses", key="#id")
public ResponseEntity<Address> getAddressByd(@PathVariable("id") Long id)

Ok, with this simple endpoint, we’re not retrieving anything by id, but a quick sanity check (removing the cacheable) shows the responses are coming back without an id. So there’s your problem:

  "addr1": "test3",
  "city": "testcity2",
  "state": "CA",
  "zip": "95616"

Taking a look at my mapping, as a test I’m running against an in memory h2 db, and was mapping my id with AUTO:


It seems this doesn’t generate any id at all for the h2 db, so let’s switch to an identity table (does h2 have sequences? I’m not sure, but feel like this is going too far down a rabbit hole)


Now what do we get in responses – the same response. Trying /addresses/1 vs /addresses/2 I can see that I have a response with id=1, but not for id=2

Looking at the mappings in the entity, I don’t have a getter/setter for the id, so let’s add one and try again. Ok, now we’re good:

  "id": 1,
  "addr1": "test3",
  "city": "testcity2",
  "state": "CA",
  "zip": "95616"

Let’s put the @Cacheable back and try again.

This cache config:

@Cacheable(cacheNames = "addresses", key="#response?.body?.id")

Still gives:

"Null key returned for cache operation"

Let’s try id directly:

@Cacheable(cacheNames = "addresses", key="id")

Now we get:

“message”: “EL1008E: Property or field ‘id’ cannot be found on object of type ‘org.springframework.cache.interceptor.CacheExpressionRootObject’ – maybe not public?”,

key=”#id” is back with this error, so it seems like this is the right config, but we’re still struggling with the ResponseEntity not being Serializable:

“message”: “Cannot serialize; nested exception is org.springframework.core.serializer.support.SerializationFailedException: Failed to serialize object using DefaultSerializer; nested exception is java.lang.IllegalArgumentException: DefaultSerializer requires a Serializable payload but received an object of type [org.springframework.http.ResponseEntity]”,

This issue on the SpringBoot project is exactly what we’re seeing, but the OP was told to post the question to SO:


This might the same question, if not then it’s pretty much the same question, but the only answer so far says what we know so far: ResponseEntity is not Serializable, but we don’t know yet how to fix this. It seems we need a custom Serializer configured, but not sure how we do that yet.

This post suggests how to change/set the serializer and key serializer in use and suggests:

template.setDefaultSerializer(new GenericJackson2JsonRedisSerializer());
template.setKeySerializer(new StringRedisSerializer());
template.setHashKeySerializer(new GenericJackson2JsonRedisSerializer());
template.setValueSerializer(new GenericJackson2JsonRedisSerializer());

Trying with this in place, we’ve got a different error, and this seems an issue with the Long key, instead of a String key.

It doesn’t look like there is a LongRedisSerializer, but searching around I found this question which suggests to use the GenericRedisSerializer for other key types, so replacing StringRedisSerializer with this:

redisTemplate.setKeySerializer(new GenericToStringSerializer<Long>(Long.class));

Now we’re making some progress. Calling the get by id api the first time, the sql is executed and the json response is returned from the endpoint. Calling it a second time we now get a different error:

com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.springframework.http.ResponseEntity: no suitable constructor found, can not deserialize from Object value (missing default constructor or creator, or perhaps need to add/enable type information?)

Starting the redis-cli and doing a ‘keys *’ to see what if anything has been inserted, there’s a couple of keys, one is “1”.

Doing a ‘GET 1’ we get:> get 1


We’ve got a cached response! So the problem now looks like deserializing the cached data back out to a ResponseEntity<Address>

What if we try @Cacheable on the JPA query instead of trying to work out how to deserialize the ResponseEntity?

First attempt:

@Cacheable(cacheNames = "addresses", key="#id")
public Address findOne(Long id);

This gives us an error about the key being null:

Null key returned for cache operation (maybe you are using named params on classes without debug info?) Builder[public abstract kh.springcloud.service1.domain.Address kh.springcloud.service1.repo.AddressRepository.findOne(java.lang.Long)

Ok. This post mentions an alternative numbered param reference to the params, using the format #p0. Let’s try:

@Cacheable(cacheNames = "addresses", key="#p0")

This works! Calling the api the first time we get the select statement against the db (seen from the hibernate debug log output), but subsequent times there’s no additional select against the db, the value is returned from cache.

There doesn’t seem to be much info about how to handle deserializing the result back out as a ResponseEntity, when I have some time I’ll see if I can work out how to do this, but in the meantime, caching the responses from JPA is good enough and avoids hitting the db every time, so this works for now.

Using Netflix Eureka with Spring Cloud / Spring Boot microservices (part 2)

Several months back I started to look at setting up a simple example app with Spring Boot microservices using Netflix Eureka as a service registry. I got distracted by other shiny things for a few months, but just went back to finish this off.

The example app comprises of 3 Spring Boot apps:

  • SpringCloudEureka: registers the Eureka server using @EnableEurekaServer
  • SpringBootService1 with endpoint POST /service1/example1/address
    • registers with Eureka server with @EnableDiscoveryClient
    • uses Ribbon load balancer aware RestTemplate to call Service2 to validate a zipcode
  • SpringBootService2 provides endpoint GET /service2/zip/{zipcode} which is called by Service1
    • also registers with Eureka server with @EnableDiscoveryClient so it can be looked up by Service1

SpringBootService1 and SpringBootService2 both register with the Eureka server using the annotation @EnableDiscoveryClient. Using some magic with @EnableFeignClients, SpringBootService1 is able to call SpringBootService2 using a regular Spring RestTemplate, but it is Eureka aware and able to lookup SpringBootService2 by service name inplace of an absolute ip and port.

This allows the services to be truly decoupled. Service1 needs to know it needs to call Service2 to perform some purpose (in this case validate a zip code), but it doesn’t need to know where Service2 is deployed or what ip address/port it is available on.

Example code is available on github here.

Configuring nginx to proxy REST requests across multiple Spring Boot microservices in Docker Containers

Using nginx to proxy requests across Docker containers is a common use case for nginx, and covered in many posts and articles. Trying to get it working from scratch is a not-so-trivial task. Following many articles, I got confused about whether I was needing to load balance requests across identical container instances, or whether I needed nginx to proxy requests across my different containers. I ran across quite a few configuration issues and challenges, but mostly I think because I didn’t understand what I was trying to do (and ended up with configuration trying to load balance and proxy at the same time which was not what I needed) 🙂

Articles like this one and this one show how to configure ‘upstream’ servers that you can load balance requests across. My understanding of this approach is that this is what you need if you have multiple server or container instances, and you want nginx to load balance across the instances. After a while of trying to get this configuration working what I realised I needed was just the proxy_pass config for nginx, telling it to proxy requests for a matching url to a given Spring Boot service in a container. This question captures this approach well.

To explore a typical configuration, I have 2 simple Spring Boot REST services, springbootservice1 and springbootservice2, each will be in their own Docker container. When run in individual containers, they are accessed via:




Without additional configuration to run them on different ports, as standalone Spring Boot services they can’t obviously be run on the same host at the same time. This would be trivial to do with just two services, but once you scale up this approach for many different services, managing different ports with manual configuration is not particularly practical.

Out of the box, docker lets you manage the exposes external ports easily when you ‘docker run’ a container with the ‘-p’ option. What I was interested in was a solution to run them in different containers, without manually defining and managing ports by hand, and to allow a web app to be able to call any of the services on port 80 so an app consuming these services has no need to know what ports the services are actually running on.

The Dockerfile for each looks like this (replace Service1 for Service2 for the second service):

#Official JDK8 Alpine-based image
FROM java:openjdk-8-alpine
ADD target/SpringBootService1-0.0.1-SNAPSHOT.jar /opt/SpringBootService1-0.0.1-SNAPSHOT.jar
ENTRYPOINT ["java", "-jar", "/opt/SpringBootService1-0.0.1-SNAPSHOT.jar"]

The Dockerfile for nginx looks like this:

FROM nginx
RUN rm /etc/nginx/conf.d/default.conf
COPY conf /etc/nginx
CMD ["nginx", "-g", "daemon off;"]

And here’s my config for nginx:

http {
 server {
   listen 80;
   location /service1 {
     proxy_pass http://springbootservice1:8080/service1/;
   location /service2 {
     proxy_pass http://springbootservice2:8080/service2/;
events { worker_connections 1024; }

The key part of this config is the proxy_pass config:

location /service1 { 
  proxy_pass http://springbootservice1:8080/service1/;

This takes incoming requests with a URI matching /service1 and proxies it to http://springbootservice1:8080/service1/ – in this case springbootservice1 is the default host name of the container running Service1. This is it’s default name from the docker-compose config which we’ll cover next. Note that the lack of trailing and trailing ‘/’s is relevant and important in order to match any incoming pattern and then forward that URI appended to the end of the target URI on the container service.

version: '3'
         build: ../nginx-loadbalancer
         #image: nginx-lb
             - "80:80"
             - springbootservice1
             - springbootservice2
             - springbootservice1
             - springbootservice2
         image: springbootservice1
             - "8080"
         image: springbootservice2
             - "8080"

To bring up all the contains in one go, ‘docker-compose up’ starts up all 3 containers, and now nginx handles requests on port 80 and proxies to the correct container based on the path. This approach could easily be scaled up to include more containers, but at some point it will become obvious with this configuration hardcoded in both the nginx.conf file and the docker-compose file, a better solution would be to use some kind of dynamic discovery of the containers as they become available. I’ll be investigating some options for this kind of approach next.