Journeys in Java, Level 9: Docker compose all the things

Our microservices project contains quite a few pieces now.

We have two databases, three API services, a user-view service for books, and a service to host our configuration.

With so many pieces to manage, it would be nice to have something that orchestrates the individual services into a system, such as Docker Compose.

Back in our Level 5 rendition, we did exactly this for our smaller version of the project.

Now that we have expanded our services, we need to add those new pieces into the existing Docker Compose management umbrella.

I expected this step of the process to be much quicker, especially since I had done it before.

However, there were several obstacles I encountered that took much longer to solve than anticipated.

I’ll do my best to cover those thoroughly here and avoid future problems for both myself and others!

Architecture

We built this project from scratch with a rudimentary scope and functionality and have slowly expanded the complexity.

In the most recent step, we had a few services managed by Docker Compose and a few managed manually.

This is because we wanted to avoid adding unnecessary complexity with an orchestration layer until we were certain service communication was working without it.

Now that things are operating well independently, we can migrate the working services so that everything is handled by Docker Compose.

Updated architecture:

There is a small grey border around each service indicating a containerized application. Services are tied to other services through arrows with some boxes to show the type of objects passed back and forth.

The larger grey box encompassing everything represents how Docker Compose is orchestrating all of services as a single unit.

Prepping applications

The first step was to integrate the config-service to Docker Compose. This service handles database credentials to both MongoDB and Neo4j.

I expected this step to be the trickiest, but in reality, getting service1 to interact with the config service proved to be the biggest hurdle, and the one I spent the most time on.

Once that was operating smoothly, adding the rest of the services took very little time.

Let’s walk through the steps!

Goodreads-config

First, we need to containerize the application using a Dockerfile. After copying/pasting from another service, I found out that my base openjdk image was deprecated. There were a few suggestions for alternatives, but went with Azul’s image as the new base image.

Next, we need to package the application from the service’s directory using mvn clean package and build the Docker image with the containerized application with docker build -t jmreif/goodreads-config . at the command line.

Our application is packaged and the Docker image built, so we need to add the service to our docker-compose.yml file.

version: "3.9"
services:
  #goodreads-db...
  goodreads-config:
    container_name: goodreads-config
    image: jmreif/goodreads-config
    # build: ./config-server
    ports:
      - "8888:8888"
    depends_on:
      - goodreads-db
    environment:
      - SPRING_PROFILES_ACTIVE=native,docker
    volumes:
      - $HOME/Projects/config/microservices-java-config:/config
      - $HOME/Projects/docker/goodreads/config-server/logs:/logs
    networks:
      - goodreads
networks:
  goodreads:

I temporarily commented out the previous service1, service2, and service3 blocks, so that I could focus on adding each piece individually.

The goodreads-config service contains many of the same fields we have used before. I added a commented-out build option test and make changes locally, rather than pulling from the remote image as the parameter above it does.

The depends_on option specifies that the database container must be started before this service can start. Note: This does not mean that the database container is in a "ready" state, only started.

Under the environment block, we have a variable set up for a Spring profile. This allows us to use different credentials, depending on whether we are running in a local test environment or in Docker.

The main difference is the use of localhost to connect to other services in a local environment (specified by native profile), versus container names in a Docker environment (profile: docker). The next option for volumes sets up the location of the config files (for now, in a local config directory) and log files.

*Note:* I played around with moving the profile variable into the config file, but found out that we need it as a variable in the container (either from the application in application.properties or in docker-compose.yml). Sourcing it from outside the container didn’t cooperate.

Let’s test what we changed so far!

Round 1: Put it to the test!

We can run this much of the system with a single command.

docker-compose up -d

*Note:* If you are building local images with the build field in docker-compose.yml, then use the command docker-compose up -d --build. This will build the Docker containers each time on startup from the directories.

Next, we can test our goodreads-config service by accessing the configuration file it is hosting. We can do this at the command line with curl localhost:8888/mongo-client/docker or using a browser with the URL localhost:8888/mongo-client/docker. This should show something like the screenshot below.

Figure 1. Config server results

Bring everything back down again with another command.

docker-compose down

Goodreads-svc1: Interact with goodreads-config

As mentioned above, this was the toughest part to get working, but I learned a few things along the way.

We already had Docker Compose managing this service in the previous Level 5 version of the code, so we can use that setup with some adjustments.

version: "3.9"
services:
  #goodreads-db...
  #goodreads-config...
  goodreads-svc1:
    container_name: goodreads-svc1
    image: jmreif/goodreads-svc1:lvl9
    # build: ./service1
    ports:
      - "8081:8081"
    depends_on:
      - goodreads-config
    restart: on-failure
    environment:
      - SPRING_APPLICATION_NAME=mongo-client
      - SPRING_CONFIG_IMPORT=configserver:http://goodreads-config:8888
      - SPRING_PROFILES_ACTIVE=docker
networks:
    - goodreads

The first several options are the same as our previous services, although I added a tag to the image name to keep a separate image with these updates.

The next change is on the depends_on option. Instead of waiting directly for the database container, service1 actually depends on the config service (goodreads-config) for the database credentials. The config service then forwards the call with appropriate credentials.

On the next line is a new option - restart. This took the longest time for me to debug, but you might remember me mentioning that depends_on only waits for the container to start, not for the service to be ready. It turns out that service1 was starting too early, so it would fail to find the configuration.

After trying a few different methods, such as building in request retries in the application itself, I discovered that the only working solution was to restart the whole container (or at least the entire application within the container). The most straightforward way to do this was through the restart option in Docker Compose. This solved the startup and configuration issues I was seeing.

Lastly, the environment variables specify the application name, location of the config server, and Spring profile. The application name and active profile help the application find the appropriate configuration file on the config server.

The SPRING_CONFIG_IMPORT variable tells the container where to look for the config server. I also noticed that these properties did not work correctly if I put them in the config file itself. The values must be accessible within the container, or it would not know where to look.

Service1: Application Changes

As I was debugging the restart issues mentioned above, one suggestion to add resiliency to the application and assist with determining errors was to add Spring Retry capabilities.

This allows us to set up guidelines for automatically retrying requests, which is especially helpful when safeguarding against situations like network interruptions.

While it didn’t solve the Docker Compose container startup issues, I kept the code to make the application more robust and assist debugging.

There wasn’t much code to add, and I followed a colleague’s advice, alongside Baeldung’s article.

	org.springframework.retry
	spring-retry


	org.springframework.boot
	spring-boot-starter-aop


	org.springframework.boot
	spring-boot-starter-actuator

First, we need the retry dependency, alongside the Spring Boot starter for Aspect-Oriented Programming. I also added Actuator, which will set up endpoints to inspect application health, metrics, and more.

Next, we need to comment out the local properties in the src/main/resources/application.properties file so that the config server variables don’t conflict with ones we are setting in the container (via docker-compose.yml).

Otherwise, it would connect to the container config server, but also try to connect to a local config server. It would continue to retry until it failed, causing the application to crash searching for irrelevant, backup property values.

Lesson learned: the retry applied to any properties set, whether in the application or environment variables.

Finally, I need to add a few annotations to the application class.

@SpringBootApplication
@EnableRetry
public class Service1Application {
	....
}

@RestController
@RequestMapping("/db")
@AllArgsConstructor
class BookController {
	....

	@Retryable
	@GetMapping("/books")
	Flux getBooks() { return bookRepository.findAll(); }

	@Retryable
	@GetMapping("/book/{mongoId}")
	Mono getBook(@PathVariable String mongoId) { return bookRepository.findById(mongoId); }
}

The @EnableRetry annotation enables retry functionality in the application. The @Retryable annotation on the two methods tells the application which methods should utilize retry logic. Default configuration for retries is set to try the request up to three times with a delay of one second between each retry. However, we can customize the defaults through properties, annotation parameters for each method, or template configuration.

Notice that I did not add retry logic to the liveCheck() method. This is because we don’t interact with other services or have other dependencies that might potentially cause flakiness in the requests. If the liveCheck() method does not work, then our application or container is not running.

While I was here, we can upgrade the Spring Boot project to use the latest versions of everything, so I modified the Spring Boot version, Java version, and Spring Cloud version in the pom.xml. I also updated the Dockerfile’s base image to the Azul JDK 17, as well.

With those changes in place, we will need to re-package the application with mvn clean package -DskipTests=true and rebuild the local Docker container. I also made some tweaks to the config file.

*Note:* the -DskipTests=true is necessary because it will look for a config server and fail when/if it doesn’t find it.

Config file: mongo-client

In the mongo-client configuration file hosted by the config server, I added a couple more properties for opening up Actuator endpoints (for application debugging) and the application name

# Enable all actuator endpoints FOR DEMO PURPOSES ONLY!
management:
  endpoints:
    web:
      exposure:
        include: "*"

spring:
  application:
    name: mongo-client
....

Round 2: Put it to the test!

Let’s run all of the pieces so far - goodreads-db, goodreads-config, goodreads-svc1.

docker-compose up -d

*Note:* If you are building local images with the build field in docker-compose.yml, then use the command docker-compose up -d --build. This will build the Docker containers each time on startup from the directories.

Next, we can test our services with a few endpoints.

• Goodreads-config: command line with curl localhost:8888/mongo-client/docker.
• Goodreads-svc1: command line with curl localhost:8081/db, curl localhost:8081/db/books, and curl localhost:8081/db/book/623a1d969ff4341c13cbcc6b or web browser with only URL.

Figure 2. Test service1 for a book

When we are done testing this, we can bring down the system with docker-compose down.

Goodreads-svc2: Interact with service1

This service is our user-facing service for interacting with book data. All of the changes made to this service will look familiar because we made the same changes in service1!

version: "3.9"
services:
  #goodreads-db...
  #goodreads-config...
  #goodreads-svc1...
  goodreads-svc2:
    container_name: goodreads-svc2
    image: jmreif/goodreads-svc2:lvl9
    # build: ./service2
    ports:
      - "8080:8080"
    depends_on:
      - goodreads-svc1
    restart: on-failure
    environment:
      - BACKEND_HOSTNAME=goodreads-svc1
    networks:
      - goodreads
networks:
    - goodreads

We only add the restart: on-failure option here. Because we could have network interruptions that cause services to miss requests or other flaky behavior, I wanted to build the same resiliency into my other applications and containers that I did with service1. For a refresher on the BACKEND_HOSTNAME environment variable, check out the Level 5 blog post.

Service1: Application Changes

First, we want to add the Spring Retry logic to service2 by adding the three dependencies in the pom.xml (lines 29-40). Next, we can add the property to the application.properties to open all Actuator endpoints. Note: We only do this in development or local testing - not production! We can comment out that property here for now. Finally, we can add @EnableRetry to the main application class and @Retryable to each method we want to use retry logic. In our case, only the getBooks() method.

We have already built in flexibility to service2 for local or remote testing with the hostname property. More detail on how and why we did that is in the Level 5 blog post.

While we are here, we can update the Spring Boot project to use the latest versions of everything for Spring Boot and Java. I also updated the service’s Dockerfile to use the Azul JDK 17 as the base image.

We need to re-package the application and build the local container, just as we did in service1. Because this service interacts only with service1, it doesn’t need any ties to the config service or database. Let’s test it!

Round 3: Put it to the test!

We’ll use the docker-compose up -d command, as we did before, to spin up Docker Compose.

Then we test our endpoints.

• Goodreads-config: command line with curl localhost:8888/mongo-client/docker.
• Goodreads-svc1: command line with curl localhost:8081/db, curl localhost:8081/db/books, and curl localhost:8081/db/book/623a1d969ff4341c13cbcc6b.
• Goodreads-svc2: command line with curl localhost:8080/goodreads and curl localhost:8080/goodreads/books or web browser with the URL.

Figure 3. Test service2 for books

And docker-compose down will shut down everything gracefully.

Goodreads-svc3: Backend service for Authors

Our third service is a near copy of service1, but it hosts author data (rather than books). Pretty much everything we learned before is also applied to this service3. We will list the changes for review.

• Docker Compose: add configuration for service3 (values match service1).
• Application pom.xml: add three depedencies for retry and monitoring (lines 38-49). Also upgrade versions for Spring Boot, Java, and Spring Cloud.
• Application application.properties: add Actuator endpoints property and comment out several (for local testing only).
• Application Service3Application: add @EnableRetry to main application class and add @Retryable to desired methods (getAuthors() and getAuthor(id)).
• Dockerfile: use Azul JDK 17 as base image.
• Application: re-package app with mvn clean package -DskipTests=true and build local Docker container
• Config file: no changes because it will use the same values that service1 uses.

Let’s test!

Round 4: Put it to the test!

As usual, use docker-compose up -d at the command line to spin up our microservices system and test the endpoints with the commands listed below.

• Goodreads-config: command line with curl localhost:8888/mongo-client/docker.
• Goodreads-svc1: command line with curl localhost:8081/db, curl localhost:8081/db/books, and curl localhost:8081/db/book/623a1d969ff4341c13cbcc6b.
• Goodreads-svc2: command line with curl localhost:8080/goodreads and curl localhost:8080/goodreads/books.
• Goodreads-svc3: curl localhost:8082/db, curl localhost:8082/db/authors, and curl localhost:8082/db/author/623a48c1b6575ea3e899b164 or web browser with only URL.

Figure 4. Test service3 for an author

Close the system with docker-compose down.

Goodreads-svc4: Backend service for Reviews

Similar to services one and three, service4 is a backing service for book review data. However, this service interacts with a graph database in the cloud (Neo4j). More background on this service is in the Level 6 blog post. Let’s list out our changes to bring service4 into Docker Compose.

• Docker Compose: add configuration for service4.
• Application pom.xml: add three depedencies for retry and monitoring (lines 38-49). Upgrade versions for Spring Boot, Java, and Spring Cloud.
• Application application.properties: add Actuator endpoints property and comment out several (for local testing only).
• Application Service4Application: add @EnableRetry to main application class and add @Retryable to desired methods (getReviews() and getBookReviews(id)).
• Dockerfile: use Azul JDK 17 as base image.
• Application: re-package app with mvn clean package -DskipTests=true and build local Docker container
• Config file neo4j-client: add Actuator endpoint property and application name.

We now have all of our services migrated to Docker Compose. Time to test the entire system!

Round 5 (final!): Put it to the test!

We can run our system with the same command we have been using.

docker-compose up -d

Next, we can test all of our endpoints.

• Goodreads-config (mongo): command line with curl localhost:8888/mongo-client/docker.
• Goodreads-svc1: command line with curl localhost:8081/db, curl localhost:8081/db/books, and curl localhost:8081/db/book/623a1d969ff4341c13cbcc6b.
• Goodreads-svc2: command line with curl localhost:8080/goodreads and curl localhost:8080/goodreads/books.
• Goodreads-svc3: curl localhost:8082/db, curl localhost:8082/db/authors, and curl localhost:8082/db/author/623a48c1b6575ea3e899b164.
• Goodreads-config (neo4j): command line with curl localhost:8888/neo4j-client/docker.
• Neo4j database: ensure AuraDB instance is running (free instances are automatically paused after 3 days).
• Goodreads-svc4: curl localhost:8083/neo, curl localhost:8083/neo/reviews, and curl localhost:8083/neo/reviews/178186 or web browser with only URL.

Figure 5. Test service4, reviews for a book

Bring everything back down again with the below command.

docker-compose down

Wrapping up!

We have successfully created an orchestrated microservices system with Docker Compose!

We saw how Docker Compose can throw some tricky environment issues at us related to startup order and dependencies between microservices, but we were able to navigate those with additional configuration options. We also built resiliency into our application services with Spring Retry to help us handle service or network interruptions.

The microservices system now includes a database container (MongoDB), configuration service, three backing services (service1, service3, and service4), and a user-facing service (service2). We also connect to an external, cloud-hosted Neo4j database.

We have grown this system, and we hope to continue adding functionality and learning along the way. Happy coding!

Resources

• Github: microservices-level9 repository
• Blog post: Simple Fix If Your Dockerized App Crashes…​
• Documentation: Docker Compose - restart
• Neo4j AuraDB: Create a FREE database