K8s End2End & component testing

Recently, I set up my first ever component testing infrastructure. It all started because my Sprkl team realized we needed to have End2End & component testing for our microservices so we can make sure they actually work as expected. It was quite a learning process. Until then, my experience was only with unit testing, writing the unit tests, and mocking everything up on the function level.

The context

I decided to test our microservices as black boxes using a real infrastructure. An experience I thought might be worth sharing.

Good luck with that.

The challenge

The first attempt was a failure. We used docker-compose and dockers to create the environments for running the tests. At first, I thought it would be easier because I had no experience working with Kubernetes. But I must admit that creating these testing environments was time-consuming, leading to a bad DX (developer experience). The experience was terrible because the process was slow and took very long to spin up and down the envs. Also, managing networks for the dockers was a pain in the ass.

The tale

At that point, I understood that using dockers was not fun. And said to myself, hey, my team is already using Kubernetes; why not utilize it for the testing? The problem was that I didn’t know how to set it up.

Before diving into my component testing tale, I’d like to thank Martin Fowler for his initiative and a great website that I often refer to when researching patterns and practices that make building useful software easier. The” Testing Strategies in a Microservice Architecture” article, written by Tobias Clemson, was the first step in clearing out the mess. A highly recommended read.

The solution: Using Kubernetes/Tilt

The objective:

• Set up the testing environment quickly and easily 
• Substitute actual services for mocked ones to remove some of the complexity even quicker
• Be able to change environments fast

After some consideration, I decided to run the test infrastructure using Tilt to spin up different clusters by injecting different values files to our Helm chart. 

I.e., I have A and B services. I want to run a component test on A without having B involved. That’s because service B is still unstable, and applying it in the tests would be complex and will require test changes soon. So I spun service A by having Tilt inject a YAML file to Helm, which then lets YAML feed Kubernetes. I will use its YAML file with a mock-up of service A when I want to test service B. 

BTW, Tilt is awesome. It allowed me to create similar environments to what we’ll have in production while mocking up some of the pods allowing simple test writing and a positive development experience.

The idea of using Tilt was a great pick because it also allowed us to automatically update the running code in the pods when changes were made locally. 

 

Rapid Kubernetes development with Tilt

The actual tests were running from what we call a “Tester pod” from within the cluster using the services APIs (Kafka, gRPC). And everything was running fast and as expected. 

The result 

So the result of this design is that when developing a new component, we can very quickly define in a YAML file a new testing env and start running tests using real infrastructure within seconds. 

Final thoughts 

1. Running component tests is very easy using some bash scripts
2. Attaching a debugger to the test using delve is also very easy
3. Changing between envs is very easy

 

Hope that some of you found this post helpful, and I’d love to hear about your testing experiences. Also, I’d highly appreciate any recommended reading materials on this subject. We’re currently enrolling our closed beta program; you can sign up for our beta if you’re using VSCode: JS/TS over Node.js.

Any questions? Ping me