Goroutines to improve Aurora Serverless V2 performance via Lambda

This was perhaps the first time I ever experienced the trill of using Goroutines. I enjoyed writing multi-threaded applications in Java back in 2011, and later I spent my time cherishing the async programming paradigm. Having practicing Golang for quite some time now, I now believe that I am spending my time in a good way.

The problem by itself is very simple — insert 10k records in a relational database in least amount of time and in an inexpensive way. The time and money aspect made it more tricky.

What made it tricky?

We have written most of our Lambdas using Golang. That is not a problem. In a super-cloud-native environment, where we boast our adoption of serverless technologies and serverless-first approach — doing such database transactions raises the eyebrows to say the least.

I wrote a simple Go function to do this job in a basic for loop. The data source was an external API which had it’s own issues with pagination. The pagination was sorted, but then now this simple for loop took around 1.5 hours to sequentially insert these 10k records.

As soon as the execution passed 15 mins mark, I immediately thought of Lambdas, and I said to myself — this won’t work. Additionally, to keep this data fresh there were talks…