Simplified billing system with AWS Lambda. Comparision: Kotlin + Micronaut vs Kotlin + Dagger vs vanilla Kotlin

Serverless architecture is a relatively new approach. Good practices and proven toolkits for implementing solutions in this architecture have not yet established in the market. Some people are trying to adapt the tools and practices they used every day to build microservices, or earlier to develop monolithic web applications. Others try not to use frameworks and minimize the number of dependencies on external libraries. We, Java developers, have to rely particularly on dependency injection solutions such as Spring Framework. In this article, we will examine whether it is worth using this type of tools when creating serverless solutions on the AWS Lambda platform. We will look at different scenarios:

• using the Micronaut framework
• using the Dagger 2 library
• creating functions without the use of additional libraries

We will analyze the impact of the chosen approach on the ease of code creation and the speed of its execution on the AWS Lambda platform.

Test example architecture

Our tests were carried out on a simplified version of the billing system.

How does it work?

The user or another system uploads a CSV file with data into a dedicated S3 bucket. This event launches the first function – Billing Items Generation. This function calculates the accounting items to be invoiced on the basis of file data, file name and price table data from the DynamoDB database. The resulting items are stored in the database and a message is issued on the SQS queue, containing information about the need to generate an invoice.

When this message appears in a dedicated queue, it triggers another function, Invoice Generation, which generates an invoice object based on position data and customer data. This object is stored in the DynamoDB database and two messages are sent to specific queues: one about the need to prepare the printout, and the other – about the need to notify the recipient of the invoice.

The message about the need to print the invoice triggers the Invoice Printing function. This in turn triggers integration with an external JS Report system, which, based on the object sent, generates a PDF file and then returns it in response. The resulting PDF file is stored in S3.

The second message (the one about the need to notify the recipient) runs Notifications Sending function and using SendGrid and Twilio services sends an e-mail and an SMS with information about the prepared invoice.

The full code can be found on our GitHub.

Tested implementations

We have prepared 3 versions for comparison. All of them are based on the same code written in Kotlin. None of the frameworks offered ready-made integration with services provided by AWS, so we relied on tools provided by Amazon for S3, SQS and DynamoDB.

To integrate with DynamoDB, we also used DynamoDBMapper, which allows for easy mapping from and to POJO objects.

For the BillingItem table with the following fields:

key: String,
billingKey: String,
amount: Number,
beneficiary: String,
productCode: String

It is sufficient to create a simple class:

@DynamoDBTable(tableName = "BillingItem")
data class BillingItem(
       @DynamoDBHashKey(attributeName = "key")
       var key: String,
       @DynamoDBRangeKey(attributeName = "billingKey")
       var billingKey: String,
       var beneficiary: String,
       var productCode: String,
       var amount: BigDecimal
)

Then we can use the character repository:

class BillingItemRepository {
    private val client = AmazonDynamoDBClientBuilder.standard().build()
    private val mapper = DynamoDBMapper(client)

    fun save(billingItem: BillingItem) {
        return mapper.save(billingItem)
    }
}

DynamoDBMapper will take care of data conversion itself.

Integrations with SendGrid and Twilio in the NotifyInvoice function are realized with the use of ready-made clients provided by each of these companies.

For Twilio (SMS) it is enough to execute the following code once:

Twilio.init(accountSid, authToken)

And then each time:

Message.creator(
                PhoneNumber("+15005550006"), //from test number
                PhoneNumber("+15005550006"), //to test number
                "You have new invoice ${request.invoice?.invoiceNumber} for ${request.invoice?.totalCost}.")
                .create()

For SendGrid, we initiate the customer once by:

val sg = SendGrid(apiKey)

And we send each message by simply using:

val from = Email("asc-lab@altkom.pl")
        val to = Email("kamil@helman.pl")
        val subject = "New Invoice - ${request.invoice?.invoiceNumber}"
        val content = Content("text/plain", "You have new invoice ${request.invoice?.invoiceNumber} for ${request.invoice?.totalCost}.")
        val mail = Mail(from, subject, to, content)
        with(Request()) {
                method = Method.POST
                endpoint = "mail/send"
                body = mail.build()
                val response = sg.api(this)
        }

We carried out integrations with JS Report Online by ourselves, because we did not find a suitable client in Java. However, the integration was very smooth and consisted in simple POST call with JSON with several settings and data elements.

For the sake of simplicity we have created a query model:

data class JsReportRequest(
        val template: Template,
        val templateOptions: TemplateOptions,
        val data: Invoice
)

data class Template(
        val name: String
)

class TemplateOptions

And the request itself is realized via standard HttpURLConnection:

        val request = JsReportRequest(Template(invoiceTemplateName), TemplateOptions(), invoice)
        with(URL(jsReportUrl).openConnection() as HttpURLConnection) {
            val enc = BASE64Encoder()
            val encodedAuthorization = enc.encode("$username:$password".toByteArray())
            setRequestProperty("Authorization", "Basic $encodedAuthorization")
            setRequestProperty("Content-Type", "application/json")

            requestMethod = "POST"
            doOutput = true

            val wr = OutputStreamWriter(outputStream)
            wr.write(Jackson.toJsonPrettyString(request))
            wr.flush()

            return inputStream
        }

AWS Lambda does not impose any special requirements on the implementation. Everything always starts with calling a single method from our code.

We tried to prepare the whole in accordance with AWS Best Practices.

The first version is the richest in terms of tools. It uses the Micronaut framework both for Dependency Injection and for our mechanism of generating simple SQS queues clients with the use of Introduction Advice. The whole thing was initiated using Micronaut CLI and the create-function command.

A positive aspect is the RequestHandler built-in to Micronaut, which has a context caching ready. This simplifies the creation of Lambdas, as the framework takes on the entire process of linking its infrastructure with the mechanisms of AWS lambda.

The second version is based on Dagger 2, which allows us for compile-time dependency injection. Its use is limited to building the object of our function together with all the dependencies.

It is possible to create the mechanism of generating SQS queue clients using Dagger, but it would be more labor-intensive. Staying in the spirit of Dagger and avoiding reflection will result in the conclusion that the best option is the automatic generation of implementations using an own annotationProcessor.

The lack of ready-made integration forced us to write our own RequestHandler – fortunately, this boiled down to a few lines of code.

The third version uses Kotlin without any frameworks. Everything is implemented manually. There is also no automatically generated code.

Assumptions

We will compare the monthly cost in two variants.

The first variant is one million invoices per month and each invoice will have 10 items. We assume that this will spread evenly over 21 working days per month, 10 hours per day. As a result, we will have to process 4761 invoices every hour.

This will result in the emergence of many instances of Lambda, but there will be relatively few cold starts. We will come back to the last problem later – at the time of comparing the execution times of particular functions in different stages of Lambda’s lifecycle.

The second variant is 1000 invoices per month. As before, we assume that there are 10 items per invoice. In this case we reduce the time constraints a little. We assume that invoices will flow in such a way that every 10 of them will cause the start of a new instance of Lambda.

In both cases we assume that the size of the CSV file with data is 10KB, and the size of the resulting PDF is 1MB. Additionally, we assume that each PDF file will be downloaded once.

Costs other than AWS Lambda itself will be identical, regardless of the implementation.

Interesting fact: Since October we have observed that the Lambda triggers associated with the SQS queue generated about 15 “empty requests” per minute. With three queues and three triggers it can generate a cost of about $0.80 per month.

Test results

Micronaut + Kotlin

Dagger + Kotlin

Kotlin

As you can see, the biggest differences in call times are in the case of instance initialization, i.e. cold starts.

In case of using previously initiated instances (so-called warm starts) the differences practically disappear. The only difference we can see here is in the NotifyInvoice function, which has the greatest fluctuations in execution time. This is due to the different response times of the services with which we integrate.

At the beginning of our research, we used the latest available version of Micronaut, i.e. 1.0.0. Unfortunately, the example with its use had clearly worse results (longer execution times), also at warm starts. We started to investigate why this happened. As it turned out, with each query, Micronaut unnecessarily re-created the whole context of the function. We reported the problem problem and it was fixed literally in a blink of an eye. Thanks to this, Micronaut 1.0.1 tests give much better results.

The total cost of AWS Lambda calls including Free Tier:

This translates into the following Lambda’s cost per invoice:

Therefore, adding this to the previously calculated costs, we get:

As you can see, this type of installation (for up to 1000 invoices per month) is free of charge regardless of the chosen framework.

What else can be improved?

The size of the jar or zip file has a very large impact on the time of the first launch of the instance. This can be seen by comparing the time of the first execution of GenerateBillingItem (17MB) and NotifyInvoice (25MB). In this case, thorough elimination of all unnecessary libraries from our functions is very important. Of course you have to be careful not to erase something that is required by the framework.

The second topic to consider is the packaging method. In our tests, collecting everything into a zip file with jars inside saved about 6-9% of the cold start time. The above tests are based on Micronaut’s default packaging using the maven-shade-plugin, which creates a single jar with all classes.

Benefits of using DI frameworks

Dependency injection makes it much easier to create tests. This is particularly convenient with the use of Micronaut and its @Replaces annotation. Dagger requires a little more configuration, but it’s still quite convenient.

In the case of Micronaut, it is sufficient to perform the following:

@Replaces(bean = S3Client::class)
@Singleton
class S3ClientMock : S3Client() {

And we can create any mock object we want. In the test code, we no longer need to change the tests. We will get the context with a new implementation.

In the case of Dagger, apart from defining the mock object, we have to overwrite the module, which will return the instance of a new class instead of the original one and set this module in the tested component.

class FunctionTestModule : FunctionModule() {
    override fun provideAmazonS3(): S3Client {
        return S3ClientMock()
    }
...

And in the test itself, for example, we have to call:

val function: GenerateBillingItemFunction = DaggerFunctionComponent.builder()
            .functionModule(FunctionTestModule())
            .build()
            .provideFunction()

Micronaut also allows us to test functions by running them in full context and checking them all the way from request deserialization to response serialization. Then we have greater certainty whether our code will work after installation on AWS.

Using frameworks brings writing functions closer to what we have done in previous architectures. We have the same set of tools and conventions, so the transition is smoother. We can equally efficiently manage the lifecycle of beans or the way they are created.

A very pleasant aspect of using Micronaut was the ability to create a hello-world project in one minute using Micronaut CLI. This allows us to very quickly create the backbone of the solution and start writing without thinking about how to build the package properly, or what to configure on the AWS side.

It is also important to have the ability to easily transfer logic from existing solutions. Both Micronaut and Dagger support the standard JEE API for dependency injection (JSR-330).

Micronaut also provides very convenient tools for integration via REST interfaces. If we integrate with external resources or other Lambdas, this can make our work much easier.

Disadvantages

The use of the Micronaut framework causes that, by definition, small packages with functions are expanding. This causes several inconveniences. There is a clear correlation between the size of the package and the time of cold starts.

The second negative effect of larger jars or zips is the need to allocate more memory to functions. In Lambdas, Metaspace is limited and its size is proportional to the total allocated memory.

Even the hello-world function generated with the use of Micronaut CLI could not be activated without allocating at least 320 MB of memory. This may not be cost effective, especially when we do not care much about speeding up the execution of our code while we mainly wait for external services. A good example are the PrintInvoice and NotifyInvoice functions, in which the main component of the execution time is communication.

Frameworks are known for their appetite for different dependencies. Micronaut, while giving us many ready-made mechanisms, adds many libraries. Spring would have added much more of them, which disqualified it already at the beginning of our journey with AWS Lambda. Google Dagger defends itself against this and does not add anything apart from the CDI API.

All these disadvantages translate into higher costs of using Lambdas. Extended cold start time translates not only into a one-time cost when first used. If we receive many queries simultaneously, then, while waiting for the initialization of the first instance, AWS will create another ones. As a result, costly instances will be created. Most of them will be used only once, because later execution times will be even several dozen times shorter and there will be no need to maintain as many copies at the same time. Let’s also remember about the limits imposed on us by AWS. We can only have 1000 function instances and unnecessary consumption of this limit can cause us problems with larger infrastructures.

Conclusions

The above results and findings show that we have to compromise.

The more advanced the framework, the easier and faster we will be able to create function code and tests for the functions. We also get ready-made tools for integration and construction. Unfortunately, this will happen at the expense of slower operation and higher infrastructure costs.

None of the described tools provided ready-made integration into AWS services, so we still have to manually use the libraries made available by AWS.

In some extremes, such as functions that occasionally accept multiple queries at the same time, the answer is simple. In such a case, we should stick to the AWS Lambda guidelines and keep the use of libraries to a minimum.

On the other hand, if we do not care so much about the start-up time and our application is computationally complicated, then there are no obstacles in using Micronaut. This will have no significant impact on infrastructure costs or efficiency of the solution, and thanks to the capabilities of the framework (e.g. generating HTTP clients) we will simplify our implementations.

Google Dagger seems to be a good compromise. It provides us with an IoC container, but at the same time it does not impose its own tools and only slightly affects the size of the package and the time of execution.

Both Micronaut and Dagger will allow us to transfer pieces of logic from the existing code at a small expense. Even the transition from Spring should be relatively easy.

According to best practices, we should strive to make our Lambdas as light as possible. We will achieve this, among others, by minimizing the complexity of our dependencies. Therefore, a simple profit and loss account should be created for each case.