Benchmarking Fly.io vs Render vs Heroku

With the ending of Heroku free tier at November 28th 2022, devs are in search of a better PaaS alternative to host their apps. The things I need in a PaaS are:

• Low latency
• Ease of deployment
• Generous free tier

Luckily, there’s a lot of available options, from Render, Fly.io, Railway, to Doku. Unfortunately, Railway cannot guarantee that it’s going to be free after the $5 credit is used, so I benchmarked Heroku, Render and Fly.io.

Method

Render and Fly.io App is deployed in Singapore Region, and the Heroku App is in US-East because SG Region is not available in Heroku.

• /health endpoint only returns “OK” as string. This should show the performance of the server alone. Tests for this endpoint are done in the terminal and REST API Client.
• /blog endpoint has query to PostgreSQL Database, hosted in the same service and same region. This should show the performance of the server and the database. Done in terminal and REST API Client.
• Google Authentication testing is done in Chrome browser, and should show the performance of the real world usage.

I used Baton, a Go based load testing CLI, and Insomnia, a popular and lightweight API client, to do the http request from my machine, requests are sent from my ISP, located in Jakarta. For Baton, I sent 30 requests with no multithreading to standardize the requests. Plus, my server has a rate limiter so I won’t try to get past that limit.

baton -u https://{{API_URL}}/health -c 1 -r 30

• -c 1 shows that we are sending 1 concurrent request
• -r 30 shows that we are sending 30 requests

Result

Here are the results with Baton CLI:

| Endpoint    | Heroku | Render | Fly.io | DigitalOcean |
| ----------- | ------ | ------ | ------ | ------------ |
| /health     | 600ms  | 2.1s   | 34ms   | 22 ms        |
| /blog       | 900s   | 2.7s   | 77ms   | 64 ms        |

As you can see, Fly.io is so much faster than Heroku and Render. For the Heroku, it’s understandable because of the regional latency. For the Render, it’s basically unusable.

Here are the results for the requests done in Insomnia REST Client and Chrome browser (for the Google auth).

| Endpoint    | Heroku | Render | Fly.io |
| ----------- | ------ | ------ | ------ |
| /health     | 1.1s   |  1.1s  | 187ms  |
| /blog       | 1.393s | 2.83s  | 372ms  |
| google auth | 600ms  | 5s     | 400ms  |

The results are the comparably the same, with Fly.io in the first place, Heroku in the second (despite being in US-Region with obvious regional latency), and Render being the last. So I chose Fly.io as my next PaaS to host my APIs, but deploying to Fly.io is trickier than Heroku and Render. I will write about deploying apps to Fly.io in the future so stay tuned to my blog.