Writing Sustainable Code

In this blog post, we will cover:

1. The Imperative for Change
2. Performance Enhancements
3. Tech nerd stuff
4. Migration / Release Outcome Summary
5. Conclusion
6. About The Author

In the digital world, an outdated backend system is full of inefficiencies and security vulnerabilities. After three years of stable operation, I've decided it is time to upgrade our backend infrastructure and skip NodeJS 18. This upgrade wasn't just about keeping pace with technology — it was a strategic move to align with Fluentos' core values of sustainability and privacy.

The Imperative for Change

Our backend, powered by NodeJS 16 LTS, had reached a critical point where stability intersected with stagnation. The upgrade to NodeJS 20 LTS was not just necessary; it was overdue, primarily driven by the need for enhanced security and performance. However, this migration set the stage for a broader change, encompassing authentication services, server frameworks, and even our approach to handling data and deployment processes.

From Auth0 to Supertokens

A pivot towards Supertokens from Auth0 for our authentication services marked our first major step toward reducing JWT token validation latency. Auth0 served us well; it's a great platform and very flexible. However, we often noticed that initial load times were slow due to token validation. Sometimes, it wouldn't load at all, which was very strange. The self-hosted solution from Supertokens solved this issue. By staying in the same region as our other EC2 servers, latency was reduced significantly, and the issue was resolved. As a bonus, we now use MagicLinks. No more usernames/passwords! It's as simple as it gets.

ExpressJS to FastifyJS

The transition from ExpressJS to FastifyJS was somewhat mentally hard. Initially, it was chosen due to its maturity, but not seeing any major updates three years later started to raise concerns. There's no doubt that ExpressJS is a stable framework with a large community surrounding it. However, upon evaluating alternatives, FastifyJS emerged as the JavaScript framework we will be working with going forward. It is lighter, faster, and has everything we need. TypeScript support was another essential requirement for us in choosing a framework. Additionally, the way FastifyJS is built is simply a joy to work with.

Performance Enhancements

Cloudflare Workers

The ever-growing complexity of our campaigns unveiled a critical performance bottleneck within Cloudflare Workers, primarily due to the inefficiencies of JSON.stringify and JSON.parse. By shifting end-user country data to HTTP response headers, we achieved a 4x decrease in CPU processing time and a 3x decrease in browser response times from 140ms to 50ms. Here, you can see the Cloudflare Worker CPU time before and after deployment. There is place for improvement, but I like where it's going.

Data storage on Cloudflare R2 was also changed to support a leaner and more decoupled setup. Previously, storage contained all domain and campaign information for both production and test environments. The new setup has domains, production campaigns, and test campaigns split. Additionally, file names are much easier to work with. Enforcing EU jurisdiction ensures less data travel between end-users and edge locations.

EC2 + FastifyJS

Fastify's lightweight architecture allowed us to utilize EC2 micro instances for our development environment, solving the memory constraints and crashes we faced with ExpressJS. This transition not only boosted our development efficiency (faster boot time, build time, and deployment time) but also reduced ongoing resource consumption. Furthermore, moving EC2 to the same region as RDS reduced latency. Read and write times were halved, and server response time decreased from 60ms to 30ms.

NX Monorepo

The backend was missing migration to a monorepo setup. The existing setup facilitated faster dependency management, streamlined builds, and deployments. NX is a great monorepo management tool, and I am very glad to utilize it for the backend as well. It finally allows all the apps to run in parallel without any of the unusual tooling that was previously required to run the whole infrastructure locally. It was a real hack-and-patch method to run locally, and I had wanted this for the last 3 years.

Database and Security Enhancements

Replacing auto-increment IDs with UUIDs addressed a latent security concern, eliminating the potential for unauthorized data access through URL manipulation. Furthermore, the overhaul of our permission model fortified our API's security, ensuring robust access control and data integrity.

Tech nerd stuff

If you've read this far, keep reading the nerdy stuff I crafted along the way.

The migration was planned carefully to ensure a zero-downtime release. Customers were informed about the upcoming release and potential disruption. The release plan was crafted with all the steps involved and their sequence.

Migration Plan Step by Step, in This Order:

1. Set up Supertokens self-hosting server with permissions and logging.
2. Add backend apps to an existing monorepo setup.
3. Add TypeScript 5.
4. Add Vite 5.
5. Set up a code baseline for FastifyJS.
6. Add a DB layer.
7. Add a Memcached layer.
8. Add a Marketing (Mailerlite) layer.
9. Add an Auth layer.
10. Add a Logging layer with NewRelic.
11. Add a Cloudflare (firewall, workers, etc.) layer.
12. Add an AWS SDK layer.
13. Add a Payment layer.
14. Add a Base Integration for the Campaign Integration processing layer.
15. Add PM2 for internal load balancing and graceful deployments/restarts on a multicore server.
16. Add a single route to check load times.
17. Set up a new EC2 to benchmark against the currently running live development environment.
18. Build tooling for deployment to EC2.
19. Start migrating routes from ExpressJS to FastifyJS.
20. Refactor Managers to be single-purpose and decoupled from one another, and move business logic into Controllers.
21. Refactor the DB data structure to use UUIDs and an enhanced Permission model to support Team Setup.
22. Refactor the Dashboard APP to use new Data Models from FastifyJS API endpoints.
23. Write a DB migration script for Go Live.
24. Do temporary mapping on existing Cloudflare Worker to facilitate old setup and new setup.

Release Plan Step by Step, in This Order:

1. Do not merge the working GIT branch into the master branch (to have a full revert if needed).
2. Create a DB snapshot and create a new DB instance.
3. Start a new EC2 production server with a different subdomain to avoid any downtime.
4. Set up NewRelic Monitoring for the new production server.
5. Deploy the production backend pointing to the new DB.
6. Deploy the Vue3 App Dashboard to production and use the new production subdomain.
7. Verify that it works.
8. Re-run campaign saving job to put all campaigns into new Cloudflare R2 storage (EU Jurisdiction).
9. Deploy Cloudflare workers pointing to the new endpoints.
10. Deploy Customer Facing Assets with changed country retrieval implementation.
11. Deploy Cloudflare Cron Jobs to point to the new endpoints.
12. Update Stripe webhooks to use new endpoints.
13. Update NewRelic synthetic health checks URLs to new APIs.
14. Update Mailerlite URLs to point to the new URL structure in the dashboard.
15. Monitor in NewRelic both the previous API and the new API to see if traffic is directed and for any anomalies.

Post-release plan:

Send out a legal update regarding Terms and Conditions and Data Processing Agreement which has been changed to support team setup.

After 2 days if all goes well:

1. Delete the Auth0 account and remove Customer data.
2. Delete DNS records pointing to old servers.
3. Delete old EC2 servers and RDS instances.
4. Merge the GIT working branch into the Master.
5. Get a beer!

Migration / Release Outcome Summary

1. Three months of intensive work.
2. Over 10,000 lines of code were migrated.
3. Simplified deployments.
4. Faster development environment build/startup times.
5. Leaner setup.
6. Deployment took six hours.
7. Zero downtime during deployment at a peak of 40,000 requests per hour.

Conclusion

The post-release phase was not just about consolidating our gains but also about future-proofing our system. Legal updates, cleanup of obsolete infrastructure, and updates to third-party integrations marked the final steps of our migration journey. The success of this migration reinforced our commitment to providing a secure, efficient, and sustainable digital ecosystem for our customers.

About The Author

Hi, my name is Ricardas Risys, and I am the CEO and Co-Founder of Fluentos. I'm also a husband, father, and indie hacker, very passionate about code efficiency and performance.