An overview of my tech stack

When first setting up my NSW Property Sales Insights app, it took me several days to put together the overall tech stack. There was a lot of time spent refining the original Python code which read and transformed the source data, and then further time spent configuring my AWS environment and learning the Chart.js library to make it presentable.

When this was completed, I decided to try a new dataset through the framework and started working on my Australian and NSW COVID-19 Insights app. This was overall significantly quicker to build than the Property Sales Insights App, even though the charts are much more complex. In fact, the bulk of the time spent on the COVID-19 Insights project was customising the charts.

I decided to write this post about my tech stack in order to support others who might be interested in setting up a similar framework for themselves.

This is a high level view of all the tools that I use to run my website, and how they are used together:

The HTML and CSS is all written by hand (I detailed in a previous blog post about my minimalistic rebuild why I prefer simple HTML/CSS over frameworks for my personal site) using Visual Studio Code, and JavaScript is only used where it is absolutely required. Python code is written using the Anaconda platform.

This is an overview of each of the different types of tools:

1. Python - Used to extract and transform data in a way that can be used quickly and easily by my web applications. Libraries used include pandas and NumPy. Specifically using the Anaconda platform. Data output files are uploaded to S3 using AWS CLI.
2. AWS Command Line Interface (CLI) - Used to quickly upload data files to AWS S3 from a local directory (refer to my previous blog post on running an Angular site on AWS for more info).
3. S3 - Storage for data files and website files. Files are stored in "buckets" based on their content, i.e. all web files are stored in the same bucket.
4. Lambda - Severless platform that reads the data files from S3 and performs further on-the-fly transformation required for front-end presentation. Functions were coded in Python. I chose Lambda over other options available (like EC2) due to the on-demand nature of the access required, and the cost efficiency of Lambda. See my previous blog post on AWS Lambda and Angular on other advantages of serverless.
5. API Gateway - Manages APIs that the front end uses to call Lambda to get the data.
6. Visual Studio Code - Integrated Development Environment (IDE) used to write the HTML, CSS, and JavaScript (excluding external libraries) used in this website. Content is committed to the CodeCommit repository using Git.
7. Git - Used to track changes to my web files and commit them to a repository on CodeCommit.
8. CodeCommit - Source control which hosts the Git repository for my website.
9. CodePipeline - Watches for any commits to the repository, then automatically deploys the files to S3.
10. CloudFront - Content delivery network which allows secure presentation of website files stored in S3.
11. Google Fonts - External JavaScript library. Provides the very nice fonts used across the website.
12. Chart.js - The JavaScript charting library which renders charts using HTML5 Canvas.

Some tools are admittedly overkill for a personal site. For example, CodePipeline isn't technically essential (I could just use AWS CLI to sync my web files to the S3 bucket), but since I am committing my code to a repository in order to track changes anyway, CodePipeline is a seamless way to immediately upload the files to S3 after each commit. It also gives me the opportunity to add testing capabilities and manual approval gates before deploying live if I'm making significant changes.

Now that the tech stack is established, it is something I can reuse for future insights projects.

Diagram made with diagrams.net with additional icons from AWS Architecture Icons and Visual Studio Code. Icons labelled in the diagram for the purpose of describing the tools.