2023 EOY Review

It's the last week of 2023, a good time to reflect on one's personal growth, accomplishments, and skillsets acquired this year. For me, it had much to do with Go, Cloud Native development, and prompt engineering.

In this post, I'll highlight some of the personal projects, concepts, and tools I honed in on in 2023. With no shortage of new developments in the horizon, I'll also detail on what I've planned for 2024.

Three hundred and twelve days ago, I wrote my last blog on Go vs Rust, where I debated on which of the two languages to learn. As I stated, I opted for Go, the simpler language of the two. Ultimately, I wanted additional bandwidth to learn the broader, more language agnostic concepts of distributed programming.

In my blogging hiatus, I published Gopher, a Golang project for learning and implementing the different facets of web development with cloud native principles.

Gopher is still an early work in progress, but it already includes viable examples on how to implement various CNCF technologies. At its core, the project builds the brains-api REST service and deploys it to a Kubernetes cluster. The Makefile highlights how the service deployment layer is instrumented with the use of docker, helm, and kubernetes. Assuming that all of the dependencies are set up, the service can be cloned, spun up and spun down using the commands make dev-up and make dev-down, respectively. Putting all of this together has given me a greater insight into platform engineering, devops, and CI/CD development.

Under the pkg directory, I created many stand alone packages to act as my foundational/re-usable packages for building distributed services. Each package embodies a concept within my learning experience, where I sought to build a component of the total project from the ground up. While each package is deserving of a proper write-up, here is a quick overview of the most notable ones; keep in mind, these are still works in progress.

• brain: A package for building neural networks from scratch. It includes the inception of packages used for activation algorithms, as well as a package for generating random test data.
• config: A convenience package for dynamically building an application's configuration struct with the option for defaults, making use OS ENV variables, and flag options.
• web: A very minimal web framework embodied with the standard net/http Server. Rather than using an established 3rd party framework, I intended to build features only as needed.

In a time not so long ago, engineers who mastered the art of Google-Fu had an edge in the acquisition of knowledge. In similar fashion, learning how to write effictive LLM prompts is a skillset in itself.

At the start of the year, I was extremely hesitant and skeptical of ChatGPT and generative prompts in general. For one, there was a self-sense of pride and integrity - using ChatGPT was somehow 'cheating' my learning. After much internal debate, I couldn't deny the inevitable: we are in the dawn of an era of 'pair-programming' with AI copilots.

Unsurprisingly, my productivity sky rocketed after a few weeks of using ChatGPT. The fact that I could generate examples, ask questions, and validate my understanding on any particular subject spared me hours of searching and reading. I found myself moving from concept to concept relatively quickly, allowing me to build more in a shorter timespan. While I still refrain in copy-pasting my code into ChatGPT (or integrating it within my codebase), prompt engineering has become my new age search engine.

In my current role, I specialize in Observability, building and sustaing platforms that collect, clean, and present data. Part of my job also includes researching and defining standards for telemetry, for which I test, document, and present to developers and engineers.

Some of the common challanges that I've seen across multiple organization are, but not limited to:

1. How data is created.

   • Different teams and/or languages tend to follow different approaches for generating telemetry.
   • What to log and what metrics to collect is never easily agreed upon, especially when the teams are large and responsiblities are shared.

2. How data is transported.

   • Many times, how data is sent from a system to a backend storage is determined by the analytical platform used; these platforms often define product specific processes, creating vendor lock-ins.

3. How data is correlated.

   • Distributed services, especially those that cross different programming boundries, tend to differ in the data that they generate.
   • Teams often have to define schemas for internal telemetry data and sometimes need to build the data transformation pipelines before the different data sources across services can be properly correlated in analytical and monitoring platforms.

As one can imagine, these are tedious, time consuming efforts that are urgent, but often de-prioritized in favor of new features. That is where OpenTelemetry comes in: an open-source Observability framework for creating, processing, and exporting logs, metrics, and traces. The framework provides an API, which includes defined standards for telemetry data that is both vendor and programming language agnostic. Furthermore, it provides an SDK, which seperates the logic of creating the data from the export of data.

Much of what I've done with OpenTelemetry has been for my job, so personal work on the matter is quite limited to the public. None the less, I've created a foundational telemetry package under the Gopher project that I hope to continue building on.

As stated earlier, I made the decision to focus on Go over Rust at the start of 2023. In hindsight, it was the best choice I could have made at that point in time in my career for many reasons.

Coming from Python, Go taught me the importance of data integrity through a statically typed language. I learned to appreciate code that was idiomatic, easy to read, and easy to write. Go's prevalence within Cloud Native helped me better understand distributed systems at a broader level.

To be frank though, I could never fully convince myself on the argument I conveyed against learning Rust. The pros of why I should priotize Rust resonated to me the more I learned Go; I just couldn't persuade myself to make the switch.

It wasn't until I stumbled on the book Zero To Production in Rust this past November, when I started to change my mind. The book encapsulates the core ideas on backend development that I've sought out, but in Rust. Having already learned backend development with Go, I now feel more confident in approaching it again but in a more challenging language.

Therefor, if you haven't already guessed it, I'm opting to make 2024 the year of Rust! As such, here is a quick list of what I have planned:

1. Build a distributed backend service with Rust.
2. Start and build out my personal project crab, a rebuild of gopher, but in Rust.
3. Master asynchronus and concurrent programming in Rust.

I wouldn't be remiss to also include more blogging about Go, Rust, and machine learning in 2024, so stay tuned.

As I reflect on 2023, I am proud of what I've accomplished and what I've learned. Looking forward, I'm excited for what challenges 2024 has in store.

What are your goals for 2024? Leave me a comment to let me know!

Thank you for reading. Have a happy New Year!