Don't YAGNI Your Code, Ask Your Code WAYGNI.

A look at how we reduce scope, what we lose along the way, and what we could gain if we didn't. With a short story from the startup trenches.

Back in 2018 I was engineer #1 in a B2B startup doing machine learning and data processing. On my first week at work I was given a Jupyter notebook prototype of what the product had to do. For readers unfamiliar with notebooks - think of a list of commands to run one by one on a computer. The gist: with my arrival the founders will focus on sales full time. Meanwhile, I will be responsible for the technical execution. Which at first meant I would be running that notebook for each client.

With the founders doubling down on getting clients, I did not want to be clogged up with running the notebook many times over. I set out to write a more robust MVP (Minimum Viable Product). In effect, join terabyte sized datasets, filter them out, train and validate a model, predict some things. Sounds like a data pipeline and workflow orchestration. Both problems that are solved many times over. I pulled my sleeves, started researching and found two frameworks to build the workflows: Apache Airflow from Airbnb & Luigi from Spotify.

By the end of the day I had running prototypes using both - not a bad start. Airflow was gaining popularity, but it lacked tests. I was concerned that over time I would end up having to fix someone else's bugs. On the other hand, Luigi did not have a scheduler - I was not sure if it would become a problem later. I also had a nagging feeling that something was not right with my whole approach.

The same evening my friend swung by the office and, as we were walking out, I told him about my dilemma. I remember he asked a simple question that reframed the whole problem for me: “What is it that you need your code to do right now?” I realised I have forgotten about YAGNI (You Ain’t Gonna Need This).

Next day I came to the office, took the prototype code, tidied it up, wrote some tests, and made them into three command line tools. Build dataset, train the model, process client input. No fancy pipeline or workflow orchestration - three flat Python scripts with test coverage. With time we added new features and built a website. The code evolved, but these three files remained the core components of the product for the next few years.

I saved myself some maintenance time and infrastructure cost, both valuable for the business at the stage we were in at that time. Eventually, we did pick up a workflow orchestration framework, but let me tell you about the lesson I got from the experience.

WAYGNI to your YAGNI

You could ask - isn't this just another story about YAGNI (You Ain't Gonna Need It) or YDNIN (You Don't Need It Now)? In my experience they are both used akin to trump card. They are put down to reduce scope and move on. I think there is something missing there. In my example the scope reduction did happen, but I reflected about moving on.

Upon reflection, I realised that with the focus on cutting off the excess scope I got the result I needed, but I also lost something. I have done preliminary work on how to use Airflow and I threw it out. Even though, realistically, a workflow orchestrator would become a viable investment later. Which it did and we had to rush research and decide on a suitable one. But we shouldn’t have. While the code would definitely had been outdated, had I documented some thoughts after cutting the scope I would have had an early warning light to start the research early. Which got me thinking about WAYGNI (When Are You Gonna Need It?)

The core idea behind ‘When Are You Gonna Need It’ is this: for any significant reduction in scope, document what metrics, observations, or product requirements would warrant bringing that scope back, in full or in part.

This WAYGNI exercise is a thought experiment, that brings clarity and establishes thresholds for when a more complex solution would be viable. Doing this exercise builds awareness about the decisions, and if diligently documented, could equally lead to the moments of invention I have mentioned in a previous article.

Moments Of Invention

Is this not RFC, Design Doc, or ADR?

Many companies have some sort of Request For Comments, Design Docs, or Architectural Decision Records, that most often include alternatives. Pragmatic Engineer newsletter has an excellent (paid) article about them:

Engineering Planning with RFCs, Design Documents and ADRs

I think WAYGNI is different, because it is not about the large scale decisions (which definitely need an RFC, Design Doc, or ADR). It is about the smaller scale stuff. For example, where you may have already invested some work and are told to reduce the scope. Or when the company or team is at the stage where spending time on formal design docs is hard to justify, a.k.a. an early stage start-up.

Scope control is a difficult part of software engineer job. It is impossible to catch all instances of YAGNIs, but when you do - it is equally important to document WAYGNIs. The extra context and thresholds channel the premature optimisation energy into something that can be helpful later. A clearer picture of how and when some complexity might be introduced into a system.