Leverage Part 1 - Open Source
Yep. Yet another blog post about Open Source software. You might say “There’s nothing new you could possibly say about this”, and you would be 100% correct. However, the way we design and build software is continuously evolving, so these topics need to be revisited from time to time. Also, this is based on my experiences, which might have a different perspective and thus be helpful (at least…. that’s the intent).
Build vs Buy?
This is a classic decision, and it’s something we should revisit regularly, but when the “buy'“ side of the equation is $0 what happens? Since this shift happened after I joined the engineering workforce, I was (and continue to be) very aware of it.
Costs
First off, it’s probably common knowledge that using Open Source has other, non-monetary costs. I’m showing some costs here that I think exist in one form or another, regardless of the organization size (which I’ll over in a bit). Here are some of the more significant costs that I have seen:
Maintenance - Open Source teams put most of their efforts into supporting their latest release. Some larger teams may patch older versions occasionally, but this is only the case for projects with large communities and good processes for managing contributions to older versions. For Open Source projects with small teams, they may only support the latest version, so this is something to consider.
Selection process - The process of selecting which open source products to use is in and of itself a cost. It’s work, and it often involves many representatives from the teams that intend to consume it. Building consensus and documenting the decisions, regularly reviewing prior decisions, etc.
Learning curve - Frequently the Open Source products are frameworks or libraries that we intend to use to abstract away some details that are not core to what we’re trying to build, or core competencies of the company. To properly leverage any framework or library, the engineers must understand the concepts and features, so there is a learning curve that they must climb. I’ll write another post about frameworks specifically, but this cost can be significant and is often overlooked.
Fragmentation - Containerization, microservices and other ways of splitting up and isolating parts of a system allow incompatible frameworks to coexist, which is wonderful in a lot of ways. However, it can lead to a proliferation of frameworks and tools that can multiply the learning curve and maintenance costs. Additionally, this can limit the engineer mobility, which increases the cost of re-allocating resources within the organization.
Context is Important
The process of leveraging Open Source products can be very different in a larger organization than in an early stage startup.
Adoption
This can vary dramatically with the size of the organization. An example of the extremes:
Small, Early Stage - The small team of engineers goes up some learning curve over a month. There isn’t a lot of existing code, so you just re-implement something in a week or so. Maybe even just a weekend. I experienced this when adopting ElasticSearch as a way to serve the query behind the main page(s) of app, that was a few years old. I saw the same thing replacing a Rails BEFE (back end for the front end) with Node/TypeScript.
Large organization - Lots of existing code, so you have to make sure that the old / new tech can interact and co-exist. One of my favorite stories is Microsoft adopting Rust as an option for system programming - they had to re-build the Rust compiler from scratch. Ryan Levick has a great story about that.
I don’t intend this as a knock on large organizations. Large orgs have resources to do things that small organizations can’t even dream of.
Build vs Buy (reprise)
The two biggest differences in the small vs large organization in terms of a build vs buy decision are:
Resources - Small organizations must remain focused on their core competencies, and are very concerned about spend, so they will naturally choose “buy”, especially if the monetary cost is zero.
Existing systems - Small organizations don’t have as much stuff, so they don’t need to think as much about co-existence and compatibility.
Compliance - Open Source licenses may have stipulations that are not compatible with compliance requirements for the company.
More about each of these follows.
Resources
A larger organization may be able to afford an entire team to research building a new framework or system, and they might be able to absorb the cost if it doesn’t pan out.
A small, early stage organization is often playing “beat the clock'“, and does not have the time for the “build” option.
Existing Systems - Compatibility and Co-Existence
A larger organization may decide to build their own when the open source alternatives would require extensive adaptation. This can come in several forms:
Integration layers - Layers on top of the open source product to make it compatible with the existing systems.
Modifications to the product - Fork and modify the Open Source product, but you potentially have issues with the stipulations in the Open Source license (e.g. LGPL requires you to contribute those fixes / changes back to the project in certain cases). At the very least, you are going to be maintaining the fork.
Refactoring the existing code - This may take years. Possibly even decades for large code bases. That’s a huge bet.
For these reasons, sometimes a larger organization will build their own internal product. Hopefully, they will consider launching it as Open Source. There is a cost to this, but there can be huge benefits in terms of promoting the brand and the company. Sounds like a good topic for another blog post.
Also… Compliance
If you’re reading this blog post, you probably know about GPL and other “transitive” Open Source licenses. There are many variants, and they all bear examination. In a smaller, startup context there often isn’t a whole lot of thought about this, but it can potentially cause serious problems later on as the startup grows. A larger organization will have compliance concerns and a legal/compliance team that helps the company navigate this.
Again, this is not a bad thing about large organizations. In fact, I would argue that small organizations mostly gloss over this because they don’t have the bandwidth to really deal with it. It depends on the domain too - the finance domain mostly has more oversight (and that’s a good thing).
Summary(ish)
So what’s the point of all this? That “it’s complicated”? Well… yes. Open Source has transformed things and can provide benefits for both small and large organizations. The main thing I wanted to touch on here was that the way that that open source is leveraged can be very different depending on the context.
