While what you’re saying is, of course, fair, the other side is also true: people often choose to complex a technology “because it’s future-proof!”, while they need just a simple system. There are tons of examples: CQRS over simple databases, Hadoop while a single server suffices, or even people choosing JIRA over a simple trello board.
As such, the obvious answer is “it depends”, and making the right trade-off is rarely properly captured in a single rule of thumb such as “always choose simple”.
> CQRS over simple databases, Hadoop while a single server suffices, or even people choosing JIRA over a simple trello board
I've heard this from many engineers over the years.
But what I've often found is that they were never involved in the decision making process and so aren't aware of all of the business requirements. Once they are made aware usually they agree with the choice.
Common example being the business requirement to have high availability.
It seems to me that the concepts of high availability, scalability and resilience are things that are just not taught or emphasized at most companies average at all. As Ops, this is a daily battle of education and prod readiness checklists.
It is even the simplest things that are ignored such as:
- what happens to your db query when you have 1 million records?
- what happens when the cache goes away?
- do you even use a cache?
- why are you treating your cache as a DB?
- can we run 2 copies of your service?
- can your service scale up and down?
- does your service have state?
Kubernetes has been a boon for us in this department. Since pods can be moved/deleted at a moments notice we have had to make our software resilient and stateless. Yes, kubernetes is complex at first but it pays off hugely when you are no longer getting paged because 'server X hung and needs to be rebooted'.
A genuine question that I wonder about: How does a single server protect you against someone in your DC pulling a plug by mistake or your cloud provider being unreliable? If it’s a database it seems like a huge potential for going out of business.
I guess if you are running single-server databases you better have a good backup strategy anyway, but is that a risk people and businesses just accept?
If you have a single server pulling of a plug (or even any HW failure which cannot be fixed quickly) is a very rare even (unless you use cheap desktop hardware instead of a decent server with ECC RAM, hot-swap HDD, redundant PSU e. t. c.).
If instead of using this single server (with manual fail-over in case it would fail) you'll start to create complex HA cluster from multiple servers not having enough people to properly design, test and maintain it you can end up having a less reliable system.
I think what you are missing is that simple databases in a well run data-centre are surprisingly reliable.
The tradeoffs involved in avoiding some of the simple "the cleaner unplugged the DB server" introduce complexities which may themselves cause outages which are much harder to solve.
The recent fire in a French DC saw serious disruption across all kinds of industries in many European countries. I know a sales engineer in a hardware manufacturing place, who was back to pen and paper until the IT provider figured out restores. But it wasn't that bad, as both the clients and competitors were decimated too.
I've never understood the hype about CQRS. To me it looks like a nightmare for a tradeoff that you "may" only need at some point in time. Particularly given GDPR, HIPAA, PCI and other compliance frameworks that require you "the right to forget". All books about CQRS talk and talk about the wonders of it and when it comes to that detail they just limit themselves to say "yeah, you should consider it"... but the reality is that the implications are huge.
As such, the obvious answer is “it depends”, and making the right trade-off is rarely properly captured in a single rule of thumb such as “always choose simple”.