I like tolerant people, and believe I am quite a tolerant person myself. Here's a thought (*) from today.

Personally, I don't pay much attention to holidays, or even weekends, in my own life. My philosophy is and remains that if what I'm dedicating my life to doing professionally is not worth my time and attention on a weekend, why should it during the "standard" business hours?

That said, I relentlessly and unconditionally respect the personal time of others.

Vacation? You have fun, I won't bother you at all. Weekend with kids? Enjoy, and, if you need a break and you see me online, I'd gladly chat with you about what you have in mind, but, from my end, you'd never receive any ask to have some "urgent" work done.

Want to trade some biz day to get an extra day off? Sure thing, I'd cover you best I can, and, rest assured, if anything goes wrong it'd be my responsibility — just make sure to get the job done, and we'll handle everything else in a BAU fashion.

Taking this further, I actually enjoy it quite a bit that most of the people I work with are very much used to taking weekends off to disconnect from work 100%. (Most of my peers these days are Canadians and Brits, which most certainly helps.)

In a way, I like to think I'm a big contributor to the culture where anyone can be off at any time, and the whole hive would still perform as it should. It just never crossed my mind that such a mindset should apply equally to myself, because, yes, as I said above — I like what I do, and I don't see the point in working on something that would not excite me enough to get [parts of] it done over a weekend or on a holiday.

Interestingly, this topic relates to an old debate about hiring "superstars".

On the one hand, from a purely corporate setting, hiring superstars is a terrible idea, as they are hard to replace, tend play by their own rules, and would, at times, strongly demand something that the organization has committed to provide, but, for organizational and other reasons, is unable to provide at the time. From these grounds yours truly is making a terrible mistake by refusing to "lay low" and pretend to be part of the "tribe", quite valuable yet perfectly replaceable. (**)

On the other hand, superstars are who help shape the culture, after all, and I firmly believe that good leadership consists of people who can not only find and hire the "right" superstars, but also a) trust them enough to form and sustain the "lead by example" culture, and b) retain them, so that those people feel empowered to keep making those changes, in ways that, quite frankly, do not always follow the chain of command, or the playbooks, or the overall "best corporate practices" of the company.

Thanks for all the congrats btw, two years and flying smoothly at PokerStars!

(*) prompted by the fact that in the past two weeks, more than once, I had calls with someone in their pre-5am. "You sure? It's damn early on your end of the world?" "Hey, no worries, I'm already enjoying my coffee at the computer, so why not, it'd make my morning."

(**) just realized while writing this that last two times I actually interviewed for a job (~2 years ago and ~3 years ago) I sincerely argued that my attitude is to never try to be irreplaceable, but rather to be perfectly replaceable — just in the right cohort of people, who are relatively high paid and relatively hard to recruit.

The next planned system design meetup is about storage and databases.

I'm preparing quite seriously, and this is perhaps the first time the slides are really worth reviewing way ahead of the event: link.

Comments are more than welcome, here in Telegram, or just in the Google Slides documents.

Looking forward to the real meetup, and I'll announce the date once there's a bit more certainty of my calendar (we're planning a team event for the whole next week, hard to say if I might be able to find time, but I'll do my best).

Thanks!

If you think Facebook's UI/UX and/or moderation policies are reasonable and without flaws, think again 😊

Also, an admin or a moderator of my own feed is yours truly, right?

We recorded and released a System Design Meetup episode about databases, specifically for distributed OLTPs. It's a longer one, so enjoy responsibly 😊

Kafkaesque problems require Kafkaesque solutions.

An educational System Design Episode on TinyURL: video, slides.

I'm trying to find the balance between in-depth conversations on the one hand and making sure our meetups are useful to less experienced engineers on the other one. Hope this is a good stare; we sure enjoyed it a lot, although it's nearly two hours 😉

The true complexity of this world is in explaining and understanding stuff.

A lot of things follow from this.

Corollary one, on science/engineering/technology and on engineering jobs. Engineering, especially software engineering, may well be the essence of the art of explaining things in formal terms. After all, computers are just that: ultra-performant entities with zero "prior knowledge" of how to interpret our "instructions". Thus, programming has to be absolutely precise. The ultimate challenge in formal explanations, IMHO, are computer science algorithms. When one can formulate a CS problem well, chances are they can explain its solution well. Even more true the other way: once one can clearly articulate how a CS algorithm works their explanation of the problem it solves tends to be sharp and concise as well.

Corollary two, on replacing jobs with AI. There is an emerging trend of "no code" in my industry, the flagship of which today is the demo of Codex by OpenAI. Outside software engineering self-driving cars are a notable example, along with cashiers or clerks. Here I am going to postulate an unpopular opinion that Codex would not change the world substantially, at least not any time soon. The reason, I think, is simply that the set of assumptions to keep in mind grows exponentially once the problem domain gets broader than putting elements into a DOM tree. Any human being, who went through a few dozen hours of learning how HTML + CSS + JavaScript result in stuff emerging on the screen, would be superior to today's "AI" the moment the problem becomes even a tiny bit less straightforward. My experience of building an NLP engine to convert English utterances into database queries proves this beyond reasonable doubt: the human language is inherently ambiguous, and the challenge an automated system faces is not in understanding every word, but in capturing their meaning, and this meaning is a) far greater than the sum of the words, and b) is very much context-dependent.

Corollary three, on implicit defaults. One of the biggest killers of quality explanations is when the subject matter itself contradicts our intuitions. This alone may be a good reason why theoretical physics attracts people of certain mindsets: one has to have their intuition calibrated in a certain way to work with seemingly contradictory observations in order to see the big picture. When humans communicate there are two "successful" modes and a death valley in between. The first successful mode is solving a math problem: the conversation is analytical, the arguments clear, and all the parties effectively incorporate them into their mental models. The second "successful" mode is discussing something ambiguous, such as what qualifies for oppression, with the adepts of certain mindset: their default state is to "agree", the people tend to agree with virtually everything, as the sense of mutual agreement is more important to them than making sure the parties understand what exactly they are claiming to agree with. Oftentimes their defaults are close enough for the differences to not matter. The death valley in between is when two people are communicating in two different "successful" modes and their defaults are out of sync. This is what leads to disasters down the road, and that's why big decisions are best to be formalized, documented, and cross-checked.

This also happens to be exactly why I believe the AI today can neither understand nor explain stuff; unless it's a very narrow AI, but we're talking AGI here, of course. GPT-3 can easily talk about what qualifies for oppression in the modern day. GPT-3 would fail miserably at solving anything nontrivial, as "just guessing" would not get one very far there. Try asking it to only talk to you in sentences of an even number of words.

The bright conclusion is that humans still matter, and we would for quite a while from now on.

Many years ago, back at Google, the junior me suggested a crazy idea: every line of code has a half-life.

In other words, the company is committed to keeping no legacy code, by constantly updating its codebase.

Say, your team shipped 20K lines of code in late 2008. At most half of them are allowed to still be part of the codebase by the end of 2010, at most half of the half can remain by the end of 2012, etc. If your code is business-critical, then make sure you re-implement at least half of its logic every two years.

Back then we concluded that it's not too bad of an idea, especially if we can keep the tests. Which we, of course, can.

Obviously, companies (and, especially, managers!) would not endorse such an approach. They are mostly in the "move fast, break things" mode, regardless, with tech debt piling up year after year being the rule, not an exception.

I still think it's a good idea. If your friends are building a company this way, let me know — I'd love to talk to them, at least.

~ ~ ~

These days I am having a déjà vu when it comes to infrastructure as code.

How about introducing the term half-life for infra? At least half of the servers you have configured for prod use today must be re-configured a month from now.

If you are cloud-first, you are, in a way, already half-way there. Servers die, or get decommissioned, and you have to have the process by which your code can be re-staged on fresh machines, being fully connected to the rest of the fleet.

Admittedly, in most companies there will be a few servers with fine-tuned manual configuration, for which this job is more manual than just a few actions. Those are exactly the points of impact that are not getting the attention they deserve most these days, and those are exactly the ones, IMHO, which a new solution, once it emerges, would improve the most.

A big reason I like the cloud is that it's easier to be disciplined if your deployment has to follow certain rules and constraints.

For on-prem deployment, this "luxury" is not there. In a way, it is making the DevOps job much harder in the long run.

~ ~ ~

Curiously, this is where the crypto community might be ahead of the curve.

The crypto folks don't have the luxury of relying on some cloud infra. At the same time, crypto solutions can not afford manual configuration, mostly because they are, well, decentralized by design.

But designing software to run in a 100% decentralized fashion doesn't necessarily solve the problems right away. Sure, decentralized solutions are a lot harder when it comes to reliability, low latency, and safety from bad actors. Still, a lot more down to earth problems, such as fault tolerance and node discoverability, have to be taken care of.

~ ~ ~

I am going to make a strong claim — and a falsifiable prediction! — that if a new and effective configuration / server management solution for one's own cloud, or for a hybrid cloud, would emerge, it would come not from the people who are thinking 24/7 of how to improve Chef or Ansible or Kuberbetes or CloudFormation.

Rather, it would be designed by those looking for better ways to speed up their smart contracts by offering users to run their own support nodes, for a modest income in certain tokens.

Time for my first Twitter thead.

Prompted by Figma's LiveGraph whitepaper. Was hoping it to be just a few tweets, but couldn't resist.

Consistent Hashing is the second episode of the Educational Series of the System Design Meetup.

An educational System Design Meetup episode on Consistent Hashing: video, slides.

We squeezed it into 1.5 hours, and managed to keep it fairly educational as opposed to a deep dive. Thanks folks!

Monoliths train strong engineers
Strong engineers design microservice architectures
Microservice architectures create weak engineers
Weak engineers build monoliths

(c) paraphrased from Slack comments of System Design Meetup

A crazy thought: Why aren't software decisions made in jury-style?

Imagine the question being discussed is to pick product A, vendor B, and option C it to build a solution in-house. Here's how the "pure" solution could look like.

• Screen developers who don't make the ultimate decision from all the conversations about the decision. They could run experiments, measure the numbers, read the code and documentation, make estimates, etc.

• Dedicate (twelve) jurors to ultimately make the decision. Make sure they are not exposed to any part of this decision at all. I.e. make sure they are not having lunches or drinking with those developers running the experiments.

• Have the advocates a.k.a. prosecutors/defendants for each option. As good senior engineers and architects should be able to advocate for an against any idea, these roles can and should rotate over several days (i.e. Alice advocates pro B on Monday, then pro C on Tuesday).

• Have the high-level moderator (a Head of Engineering or Data, or a CTO, acting as the "judge"), whose job is limited to a) making sure the right questions are asked and answered, b) making sure the thresholds against which the numerical results of engineering experiments are compared do get discussed and agreed upon beforehand and communicated to the jurors accordingly, and c) preventing off topics, and overall biased / under-justified opinions from becoming too vocal in the [court] room.

• Hold the process of making the engineering decision (or "the trial") over several days, making sure that all the decisions are ultimately made by the jurors, with the jurors only receiving the relevant input datapoints, and no noise.

Obviously, for simple decisions the above is unnecessary. But when it comes to company strategy, when people are biased, and when the technological future five years ahead is at stake, I'd love to watch such a process, participate in it, or even moderate one!

Those Dreaded Microservices: video, slides.

The programming language of mid-21st century would:

⒜ Have a primitively-recursively-evaluated static type system.
⒝ Focus on the very business logic / application code being imperative.
⒞ Enable the tests to be functional and/or declarative.

On ⒜, the point is that strong typing doesn't have to be limited to numbers vs. strings and objects vs. arrays. Whatever invariant can be enforced statically is worth enforcing statically. Just think of an IDE highlighting that you forgot an important corner case, not on the level of forgetting one switch / match option, but on the level of breaking some one-to-one contract as part of a presumably atomic mutation / transaction.

For ⒝, I'm becoming more pragmatic as I get older, and we just won't get functional any time soon. The hardware that that runs our code is imperative (don't get me started on OOP, it really mostly is syntactic sugar in its present shape & form, and actors-reactors or lock-free queues are imperative AF). Besides, the complexity of code is no longer within individual "app features", but rather in the contracts between various components (a.k.a. microservices and DB/cache/LB layers). So, keep it simple, and write strongly typed and easy to follow imperative code.

Finally, ⒞ is where we need most help, I believe. We've had some golden age of testing ~10 years, with the explosion of unit testing frameworks, mocking, shimming, model-based testing, etc. All these things are hopelessly behind these days, IMHO. Not to say they are bad, they just haven't adjusted to the modern day's needs. Most true production problems these days are nearly impossible to be caught with unit tests, unless the amount of thought and work put into testing is far greater than that put into development. Configure your message broker wrong, don't handle idempotency correctly in one place, or rely on the order of messages being as expected — and, voila, a few years later, under an increased load, those pesky production issues emerge like mushrooms after rain.

I wonder how many people are thinking along the same lines. The nerd in me would love to help the world to get there, but it's not clear how do I best start, aside from aligning my worldview with what other smart people have to say, and educating people who work with me.

Figured I could start executing on 2022 plans for the System Design Meetup before 2022 is started: https://dimakorolev.substack.com/p/happy-2021-holidays

TL;DR: Yours truly and the System Design Meetup now have our own Substack.

Got the permission to publish some of our real work's research, and decided my Substack would be a good place for it: https://dimakorolev.substack.com/p/open-policy-agent-musings

If you know about the topic, or know someone who knows about the topic, or know someone who knows someone, I'd appreciate a forward — the text and the research should be deep enough to be worthy of a few minutes of an experienced person who's in this space. Thanks!

Re. the Lex Fridman & Peter Wang podcast.

Folks, you both are missing the point.

There is inherent beauty in the fact that human-to-human communication is messy and ambiguous. This beauty is due to that when we communicate with humans we develop an emotional connection.

There's certain non-fungible pleasure in "clicking", with another person or with the audience. Over time, assuming two humans, or a group of humans, choose to and do communicate more and more, important higher-order effects emerge, ref. "finishing each other's sentences". These higher-order effects largely are what, in my book, qualifies for "building a relationship", and the way the humans are wired makes (most of) us long for quality relationships.

A seasoned dog owner would tell you their relationship with their dog follows a similar pattern, and activates similar neural and hormonal "pathways" in the dog owner's self.

The irrevocable property of this relationship being built and developed is that not just one human, but all the parties change over time; not only in our ideas and how we express them, but also in how do we align the higher-order concepts in our heads, which topics keep popping up, which are best to be avoided, etc., etc. It's hard to imagine a dog owner sharing their beloved dog with a hundred more dog owners -- there would just not be enough room (enough "intimacy" if you wish) for this dog to "contain" the nuanced relationships with all those humans.

Now, the argument that made me write this post is that the "ML-powered" (machine-learning-powered) programming is the approach that takes software development further away from the "formal", "robotic" process that it is today, and thus closer to the "human to human" collaboration.

I see a lot of wrongs with this argument, but the major point I have in mind is the following. If you want to build software by means of a language that is inherently ambiguous, you'd have to build a human-level relationship with the "ML-powered tool" that you are partnering with in order to build that software.

In other words, the very possibility of ML-based "messy" programming languages taking off is predicated on the critical mass of human beings who are both capable of and willing to establish human-style emotional bonds with the non-living entities.

Which I personally do not view as something impossible. On the cybernetics/computability level, I would argue that it already is relatively easy to "engineer" and then "simulate" a "partner" that would be able to successfully act as a high-quality replacement for, say, a pet.

But, on the grand scheme of things, it is my understanding that very few people would truly be comfortable taking such a "replacement" over a "real thing". In fact, the primary reason humans like pets is to more effectively fulfil their emotional needs.

To continue with the dogs analogy, think of K9 dogs that help patrol our airports. The human being who is working in a tandem with that dog has to establish a strong emotional bond for this partnership to work. How many humans are capable of acting out this role?

So, I think with ML-powered "programming" the state of the art would be about the same. In some specific cases — similar to K9 dogs helping to keep our airports safe — tandems of humans + adaptive ML models would emerge as a more effective solution, on the social scale. Much like the customer support department of virtually any large company is now augmented with robots answering voice calls and with chatbots that take a good share of the job away from human representatives.

But we all know those chatbots can only take us that far. *Even* for the domains where the scope of what can be done is remarkably limited and well mapped.

And this is far before we enter the the territory of thinking about the state of a piece of software evolving over time. Which requires the brain wiring of the level absolutely unreachable for even the top-notch ML systems of today. The example of asking GPT-3 to talk to you in sentences of an odd number of words comes in handy here; and it's still about "imagining" one step ahead, not a potentially infinite & unbounded evolution of the data in some DB and the UI/UX of some application.

And any nontrivial piece of software absolutely requires reasoning on this "infinite evolution of data" level, where the mutations of this data form chains and sequences, and inevitably depend on one another.

Now, maybe you can imagine a machine learning system that can "pretend" to "understand" you (by correctly reading between the words and "completing your sentences"), and thus convert your unfiltered & unstructured thoughts into fixed-form software instructions -- all the power to you! Maybe we'll get there eventually.

But, personally, I just can't imagine us, the humankind, developing such a technology in the next 20+ years.

Because, honestly, building correctly functioning software from incomplete specifications is literally the hardest possible AGI task I can think of.

Before we can get there, show me virtual drivers and virtual teachers or coaches and virtual doctors who consistently outperform humans. The experts in those professions — who most certainly are extremely talented and intelligent and hard-working — still do operate in far more constrained environments compared to software developers. Simply because the spaces of all possible driving conditions, or of all possible ways a student may have troubles grasping some topic, of human body malfunctions — these spaces are still rather bounded in their nature, while the space of all possible algorithms and data structures is literally unbounded, and it is expanding at an increasing pace as we speak.

To my taste, this is an extremely important post: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/

<rant>

First, it talks about a real problem in a real company, that affected millions of users for an extended period of time.

Second, at least to my taste, it's quite poorly written. Interestingly, at the same time, I believe the authors of the post do not think so!

It's easy to see through the post that the people who were debugging the issue are not seeing the forest, but just the trees.

The median latency that used to be 500ms and now is 2 seconds? Why is it even important? Should 2 seconds, or even 10 seconds, not be good enough for service discovery? How often do you even have new nodes to discover, to begin with?

Shouldn't the whole fleet work just fine if the service mesh / service discovery subsystem is down and not accepting updates? At least while no machines are failing, the previously "discovered" ones should just talk to each other as if nothing happened, no?

Shouldn't there be a read-only cache present for the cases where the mutation path (the write path) is slow? Shouldn't the Consul folks think about this? Especially when introducing the "optimized" access pattern, being fully aware that, under certain circumstances, that are obviously not well understood by most users, this "optimized" access pattern would inevitably be a disaster in the making?

The folks who wrote this blog post seem to not even consider the above questions important; at least it doesn't show through the text at all.

I get it, it is important to keep it cool, to praise the players who "DNS-hacked" the system to begin playing a bit sooner than others. And it is important to keep a good relationship with the Consul team. And it is important to maintain the image of a transparent, developer-first organization.

Nonetheless, to me, the post does read as an affirmation that software system design at scale is experiencing a major crisis. It's the kind of a crisis that can have a cascading domino effect of failures. Across multiple companies.

<more_rant>

Heck, in my professional life I do system design interviews and coaching. And a lot of people sincerely believe Mongo and/or Cassandra guarantee data availability and consistency across the whole spectrum of possible failure modes, while local hash maps are a big no-no "because Redis is better". Now Consul, that, presumably, does offer stronger consistency guarantees compared to Mongo or Redis, is behind a massive outage. And nobody appears to want to ring a bell; people are just praising each other and the overall spirit of teamwork.

I get it, if Roblox et. al. get down for a few hours every other year, the world would not come to a stop. But it's the same people who design the air traffic control systems, or our healthcare or elections software, or the semi-autonomous weapons that our military is planning to adopt as we speak. Doesn't this concern you at all?

</more_rant>

Okay, maybe it's me whining. But I seriously don't know what big conclusions are right to be drawn from this Roblox post. To me it does read as a poor post where the authors are not only explaining the situation well, but are genuinely clueless about how to build systems of the scale they happened to have built.

I do hope to be wrong; I do hope that at Roblox there are more people, behind the scenes, who have nothing to do with this post, but have a lot to do with helping clean this particular mess and prevent the future ones. But, from my experience, I can assure you that it is not the case generally, even if in Roblox in particular the situation is not as bad as I'm reading between the lines.

Because the hipsters do appear to run the show, at least when it comes to microservices / SOA these days, and the overall system design situation does look like the early stages of Idiocracy to my taste. With the conversation about using Mongo and Redis as a DB and a cache resembling the "it's got electrolytes" scenes more and more.

</rant>