Spoke a bit about NLP and related topics last week, at an ABD Society — Data Analytics Speaker Series event. The video is now up.

Looks like all good engineers around me have developed a strong belief that, when it comes to software architecture, those who use the terms "best practices" or "industry standard" are at best underqualified, and at worst are better off being removed from engineering ASAP.

— from 2 years ago

The tech scene has largely converged to the understanding that configuration as code is a good mantra.

I wonder when would we embrace the idea of management as code?

~ ~ ~

As an engineer, I think the largest win of configuration as code over the previously existing practices is that we have killed drag-and-drops and other UI-based operations in favor of configuration files.

And when configuration is stored in human-readable files, it allows us to follow best engineering practices, most importantly reproducibility, code reviews, and integration testing.

If some configuration, say, a new environment or a new deployment, has succeeded once, it can be projected to succeed the next time with, ideally, zero extra cost. If some configuration has succeeded at altering the state of three servers, it's a great start on the path to correctly altering the state of 10, 100, and 1000 machines.

Here I am not saying scaling is a straightforward problem. It never is. But reproducible individual steps most certainly lay the foundation on which it is a lot easier to predictably build large-scale systems.

I think it's time we, the engineers, do the same to Jira.

~ ~ ~

With configuration as code as the best practice with us for just a few years, we have collected more than enough data to know exactly what is easy and what is complicated. We know exactly where time is lost, and we know exactly who and how managed to get things done a lot faster than usual.

We no longer live in the world of “I'm a believer in approach X”. The world today is a lot closer to "we know for a fact that approach Y is superior to approach Z”.

And the world of today is a lot more meritocratic, as we don't have to rate candidates, for example, by how confidently they speak, or by how many configurations they have built in the past several years. Instead, we can accurately judge whether their understanding of the process of building and maintaining production services is correct. Much like algorithmic problems and big-O notation enable us to ask questions with definitively right and definitively wrong answers, in the world of configuration we have dramatically shrink the space where one's hand-waving and a "strong" resume can take them places; we're looking at the true substance these days.

Knowledge is the reduction of room for uncertainty, in science, and, especially, in engineering. And with configuration as code we, as the industry, have made a big step forward.

~ ~ ~

Now, when it comes to project management, imagine if each “permission obtained”, “access granted”, “cross-team dependency resolved”, or some “project successfully derisked by eliminating an unnecessary costly dependency” is no longer a vaguely recalled poorly told story, perceived and judged differently by different people.

Imagine it’s a well-traceable trail of pull requests, each of which visible to all or most of the organization, and each of which can be used later. Either as a reference of how one can best get things done in an effective way, or an example of how lost productivity and wasted time look like.

Won't this be a magical world?

Let’s make it happen.

For once Quora’s eng team has released a useful piece of research: Choosing Quora’s GraphQL client.

Great job Michael Yong. Also check out Ben Newman's comment, about Apollo Client 3.0 and its improvements over 2.6.

I like the general direction the world is taking at postulating the data access problem. At solving it — not quite.

I made a meme.

Realization of the day:

⒈ I actually enjoy building products.
⒉ If the backend for a product would take me less than a week, I don’t believe in such a product.

When a conversation about “product development” is exclusively about appearance & presentation, and not a single thought from, say, a ~15 minutes long part of this conversation has to do with the data that this product is about, I’m out.

Examples of good products: Google search, maps, GMail, Uber, Mixpanel, Tableau, AWS, online brokers, Kubernetes, multiplayer online games, programming languages.

Because building these requires carefully thinking through the workflows, most of which are far from trivial data-wise.

Examples of what I don’t even consider products: weather-tracking chat bots, DocuSign, Blind, modern messengers, Twitter, LiveJournal, Dropbox, Salesforce.

Because in these projects the major concern is not how to build them, but “how do we best present the user with feature X, which we believe they need”.

~ ~ ~

Simply put, I want to work on products where it’s clear a) what is that the user needs, and b) why it is hard to make happen from the technological standpoint.

And I do not want to work on products where the main “challenge” is to make the user want to do some obvious thing with a slightly better UX, and to have them enjoy the process slightly more.

~ ~ ~

I can totally see myself working on the technology side of, say, Telegram. That would be fun and exciting. But I won’t be the guy designing the “product” part of it, as it’s, well, too simple and thus too boring to my taste.

Actually, if I'm the CTO of Telegram, I’d make the product part of it open source, and let individual developers compete to build the best UI/UX part. Because the “product” part of Telegram can be handled by a bunch of teenagers; to my taste, it’s the tech behind it that matters for this particular product.

~ ~ ~

And to this day I don’t understand how come DocuSign is even considered a product. After all, it only exists because its “users” were too lazy to figure out which five features does their business process need, and then hire two good engineers to build the tech for that five-features process in a bulletproof and effective way.

I’m wondering why is “Zero to One” such a widely used term these days, and yet its counterpart, one to zero, is hardly discussed.

I want zero extra actions to have to be performed so that:

• the tests are run,
• the code deploys,
• the schema is respected,
• the paycheck is sent,
• the event is in my calendar, with the proper link to join the meeting,
• access permissions are granted upon being asked for, and
• non-engineers are not messing with well-functioning engineering processes.

Each of the above billet points by itself deserves a good book.

The pros and cons of starting a company.

Via Spencer Greenberg.

1. Autonomy
Pro: you're the boss and decide what to do.
Con: you HAVE to always decide what to do. There will be a huge array of options at any given moment, and you'll never know for sure which to work on. You can seek advice, but ultimately YOU are the one who must decide.

2. Lifestyle
Pro: since you're the boss, you'll have flexibility in your hours.
Cons: you'll inevitably be working a lot of hours. It takes a lot of work to succeed as an entrepreneur.

3. Resilience
Pro: it's an incredible way to train resilience, persistence, and problem-solving skills.
Con: the world will punch you in the face between 5 and 100 times, and if you ever give up, you lose. This is stressful, and most humans would give up after 5 or 10 face punches.

4. Expected Value
Pro: if you're well suited to it and work on a good idea, the expected (mean) value in terms of potential impact and monetary reward can be REALLY high. Some companies have truly altered the course of history.
Con: the probability of failure is high, and luck is a significant factor. And unless you have substantial savings, you'll likely be living frugally at first.

5. Learning
Pro: you will learn a tremendous amount. Even if the startup doesn't work out, this valuable experience will apply to MANY other things. I don't know of another way to learn so many things so quickly.
Con: you will inevitably make many mistakes (ouch), and you have to face up to them (double ouch) in order to learn fast enough. It forces you to acknowledge and improve (or develop workarounds for) your weaknesses.

6. Meaning
Pro: you can choose to work on an idea that is DEEPLY meaningful to you. Most jobs don't provide this level of meaning.
Con: if you fail, you will have invested a lot of time in (and then failed at) something deeply meaningful to you.

7. Respect
Pro: many people have a lot of respect for entrepreneurs, and it's considered cool in plenty of circles.
Con: this respect increasingly kicks in as success increases, and before that, some people won't even respect it as a career choice.

8. Relationships
Pro: you will likely meet lots of interesting people and build meaningful relationships with your team members.
Cons: you may have to deal with difficult personalities or navigate complex human dynamics when it comes to employees, investors, and/or co-founders.

9. Responsibility
Pro: it teaches you a deep form of responsibility, which strengthens your character.
Con: you are ultimately, at the end of the day, responsible for everything. You're the captain, the last line of defense, and the goalie.

10. Adaptability
Pro: it hones your creativity and adaptability, and can really build self-confidence and help you develop a sense of how much you're capable of (probably more than you think!), as you figure out solutions for complex challenges, develop new ideas, and map out how to make them a reality. It pushes your boundaries in new ways and makes you grow.
Con: it can stretch your creativity and adaptability to the limit. Plans, essential as they are to make, rarely survive their collision with reality. It sometimes pushes your boundaries more than you would like.

So, why NOT be an entrepreneur? Because it's a high risk, chaotic, stressful, responsibility filled, boundary-pushing, challenging life. And it's hard work.

Not everyone is suited to this path, and it's irresponsible to pretend that everyone is.
BUT, if you're well suited for it, it can be one of the most deeply meaningful, high value, high impact lives to lead. You'll meet lots of people, build resilience and adaptability, push your skills to new heights, and learn a STAGGERING amount.
For some people, it is absolutely their best life path.

You can learn more and get in touch with us at: https://www.sparkwave.tech

Source: https://www.facebook.com/spencer.greenberg/posts/10105424972822772

From a work chat:

— What is data warehouse? Is it a database?
— [ me explaining what MapReduce, Hadoop, HiveQL, Spark, etc. are ]
— Oh, I got it. Mongo can do it, right?

I'm conflicted between thinking whether it's our folks got something very wrong, or the Mongo folks got something very right.

Full disclosure: Religiously, I'm inclined to believe it's the former. Mongo hasn't done much right, to be honest, not to mention their marketing and product placement is misleading to say the least.

But the very fact that in 2020 the new generation tends to view the world of big data from an entirely different point is certainly something to ponder on.

How can programmers keep track of their design decisions?

Let me open up by quoting an okay joke from memory:

If you change random things in your code until it works it’s “bad debugging”, but if you can do it several thousand times a second it is called machine learning and pays three times what you are making now.

One way to understand this joke is to appreciate the power of the simple idea that in the right environment, that embraces natural selection, a bunch of well-tested heuristics generally do the job quite well.

If the above does not look anywhere near the answer you are looking for, kindly allow me to rephrase it in the following way:

• Design decisions are what shapes the implementation in code.

• Good design decisions are the ones that yield implementations that last.

• People with good design decisions get to make more of them later on.

Still not looks like an answer? Okay, let me set the record straight.

It’s not that programmers contemplate a lot on which design decision to make, then make this decision, and then adhere to it. The real world works pretty much the other way:

• The process of settling on a design decision involves multiple proposals and multiple arguments pro and against them.

• Throughout making the design decision, the proposals that were countered successfully are eliminated, with all the arguments against remembered by everyone involved.

• Ultimately, one proposal remains, and is being executed upon. If multiple competing proposals make it to the final round, an authoritative decision is generally being made, with people not agreeing with this decision often times leaving the team or the project or the company.

• It is at this point, when the decision is largely made, optionally, the design doc is written. It may well share some 90% of the contents with what it was during the proposals phase, but, generally, when the design doc is being finalized, the decision has been already made, and, often times, the work has already started.

Now, I hope, the above paints a clear picture that does constitute the answer to the original question.

Software engineers (a.k.a. programmers) don’t really keep track of their design decisions. They don’t have to, because given the information they have obtained during all the discussions before the decision was made, they would make the same decision right away in an eyeblink.

And someone with a strong opinion against the decision that hasn’t been made has either already changed teams, or remembers very well what exactly was the decision they disagreed with, and yet chose to stay on the team despite.

The beauty of engineering, and, especially, software engineering, is that the designs tend to get battle-tested. There’s a real-life feedback look. There’s Skin in the Game, to quote Nassim Taleb.

Software engineering, generally speaking, is no peace-time policymaking game, where arguments are about as strong as they sound to be. Software engineering is closer to a series of war-time campaigns, with the enemy being the laws of nature, systems complexity, and the strength and the dynamics of the team working to implement and later support the designs based the decisions made.

So, most of the time, there’s really no need to keep track of design decisions. What works a lot better is to surround oneself with experienced people who can readily provide plenty of arguments that both stick to everyone’s mind right away, and explain beyond reasonable doubt which decision is and remains the best one given the constraints.

"Read the last line of a huge text file" is my new favorite coding task for those who self-identify as system programmers in Python.

Our first system design meetup video is now up.

I enjoy interviewing, and enjoy helping people prepare for interviews. This year I realized the system design format is taking off, and I especially like conducting interviews of that kind. At the same time it appears there's not much quality content available online, so, without further due, we recorded this piece and are presenting it to you.

Kudos to Tilek Mamutov and the Outtalent.com team for helping to make it happen.

The format is similar to the practice sessions we ran before, with a few tweaks to make it more relaxed. It still is an experiment, to the degree that I haven't even wrapped my head around whether it should be called a session, a colloquium, or a meetup. Let me leave it for you guys to judge, and hope you like it.

I plan on making this a regular thing, and if you want to join one of the next ones, here's a form for us to pick the best date and time.

Thanks for all the interest to the previous post!

I've put together a Slack "workplace" (here's an invite link to it), and a Twitter handle. We are planning to host the next event this coming Friday or Saturday.

I also did some magic with the "connected discussions account" here on Telegram, so, if we're lucky, from this post on there'll be comments enabled here.

Next meetup video: Designing user authentication as if it's 1998 (spoiler: not really 1998).

I've enjoyed it quite a bit — thanks everyone who joined and helped! — and am looking forward to the next ones.

As a friendly reminder, here's our Slack, and here's an invite link to it.

A very valuable 16 minutes piece. Kahneman talks about intuition and how to use it effectively, as well as about the role of optimism, both for an individual and for our society as a whole.

Web Crawler: today's #meetup video.

With interesting detours into message queues and brokers, as well as into whether Redis can be used as a queue, and if FIFO is that important when it comes to managing the crawl queue.

(Let me know if you'd rather not have me post these links in this channel, still calibrating on this social media thingy. Thanks!)

Can't resist :)

Today's meetup: A distributed filesystem and much more.

And our next system design meetup video: designing WhatsApp / Telegram / FB Messenger.

With online whiteboards this time, hope you enjoy it! We had a lot of fun for sure, and, as always, we've learned something too.

As a fly in the ointment, my Ubuntu screen recording at 60FPS stared dropping frames, effectively freezing the video in the last quarter or so of the recording. The audio is fine. Lesson learned, I hope to do better next time.

And yesterday at the meetup we talked about reactive streaming.

I'm experimenting with formats tirelessly =) and this one was not really about discussing the design of a particular large-scale system, but more of a casual conversation on one of the important topics in modern software architecture.