New meetup video: friends & content suggestions. Two of us present two designs, one mostly based on graph traversal / hardcore CS, and the other one mostly based on embeddings / real-time ML.
YouTube
System Design Meetup 2021-Feb-27: Friends & Content Suggestions
We present two possible designs of "people you may know" or "content you may be interested in": the CS one based on graph traversal and the ML one based on embeddings.
I am cautiously optimistic for the world where the reopening does happen, but quite a few people still enjoy working from home.
This would give those of us who are both tech-people and people-people a tremendous opportunity: to live and work literally anywhere, and have business trips of a few days to ~1.5 weeks long once or twice a month.
With all my dislike of the modern corporate structure, if most of my team is remote, we gather often via video chats, and yet some ~half of us enjoy to see each other in person, which we do by gathering in various places for mini-offsites on the company's buck, as the company clearly understands the cost-effectiveness of such an approach ...
... that's the world I might enjoy living in.
~ ~ ~
Although it's all a distant dream so far. As of today, most places can't even be traveled to, and only the most adventurous of us are still up to asking the very "where to work from next month" question.
It is incredibly sad to see how herd-able by fear the people are. But, much like in 2008, a sober reminder is that in Mandarin "crisis" consists of two hieroglyphs: dangerous + opportunity.
Be safe & sane, everyone.
This would give those of us who are both tech-people and people-people a tremendous opportunity: to live and work literally anywhere, and have business trips of a few days to ~1.5 weeks long once or twice a month.
With all my dislike of the modern corporate structure, if most of my team is remote, we gather often via video chats, and yet some ~half of us enjoy to see each other in person, which we do by gathering in various places for mini-offsites on the company's buck, as the company clearly understands the cost-effectiveness of such an approach ...
... that's the world I might enjoy living in.
~ ~ ~
Although it's all a distant dream so far. As of today, most places can't even be traveled to, and only the most adventurous of us are still up to asking the very "where to work from next month" question.
It is incredibly sad to see how herd-able by fear the people are. But, much like in 2008, a sober reminder is that in Mandarin "crisis" consists of two hieroglyphs: dangerous + opportunity.
Be safe & sane, everyone.
Sorry in advance for Russian, promise to keep this rare — интервью, которые мы с Тилеком Мамутовым (Outtalent.com) записали в Ололо в Бишкеке. Спасибо, было офигенно, и ты отличный интервьюер!
Forwarded from Грамм Тилека
Дима Королёв (@BoreMeNo) - архитектор программного обеспечения в Poker Stars, преподаватель, основатель Systems Design Meetup. Кроме этого он работал программистом в Google, Microsoft и многих других компаниях.
В этом интервью мы поговорим о собеседованиях по алгоритмам, systems design и о том, как можно развивать карьеру программисту.
https://youtu.be/iYvWLflZ9E4
В этом интервью мы поговорим о собеседованиях по алгоритмам, systems design и о том, как можно развивать карьеру программисту.
https://youtu.be/iYvWLflZ9E4
YouTube
Дима Королев — о собеседованиях по алгоритмам, systems design и как развивать карьеру программисту
Дима Королёв — архитектор программного обеспечения в Poker Stars, преподаватель, основатель Systems Design Meetup. Кроме этого он работал программистом в Google, Microsoft и в многих других компаниях.
В этом интервью мы поговорим о собеседованиях по алгоритмам…
В этом интервью мы поговорим о собеседованиях по алгоритмам…
A system design meetup episode on MapReduce: video, slides. Thanks folks!
And sorry my OBS Studio didn't record your questions this time — good thing I have a habit of repeating them from offline conferences :-) also, I should probably not forget to hide the taskbar next time.
#incrementalimprovements #practicemakesperfect #thanksforhangingalong
And sorry my OBS Studio didn't record your questions this time — good thing I have a habit of repeating them from offline conferences :-) also, I should probably not forget to hide the taskbar next time.
#incrementalimprovements #practicemakesperfect #thanksforhangingalong
YouTube
System Design Meetup 2021-Apr-10: MapReduce
A short episode on MapReduce.
0:00 Warmup
0:29 Intro
1:34 Agenda
2:15 Goal
4:51 Practice 101
9:58 Implementation 101
14:24 Theory 101
29:29 Implementation 201
36:44 Practice 201
38:35 Implementation 301
Slides: http://bit.ly/sysdesignmeetup-mapreduce-slides
0:00 Warmup
0:29 Intro
1:34 Agenda
2:15 Goal
4:51 Practice 101
9:58 Implementation 101
14:24 Theory 101
29:29 Implementation 201
36:44 Practice 201
38:35 Implementation 301
Slides: http://bit.ly/sysdesignmeetup-mapreduce-slides
This important design paradigm has reached me twelve days too late: https://github.com/zhuowei/nft_ptr
Also, copy-pasting and paraphrasing from a private group: The history of C++ repeats itself, the first time as tragedy, the second time as farce, the third as blockchain.
Also, copy-pasting and paraphrasing from a private group: The history of C++ repeats itself, the first time as tragedy, the second time as farce, the third as blockchain.
GitHub
GitHub - zhuowei/nft_ptr: C++ `std::unique_ptr` that represents each object as an NFT on the Ethereum blockchain
C++ `std::unique_ptr` that represents each object as an NFT on the Ethereum blockchain - zhuowei/nft_ptr
I'm experiencing a rare déjà vu.
As in the dev part of my career is, again, slowly expanding, from encompassing just the realm of C++ before, to now building JavaScript wrappers over it, enabling more and more people access the sacred performance of The Language.
It’s almost like my Java/JNI stint 2.5 years ago, but a lot better actually. Better because JavaScript is in fact a relatively good language to wrap C++ into, at least once the initial blockers are behind.
If all goes well, I’ll be generating (DSL, *.inl) C++ as part of the build step some time in the next few weeks, after which point shipping C++ code as npm modules, bundled with their TypeScript interfaces, may well become my second C++ nature, as long as the tasks I have to work on these days allow us to explore this direction.
Links: experiments, and my clean repo with example usage.
Fun times.
As in the dev part of my career is, again, slowly expanding, from encompassing just the realm of C++ before, to now building JavaScript wrappers over it, enabling more and more people access the sacred performance of The Language.
It’s almost like my Java/JNI stint 2.5 years ago, but a lot better actually. Better because JavaScript is in fact a relatively good language to wrap C++ into, at least once the initial blockers are behind.
If all goes well, I’ll be generating (DSL, *.inl) C++ as part of the build step some time in the next few weeks, after which point shipping C++ code as npm modules, bundled with their TypeScript interfaces, may well become my second C++ nature, as long as the tasks I have to work on these days allow us to explore this direction.
Links: experiments, and my clean repo with example usage.
Fun times.
Exactly Once: today's system design meetup video. And the slides.
We covered the narrow edge between the easy problems and the hard one, touched on message sequencing / "serialization", Kafka, Lamport clocks, the CAP theorem (once again), and how to deal with them in practice, not just in theory.
I truly enjoyed it today, and hope the recording captures the spirit well enough 😊
We covered the narrow edge between the easy problems and the hard one, touched on message sequencing / "serialization", Kafka, Lamport clocks, the CAP theorem (once again), and how to deal with them in practice, not just in theory.
I truly enjoyed it today, and hope the recording captures the spirit well enough 😊
YouTube
System Design Meetup 2021-May-8: Exactly Once
Exactly once delivery.
0:00 Welcome
2:14 "Fire and forget" and other techniques
3:52 Message queues
4:31 Example problems
8:16 Idempotency
12:58 A real-life example
14:59 Possible (bad) solutions
17:02 Timestamps
18:00 Idempotency tokens
19:49 Ledger
21:19…
0:00 Welcome
2:14 "Fire and forget" and other techniques
3:52 Message queues
4:31 Example problems
8:16 Idempotency
12:58 A real-life example
14:59 Possible (bad) solutions
17:02 Timestamps
18:00 Idempotency tokens
19:49 Ledger
21:19…
Today's System Design Meetup is about monitoring production services, at scale, about detecting possible issues early on, and about reacting to them promptly and correctly. Here's the video and the slides.
YouTube
System Design Meetup 2021-May-29: Monitoring and Alerting
This episode is about monitoring production services, at scale, about detecting possible issues early on, and about reacting to them promptly and correctly.
Slides: https://tinyurl.com/sdm-pagerduty-slides
Slides: https://tinyurl.com/sdm-pagerduty-slides
After drinking the C++ metaprogramming cool aid for several months, we are now seriously debating whether it's a good idea to have a universal wrapper that would make functions accept arguments in any order.
They are all of different types, after all, and we use enum classes and user-defined literals everywhere, for readability reasons. So, there are no
Spoiler alert: I am now the one advocating against this idea.
Although the chaotic part of me wants to go full Haskell mode and implement an even stronger wrapper. So that "partial specializations" would result in "Currying 2.0": returning functions of lower arity, with some argument values already preset.
(The chaotic evil part of me is even thinking about compile-time checks that we should be able to associate with these wrappers.)
They are all of different types, after all, and we use enum classes and user-defined literals everywhere, for readability reasons. So, there are no
bool`-s left, and any number or string would likely be suffixed by some `27960_container_port or "/tmp/foo/bar"_host_dir.Spoiler alert: I am now the one advocating against this idea.
Although the chaotic part of me wants to go full Haskell mode and implement an even stronger wrapper. So that "partial specializations" would result in "Currying 2.0": returning functions of lower arity, with some argument values already preset.
(The chaotic evil part of me is even thinking about compile-time checks that we should be able to associate with these wrappers.)
Embarrassingly, I knew it since forever that
So, even when we all knew a git proxy would help speed up cloning the repos, I was somehow reluctant to look into this.
Well, git upstreams can be
• Create a
• Clone everything you need there.
• (You may even "clone" from your local, already cloned, dirs, just make sure to change their upstream to the true ones.)
• Write a cron script to do
• For your new repo, add this
• (You may also want to mark it read-only by preventing pushes to it, just to be safe).
• Now, establish a habit of saying
Voila. Time saved: hours and hours. A rewarding git experience: priceless.
git upstreams can be local dirs, but it always felt a tricky feature to use.So, even when we all knew a git proxy would help speed up cloning the repos, I was somehow reluctant to look into this.
Well, git upstreams can be
/home/dima/.my_git_cache/${repo_name}. In other words:• Create a
.my_git_cache local dir.• Clone everything you need there.
• (You may even "clone" from your local, already cloned, dirs, just make sure to change their upstream to the true ones.)
• Write a cron script to do
git fetch in all those .my_git_cache/* repos every few minutes.• For your new repo, add this
.my_git_cache as one of the upstreams, e.g. cache.• (You may also want to mark it read-only by preventing pushes to it, just to be safe).
• Now, establish a habit of saying
git fetch cache every time you need to sync with the "true" upstream.Voila. Time saved: hours and hours. A rewarding git experience: priceless.
The Kafka episode of the system design meetup is ready: https://www.youtube.com/watch?v=zNdRUTMX1ao&t=15s
Enjoy responsibly!
Enjoy responsibly!
YouTube
System Design Meetup 2021-Jul-16: Kafka
Today we had a conversation about Kafka: from its internal organization and its power all the way to when it is worth considering to look elsewhere. Thank you António!
Slides: https://tinyurl.com/sdm-kafka-slides
All the links are clickable, at the time…
Slides: https://tinyurl.com/sdm-kafka-slides
All the links are clickable, at the time…
This is hilarious: a robotic cue to take pool shots. I was dreaming of something like this 10+ years ago, and the guy has implemented pretty much most of it =)
YouTube
Automatic pool stick vs. strangers
Get a Free 32GB Flash Drive and 32GB Micro SD Card: https://micro.center/dbfba
Check out the Micro Center Custom PC Builder: https://micro.center/4879a
Join the Micro Center Community: https://micro.center/15dd5
Get merch + support the channel: https://s…
Check out the Micro Center Custom PC Builder: https://micro.center/4879a
Join the Micro Center Community: https://micro.center/15dd5
Get merch + support the channel: https://s…
In the meantime, the System Design Meetup is growing, and we are now 99 people in Slack.
If you are interested in making it a three-digit number, or in joining the meetup, or both, here's the invite link: https://tinyurl.com/sdm-slack-invite
To also take part in the on-the-record episodes, which we release on YouTube, please also fill the https://tinyurl.com/sdm-signup-form form.
If you are interested in making it a three-digit number, or in joining the meetup, or both, here's the invite link: https://tinyurl.com/sdm-slack-invite
To also take part in the on-the-record episodes, which we release on YouTube, please also fill the https://tinyurl.com/sdm-signup-form form.
As part of doing something pure and useful every once in a while, I've spent some ~1.5 hours to confirm I speak
Code: http://github.com/dkorolev/cmake_playground/, with the README describing what does it accomplish.
I've even added a
On both Linux and Windows, in Qt Creator and Visual Studio respectively, this code can be "opened" (as a directory and/or as the
Back to the real work now =)
cmake sufficiently well.Code: http://github.com/dkorolev/cmake_playground/, with the README describing what does it accomplish.
I've even added a
Makefile to shortcut the popular commands, so that you won't have to worry about cmake at all.git clone https://github.com/dkorolev/cmake_playground
(cd cmake_playground; make)
(cd cmake_playground; make test)
On both Linux and Windows, in Qt Creator and Visual Studio respectively, this code can be "opened" (as a directory and/or as the
CMakeLists.txt file), and the IDE would do the magic down to identifying individual googletest cases. Heck, I can even debug them one by one — it's been a while since I used anything but gdb on Linux for these purposes.Back to the real work now =)
GitHub
GitHub - dkorolev/cmake_playground
Contribute to dkorolev/cmake_playground development by creating an account on GitHub.
So, I really love open source.
To the extent that I often mention, in private conversations, that my ideal “engineer” life is to be paid for working in a 100% open fashion, so that my business hours could be screencast.
And that I’d need to do nothing to configure a new laptop or desktop but to copy a bunch of dotfiles and have its public key populated on the repos I work with.
As of now (2021) this, technically, is attainable. But the question already is in the air: what if my open source work benefits the wrong people?
My answer to this concern remains the same: “none of my problem”.
Because if that’s what you are worried about, why don’t you focus your energy on making our world more united and more positive-sum? As opposed to contributing to its divisiveness by erecting even more barriers, such as constraining what general-purpose technologies should be open source by the reasoning that goes along the lines of “intellectual property”, “corporate secrets”, and “national security”.
Technology, after all, is that high tide that lifts all boats. And, when it comes to software, the cost of making it possible for everyone to contribute to it is literally zero. In fact, this cost is negative — as controlling access to the code is by itself a nontrivial task that’s quite an expense for the entity that “owns” this code.
And we, each and every human being, undoubtedly benefit from better maps, better messengers, and better wealth preservation and transfer solutions that are near-instant, know no borders, and are mathematically guaranteed to do their job well.
~ ~ ~
And it occurred to me just today that such a point of view may well be cancel-worthy in some not too distant future.
Which is a very sad thought. Because it means I myself do not believe 100% that my dear humankind is interested in playing more and more positive-sum games moving forward.
Seriously, the Golden Age of technology may already be past us. To the degree that I feel the urge to publish this post today, while it is still safe to hold such a point of view publicly.
To the extent that I often mention, in private conversations, that my ideal “engineer” life is to be paid for working in a 100% open fashion, so that my business hours could be screencast.
And that I’d need to do nothing to configure a new laptop or desktop but to copy a bunch of dotfiles and have its public key populated on the repos I work with.
As of now (2021) this, technically, is attainable. But the question already is in the air: what if my open source work benefits the wrong people?
My answer to this concern remains the same: “none of my problem”.
Because if that’s what you are worried about, why don’t you focus your energy on making our world more united and more positive-sum? As opposed to contributing to its divisiveness by erecting even more barriers, such as constraining what general-purpose technologies should be open source by the reasoning that goes along the lines of “intellectual property”, “corporate secrets”, and “national security”.
Technology, after all, is that high tide that lifts all boats. And, when it comes to software, the cost of making it possible for everyone to contribute to it is literally zero. In fact, this cost is negative — as controlling access to the code is by itself a nontrivial task that’s quite an expense for the entity that “owns” this code.
And we, each and every human being, undoubtedly benefit from better maps, better messengers, and better wealth preservation and transfer solutions that are near-instant, know no borders, and are mathematically guaranteed to do their job well.
~ ~ ~
And it occurred to me just today that such a point of view may well be cancel-worthy in some not too distant future.
Which is a very sad thought. Because it means I myself do not believe 100% that my dear humankind is interested in playing more and more positive-sum games moving forward.
Seriously, the Golden Age of technology may already be past us. To the degree that I feel the urge to publish this post today, while it is still safe to hold such a point of view publicly.
Mobile phones have a battery life of about one day.
A bit less if you're lucky, a bit more if you prioritized battery life when choosing the phone, and some 5x more if you have a power bank on you.
Life changes with a charger in your pocket. Your "battery life" is effectively infinite now. In the vast majority of places these days total strangers would let you charge your phone, for free. The cost of power, after all, is miniscule.
In fact, if your phone needs a standard charger, there's a good chance you don't need your charger with you at all; in many, if not most places you could find yourself at people would be happy to have your phone charged.
If you are a semi-frequent Starbucks, or virtually any, coffee shop visitor, quite frankly, you don't have to worry about charging your phone at all. A daily cup of coffee, virtually by definition, does pay this bill.
~ ~ ~
Why can't we, as the civilization, do the same for the Internet?
Forget video. Forget streaming. Three very basic Internet usecases are:
⒈ 𝗔𝗰𝗰𝗲𝘀𝘀 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻. To read Wikipedia, or my favorite book, or today's news.
⒉ 𝗠𝗲𝘀𝘀𝗲𝗻𝗴𝗶𝗻𝗴. End-to-end encrypted, and unconditionally free from censorship of course.
⒊ 𝗣𝗮𝘆𝗺𝗲𝗻𝘁𝘀. A healthy mix of Venmo / Zelle and crypto. Banks are welcome.
Banks, in (3) are, of course, welcome, as are third-party messenging platforms in (3), as long as they provide the basic guarantee of unconstrained wealth transfer and unconstrained message passing.
In other words, most of today's banks and today's messengers are not welcome (the former have a tendency to block innocent payments, the latter have been caught multiple times failing to send the messages which the platform "does not endorse"). They are still welcome. On the proper API terms.
Let's get this Net Neutrality thing working. I don't care about the term; it would likely need to change. Still, there's no reason at all that poor kids all over the planet are deprived from a feature as simple as sending a "Hello" message, or sending $10 to a friend, or reading today's news.
~ ~ ~
Maybe this would be the largest impact Elon Musk makes all in all. Make Starlink available 24/7 for ~1KB/s traffic, unconditionally. Simple as that.
If he attaches a cryptocurrency to this (a possible future for DOGE, right?), this alone could be cash-positive AF. I sincerely hope this, or something like this, is about to happen some time very soon.
A bit less if you're lucky, a bit more if you prioritized battery life when choosing the phone, and some 5x more if you have a power bank on you.
Life changes with a charger in your pocket. Your "battery life" is effectively infinite now. In the vast majority of places these days total strangers would let you charge your phone, for free. The cost of power, after all, is miniscule.
In fact, if your phone needs a standard charger, there's a good chance you don't need your charger with you at all; in many, if not most places you could find yourself at people would be happy to have your phone charged.
If you are a semi-frequent Starbucks, or virtually any, coffee shop visitor, quite frankly, you don't have to worry about charging your phone at all. A daily cup of coffee, virtually by definition, does pay this bill.
~ ~ ~
Why can't we, as the civilization, do the same for the Internet?
Forget video. Forget streaming. Three very basic Internet usecases are:
⒈ 𝗔𝗰𝗰𝗲𝘀𝘀 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻. To read Wikipedia, or my favorite book, or today's news.
⒉ 𝗠𝗲𝘀𝘀𝗲𝗻𝗴𝗶𝗻𝗴. End-to-end encrypted, and unconditionally free from censorship of course.
⒊ 𝗣𝗮𝘆𝗺𝗲𝗻𝘁𝘀. A healthy mix of Venmo / Zelle and crypto. Banks are welcome.
Banks, in (3) are, of course, welcome, as are third-party messenging platforms in (3), as long as they provide the basic guarantee of unconstrained wealth transfer and unconstrained message passing.
In other words, most of today's banks and today's messengers are not welcome (the former have a tendency to block innocent payments, the latter have been caught multiple times failing to send the messages which the platform "does not endorse"). They are still welcome. On the proper API terms.
Let's get this Net Neutrality thing working. I don't care about the term; it would likely need to change. Still, there's no reason at all that poor kids all over the planet are deprived from a feature as simple as sending a "Hello" message, or sending $10 to a friend, or reading today's news.
~ ~ ~
Maybe this would be the largest impact Elon Musk makes all in all. Make Starlink available 24/7 for ~1KB/s traffic, unconditionally. Simple as that.
If he attaches a cryptocurrency to this (a possible future for DOGE, right?), this alone could be cash-positive AF. I sincerely hope this, or something like this, is about to happen some time very soon.