How Can We Reach The Desirable State Where Most Work Is Done By Robots And Humans Work Short Hours At High Wages?

What most people don’t get is that “high wages” is subjective.

I could write a long paragraph here, but the gist is simple. Assuming we are speaking about the state of the art that is substantially different from what we live in today the “humans work short hours at high wages” statement is indistinguishable from some “what humans need and what is produced by robots costs virtually nothing”.

Simply put, a cup of coffee at some Starbucks, where the barista is a human. It was, is, and will remain expensive. Because it’s human labor, and, from my experience, the vast majority of people who are pro robots and automation are also pro shrinking the wage gap, reducing the Gini coefficient, and overall improving the equality of equity, not the equality of opportunity.

So: a Starbucks coffee would still cost a lot. Lowest-earning humans would still not be able to afford some ~20 lattes per day. Because to “fix” that we’d have to pay Starbucks baristas less, which contradicts the very goal of having short hours and high wages for everyone.

At the same time, whatever can be done with little or no human intervention would be getting progressively cheaper. A decent example could be our modern computers and smartphones: they are indeed getting a lot cheaper day after day, especially if normalized by FLOPS or something. If FLOPS are too hard for you, think of gigabytes on your memory cards. All electronics, pretty much.

Further along this axis would be some delivery services. Ever thought how can shipping from China to the US cost cents? Well, it’s because there’s little to no human involved, making it easy for your $3 piece of jewelry or whatever travel you way, effectively for free.

As automation continues, we will have more and more “robot-provided” services go down in price. This process would be governed by big money and corporations, it will be slow, yet inevitable. A canonical example here is self-driving cars; imagine the world where an Uber costs less than a tram — nothing extraordinary, the transportation just becomes available to everyone (much like some cities/countries in Europe are already working towards making it free). Quite frankly, it’s not that dramatic of a change for people living in most parts of the “developed” world, where neither latte expenses nor Uber expenses are a considerable burden on one’s budget.

A counter-example could be Internet access in the US. It’s terrible quality-wise, and it still is ridiculously overpriced. Not that there are tons of humans involved; it’s the companies that can still overcharge people due to their positions of monopolies and oligopolies.

Painful? Yes. But it will end some time soon.

On a positive note: We kind of already are some half way to the world where humans work “short hours at high wages”.

It just requires some creative view of what short hours and high wages are.

Where pure market rules, such as better phones and computers, or such as trading your online time to bananas in countries that produce bananas, the world is already in good shape. Most people who are capable of getting some of their work done remotely are more or less in position to afford themselves enough food and shelter by moving to certain locations on our cozy planet, where some $150 a month would cover both easily.

Where governments and corporations rule the game, even with extreme automation, for a while from today we will be living in a world that is a lot like today’s.

Take a moment to look at the "Productivity and average real earnings" plot from the Wikipedia page on Workforce productiviy.

Is car manufacturing an industry that is being automated as we speak? Hell yes. For many decades by now.

Consider a simple metric: a number of months a car factory worker has to work in order to afford a car they are assembling. Well, this metric didn’t quite go down; pretty much at all. With the trend towards automation, if we extrapolate from some 1970, this metric should be closer if not better than “one month one car” by today; but it’s not.

What I’m coming to with this story is that “short hours, high wages”, economically speaking, is not what the “ruling class” wants for us, humans. And I’m not making a value judgement here; I’m merely noting an observation. That’s why in the “developed” countries the costs of housing, education, or healthcare are and will remain extreme, so that even the top earners don’t feel like they can “work for a year or two more, and then chill for the next decade”. Admit it: had this been possible, and had enough people decide to take such a “career path”, our world might be on the verge of collapse today; so maybe, just maybe, it’s a good thing there are quite some forces that are making sure we, The People, keep working at about the same pace.

I can’t predict what the future will bring.

But the sustainable equilibrium appears easily attainable. As long as we don’t procreate like crazy, the worldwide trend of eradicating poverty and major diseases is well set up to continue.

Fewer and fewer humans would starve, thanks to many bits and pieces, robots and automation included. More and more humans would be in position to consciously choose a lifestyle where they can work fewer hours.

As for wages … well, as I’ve said above, don’t expect some multiplier to your (inflation-adjusted) “effective earnings” even if the robots to make it possible are widespread available tomorrow. Don’t expect the state of the art where healthcare/education/mortgage would suddenly stop bugging you. But do expect to have to worry less and less about the most basic things, such as water, food, basic medicare, powerful enough electronic gadgets, and Internet access.

Because this does align well with all three: with the state of the art in technology and engineering, with what most people want, and with what most governments and activists want people to have.

Source: https://www.rebellionresearch.com/blog/how-can-we-reach-the-desirable-state-where-most-work-is-done-by-robots-and

I keep thinking why do nerds and ex-nerds generally get happier after thirty. And have a theory that is simple enough and may well be true.

The theory is that there's some "step size" in one's head, that determines by how much above one's head they should be jumping. And thirty is about the age by when the two processes of our brain and hormonal chemistry are mostly settled:

• Gauging our own potential in this world, and
• Making sure to not engage in reckless gambles.

In other words, there are two "mistakes" we tend to make repeatedly in our teens and twenties, and we generally outgrow them by our thirties. They are:

• Believing we can do a lot more than we are objectively capable of, and
• Thinking we should keep shooting for the stars, disregarding the risk/reward analysis.

So, constant re-evaluation along these two axes appears to make us think, by the time we are about thirty, that our pessimistic prognoses are actually quite close to the reality, and that quite a few of our ambitions are generally unjustified.

Multiply this by the beliefs of "oh, I should just try this one more time under a slightly different sauce, and my dream life will come true", which humans have hard time parting with in their twenties — and here we arrive to what is commonly called the mid-life crisis.

However, [ex-]nerds and normal people reach this point with a very different amount of baggage and momentum.

For a “non-nerd” the above realization is generally the re-evaluation of most major life plans; what they would see is that no big career changes are likely any time soon, and their skills set is such that a noticeable improvement in the quality of life — such as moving to a substantially more modern house in a better location, or sustaining one's life by working fewer hours a week, for example — is some dozen years away at best.

[Ex-]nerds, on the other hand — and I don't only mean software engineering nerds, just people of the rationalizing mindset — would just realize that all they need to adjust in life is to strive for a bit more stability, which generally translates to evaluating risks in a more conservative way and saying "no" to one or two more of many ideas this and next year. Because pretty much everything else — friends, the surrounding, life satisfaction both with what one does to make money and outside this realm — stays the same.

And a sober re-evaluation of those aspects of life would generally lead [ex-]nerds to a conclusion along the lines of "well, I must have been doing something right; time to shift gears, dedicate more time to myself, and figure out how to best capitalize on all those crazy ideas I was working on for the past fifteen++ years".

What do people who lack empathy act like?

I've worked with a lot of people who hold a very simple belief:

"What others feel now is temporary, so it's best to disregard it and look ahead. If they are unable to acknowledge this, it's their weakness and we should re-evaluate whether and how to deal with them moving forward." It's not lack of empathy, but somewhat close, if you ask me. Myself, I at least half-agree with such a statement, which makes me somewhat more qualified to answer.

When making decisions and taking actions, they think long term. Their intuition and gut feeling is optimized to make the world around better in the long run, not focus on one local, "incremental", improvement. This long run improvement also involves future emotional state, both of people affected by their decisions, and of themselves.

Effectively, their definition of empathy goes along the lines of "This person knows it's the right thing to do. They are shy or hesitant to make it happen; I am not, so let's just get over with it and move on."

I personally find it very rewarding to work with such people; rewarding both professionally and emotionally. When someone points out that my own perception of something is a temporary emotional reflection rather than an actual representation of the state of the art, I can not but thank them.

Especially if they have a track record of being right.

What are some best practices in Feature Engineering?

The list below is by no means complete, but I hope it provides a good enough starting point.

Understand the data.

Feature engineering is an art. Understanding where do the features come from helps a lot.

Key questions:

• Are the features continuous, discrete or none of the above?

• What is the distribution of this feature?

• Does the distribution largely depend on what subset of examples is being considered? Time-based segmentation? Type-based segmentation?

• Does this feature contain holes (missing values)? Are those holes possible to be filled, or would they stay forever? Is it possible to eliminate them in the future data?

• Are there duplicate and/or intersecting examples? Answering this question right is extremely important, since duplicate or connected data points might significantly affect the results of model validation if not properly excluded.

• Where do the features come from? Should we come up with the new features that prove to be useful, how hard would it be to incorporate those features in the final design?

• Is the data real-time? Are the requests real-time? If yes, well-engineered simple features would likely rock.
If no, we likely are in the business of advanced models and algorithms.

• Are there features that can be used as the "truth"? If the problems to be solved would be of the supervised learning type, are there features that can be used as golden measures to be predicted? Example truth features: prices, yes/no decisions that have been recorded, recommendations followed or not followed.

Brainstorm and come up with as many diverse ideas as possible.

The best features are the ones that:

• Can be intuitively explained.
• Are always or almost always possible to compute, and
• Either define some way of looking at the data extremely well or connect multiple ways of looking at the data extremely well.

Examples:

• "Did the user register for the site?" is a good yes/no feature.
• "How much time did the user spend on the site in the past week?" is a good continuous feature.
• "How much time did the user spend on the site compared to the histogram of how much time did registered users spend on the site?" is a good feature combining both yes/no registration decision and continuous "time on the site" metric.

Always validate your guesses.

If there is the truth to predict, start building toy models from day one and establish a procedure to validate new models.

It would likely involve heavy cross-validation and resampling. Make sure you split the data into train and validation/test sets properly, so that potentially duplicate or similar examples are separated.

Make sure the model validation process is trustworthy.

Test it against some random features and/or random models to ensure that once your process detects a trustworthy model, you can trust in it.

Think back from the objective.

The end goal is rarely "look at this data and tell us what do you think of it".

Virtually always, one has some end goal in mind.

The goal might be estimating or predicting some parameter (supervised learning: classification, regression, ranking), or it might be understanding the grouping and anomalies in the data (unsupervised learning: clustering, anomaly detection).

Often times, the end goal might give a hint on the cost functions that fit the problem best. This, in its turn, allows for better guesses on what types of models would perform best, even before the modeling has been started.

Keep RFMVT in mind.

RFMVT stands for:

• Recency. Signals how old certain event is.
• Frequency. Signals how often does certain events occur.
• Monetary. Any numerical representation of direct of indirect business value of an example.
• Variety. How many distinct items are found for certain type of an example.
• Tenure. How much time has elapsed since the first appearance of certain example or of an example of certain type.

Don't overengineer.

The temptation is almost always high to build a big thing that can automatically assess how well certain feature, or type of features, can perform.

In the real life, during feature engineering phase, it does not make that much of a difference whether an iteration takes a few minutes or half a day.

The most productive time during the feature engineering phase is spent at the whiteboard. The most productive way to make sure it is done right is to ask the right questions about the data.

Seen in this light, feature engineering might well be the most academic and the most scientific part of what is called Big Data nowadays. There is no room for hacking; but there is no room for overengineering either.

Use the right tools for the right tasks.

Damn, my long-held secret is now public.

Don't ask for, or even long for a larger compensation package.

Instead, honestly talk about your lifestyle, well-being, and other secondary factors; and be sincerely ready to join with a lower total comp when the team offers more overall upside to your immediate and projected quality of life.

"Worst" case scenario, you will find the product and the team you really enjoy working with, join them, and be happy. At least to me, long-term happiness trumps being somewhat underpaid.

In reality, odds are in your favor that you will end up doing something you enjoy, being paid close to what a fair market compensation for your contribution is.

At the end of the day the market does work. And opening up with an argument that money does not play a big role in you making the decision only evens up the conversation between you and the people who want to work with you.

http://blog.interviewing.io/there-is-a-real-connection-between-technical-interview-performance-and-salary-heres-the-data

Can deep learning solve "average-case" instances of NP-hard problems?

Easily.

This seemingly naive question is actually very deep. It scratches the surface of The Very Question of intelligence. What is Intelligence all about?

One definition I have stumbled upon many years ago, way before the AI and “AI” eras we are living through these days, is very simple: Intelligence is the capacity to produce meta-concepts.

Take some puzzle to solve. I like to play Loopy from time to time, but Sudoku would do just fine. The concept of intelligence is surprisingly easy to illustrate based on this puzzle.

I will first outline two approaches, and then define some approximation of the criteria by which I generally distinguish intelligence from non-intelligence.

Approach one, “bottom up”:

• Come up with a set of heuristics of what is “definitely true”. Apply “definitely true”-s until stuck.

• Come up with a set of heuristics of what is “likely pivotal”. When, if stuck, when resorting to enumerating all the possibilities, focus on the “pivotal” options first, aiming to either fail fast and shrink the search space substantially, or to get lucky and ultimately solve the puzzle.

• Come up with the set of “more likely / less likely” moves. So that to, when considering a “pivotal” search [sub]tree, in the beginning of solving the puzzle try the “least likely” options (hoping to cone to a contradiction and discard them fast), while towards the end shift towards “more likely” (because it may well solve the problem right away, or provide a strong enough hint along the way).

• Come up with some “it’s [not] going to work” heuristic. So that if trying some “pivotal” candidate takes one too far both ways, instead of going deeper, perhaps considering a different “pivot” is a wiser idea.

Approach two, “top down”:

• Qualitatively, not quantitatively, broaden the set of heuristics in one’s arsenal.

• When stuck with “simple augmentations”, take a step back and try a more integral view: perhaps there’s a major hint to use.

• Ultimately — both in cases of Sudoku and or Loopy — rather efficient algorithms can be devised and implemented; and, in some form, those algorithms can be run “manually”.

For example, Loopy most likely does not have a polynomial solution, but I have an O(N*2^(2*N)*(1+log(N)) one well internalized in my head — and even implemented twice for fun — and that’s my “fallback plan” if I really feel like solving a particular tricky intsance of a puzzle on my own.

So, approach one is more heuristic-based, while approach two is more holistic in nature.

Now, the final accord: In my book, “intelligence” is the capacity of a system to seamlessly move between approaches one and two.

Yes, it’s a crude and rather vague “definition”, but I like it a lot for its intuitiveness and its capacity to be explained in relatively laymen terms.

While the above may have looked like a detour, it is actually very much on point.

Take a look at this paper or this blog post. It’s Jeff Dean outlining a practical approach exactly to the problem you are postulating. A real-life problem that affects gazillions of CPU cycles all over the world every second of wall time.

The TL;DR of his suggestion? Use ML. And it works.

Yes, life is full of solving problems that do not have perfect solutions.

In a very narrow set of cases we can outline the constraints and have an unambiguously correct closed form solution that is proven to be the best one possible.

On the other end of the spectrum we get all sort of human-related problems — where beliefs, ethics, morals, etc. come into play — which are hardly solvable at where we are right now.

Somewhere along this spectrum there exists a narrow band of problems that are formal enough to be analyzed by means of applied science, and practical enough so that we need to keep solving them in order to move forward as the civilization.

Some of those problems are NP-hard problems in the general case, but, in reality, the nature is not “trying hard” to make all possible approximate algorithms fail, but rather draws the input constraints from some “natural” distribution that is far from random, and which, thus, allows for better approximations.

Which is exactly what the original question is about: instances of NP-hard problems with somewhat “average-case” input data.

Yes, deep learning is most certainly among the most powerful tools to try when approaching those problems these days.

As it happens every several months, I'm back to the mode of alternating between C++, {Java,Type}Script, and Python.

Once every several months is about the right frequency to re-evaluate and update my views on how do these technologies compare.

* * *

Pros of C++ (and cons of node.js):

1) Strong typing.

Type errors, and generally, data-level errors, are costlier and more difficult to fix than code-level errors. While TypeScript does provide quite a few guarantees, it's by far not bulletproof. Bad things happen, and in the world of {Java,Type}Script they tend to happen in production.

While the type system of C++ is, as one may say, not as powerful as Haskell's or Scala's, it certainly does the job of making sure easy-to-miss details don't fall through the cracks. And it's paying off.

You know your system of record systematization work is a success when out of ten conversation about data schema nine have to do with product requirements being not even remotely complete. After quite some fist fights, we did strike this 9:1 nonsense-to-work balance. I also learned valuable lessons that have to do with the process to get there. Sometimes rational arguments work, and some other times the best strategy is to watch those against data systematization fail spectacularly, then quietly step in and clean those Augean stables. Apart from confirming my strategic belief that standing for the data dictionary from day one is the way to go, intuition tells me those tactical moves will keep proving themselves useful.

2) Template support.

The ability to write generalizable code is great by itself, but coupled with C++-level strong typing it gets promoted from great to fantastic.

What if you're adding another data type to the set of types to process, but the key for those objects is not in the set of types of supported keys? I'm happy to see the compiler assume the role of static code analyzer and point it out to me. What if that key does not have a hash function defined, while later on those keys are being put into a hashmap? It may fail in production (not too bad), it may revert to a less effective hash function (bad), may resolve all hashes to zero (bad), or, voila, it can fail to build. In the above list, I would always pick the last option.

On a side note, Java's generics, being largely runtime-injected, are not even remotely close to what C++ can do in this regard, where the compiler effectively is more of the correctness-validating tool than the actual machine-code-generating one.

* * *

Neutral:

1) Performance and multithreading.

You're reading it right. Performance is a "neutral" aspect to me.

Native code supporting multhtreading is a superpower, with which naturally comes super-responsibility. On the bright side, by today it's evident that single-threaded operation mode is well-suited for most production tasks, which renders it unnecessary to step on the land mine of threads synchronization. For projects where high QPS is a must, node.js is just out of question. For projects where performance _may_ become important at scale, but not quite yet, it's really the team's call which stack to choose as the foundation.

In RealtyShares we use the strongly typed append-only event log, and three major interfaces to it are pub-sub, REST API to the materialized views maintained in memory, and CQS commands and queries. The former one is threading-agnostic from the user perspective, and the latter two, for now, simply lock the log from being appended to, making developing as single-threaded as it may be.

This way we reap all the benefits of C++'s performance and strong typing, while not stepping onto the multithreading minefield when it comes to adding new features.

And it's nice to always know there's some 1.5 orders of magnitude performance reserve should we need to expose our APIs to the wild.

2) Learning curve.

Yes, I am inclined to believe that by today "learning" C++ to the degree that makes one a productive contributor is not harder than learning TypeScript after speaking only Python.

The most widely cited, and, quite frankly, most annoying issue with C++ these days are the difficult-to-read compilation error messages. But I will not get tired of stressing that ten minutes spent figuring out the types in a code snippet is an amazing time investment, especially given that the alternative is to spend at best one long and stressful day locating the issue and deploying the fix for a transient bug that is presently affecting production.

In RealtyShares the only two nontrivial pieces that require a C++ expect are schema evolution and generic data exporters. The former has to do with zero-downtime master flip while the underlying DB undergoes structural changes, and it sure _does_ need deep understanding of our templated type system to get _that_ code to build and run. And the latter _may_ need template metaprogramming, although in most practical cases a list of #define-s of included data tables and fields does the job just fine.

Needless to say, to pass the bus test our engineering team must have at least three C++ experts. I, for one, don't think it's uncommon these days.

* * *

Pros of node.js (and cons of C++):

1) Compilation takes time.

Yes. Being able to modify the code live is a huge win, which C++ does not have. I'll touch on this one a bit later.

2) A wide collection of packages.

Yes, npm is great. C++ is nowhere near that state of the art. Good news is: In a well-designed system there's hardly any need to perform node.js-level tasks in C++. That's what scripting languages are for, after all.

The only time I _almost_ had to write something that may require a Python or an node.js module was an internal dashboard page generation. After some thought, I've put dashboard_before.html and dashboard_after.html files into my C++ project, and had the backend output either a plain JQ-friendly JSON, or that very JSON squeezed in between those two files if Accept-Type of the request lists HTML. Last time I checked (just now), that's exactly the state in which this code is right now, fully functional in production, and used as the "UX" by several non-engineers.

* * *

Thus, the only real blocker is compilation and stating time. The frequency of development cycles, to be precise.

The compilation time itself is not an issue when the code is broken into compilation units (thanks Nathan!), especially since we moved to the gold linker (thanks Maxim!). The one-line change time in a CQS command is under fifteen seconds. The real challenge is: "interactively" deploying those changes to some test environment results in iterations that are still over an hour long.

We are eyeing the solution that enables dynamic on-the-fly adding of trusted just-built libraries into our test environments. It would improve the lives of the engineers working on those features dramatically. Unfortunately (or fortunately?) we never had a business need to dedicate sufficient effort to this task. Those few times when the situation warranted us to move faster, we always managed to push the C++-related changes through the door within just a few business days, leaving the remaining API-grade heavylifting to node.js or F#. Guess that's the curse of having enough C++ experts around. Or the "curse" of having good tests.

Or just the curse of a perfectionist architect in the world of modern engineering: every project becomes lo-pri the moment it's evident it does serve the business need "somewhat well". Such is life, might as well get used to.

* * *

To recap: a while ago I've made a prediction that C++, as long as it keeps becoming engineer-friendly, will be making a comeback, especially when it comes to developing backends in small-to-medium engineering organizations.

In the past two years I can confidently say we are solidly on track.

On Ethics of Applying ML/AI

There generally are two good approaches to building data-driven solutions, with a bad one in between.

Good approach #1: Formulate the problem along with measurable metrics, then employ machine learning, and be open-minded at interpreting and acting upon what that machine learning has discovered, however controversial or contradictory it is.

Good approach #2: Decide up front that the problem is too socially-sensitive, and do not use machine learning to solve it. Collect data, involve humans, and have a human, or a team, responsible for each decision it makes. Use data science in the back office to make sure humans are not making obvious blunders, but don’t mix the two.

Bad approach: Claim AI / ML is used, train and validate some models, and then, upon receiving the results that are different from what were expected, tweak the way those results are used.

https://medium.com/@Dima/on-ethics-of-applying-ml-ai-ea588432016a

Here's my top pick for the best #GoogleCloudNext talk. It's Pandora's Move to GCP With Composer by Daniel Barrett.


There were two more speakers, from Google. And they took the stage for longer than Daniel, of course, because it's Google's conference. And what they were trying to show did not always work, which, of course, was a lot of fun to watch. But the playmaker of this show is obviously Daniel.


Approximate plot:

• Must migrate to Google Cloud.

• Need to run three jobs in parallel, because otherwise it's pointless to move to Google Cloud.

• Manually determine the required machine type for each of the three jobs.

• However, spinning up the three machines of the respective type doesn't work, because ...

• ... Composer starts one job on the first machine, the other two on the second, and leaves the third machine unused. So, it overwhelms the second machine, and the whole thing fails.

• Tried several ways to walk around the above. Found the manual way to be the best one.

• The manual way is to have Composer steps to: 1) Spin up those three machines, 2) Run scp to copy the binaries to those respective machines, 3) Signal those three jobs to start on the respective machines, 4) Wait until they complete, 5) Tear down the machines. Works automagically!

• Composer also uses the mechanism to signal whether some nodes in the computation DAG are good to go; this mechanism is called Sensors.

• To utilize Sensors in Pandora's case, an additional service is spinned up. It's just an API wrapper over a pretty much one-table SQL database, and ...

• ... this very SQL database is not yet in the cloud, "but it's the next step, and we are working on it".

• Also, since we were looking for something super-simple, and Composer (given all the above) is not simple enough, we have built our own Python-based meta-language, readable enough to even teach our analysts to use it.

• This meta-language creates Composer DAGs, and here's how some fancy ones look like when visualized. (Spoiler alert: they are fancy-looking ~50-nodes DAGs, as any engineer would expect.)


Unspoken during the talk, but here is a valid interpretation of the aftermath:

"Too bad I’m not allowed to manipulate machines manually. Thus, what can be done with an under-hundred-lines Makefile, which is a common way to manage DAGs of dependencies and run the jobs in the right order, is instead being done within this beast called Composer, an order of magnitude slower, more complex, and more expensive."

After writing all the above, I'm still not convinced my text truly reflects how do I feel about this presentation. Let me try it once again, from an a bit more blunt angle. Below is one of the best – if not the only! – way a critically-thinking engineer could understand the above talk.

• Please help.

• They are torturing me.

• They are holding my family hostage.

• They have threatened to molest and kill everyone and everything I cherish in this world if I do not show absolute and unconditional obedience to Them.

• The only way for me to send out a message about what's really happening behind the scenes is this talk, where I could manage to speak in the terms opaque enough so that Their censors would not realize how badly am I begging for help, right here and right now. Don't move an eyebrow if you understand me, and instead please listen very carefully, for I'm about to share deeper details.

• Please help. Please help. Please help.

• They are forcing me to use Google Cloud.

• Not only They are forcing me to use Google Cloud, They are forcing me to use Composer.

• Composer is what Makefiles could do in the seventies.

• But I must glorify Them in any and every way possible to keep Them pleased, and to soften my suffering from how They can and will torture me unless I show how successful their brainwashing has been so far.

• Composer does indeed do what Make can do for almost fifty years by now.

• My life would become way easier had I not been forced to use Composer. I could just write scripts that spin up and tear down instances. That would be a lot less code. And a lot easier to follow and maintain. And we would not have to throw that code away in a few years after Composer outgrows its present stage of evolution.

• Alternatively, Composer would become more bearable if it could f*cking understand that if I'm giving it three manually spinned up machines for three jobs, it may well schedule those three jobs on those three machines, mapping jobs to machines one to one.

• Or maybe, just maybe, Composer could expose some interface for me to manually hint it which dedicated machines to use for which jobs.

• Hope you can see how badly am I suffering. It's just the tip of the iceberg. Please, please, please, please help. Rescue me from here. I beg you.

• Obviously, I can not tell you openly, here and now, how terrible the state of the art is. And, remember, I am forced to use Composer. So, despite all the handicap that Composer imposes on me, I've done some Real Software Engineering to actually have the Google Cloud do what the cloud should do in order to help me solve the engineering problem I'm facing: to reliably and repeatedly run three jobs on three Cloud machines in parallel. The best way I have found involves spinning up three machines manually, as well as another service and another DB to get the job done. Good thing, They – thank God! – have allowed me, for this another service I have to run to please Composer’s Sensors, to, at least for now, keep this one-table DB outside the Google Cloud, just so that I have one fewer thing to worry about so far.

• With the above being said, and with Them watching: All hail the Google Cloud, the Google Cloud is the future. I am grateful beyond comprehension for an opportunity to use it. Please don't kill or torture me or my family. I'll keep using Google Cloud for as long as you keep feeding me, and thanks for that health insurance too. Amen.


Fantastic talk indeed.

Even better than that keynote offtopic by a guy from Chase about coming from the family of slaves when asked to tell the audience more about how does Chase leverage Google Cloud.

Which language should I use for my interview, Java or Python?

One says Java is better because it is more useful, and interviewers prefer that. Another says in Python you will make fewer mistakes on the white board, and can focus on problem solving.

Language selection 101 for a coding interview.

• Use the language you are most comfortable with.

• If the decision is close, use the language your interviewer is most comfortable with.

• If still close, use the language the code in which is easier to explain and less likely to contain bugs.

• If still close, use the language that fits the task best. (Unlikely for interview questions except for some coding tricks though.)

• If still no clear winner, use the language the company you are interviewing for prefers.

• If still unclear, Python is better than Java. :)

Re. OpenAI and not releasing the new language model.

Here is my doubly controversial opinion.

First off, despite leaning libertarian, I can actually see some pros behind the decision to not release such a powerful tool into the wild. It would obviously be [ab]used against the very forces that hold Alphabet together, to begin with. On the other hand, not keeping these results secret is already a big step, and we have to give OpenAI a lot of credit for not being quiet about them.

And, yes, other companies would obviously create models of comparable or higher quality. It may well even happen within half a year, which would give OpenAI a pass with respect to actually releasing their very own results.

Here comes the really controversial opinion.

What would I do if I have such a model at hand?

I know what I would do. I would apply it to a lot of sensitive domains with a precise goal in mind: To help me or my team change our minds.

Take literally any controversial topic. Climate change, for instance, or illegal immigration, or religious controversies on whether book X actually says Y.

We know for a fact those topics are controversial largely because quite a few forces want them to remain so. And I am in possession of the technology to quickly and effectively summarize tons of information, provide quotations and citations, ultimately tell the real source from the derivatives, and detect clusters of presumably paid/promoted/fake content aimed at swaying the public opinion one way or another.

I would target a broad range of those controversial topics. I would make a survey within my team on what our views are; an anonymous survey, at least initially. I would have my "AI" gather the facts to form an "unbiased" opinion. I would present those facts to my team to study. I would look on what topics do we, presumably smart humans, change our opinion.

And I would release that. Along with the results of those anonymous surveys, and along with the facts and citations my "AI" has discovered.

Ideally, what we would release would go against the majority public opinion on some topic. Ideally, what we would release would publicly uncover some cluster of resources that we have evidence are artificially biased pro or against certain point of view. Ideally, what we would release would come with [anonymous] testimonials from our very team members, that "they used to believe X, and, as smart people, we used to believe X has to be the truth, but, evidently, this OpenAI model has presented us with clear proofs that the opposite of X is true instead."

Now, this would be a clear step forward. This would open up a real conversation about the future of our civilization. Can we trust ourselves? Does voting really convey the will of the people, or is it just too easy these days to manipulate?

In a way, the above idea is as simple as applying the technology to where it should be applied first and foremost: to help us uncover the universal truths about the world we live in.

The laws of, say, physics or chemistry, are largely understood by now. Some "laws" of how does the human body function — and how to "fix" it if it's "broken" — are being understood better and better. The "laws" of psychology and sociology, on the other hand, are just too hard to uncover for us, humans. Large-scale experiments are either taking too long, or are too immoral to run. We have to rely on scarce and possibly erroneous bits and pieces gathered so far, and our picture is far from perfect. Thus, when making the decisions that are mission-critical for our civilization, we conservatively rely on certain ideas that we only vaguely know are the valid ones.

I want to see the "AI" applied to bridge this gap. This is a noble goal that I believe is worth pouring resources into.

...

It's funny people seem to accept huge spendings on the Large Hadron Collider, on religious institutions, or on various hot and cold wars, while the very idea of directing even a one-digit percentage of our budgets and of the humanity's wealth towards the very applied research of understanding what manipulations are we most prone to, where our weakest spots are, and how to best structure the society so that those weak spots are worked around, not amplified — these essential fields of studying the very us, humans, are almost non-existent today, to the degree that individual corporations are rightly afraid of releasing practical results of certain experiments, justifiably fearing those results are too ahead of our time, and would thus likely be abused.

What Is The Expectation Out Of Each Software Engineering Level At Facebook?

Full disclosure: Former full time employee in Facebook, Google, and Microsoft MSFT.

All examples are hypothetical (they used to erase my memory before letting go).

Levels add on to the previous ones, with lower level expectations slowly fading away. For example, an engineering director is not supposed to know how to fix the build in one of their teams from the top of his head.

Also keep in mind that, because of career-driven development and other completely unrelated factors, people would not necessarily work on their level all the time. Naturally, they will try to get to the tasks slightly above their level to grow faster skills- and recognition-wise. Examples in the other direction happen as well.

https://www.forbes.com/sites/quora/2013/12/09/what-is-the-expectation-out-of-each-software-engineering-level-at-facebook/