It’s a Trap
The AI marketing train has been going full steam for a while now and many have been caught up in the hype. Some have lost hope in humanity, lost faith in the future, and sunk towards depression with thoughts of a purposeless life in a world without a job to be had. These experiences are deeply personal; they are most uncomfortable.
The marketing train goes something like this: “AI is here! Look at these benchmarks! It’s already smarter than us! It is the last human invention and will automate all jobs, but you’ll lose your job before then to those who use AI… unless, of course, you use our AI first!” What a wildly productive message! The marketing hype uses some modicum of truth (we have some version of AI), with false equivalency (as if benchmarks resembled real life), and quickly launches the imagination into a possible future anchored in a claim about intelligence in the form of prophecy about jobs.
That message creates the emotional context of fear. Fear of loss. Fear is incredibly motivating. Daniel Kahneman showed that the motivation to avoid a loss is 2x stronger than the motivation to gain the same thing. Fear even sells better than sex.
Beware the Corporate Agenda
Let’s take a look at GitHub and its Copilot product in the early stages:
GitHub took on an enormous investment to develop Copilot using unproven generative AI tech;
Copilot launched as an unpaid technical preview in 2021, generating zero revenue;
They conducted a study and published an article in 2022, spending even more time, effort, and money; and
They wrote a follow up blog post to show just how “good” their tool really was.
That was a lot of money tied up in unrealized inventory. How badly would you want a gigantic investment to pay off? What do you think GitHub would have done to make that happen?
Skewing the Subjective
GitHub shared that productivity is “difficult to measure”, productivity to a developer is like “having a good day”, and “satisfied developers perform better”. They painted a picture where objective measurement was too hard, thereby elevating the importance of the subjective. They alluded to the objective by using the SPACE framework to measure productivity, but with a stated focus on Satisfaction and well-being and Efficiency and Flow, both of which are subjective. They provide charts with stats and numbers, which imply objectivity; however, they are merely aggregations of more subjective self-reports. They even admit what they did in their paper:
“We will thus focus on studying perceived productivity”
That emphasis is theirs. They didn’t study productivity, they studied perceived productivity. GitHub found a group of early adopters who were predisposed to a positive emotional state, crafted a study which leaned on cognitive bias instead of measuring objective value, and ran a marketing campaign with a crafted “study” to capitalize on a gigantic investment. We all have biases; they took advantage of them. We simply feel more productive because it takes less effort to offload our skilled work, but how do we know if we are actually more productive?
Their chart in their first section of the blog post reveals much. Research conducted to uncover the truth uses neutral phrasing to avoid bias, like “how did this impact your productivity?” Yet their questions were written to psychologically prime the reader for a positive answer. They are still using the same tricks today. You can see it in their survey engine in questions like, “How much less time did the coding take during this PR with Copilot?” (Emphasis mine.)
So much effort was spent measuring the subjective and advertising it as objective. Yet despite all of their claims about the difficulties in conducting objective measurements with disagreed-upon metrics of what constitutes productivity, they proved they knew exactly how to do it. It’s how the industry has been doing it for ages.
Skewing the Objective
The blog post has a single objective measurement, and it did not appear in the "perceived productivity" study. They measured that it took developers 55% less time to complete the task with Copilot! What an amazing find! Dollars were sure to roll in! The chart in that section is flashy and draws attention. It’s easy for us to gloss over the setup of the study:
“…write an HTTP server in JavaScript.”
What are the most difficult things for an AI to accomplish? Everything that was stripped out of that test:
Understanding the existing code;
Correctly rationalizing complexities of the business rules and its domain; and
Knitting new changes together with the old code and logic in a manner which increases quality.
None of those difficulties were in their highly synthetic and ridiculously simple test case.
What have we seen AI generally succeed at? Greenfield development of a tightly-defined and well-known problem which is repeatedly documented in training data. That one test was specifically designed to showcase Copilot in the absolute best light possible, which they had already discovered through use of Copilot in the technical preview.
It is all a marketing ploy.
GitHub is not alone in this. In general, companies pushing AI avoid legitimate objectivity, mask the subjective as objective, and use existential fear to drive sales. How do we feel about that?
Where Do We Go From Here?
We encourage every developer to capitalise on the absolute best tool they have: the human brain. Your brain has been engineered to learn, which enables you to increase your rate of progress and accuracy over time. It is the practice of our skills which stimulate growth. How much practice of real software development do we get when we offload our skills to an AI?
The first step is to do the work ourselves to stimulate our own growth. The next is to find tools which boost the right kinds of measures. We even have a tool for you: Easely. Easely solves common issues with modern software development.
It is designed for your brain; the churn of cognitive load is constrained to the tool, allowing you to allocate those extra mental resources to the task at hand;
It transforms the process of development into a system of learning to uncover the next right step as quickly as possible; and
It streamlines the process of pulling a solution from our mind.
One step at a time, you’ll write the only code that is needed.
