When building software, there are many pieces that you need to assemble for your product to function. Like Lego pieces, you put the pieces together. But Legos come with a set number of pieces in a bag, and a guide on how to put them together. Software is. To my own detriment, not that. Software is closer to having a bag of lego pieces, instructions on what these pieces do. You read the guides, try to put these pieces together in hopes that your product works, but then halfway through, you realise you got the wrong pieces, or you got the right ones, but you’re missing a whole lot more. Software building is Legos without the guardrails, and the lower you are on the knowledge curve, the more challenging it is. But when you’re new to it, you’re filled with hope and confidence. "I'll just read the documents and put the pieces together. How hard can it be?" you tell yourself.
The Darien scheme, Scotland's attempt to establish a colony on the isthmus of Darien (what is now Panama), was born out of Scotland's exclusion from the riches of exploration and the resulting economic hardship during the Age of Discovery, a time when the Spanish and British Empires flourished. The scheme was planned and championed by William Paterson, who founded The Bank of England.
Maybe it was a case of Hubris, or a willingness to survive the famine and poverty. The reasons for failure are contested and multifaceted, but one popular angle is the string of miscalculations that went into the planning of this venture. I’d like to imagine if William relied on AI to plan this venture, it would probably go something like this:
William: I am embarking on a plan to colonise the isthmus of Darien. What do I need to do?
AI: Sounds exciting! Here’s what you’re going to need: You’re going to need capital to finance this ambitious project, ships to carry the settlers, and food supplies
William: Great, I have managed to raise the capital from proud Scotsmen, I have secured the ships, and we’re now on our way to the Isthmus
AI: Perfect, let me know if you need anything else
But the venture encounters unforeseen troubles, ones William did not even think to ask about.
William: our supplies are low, and the natives do not want to trade with us. We brought them… combs? For their hair
AI: You should actually consider trading essential goods with them. Ask for a resupply and make sure to onboard goods you can trade
William: Oh great, now the Spanish are attacking us and the English are not coming for our help
AI: Oh, political espionage is an interesting topic! But you only asked me about the logistics of this scheme, and that’s on you buddy
Linear Thinking
The fastest way to get from point A to point B is a straight line. That is theoretically true, but it doesn’t translate in real life applications. Large language models tend to think linearly, which can be deceiving.
I spend a meaningful amount of time planning out the architecture of the knowledge engine I’m building using these LLM systems. They can reason, and they can develop a holistic understanding of the project’s scope, but they’re often unencumbered by doubt or skepticism. I come to appreciate these human qualities when asking these systems to execute on our plan,and, to go back to my lego analogy, put the disparate pieces together. The systems exude the confidence and naivete of an intern that’s aced all their undergraduate classes, but has not dealt with the curve balls thrown around in a work environment. So I end up being the skeptic partner and asking questions like “are you sure we have all the lego pieces, do we maybe need another bag?” when we hit a wall, which is often the case.
This behavior is baked into these systems, they’re designed to find the most straightforward path to a solution. I find that concerning at a time when there are conversations around AI agents replacing software engineers at big tech companies. I can only imagine an autonomous, mid-level engineer working arrogantly, not asking coworkers any questions, and wastefully spending company time and resources trying to retrofit their linear solution to a project it's tasked with.
For a venture in the early innings of its journey, much like this one, displacing human labor and delegating to the AI overlords is a tough terrain to traverse. Before you know it, you sunk one fifth of your capital on a failed project implementation, and your only option is to join a union with England, sorry I mean, your larger, better-funded competitor, if you’re lucky.