Raphael's workbench

Context Rot, Corporate Feudalism, and the Art of Weakening Belief Edges

Last month’s active lifestyle left me less time for reading. I was only able to read one full book. But what a book!

Influence: The Psychology of Persuasion - Robert B. Cialdini
I started reading this as it was recommended to me as one of the main books in the library of expert sales people. For that purpose, it certainly does the trick, and it shifted my belief system quite significantly from a throbbing “all salespeople are sleazy scumbags” to “wow, human psychology is kinda neat, and it’s sad that some salespeople are just not very competent at their job.” In my quest to become better at sales, I would say reading this has helped quite a bit (and will likely continue to help). However, meanwhile, I was also able to look at this as a practical treaty on social psychology. I find that it complement well my previous studies on social trust and power brokerage, and will fit it in my “political science” bibliography shelf.

❦❦❦

Let me also share a few links to important AI/LLM news. I still find it important to stay on top of recent developments. That said, I have decided to lower my involvement with LLM tech last month, and I hope to reflect this in the distribution of my comments below.

Last month, we looked at the “accuracy collapse” that LLMs encounter with growing problem complexity. This month, the big scientific splash came from a team at Chroma: in Context Rot: How Increasing Input Tokens Impacts LLM Performance, Kelly Hong et al. report that LLM performance also decrease sharply past some threshold context size. This is significant because the commonly held belief in the community was that performance of LLMs is currently limited by context sizes, and that larger contexts would lead to better results. According to this latest research, this may not be true. It is unclear if this new result is a variant of the complexity problem studied by Apple, or a new problem entirely.

Meanwhile, in Not So Fast: AI Coding Tools Can Actually Reduce Productivity, Steve Newman reviews a recent scientific result on developer productivity. In a nutshell, the authors discovered that experienced developers estimate a productivity gain of ~20% ahead of time with AI coding tools compared to baseline (no tools), but end up taking ~19% more time with the tools than without. Besides the proximal result (productivity decrease), the authors emphasize the surprising result: that developers are far more enthusiastic about the perceived productivity gain than the evidence supports. This should help us look at other news with bigger grains of salt.

In yet another direction, the Sutro team explains in The End of Moore’s Law for AI? Gemini Flash Offers a Warning how the operational costs of LLMs grow quadratically with the length of the input context and that of the output, whereas the various operators currently charge customers linearly with these lengths. In other words, the more complex the work submitted by customers, the more money the operators are losing. Whether operators change their cost model or raise their prices, the writing is on the wall: the time of low-cost, VC-subsidized LLM-as-a-service might come to an end sooner than many expect. To me, the forces towards switching to open source, locally executed models will become immense.

❦❦❦

On the “AI geopolitics” news, a few notable things have happened.

There is “inwards looking news”, like The Economist’s AI is killing the web. Can anything save it? (archive). Despite the alarmist title, the journalist here is merely pointing out that the actors of the commercial web are changing (the web, as a system, is not going anywhere of course). This, like many other articles merely looking inwards, reads like fishers arguing with each other about the effects of their collective over-fishing on their own business.

To take the fishing analogy further, I find more interest in news covering how the over-fishing affects people further away from the fishing business.

For example, in Bad Actors are Grooming LLMs to Produce Falsehoods, Sophia Freuden, Nina Jankowicz, and Gary Marcus point out that LLMs are vulnerable to both intentional and unintentional propaganda. I find this type of news unsettling especially in combination with articles like The Force-Feeding of AI on an Unwilling Public by Ted Gioia, highlighting that the general public seems to have fewer and fewer options for human-to-human interactions.

One thing we would do well would be to remember to recognize The sound of inevitability (Tom Renner): it’s not because big companies like OpenAI tell us that “AI will happen” that it actually will, nor should. We do have collective agency over what kind of society we want to build forward.

❦❦❦

Now, let’s switch gears and lift our heads from the swamp of AI news. I found a few thought-provoking things to read.

In Being too ambitious is a clever form of self-sabotage, Ma Alvika points out that smart people are “too good” at visualizing what they want to achieve and thereby set themselves up for failure, because their experiential skill can never catch up to their imagination. According to the author, every creator encounters a point in their project where they intimately realize how much more work there is to do than what they were enthusiastic for at the start. In their words,

The quitting point is the moment you discover whether you want to be someone who had a great idea or someone who made something real.

This concept is closely related to the one identified in Learn to love the Moat of Low Status, by Cate Hall. This author identifies the embarrassment of appearing to be a beginner in a new field (or an endeavor) to be very hard to experience emotionally for people who already have experience in other areas. This “protects” existing players in that field from newcomers, thus creating a moat.

At a personal level, I am recognizing (and experiencing) both challenges above in my current project. Naturally, I also have other challenging circumstances to deal with (see above), but the effect of these two is real.

❦❦❦

And then there’s stuff I think about more often and consistently.

In a two-part series on tribalism (part 1, part 2), Spencer Ying offers a view that tribalism still exists in modern Western societies as a parallel system for power brokerage alongside the rule of law. The author also suggests that acknowledging this in public is somewhat “taboo” and that we are limiting our understanding of how the world works by not studying this topic more. I nearly ignored this writeup until I watched this video on morality and the power structures in modern Russia from an ex-Soviet historian called Elvira Bary. I remember distinctly thinking that some of the problematic systems that Elvira describes do not seem exclusively Russian to me; they echo my understanding of how things work in certain pockets of society where I live and across the ocean. This made me realize that Spencer Ying was maybe on to something.

Generally, I’m not personally sold on the concept of “tribalism”. Its anthropological definition seems too primitive to encompass the power structures in large organizations. However, some of the arguments Spencer Ying made also support the idea that our democracies tolerate (and, in some cases, foster) pockets of feudalism as an instrument of productivity and economic growth. What is a corporation, after all, other than a legally sanctioned governance structure that eschews democracy and institutionalizes class hierarchy and low-agency subservience? The common counter argument is that anyone is able to “leave” when they want, unlike in medieval times. To this, I would object that 1) folk in medieval times did legally have the right to leave, and were prevented from doing so through economic hardship rather than outright force and 2) this is not so different today; it is hard for workers to transition out, as the discomfort of a job you already have often trumps the uncertainty of finding any other job (especially in the current economy!). Also, it’s not just me who has this opinion. Elizabeth Anderson, a philosophy professor in Michigan, wrote a comprehensive study of this precise topic. Noam Chomsky also had choice opinions on the matter.

There are two ways to reflect on these topics. One is by using them as input when making personal choices. When building an organization, I could consider what I’ve learned on matters of governance to make things “better” in various dimensions within my organization. The main constraint however, is the Prisoner’s Dilemma: how many degrees of freedom do I truly have if I also want to remain competitive with someone who takes moral and ethical shortcuts?

(Make no mistake; I am not suggesting that individual organizational leaders have no agency on the matter. There are organizational approaches that do work in a fully local fashion. For example, in There is No Meritocracy Without Lottocracy Paul Melman recommends random selection to attribute responsibility, as a way to fight nepotism and tribalism. Many other approaches are applicable too.)

The other way to reflect on this is to recognize the inherently political nature of the challenge. People form local groups (be it e.g. a family, sport clubs or corporation) using the rules and norms set by the larger group/society. The topic of “tribalism” (or institutional feudalism) exists within a wider context of a social discussion of what we should allow as a society, and what system of incentives and regulations we are willing to put in place to get there. For example, child labor was seen as important, even irreplaceable, until we changed our collective priorities.

As to how this can happen? It feels naive (to me) for us to assume that one group/institution will discover or implement significant social change, demonstrate success, and then incite other groups/institutions to imitate its choices from their desire to imitate its success. I personally would not trust modern corporations (even my own!) to positively contribute to social development—not because there are actively bad actors, but because the incentives are perversely misaligned and the competitive equilibrium does not easily allow it.

That being said, I also changed my views since my university days quite a bit: I also do not believe that science would change anything. Most of the systems around us are entrenched based on feels, not rational thought. So, what remains?

In a way, I did experience the book I read this month (Influence, see above) as one way to work on this. We could work on carefully crafted marketing campaigns (propaganda) that advocate for better structures. However, I was also excited to discover Why Facts Don’t Change Minds in the Culture Wars—Structure Does by Vas Zayarskiy: a complex system can be forced to transition to a new equilibrium if only one edge of the associated beliefs is sufficiently weakened. This sounds much more tractable!

❦❦❦

References:

Like this post? Share on: BlueSky ❄ Twitter ❄ HN ❄ Reddit ❄ LinkedIn ❄ Email

Raphael Poss Avatar Raphael Poss is an entrepreneur who occasionally publishes field notes on systems, leadership, and the messy edge between technology and people.
Comments

Interested to discuss? Leave your comments below.

Keep Reading

Reading Time

~9 min read

Published

Category

Briefings

Tags

Stay in Touch