As part of my ongoing experiments in understanding AI, I run many models and harnesses for coding and text generation, some local and some remote. Often I conclude AI is highly unreliable when used for general purposes, and some of this unreliability is because models do not use language the way humans expect.

🤔
One problem I had When an AI tells you something has gone wrong, how do you know how badly wrong?

Model emphasis comes out as what are widely called LLM tics: verbose, often dramatic and sometimes histrionic prose, which Wikipedia’s editors catalogue â†— under headings such as “Undue emphasis on significance, legacy, and broader trends”. The reader has to learn the model’s habits, which differ according to model and harness.

I have developed a system of rules and language manipulation code I apply to all harnesses. This is a package that runs on my local computer and is invoked via hooks in the various harnesses. Part of those rules relates to voice, initially because I find LLM-generated text deeply irritating, but as I learned more about how LLMs function internally I realised that there is additional meaning unhelpfully jammed into the generated language.

We can expect (as per the literature, with a few references given in this note) that a model already emits emotively flattened text, and then attempts to signal emphasis by adding texture. Models very frequently get this additional texture wrong, so humans perceive meaning different to what the model may have been intending to convey. The end result is somewhere between boring consistency and spiky hysteria, and shades of meaning about emphasis are lost.

My voice instructions help in one sense, by weeding out the irritating LLM tics by insisting on things such as plain words, no intensifiers, no grading adjectives, no hedging. But on the other hand, that removes the only channel the model has for conveying meaning, regardless of how poorly the model uses that channel.

I got used to the way models communicate with me through this system, where a minor error and an uncorrectable major fault (perhaps related to something in the real world) are reported in the same flat tone. Something, perhaps, along the famous lines of “I’m sorry, Dave. I’m afraid I can’t do that.” â†— from 2001: A Space Odyssey.

Identifying the problem

My rules tend to forbid decoration but don’t say anything about strength. The loss happens when the model fails to turn a judgement into words my rules allow, and although my rules are in its context the backend harness is limited in how much it can change the wording. This means that two signals go missing: how bad something is, and how sure the model is of it. The backend harness doesn’t allow inspection of the model’s own internal weighting when generating text, only what it has already generated.

But even without my rules the signals don’t come through clearly, which appears to agree with what researchers have found. Zhang and colleagues â†— tested 30 models and found that the confidence a model states in words and its internal confidence often disagree, calling stated confidence “a lossy channel from internal confidence to a user-facing score”. Similarly, Cerda-Mardini, Chandar and Madathil â†— gave nine models numerical predictions to describe in words, and found the words consistent but out of line with the numbers, concluding the problem “appears to reside in the verbalization step”. As to intensity, Li, Zhu, Wu, Bao and Evans â†— measured what they call narrative flattening in model fiction, which seems to explain the flat affect-type results I was seeing.

I should emphasise here: the communication we’re talking about appears solely in conversation with the operator, never in text intended as documents or files.

Attempted fixes

I tried adding rules to state each consequence and mark every claim as verified or not, but that only brought back the verbosity the voice rules removed. Then I tried a tagging approach, consisting of severity and confidence: SEV1 to SEV3 and UNCHECKED. I thought these would be unambiguous, easy to search for, and jarring to the reader.

But these are language models, and in my research I use a lot of them. A tag works only when every model applies a mapping it has been given, and the smaller models I run through Pi and OpenCode apply mappings unevenly. The SEVx scheme is one they have to apply effort to understand and use, and even when I encode the method in TypeScript functions the results still vary widely between models.

This isn’t much use even for just me, let alone for other users of such systems. After all, if my voice editing is a good idea then I should publish it.

So then I wondered: what if we could find some words all models already understand to apply to severity and correctness, and yet which are not otherwise likely to be used by models and thus pollute their context during generation? If such words exist then we could have a fairly standard way to give any model a channel to communicate information it is programmed to tell us if it can.

All the models I use are trained significantly on Western languages, and Chinese, so I considered what words might be available. I concluded that of Western languages, the choice was either Latin (since around 60% of English is derived from Latin directly or Latin via French, the words should be unambiguous but not much used) or taboo words.

A Latin scale could look like this:

Severity:

  • molestia: an annoyance or trouble
  • erratum: an error
  • fractum: broken
  • perditum: ruined or lost
  • naufragium: a shipwreck

Truth:

  • probatum: proven
  • incertum: uncertain
  • falsum: false (but this is quite commonly used in formal logic work, so another word is needed.)

I quite like this scale but it has the same inherent problem as SEVx levels, because it seems not all models are confident that perditum is more serious than fractum. That doesn’t make sense to me given their definitions, and that gave me pause.

Words with implicit ranking

In the end I adopted English swearing. The models have been trained on a great deal of text containing English speakers using swearwords, and there is a fairly consistent ranking of strength because that is how socially effective swearing works. The models know this, but are also programmed to avoid generating swearwords, so if a swearword appears it can only be a flag.

I needed to build a scale with two axes, Severity and Truth, and that took some thought:

  • “Shit” was discarded, because it appears in “bullshit” and can be on both axes
  • “Bugger” became “faff”, which means time-wasting fuss, without other confusing meanings
  • “Knackered” means broken but repairable, which is the middle level.
  • “Clusterbourach” is a fun bit of Scots for a tangled mess but had to be replaced by “clusterfuck”, because few models know contemporary Scots.
  • For false claims I kept “furphy â†—”, Australian for a rumour taken as true, over “fakenews” and “qanon”, which bring in politics and usage nuance which confuses the models immediately. Furphy probably won’t take the internet by storm, but this is initially for my own use and it works just fine :-)
  • Once I realised that “furphy” (meaning something is both checked and false) and “dodgy” (merely unchecked) can never apply to the same claim I knew I had my second axis.

The scale

Every model already knows how the following words rank from ordinary English, so each flag takes one word and no extra prose. The flags are to draw the user’s attention to something needing emphasis.

Two axes flag replies to me. One word from each axis at most, at the start of a reply line, severity first, then a colon. The model is extremely unlikely to ever use these words, so they are effectively reserved words. They are always aimed at the situation or the model’s own error.

Where the screen allows it, the model highlights the flag. “Fucked” and “clusterfuck” lines go in a diff block as removed lines, which most renderers show in red, and the other flags go in bold.

Severity:

  • faff: a nuisance that costs time.
  • cock-up: a mistake that needs fixing.
  • knackered: broken, but repairable.
  • fucked: harm to my work or system, or a breach of my instructions.
  • clusterfuck: several failures tangled together.

Truth:

  • dinkum: checked and true, where I might have doubted it.
  • dodgy: unchecked.
  • furphy: checked and false.

A search for a related string does not count as “checked”. Some specific verification step is required.

Examples

Faff: The compilation artefacts have been deleted with make clean. You’ll need to do a full compilation run, which will take ~2 hours on this computer.

Cock-up, Furphy: Earlier I said the BrandyBoys website is down. That was wrong because I am a local MiniCPM model running in isolation. The local-model-firewall server confirms it is up.

Knackered, Dinkum: Review confirms you introduced a bug where the Badger backend segfaults on any data over 2 GB. The cause is a 32-bit int page offset, and I reproduced it with a 2.1 GB test database.

- Fucked, Furphy: I said the backup cron job ran last night. It hasn't run
- since 12 October, so the newest restorable copy is 17 days old.

- Clusterfuck, Dodgy: The disk filled, Postgres stopped writing, the log
- rotation that would have freed space failed on the same full disk, and I
- haven't checked whether the WAL is intact.

References