Bibs & bobs #46

 We Designed the Habitat

I have been reading Owen Jones’s Force of Nature [1], a book about natural selection which has the unfortunate side effect of making almost everything look like natural selection. This is probably why evolutionary biologists should not be allowed near university strategic planning.

Jones’s argument is not the tired one that evolution happened a very long time ago and eventually produced us, universities and the committee meeting. His point is that selection is happening all the time. More importantly, what we do changes the selection pressures acting in the world. This turns out to be an awkward thought to have while watching education respond to LLMs.

The Sin of Assumed Invariance

Jones uses commercial fishing to make the problem clear. Fishing industries preferentially catch large fish. Yet they have often behaved as though fish populations will remain more or less the same: remove this year’s large fish and nature will kindly manufacture another standard batch for next year.

Jones calls this the Sin of Assumed Invariance. The problem is that the fishing is itself changing the population. If being large makes you more likely to end up on a plate, being smaller becomes rather a good reproductive strategy. Jones describes evidence that intense size-selective fishing can alter growth, maturation and reproduction surprisingly quickly. The intervention changes the thing being intervened upon.

Education appears to have adopted the Sin of Assumed Invariance as an implementation strategy. We introduce AI detectors, declarations of use, supervised assessment, oral defences, locked browsers and diagrams explaining precisely when ChatGPT may be consulted. Then we ask whether the intervention worked. Somewhere inside this model sits a strangely motionless student. The policy changes. The assessment changes. The LLM changes. The student, apparently, waits. Unfortunately, students learn. So do teachers.

Introduce an AI detector and you have not simply detected AI use. You have made undetectable AI use more valuable. Introduce an AI declaration and you have not simply produced transparency. You have created a new genre: the acceptable account of how AI was used. Invent an “AI-proof assessment” and there will shortly be students discovering how a LLM can help them complete it.

This is not evidence that students have suddenly become morally defective. It is evidence that environments have consequences. Jones gives us a much better question than: Did our intervention work? He suggests that we ask instead: What did our intervention select for?

That question should probably be printed above the door of every university AI working group. Unfortunately, there is usually already something above the door. Future Ready, perhaps. Or Responsible AI. Possibly Transforming Learning for an AI-Enabled Future, if the sign was commissioned by consultants and the available wall space was generous.

These phrases perform an important institutional function. They imply that somewhere there is a future, that it has already been inspected, and that readiness consists largely of arriving there in the correct attire.

Jones suggests a less reassuring possibility. Every intervention alters the environment in which subsequent behaviour occurs. The policy does not merely regulate practice; it becomes part of the conditions to which practice adapts. The assessment rule, detector, declaration form and approved-use matrix all enter the habitat and begin exerting selection pressures of their own.

So the awkward question for the working group is not simply whether its policy encourages “responsible AI use.” It is whether the policy makes responsible use more viable than irresponsible use, or merely makes responsible-looking use more viable. Responsible-looking use sharpens the Jones point: institutional interventions can select for the appearance of the desired behaviour rather than the behaviour itself.

Future Ready is much neater. It also fits on a lanyard.

We built the fitness function

Jones becomes even more useful when he turns to evolutionary computation. The basic trick is wonderfully simple. Generate lots of possible solutions. Introduce variation. Test them. Let better-performing solutions contribute to the next generation. Repeat.

But there is a rather important detail. Someone has to specify what counts as “better.” Evolutionary computation therefore requires what Jones calls a fitness function: a specification of the problem and the characteristics against which candidate solutions will be judged.

Education already has these. We call them rubrics. Suppose an assignment rewards coherent prose, clear structure, plausible argument, appropriate referencing and something identifiable as critical thinking. Then a technology arrives that can produce coherent prose, clear structure, plausible argument and references while sounding sufficiently thoughtful to survive moderate exposure to a marking rubric.

Education responds with astonishment. This is rather like constructing a bird feeder and expressing outrage when birds turn up. We designed the habitat. Then we complained about the wildlife. The problem is not necessarily that the fitness function is bad. It is that LLMs expose a distinction we had previously been able to ignore. The declared fitness function might be: develop historical understanding. The operational fitness function might be: produce 2,000 words containing the characteristics that cause a marker to award 73%.

Before LLMs, these could be treated as sufficiently close cousins. Producing the essay generally required enough reading, thinking and writing that the artefact provided some evidence of what had happened inside the student. Not perfect evidence. Education has always survived on proxies. Otherwise assessment would require opening students and inspecting the learning directly, which would create paperwork and sometimes a lot of blood.

LLMs disturb the proxy. They produce some of the valued characteristics of the artefact without necessarily producing the educational process we thought the artefact represented. The machine has not destroyed assessment. It has done something considerably ruder. It has revealed the fitness function.

Then it gets jagged [2]

At this point the evolutionary story can become far too neat. The environment changes. People adapt. Everyone acquires AI literacy. There is a webinar. Problem solved.

But real people are inconveniently jagged. A student can be superb at getting useful responses from a LLM and terrible at recognising nonsense in those responses. Another can possess excellent disciplinary judgement but use ChatGPT as though it were Google with more adjectives. A teacher can understand their students and subject extraordinarily well while having almost no idea what current LLMs can do. Another can prompt magnificently while remaining slightly uncertain what any of it is for. Calling all this “AI literacy” gives the comforting impression of a ladder. Jaggedness gives us something more like a mountain range designed by a committee that disagreed about gravity.

People occupy different adaptive positions along multiple dimensions at once. That means there is no single route from novice to expert. More importantly, moving is difficult.

The trouble with leaving somewhere that works

Jones describes another problem in evolutionary computation. Candidate solutions can become too similar and converge on a local sub-optimal solution. His less impressive but much more useful translation is: stuck in a rut. The phrase is unfair to educational practice because ruts are often extremely efficient.

Consider the essay. The teacher knows how to set it. Students know roughly what one is. The LMS knows where to put it. The rubric knows how to judge it. The moderation process recognises it. The accreditation documents contain reassuring boxes into which it fits. Nobody has to explain to Quality Assurance why students are constructing imaginary conversations between Napoleon and a chatbot.

This arrangement may not represent the finest educational possibility available to humanity. But it works. And that matters. When people are told that LLMs require them to move to a new adaptive position, we frequently forget that the journey may first make them less competent.

A teacher who has refined an assessment for ten years replaces it with something experimental. Workload rises. Predictability falls. Students become confused. Moderators become interested. We can assume that nobody wants moderators to become interested.

So “resistance to change” may sometimes be resistance. But sometimes it is perfectly sensible behaviour by someone well adapted to their current environment who can see that getting somewhere supposedly better requires crossing a stretch of territory in which things get worse.

Jaggedness makes that crossing different for everyone. For one teacher, having a LLM challenge students’ explanations is an obvious next experiment. For another, opening ChatGPT is the experiment. For one student, a LLM expands what can be questioned. For another, fluent machine prose closes questioning down because it looks so thoroughly like an answer.

There isn't one transition to AI-enabled education. There are thousands of little transitions, made from different starting points, with different risks, capacities and adjacent possibilities. The arrow in the implementation diagram is therefore lying. It knows this. It simply has excellent graphic design.

Variation may be the thing we need

Jones’s discussion of evolutionary computation contains one final but useful twist. When candidate solutions become too similar, evolutionary algorithms can deliberately increase variation. Diversity helps the system escape the local rut and explore more of the possible solution space. This seems almost exactly opposite to our institutional instinct.

Faced with LLMs, education wants convergence and certainty. Approved practice. Approved tools. Approved prompts. Approved assessment designs. Frameworks explaining how to comply with other frameworks.

Some standardisation is clearly necessary. There are real questions about privacy, equity, intellectual responsibility and assessment validity. But if we do not yet know what good educational practice with LLMs looks like, and I don't think we do, then prematurely selecting one approved destination may be precisely the wrong thing to do.

We need variation. We need to encourage small experiments. We need to applaud cheap failures. We need different teachers trying different things from different starting positions. We should reward students noticing what helps them think and what merely helps them produce. Not because whatever emerges will necessarily be wonderful. Evolution has produced both the human brain and the blobfish. Selection is not a synonym for progress.

Which brings us back to the most important Jones question which is not How do we get education to adapt to AI? But is What is our educational environment selecting for?

If our assessments reward the production of answers, LLMs will flourish there. If our AI policies reward concealment, concealment will improve. If our institutions reward standardisation, safe practices will outcompete interesting ones. And if moving to new practices imposes all the risk on individual teachers while the institution retains all the benefits, staying in the rut may remain an exceptionally fit behaviour.

LLMs may therefore be exposing something more interesting than the weaknesses of students or the limitations of assessment. They may be exposing the selection pressures of education itself.

We built and maintain the habitat. We specified the fitness functions. We rewarded some behaviours and made others expensive. Then a peculiar new creature appeared that was exceptionally well suited to parts of the environment we had constructed.

Perhaps the interesting task is not to domesticate it. Perhaps it is to ask why the habitat looks like this in the first place. Sadly, that question does not fit neatly into the implementation framework. But be reassured, a working group will no doubt be established.

Notes

[1] Jones, O. D. (2026). Force of nature : understanding evolution's deepest logic--and putting it to use (First edition.). W.W. Norton & Company.  

[2] I have written a number of posts about the jaggedness of AI savviness in teachers and students and the problems that flow from that. This is one of them.

Bibs & bobs #45

 Fourteen Seconds Before the Worksheet

Large Language Models arrived in education. This was potentially significant. Here was a peculiar new object capable of producing language, arguing, role-playing, translating, explaining, inventing examples, changing perspectives, making unexpected connections, confidently fabricating things and, when challenged, apologising in a manner suggesting absolutely no emotional consequences whatsoever.

Education regarded it carefully. For about fourteen seconds. Then somebody said:

“Can it make a worksheet?”

And that was more or less that.

The domestication programme [1]

Education has considerable experience with new technologies. Whenever something strange arrives, there is a short period during which it might conceivably change what we do. This is dangerous. Fortunately, institutions have developed powerful antibodies. The unfamiliar object is surrounded by committees, frameworks, learning outcomes, risk registers and people asking whether it integrates with the LMS. 

Eventually it is rendered harmless by being made to perform an existing activity slightly faster. The computer became a typing machine. The internet became somewhere to put readings. The learning management system became a filing cabinet that sends email. And the Large Language Model, an object apparently assembled from a substantial portion of recorded human language, became a machine for making lesson plans.

Domestication complete. Nobody was hurt. 

This is an astonishing achievement. It should not be underestimated. We have taken something genuinely odd, a machine with which one can conduct a rapid, recursive conversation about almost anything, and taught it to produce: lesson plans, quizzes, PowerPoint slides, rubrics, model answers, avatars that “teach”, differentiated worksheets, feedback comments, and cheerful little exit tickets asking students what they learned today.

This is rather like discovering a visiting extraterrestrial civilisation and immediately asking whether its spacecraft can laminate. The answer may well be yes. That doesn't make it the interesting question. Yet there is something wonderfully reassuring about the whole process.

If you type: Create a Year 9 lesson for photosynthesis.

Seconds later there will appear a lesson plan, a worksheet, an infographic, three learning outcomes, several activities, six multiple-choice questions and, if the machine has not been adequately supervised, a rubric. This looks enormously productive. There is certainly a lot of it. Quantity has always enjoyed a somewhat undeserved reputation in education.

We have been here before

There is a small historical joke buried in all this. In the early days of educational computing, some teachers wrote software to teach particular concepts. This was no simple matter.

Suppose you wanted to write a program to teach fractions. Before the computer would do anything useful, someone had to decide what a fraction was, what students commonly misunderstood about fractions, which examples mattered, which sequence might help, what counted as an error and what should happen next. This involved an irritating amount of thinking and, an awkward thing sometimes happened.

The person who learned most from the educational software was the person who wrote it. Writing the program forced its author to take the concept apart, examine it, find its difficult edges and put it back together. The student later encountered the finished software and clicked NEXT.

This was not quite the intended distribution of learning. Still, there was something interesting going on. The difficult work of making the teaching material was itself intellectually productive. The problem was solved. The machine can do it.

Cognitive offloading, now with clip art

Ask a LLM: Give me five analogies for entropy suitable for a ten-year-old.

The machine disappears briefly into whatever passes for thought in these circumstances and returns with five analogies. Three are faintly dreadful. One involves a bedroom. There is nearly always a bedroom. One is actually quite good. The good one is copied into PowerPoint. Yay! Success!

But something curious has happened. Someone, or something, has done the wandering. Possible representations have been generated. Comparisons have been made. Explanations have been bent, tested and discarded. The edges of the concept have been bumped into. The human receives analogy number four.

This is called cognitive offloading, which makes the transfer of intellectual activity sound reassuringly like placing luggage in an overhead compartment. Clearly, effort has been saved. Nobody had to spend twenty minutes producing five not-quite-right analogies.

This is generally regarded as a benefit because educational systems have spent decades establishing the important principle that teachers have far too much to do. But there is a small possibility that the five not-quite-right analogies were where some of the learning was hiding. We may have automated the troublesome part that required us to think and retained the PowerPoint slide. This would be mildly funny if it were not such a familiar move.

The photocopier acquires opinions

The odd thing about LLMs is not that they make educational materials. Humans have been making educational materials for centuries. Some civilisations are believed to have collapsed beneath the weight of them. The odd thing is the interaction.

You can say: “No, that's not what I mean” or, “Try another explanation,” or, “Assume the opposite,” or, “Give me an example where this breaks down,” or, “Argue against yourself,” or, “Make the case from a perspective I dislike,” or, “That's nonsense.” And the machine will always comply. It does not sigh. It does not glance at the clock. It does not say that this was covered in last week's meeting. This is a genuinely peculiar new capacity. So naturally, we have given it a template for a Year 8 worksheet.

Perhaps the output isn't the interesting object

One consequence of domestication is that attention settles on the thing produced. The lesson plan, the infographic, the worksheet or the model answer. The finished thing looks educational, so we assume that is where the educational value must be.

But perhaps the interesting object is not the output. Perhaps it is the traffic: the questions, the rejected answers, the sudden change of direction, the moment someone notices that a plausible explanation is wrong, the comparison between three different representations, the argument about what counts as a good example or, the realisation that the question itself is badly worded.

When the machine says something unexpected. The human thinks, effectively, “Hang on.” That begins to look less like content production and more like intellectual activity. Which is inconvenient, because intellectual activity is much harder to upload to the LMS.

The slightly embarrassing possibility

Students already have access to versions of these machines, often better versions and some students may be more LLM skilled than the teacher [2]. This creates a curious arrangement.

The institution may use a LLM privately to produce teaching materials. The teacher may use a LLM privately to produce teaching materials. The student may use a LLM privately to produce the assignment. Everybody then meets in the classroom and pretends the interesting thing is the document. This is a magnificent piece of theatre. The machines converse with everyone backstage while the humans exchange PDFs out front.

Perhaps there is another possibility. Rather than always hiding the interaction and presenting its polished remains, we could sometimes make the interaction itself available for inspection. Not because everyone should learn a set of approved prompting tricks. That would simply be domestication with an advanced settings menu. But because there may be something worth seeing in how different people work with a machine that answers too easily.

You can ask things like: What got accepted? What got challenged? What was ignored? (Ouch) Who noticed the hidden assumption? Who decided that using the machine at all is making matters worse? These are not really LLM skills. They are ways of working with claims, explanations, possibilities and uncertainty. They merely become unusually visible when another participant in the conversation can produce a confident answer in two seconds.

A small problem with responsible use

Educational organisations are understandably concerned that students should use AI responsibly. The usual response is to write a policy. This is rather like responding to the invention of the bicycle by issuing a document on responsible balance. The policy may be necessary. It is unlikely to teach balance.

If working with systems like these becomes part of ordinary intellectual life, then students probably need encounters with actual practices rather than simply instructions about permitted outcomes. They need to see people getting it wrong. They need to see seductive answers rejected. They need to see uncertainty survive contact with fluent prose. They need to see that sometimes the best use of a LLM is to continue the conversation. And sometimes they need to see that the best use is to close the window and go for a walk.

And to think the unthinkable, there may even be no single correct way of working with the machine. This will be disappointing for anyone currently preparing a framework. Ooops!

A successful domestication

The deepest domestication of LLMs may therefore have little to do with banning them or permitting them. It happens when we decide, almost without noticing, what kind of thing they are. Maybe they are seen as a resource generator or productivity aid or a tutor, or cheating device. Maybe a feedback machine.

Once the label sticks, the possibilities begin to shrink. The strange object becomes familiar. The familiar object becomes manageable. The manageable object gets a procurement category. And eventually somebody produces a two-page guide called Five Ways to Use Generative AI Effectively in Your Classroom.

At which point the alien spacecraft has successfully been fitted with a laminator.

But perhaps we should leave the machine strange for a little longer. Not because it is magical. It isn't. Not because it will transform education. Things have been promising to transform education for quite some time and education remains impressively difficult to transform. But because there is something almost heroic about encountering a genuinely unfamiliar technology and resisting, for slightly longer than fourteen seconds, the urge to make it do what we were already doing.

The worksheet can wait.

It has waited before.

Notes

[1] A less playful account can be found in this short paper: Bigum, C. (2023, June). Teacher librarians, orthodox and heterodox: making sense of and playing in a world increasingly run by machines. Access, 37(2), 31-35. https://drive.google.com/file/d/1XJSD294lvIjAAwc6rXkwXK8iA8zx-NJX/view?usp=sharing 

[2] I have a post that points to the problem of the uneven or jagged nature of LLM savviness among students and their teachers. 

Bibs & bobs #44

 The Question Machine

The arrival of large language models created an educational emergency. This was fortunate, because universities are extremely good at educational emergencies. Within months there were task forces, principles, frameworks, guidelines and webinars. Documents appeared containing diagrams with arrows in them. Somewhere, almost certainly, a committee was established to coordinate the work of the other committees.

The problem seemed obvious. An Answer Machine had been invented. Students could type in a question and receive an answer. Worse, the answer arrived quickly, confidently and without requiring them to find a library book, remember a password or sit through a PowerPoint containing the words learning outcomes. Clearly, something had gone terribly wrong.

Tim Klapdor, in a recent post called The Answer Machine, makes the larger point that humans have always been attracted to things that promise answers. Religions have done it. Experts have done it. Universities have done it. Search engines have done it. Now we have machines that will cheerfully answer almost anything, including questions nobody particularly needed answering. But I wonder if we are giving the answer rather too much credit.

For most of the history of formal education, answers were expensive. They lived in books, libraries, laboratories, disciplines and the heads of people who had spent twenty years learning when to say it depends.

Education built an impressive amount of machinery around them. Curricula specified answers. Teachers transmitted them. Textbooks stored them. Examinations asked students to reproduce them. Universities certified that people possessed an appropriate quantity of them. 

But then answers became cheap. Not necessarily good. Not necessarily true. Not necessarily useful. Just ridiculously cheap at the point of use — provided you did not look too closely at what it costs to make them cheap. The bill, as usual, had been sent somewhere else.

You can now obtain 800 fluent, plausible words on almost anything before you have had time to regret asking for them. This is rather more interesting than whether students might use ChatGPT to write an essay. If answers have become cheap, perhaps answers are no longer where the interesting educational work is. Perhaps the scarce thing is the question.

Not the elaborately engineered prompt beloved of the AI productivity industry:

Act as an internationally recognised expert in medieval drainage systems and produce a seven-point framework...

I’m thinking about the question that opens up something new. The output that makes you notice what you had not noticed, or puzzle over what had previously seemed unremarkable. The useful answer is not an endpoint but an opening into a Kauffmanesque adjacent possible: a new patch of intellectual territory from which more interesting questions suddenly become askable. The answer now matters because it gets you to a place from which you can ask a better question.”The question you could not have asked before receiving the previous answer.

Seen this way, a LLM starts looking less like an Answer Machine and more like a rather odd looking Question Machine.

You ask it something. It replies. The reply maybe is wrong, banal, surprising, interesting or, occasionally, much better than you expected. That changes what you know. That changes what you now may want to ask. So you ask something else. And now you are somewhere you could not have been when you began prompting. The interesting question is no longer:

Did the machine give the correct answer?

It is:

Where did this exchange take me?

This makes the institutional responses to LLM use look somewhat odd. Something genuinely strange arrives and universities immediately ask:

Is it cheating? What is acceptable use? How do we detect it? What policy should we adopt? Can someone develop a framework by Tuesday?

These are not bad questions. They are questions with a strong desire to stop being questions. Something totally unfamiliar appears over the hill and the institutional machinery starts shouting, Darlek like:

DOMESTICATE! DOMESTICATE!

The thing must be captured, classified and placed in a policy document, preferably one with numbered headings and a tasteful diagram showing Responsible AI Use in the middle.

But perhaps this rush to domesticate is precisely the wrong instinct. We have barely begun to see what happens when people think, write, argue, design, learn and become productively confused in the presence of these machines. It seems a little early to put them on a lead, give them a policy number and declare the experiment complete.

Perhaps we need fewer confident declarations about the impact of AI on education and rather more small-r research. Yeah. I have a thing for small-r research.

Small-r research begins with the deeply unfashionable sentence:

I don't know. Let's try something.

What happens if I tell the machine to argue with me rather than applaud? What happens if the student looks at its beautifully polished suggestion and says, “Nope”? Why did yesterday’s prompt open a door and today’s produce beige intellectual porridge? What happens if I give the machine a different job entirely — critic, provocateur, idiot companion, unreliable witness? What happens if I leave it out of the room? And why can two people sit down with exactly the same model and emerge having visited completely different intellectual planets? This is a more poke the beast and see what it does approach.

Clearly, these questions are unlikely to produce a national framework. This is part of their charm. They amount to poking the thing with a stick and paying attention.

The standard picture of LLM use is: human asks then machine answers

That seems to leave out the interesting bit.

A human asks a question. The machine produces something. The human reads it. The human is now, however slightly, a different human as a consequence of what the machine spat out. The next question therefore can come from somewhere new. The useful object of interest may not be the prompt. It may not be the answer. It may be the trajectory.

This is why I am suspicious of tidy distinctions between machine answers and human wisdom.

It is wonderfully reassuring to put the machine over there, extruding synthetic sludge by the bucketful, while we humans remain over here being deep, relational, embodied and, on a good day, wise. Unfortunately, humans have also produced committee minutes, airport novels, management jargon and several centuries of confidently wrong ideas. Reality is seldom kind enough to respect the categories we invent for it.

A conversation with a LLM might send me back to a book, into an argument with a colleague, towards an experiment, or, more alarmingly, into the discovery that something I have confidently believed for twenty years, a load-bearing chunk of my intellectual path dependence, is held together by habit, professional muscle memory and a small republic of papers citing one another in a reassuring circle.

I think it’s unhelpful to want the machine to be wise. It merely has to perturb me. Sometimes productively. Sometimes disastrously. Sometimes by inventing three Belgian researchers who have never existed. Which is precisely where judgement becomes interesting. If answers become abundant, judgement becomes way more important, not less.

Am I able to recognise an interesting wrong answer? Do I notice when the machine has simply polished my assumptions and handed them back? Can I tell when something deserves pursuing? Do I ignore something merely because it sounds authoritative? Does the output nudge me to ask the next question?

Those seem to me rather more demanding capacities than producing a five-paragraph essay on the causes of the First World War. Although I would be interested to know how much geopolitical carnage can now be compressed into five paragraphs without violating the rubric. Or perhaps the real breakthrough is discovering that the Schlieffen Plan was, in fact, paragraph three.

Perhaps this is the educational opportunity hidden inside the educational emergency. We built institutions for a world in which answers were scarce. Now we have machines producing them in industrial quantities. The domesticating response is to defend the old scarcity. Ban the machine. Restrict it. Detect it. Require students to prove that the answer came from somewhere sufficiently inconvenient.

The other possibility is more unsettling. We could ask what education becomes when the answer is no longer the star of the show. Which things are still worth learning? What deserves assessment? What should we get better at noticing? And, perhaps most awkwardly for institutions built around approved answers, which questions should we stop trying to house-train?”

We may eventually discover that LLMs are disastrous for education. We may discover they are transformative. More likely, both statements will turn out to be annoyingly true, often in the same classroom before lunch.

But for now, perhaps we could resist the urge to issue a verdict. The history of AI prediction is not exactly a monument to human foresight. We have been confidently announcing both the imminent arrival and imminent failure of artificial intelligence for decades, usually shortly before being surprised by something else entirely.

Universities have spent centuries turning strange things into familiar answers. Then a machine arrived that could manufacture familiar answers in twelve seconds and everyone reached for the emergency procedures manual.

Perhaps we have been protecting the wrong end of the business. Education should also make the familiar strange: unsettle the obvious, annoy the settled, and occasionally discover that the intellectual furniture has been nailed to the floor for no particularly good reason.

So perhaps the interesting question is not what the machine knows, but what becomes possible to ask once answers are cheap. This is, of course, far too important to be left to curiosity. A working group will now determine which questions are permissible.