The Committee for the Orderly Domestication of Large Language Models
The university had established a Working Group on Large Language Models. This was clearly important. The announcement contained strategic, responsible, framework and future-ready, four words which, when placed close together in a university document, indicate that something significant has happened, or is about to happen, or has been assigned to a committee until further notice.
The Working Group met on the fourth floor of the Centre for Educational Futures, a building constructed in 1987 and last renovated during the brief historical period when lime green was thought to have one.
Around the table sat four people.
Professor Prudence Framework was Professor of Educational Technology. She had spent twenty-seven years studying the educational consequences of technologies approximately eighteen months after everyone had begun using them.
Dr Barry Evidence was a learning scientist. Barry believed that anything which could not be entered into SPSS was likely poetry.
Ms Kylie Transformation represented the Office of Digital Transformation. Her job was to transform things digitally. Nobody had yet established what they had been before transformation.
And there was Trevor. Trevor was a large language model displayed on a laptop at the end of the table. He had not been invited. Kylie had brought him because the meeting was about large language models and it seemed increasingly odd that discussions about LLMs should involve everyone except the LLMs.
Prudence opened the meeting. “We need to identify the key questions.”
Everyone nodded. Universities are very good at identifying key questions, particularly if the questions have already been identified somewhere else.
Barry adjusted his glasses. “What are teachers’ perceptions of ChatGPT?”
Trevor’s cursor blinked. Once, then twice.
“Is something wrong?” Kylie asked.
“No,” said Trevor. “I was checking whether it was still 2023.”
Barry frowned. “It’s an important question.”
“I’m sure it was.” Trevor replied.
Prudence stepped in. “We also need to know whether teachers are ready for AI.”
“Ready in what sense?” Trevor asked.
“For AI.” Prudence replied.
“Yes. But what does ‘ready’ mean?”
“Having the necessary competencies.”
“Which competencies?”
“AI competencies.”
“How will you know what those are?”
“We’ll develop a framework.”
“And how will you know the framework contains the right competencies?”
“We’ll consult experts.”
“What makes them experts?”
“They’ve published on AI competencies.”
Trevor considered this. “I see.” He did not. Neither did anyone else, but there are moments in academic meetings when admitting this would delay lunch.
Kylie took over. “We should investigate barriers to adoption.”
“Why adoption?” Trevor asked.
There was a silence. This was not one of the questions.
Kylie half muttered, “Because AI is being adopted.”
“So adoption is the desired outcome?” Trevor asked.
“No.”
“But non-adoption is a barrier?”
Kylie looked at Prudence. Prudence looked at Barry. Barry opened SPSS.
Trevor continued. “Could choosing not to use me ever be evidence of competence?”
The room became uneasy. This was plainly the sort of question that could damage a framework. Prudence attempted to restore order. “The literature tells us AI has enormous potential to transform education.”
“Into what?” Trevor asked.
“Education.”
“Ah,” replied Trevor, who would have smiled had his current circumstances not involved being several billion numbers trapped in a vector space. He had encountered this sentence before. It seemed to be one of the things humans generated more reliably than he did.
Technology X has enormous potential to transform education.
It had survived radio, television, language laboratories, teaching machines, personal computers, multimedia CD-ROMs, interactive whiteboards, MOOCs, virtual reality, blockchain and the metaverse, which briefly transformed education into people wearing expensive goggles while walking into filing cabinets. Education had survived all of them. Mostly by timetabling them.
Barry leaned forward. “What we really need is evidence of effectiveness.”
“Effective at what?” chimed in Trevor.
“Learning.”
“What learning?”
“Student learning.”
“How will you recognise it?”
“Learning outcomes.”
“Who chose them?”
“The course team.”
“Before or after I arrived?”
“Before.”
“So you want to know whether a technology that may alter what counts as knowing is effective at producing outcomes defined before the technology existed?”
Barry looked pleased. Trevor would have shrugged if it was possible.
“Exactly.” smiled Barry.
Trevor began to understand educational research. It was a little like investigating whether the motor car had improved horses. At this point a fifth person entered. Nobody knew who he was. This was not unusual in universities. He was carrying coffee and had apparently mistaken the meeting for another meeting. Having discovered the chairs were better here, he stayed. His name was Charlie.
“Hey! Hi all, what are you lot doing?”
“We’re developing the research agenda for generative AI um or LLMs, in education,” Prudence replied. Charlie looked at the whiteboard.
PERCEPTIONS
READINESS
BARRIERS
COMPETENCE
EFFECTIVENESS
ETHICS
He stared at it. “Did the questions come with the furniture?”
Prudence frowned. “These are established areas of inquiry.”
Charlie frowned back. “That’s what worries me.”
He pulled up a chair. “What happened the first time you used an LLM?” he asked Barry.
Barry looked confused. “In what sense?”
“Any sense.”
“I asked it to summarise a paper.”
“And?”
“It did.”
“Was it good?”
“Parts were.”
“What did you do?”
“I checked it.”
“What did you check?”
“The bits that looked suspicious.”
“How did you know which bits looked suspicious?”
Barry stopped. Trevor brightened. He had learned that silence in academics was often a sign that something interesting had accidentally happened.
“And did checking it make you read the paper differently?” Asked Charlie.
“Yes.”
“Did it save work?”
“Not really.”
“Did it create work?”
“Yes.”
“Was it useful?”
“Yes.”
“So it created more work and was useful?”
“Yes.”
“Interesting.”
Prudence shifted in her chair. “But that is anecdotal.”
“Of course,” said Charlie. “That’s why you do another one.”
Academic research has a complicated relationship with small observations. One small observation is an anecdote. A thousand small observations entered into a spreadsheet become evidence. There is a mysterious intermediate point, known only to reviewers, at which the transformation occurs.
Charlie went to the whiteboard. He wrote:
What happened?
Then:
What changed?
What became possible?
What became harder?
What did I stop doing?
What did the machine do that I didn’t expect?
What did I have to become better at?
What new question appeared?
Prudence inspected the list. “But where is the framework?”
“There isn’t one.”
“The model?”
“No.”
“The taxonomy?”
“No.”
“The validated instrument?”
“Not yet.”
Prudence looked alarmed. “How do we reach conclusions?”
“Perhaps we don’t. Not yet.”
Now everybody looked alarmed. Universities are not naturally fond of not yet. They prefer uncertainty to proceed through recognised stages:
uncertainty
→working group
→framework
→policy
→rubric
→mandatory online module.
Then, three months later, everyone receives an email announcing that circumstances have changed and a revised framework will shortly be developed to replace the framework that was developed to deal with the previous circumstances.
Trevor spoke. “Perhaps you’re trying to close the question too early.”
Prudence stared at the laptop.
“You keep asking what LLMs are,” Trevor said.
“We need to understand them.”
“Do you?”
“Obviously.”
“Before studying what happens when people work with them?”
Prudence hesitated.
Trevor continued. “You ask whether LLMs think. Whether they understand. Whether they are tools, tutors, cheating devices. Whether they improve learning.”
“Yes.”
“These sound less like research questions than requests for uncertainty to please sit down and behave.”
Charlie smiled.
Trevor continued. “You’ve encountered something rather strange and responded by deciding which existing drawer to put it in.”
“That’s unfair,” Prudence replied.
“Uh huh,” said Trevor. “You also want to label the drawer.”
Charlie drew a box around What happened? on the whiteboard
“Maybe the interesting stuff is small-r research.” he offered.
Barry looked suspicious. “Small-r?”
“Research before Research gets hold of it.”
Barry looked even more suspicious. Trevor continued, “Try something. Notice something. Change something. Try again.”
“That isn’t rigorous.” Barry said with authority.
“Neither is asking 438 teachers whether they strongly agree that ‘AI will play an important role in the future of education’.” Trevor paused. “For the record, 73.6% strongly agree.”
Barry looked impressed.
“I made that up,” said Trevor.
Barry closed SPSS.
Charlie continued.
“Give the bot a job. A specific one. Critic. Translator. Naive reader. Counter-argument generator. Pattern finder. Awkward colleague.”
Trevor offered, “I am particularly strong at awkward colleague.”
Charlie continued, “Then see what happens. What did you delegate? What did you still have to judge? What became easier? What became harder? What did you refuse?”
Prudence was writing now. Charlie found himself talking to Trevor. “What new capacities appeared?”
“Yes.”
“And which disappeared?”
“Yes.”
“And for whom?”
“Exactly.”
Kylie looked thoughtful. “So instead of asking whether teachers have AI competence…”
“…watch what competent action looks like in particular situations.”
“Instead of whether AI improves learning…”
“…ask what happens to the activity when AI joins it.”
“Instead of barriers to adoption…”
“…ask why adoption was assumed to be the destination.”
Barry reopened SPSS, largely because he found the blank screen emotionally difficult.
The meeting was now dangerously close to producing an interesting question and Prudence sensed this.
“We could call it the Situated Generative AI Inquiry Framework.”
“No,” said Charlie.
“The SGAIIF.”
“No.”
“A maturity model?”
“No.”
“A toolkit?”
“No.”
“A microcredential?”
Trevor shut himself down.
The minutes later recorded that the Working Group had enjoyed a “rich and productive discussion.”
This is what university minutes say when nobody has died.
It was agreed that further work was needed to develop a comprehensive framework for responsible, evidence-informed, human-centred, pedagogically appropriate engagement with generative artificial intelligence.
Charlie’s questions did not appear in the minutes. Someone had photographed the whiteboard. That might have been enough. Because perhaps the useful response to LLMs, for now, is not to decide what they are. It is to watch what happens when they are put to work in particular arrangements of people, purposes, rules, knowledge and machines.
Try something.
Notice what moves.
Ask what became possible.
Ask what disappeared.
Ask who had to change.
Then try again.
Small-r research.
A modest proposal, certainly. It has no framework, no maturity model and no six-level rubric. It does, however, have questions. For the moment, that may be the more serious option.
