Writing ESL Test Questions With AI: Better Distractors

Ask an AI assistant to “write ten multiple-choice questions on the present perfect” and you will have a quiz on your screen before your coffee cools. The problem is not speed — it is that the quiz almost always looks better than it works. The correct answer is obvious, the wrong options are throwaway filler, and a student who knows nothing can still guess their way to a passing mark. Fluent English is not the same as a good test, and this is exactly where most teachers get let down by AI.

The single skill that separates a usable AI-generated test from a decorative one is the ability to shape the distractors — the wrong answers. This guide walks through how to prompt for assessment items that measure real ability, how to calibrate difficulty without a pilot group, how to catch the cultural bias AI quietly bakes in, and how to build answer keys and rubrics you can actually defend to a parent or a program director.

Why AI struggles with assessment even when its English is perfect

A language model is trained to produce the most probable, most helpful-sounding text. That instinct works against you in testing. When you ask for a wrong answer, the model tends to produce something obviously, comfortably wrong — because “wrong but plausible” is a harder target than “wrong and clearly signposted.” The result is a question where the right choice glows on the page.

Assessment also has a hidden requirement that fluency does not: every question is supposed to measure one specific thing. A well-built item isolates a single skill or piece of knowledge, so that a wrong answer tells you which misunderstanding a student holds. AI does not know what you are trying to measure unless you tell it, and it will happily blend three different grammar points into one confusing question if you let it. The fix is not a better model — it is a better prompt and a teacher who knows what the question is for.

Start with the construct, not the question

Before you type a prompt, decide what the item is supposed to reveal. Testing people call this the construct — the specific ability behind the question. “The present perfect” is too broad to be a construct. “Choosing between the present perfect and the simple past when a time reference is present” is a construct. So is “recognizing that for እና since pair with different kinds of time expressions.”

When you hand the AI the construct instead of the topic, the quality of the output changes immediately. Compare these two prompts:

Weak: “Write a multiple-choice question about the present perfect for B1 learners.”
Strong: “Write one multiple-choice question for B1 learners that tests whether a student can choose the present perfect over the simple past when the sentence contains an unfinished time period like this week. Each wrong answer should reflect a real learner error, not a random tense.”

The second prompt does most of the assessment design for you. You are no longer asking for a question; you are asking for a measurement. That framing is what turns AI from a quiz vending machine into a genuine drafting partner.

gray and white click pen on white printer paper

The real skill: writing distractors that reveal thinking

A distractor is only doing its job if a student who holds a specific misconception is genuinely pulled toward it. Good distractors are the difference between a question that measures learning and one that measures luck. The best wrong answers come straight from the errors your students actually make.

ESL teacher at an English whiteboard guiding a student, illustrating how well-crafted distractors reveal learner thinking

What makes a distractor work

A strong distractor is plausible to someone who half-understands the point, and it maps to a diagnosable error. If a student picks it, you should be able to say exactly what they got wrong. In a present-perfect item, useful distractors might include the simple past (the classic overgeneralization), the present continuous (a common substitution from lower levels), and a subtly wrong participle form. Each one, if chosen, tells you something you can teach to next lesson.

Weak distractors, by contrast, are answers no reasonable student would consider: a wildly wrong tense, an obviously ungrammatical fragment, or a word from a different topic entirely. They pad the question to four options without adding any measurement. When AI gives you these — and it usually does on the first pass — that is your cue to push back rather than accept the draft.

The prompt pattern that gets you there

The pattern that consistently produces better distractors is to make the AI justify each wrong answer. Add a line like: “For each option, state in a parenthesis the learner misconception it represents. If an option does not correspond to a real error, replace it.” This forces the model to reason about why a student would choose each answer, and it exposes the lazy fillers instantly — because the AI cannot invent a plausible misconception for a throwaway option.

A second reliable move is to feed the AI your students’ actual mistakes. Paste three or four real error sentences from recent homework and say: “Base the distractors on the kinds of errors in these examples.” Now the wrong answers reflect your classroom instead of a generic global average, and the test starts diagnosing the students in front of you.

Calibrating difficulty without a pilot group

Professional test writers trial their questions on real students and throw out the ones that are too easy or too hard. Classroom teachers rarely have that luxury, but AI can approximate the process if you ask the right question. Instead of asking “is this too hard?”, ask the model to role-play your learners.

ESL classroom with whiteboard and open workbooks on a desk, representing calibrating test difficulty without a pilot group

A prompt like “Estimate what percentage of a typical B1 class would answer this correctly, and explain which distractor would trap the weakest third of the class” gives you a rough difficulty read and a sanity check in one move. It is not psychometric data, but it flags the two failure modes that matter most: a question so easy everyone passes it (measuring nothing) and a question so tangled that even strong students miss it (measuring test-reading, not English).

You can also ask the AI to produce the same construct at two difficulty levels — one for your stronger group and one for learners who need scaffolding. Because the construct is fixed, both versions test the same skill, which keeps a mixed-ability class fair while still meeting students where they are.

Catching the bias AI quietly builds in

Language models lean heavily on the cultures that dominate their training data. Left alone, they will set reading passages in American shopping malls, name characters Jake and Emily, and assume everyone celebrates Thanksgiving. For an international classroom, that is a fairness problem: a question that requires cultural knowledge a student does not have is testing background, not English.

Young ESL student in an online lesson with a teacher on screen, illustrating checking AI-generated tests for hidden bias

Two habits keep this in check. First, specify the context in your prompt: “Use neutral, internationally recognizable situations — no holidays, sports, or foods specific to one country.” Second, run a dedicated review pass after the questions are drafted: “Review these items for cultural or regional knowledge that a learner outside North America might not have, and flag anything that could disadvantage them.” The AI is surprisingly good at auditing bias when you ask it to — it simply will not do it unprompted. This same review discipline is worth applying to any AI-generated content, not just tests.

Building answer keys and rubrics you can defend

An answer key is not just a list of letters. For any question that could reasonably have two defensible answers, you need to know why the key is what it is — because a sharp student or an unhappy parent will eventually ask. When you generate items with AI, generate the rationale alongside them: “For each answer, write one sentence explaining why it is correct and why the closest distractor is not.” Keep that rationale in your files. It doubles as instant feedback you can hand back to students.

For open-ended and writing tasks, AI is genuinely strong at drafting rubrics, but treat its first version as a starting point. Ask it to build a rubric tied to the same construct — “a four-band rubric for a short opinion paragraph, describing what distinguishes each band in terms of task completion, range, and accuracy” — then edit the descriptors into your own words and your program’s standards. The AI handles the tedious scaffolding; you supply the judgment about what actually counts as good work in your room.

Happy teacher attractive matude adult is smiling using laptop in class typing working with chalkboard in background. People a

Fitting assessment back into what you teach

A test only means something if it lines up with what you taught. The fastest way to keep AI-written assessments honest is to feed the model your lesson objectives directly. Paste the week’s learning targets and say: “Write assessment items that measure only these objectives. If a question drifts beyond them, flag it.” This keeps the test from wandering into grammar you never covered — a common failure when AI free-associates around a topic.

It also works in the other direction. When you are building a unit, ask the AI to map each objective to the type of question that would best measure it. Some targets suit multiple choice; others — like producing a spoken request or writing a persuasive sentence — need a performance task. Letting the assessment shape the lesson, rather than bolting a quiz on at the end, is where AI stops being a shortcut and starts making your teaching tighter.

A repeatable routine for your next test

Pulling it together, a reliable pass looks like this. Name the construct for each item so the AI knows exactly what to measure. Ask it to justify every distractor with the misconception it represents, and reject the fillers. Have it estimate difficulty and flag the trap options. Run a bias review against an international audience. Generate rationales for the answer key and a rubric for any open response. Finally, check every item against your lesson objectives before it goes anywhere near a student.

None of this takes as long as writing the test from scratch, and the output is dramatically better than the vending-machine quiz you get from a one-line prompt. The AI is doing the drafting; you are doing the assessment design. That division of labor is the whole game — because a test that measures the wrong thing quickly, or measures nothing at all, is worse than no test. Used well, AI gives you back the hours you used to spend writing plausible wrong answers, and hands them to the part of teaching only you can do: reading what the results are telling you and deciding what to teach next.