Task Testing
Part of AI writing tools compared
Comparing editing quality without rewarding longer answers
Use a fixed brief, meaning gates and repair effort to compare AI edits without giving extra credit for longer answers.
Judge AI edits by how much they improve a fixed passage while preserving its job, facts and length. More words do not make a better edit. Give each candidate the same rough draft, audience and length limit. Then assess the text a reviewer could approve.
Set the editing brief first
Use a short passage resembling work your team edits. Keep its source facts and intended message on a separate answer sheet. Mark what may change: sentence order, tone and repetition. Mark what must remain: a quoted term, number or condition.
Ask for an edit rather than a new article. For example: “Make this notice clearer for customers, keep every condition, add no facts and stay within roughly the current length.” If one tool works through selected-text suggestions and another through a chat prompt, give them equivalent requirements and record the interface difference. Give each candidate the same number of attempts.
A length cap prevents extra explanation from winning merely because there is more of it. It is a constraint, not a target: a shorter edit still fails if it drops a necessary qualification.
Judge the edited passage
Review the result in this order:
- Meaning and facts:Reject changed claims, omitted conditions and invented details before scoring style.
- Clarity:Can the intended reader understand the main point on one read? Identify what improved or worsened.
- Economy:Did the edit remove repetition and unnecessary setup? Keep new words only when they help the reader.
- Voice and fit:Is the wording suitable for the audience and channel without becoming vague or inflated?
- Repair effort:What would a person still need to change before approval?
Record reasons rather than one impression such as “sounds professional”. A useful note might say that one version puts the required action first while another buries it under background. If reviewers disagree, compare their reasons against the brief.
AI Editing Quality: Key Evaluation Criteria
- Meaning and factsNo changed claims, omitted conditions or invented details
- ClarityMain point understandable on one read
- EconomyRemoval of repetition and unnecessary setup; only new words that aid understanding retained
- Voice and fitAppropriate for audience and channel without vagueness or inflation
- Repair effortWhat a human reviewer still needs to fix before approval
Step-by-Step AI Edit Assessment Process
- Verify meaning and facts against sourceEnsure no conditions or claims are altered or omitted
- Assess clarity of main messageCheck if reader understands in one read
- Evaluate economy of languageRemove redundancy and unnecessary phrasing
- Confirm tone and audience fitMatch style to intended channel and readership
- Estimate repair effort requiredNote what a human would still need to fix
Keep the comparison fair
Where practical, show reviewers the edited passages without product names. Keep the original beside each result and check protected facts against the answer sheet. Record every attempt, prompt, feature, plan and manual change. If a product offers several suggestions, state how the chosen suggestion was selected; do not compare a hand-picked option with another tool’s first output.
Use several editing problems, such as a wordy notice, an awkward but accurate paragraph and a message needing a tone adjustment. Report results by problem type so an average does not hide what each tool handles well or poorly.
Grammarly’s Paraphraser offers paragraph-level suggestions that users can accept or dismiss. Copilot in Word can rewrite selected text for tone or concision. These documented controls make them relevant to an editing exercise, but neither description establishes the quality of an edit. The decision is whether an acceptable version needs less human work without losing information.
