Senior Capstone Project

This past semester, I supervised a capstone project for senior Applied Linguistics students. We asked whether using AI for writing helps students write better without it.

The question is by no means trivial. Writing is one of the most critical cognitive skills, and we, humans, locate ourselves in our thinking by the conscious and frankly time-consuming process of writing, writing messily and imperfectly, rereading, deleting, editing, the beautiful art of meaning-making. Like Joan Didion put in her literary manifesto: “I write entirely to find out what I’m thinking, what I’m looking at, what I see and what it means.” So I often wonder, will my students understand what they think and what it means, before AI beats them to it? Will they be able to locate their voice? I was excited to see this project unfold.

Nuraiym Kaden, Zhansulu Mirambekova, Ayauzhan Assetova, and Aida Bekmukhan designed a pre-experimental one-group pretest-posttest study, an efficient design for generating preliminary evidence on an intervention without requiring the resources of a full RCT. Participants wrote a baseline essay independently, completed three weeks of structured AI-assisted writing tasks built on Vygotsky’s scaffolding framework (“AI as a More Knowledgeable Other” whose support deliberately fades as the learner grows), then wrote a final essay without AI.

Two independent raters scored essays across four dimensions (content and development, organization, language use and style, mechanics) using an adapted validated rubric; ICC of 0.773 confirmed acceptable inter-rater reliability. Statistical analyses ran in jamovi.

Mechanics and language use and style showed the strongest gains. Content and development and organization also improved, but not so much. This made sense to us. Higher-order skills, such as constructing an argument, developing an idea with intellectual weight, need more than five weeks and certainly more than AI feedback to internalize. They require thinking, and thinking takes time.

In practice, I see this every day in student writing. Yes, the grammar is cleaner, the vocabulary richer, the style polished. But the language is sterile and the voice uniform, any possible idiosyncrasy ironed out. I struggle to find a thought. This is how I know. And this is why the best AI detectors are AI-proficient humans, not software.

For Nuraiym, Ayauzhan, Zhansulu and Aida, it was their first time recruiting participants, designing an intervention, evaluating student writing, running statistical analyses, and defending their methodological decisions in front of the committee. The team was honest about what the design couldn’t control: threats to internal validity (history effects such as coursework running in parallel, maturation), no control group, small sample size, no follow-up. The project left us with more questions than we came with, perhaps that’s the point of research. But I know that it feels a little less daunting now. Grateful and proud!