Sunday, October 27, 2013

So, do iPads really improve student learning?

Flickingerbrad via Flickr
For three semesters, from January 2012 to June 2013, I collected student data. I surveyed my high school students weekly or biweekly and kept a record of their responses. I analyzed midterm and final exam scores, both cumulative and by skill (speaking, writing, reading, listening, grammar). I compared yearlong averages for students in both the iPad class and the traditional class. Do you know what I found? 

Integrating iPads has almost no effect on student performance.

I have to give you some caveats before I go too deeply into the numbers and my analysis. 

  • First of all, I'm not an expert. I don't have any formal training in data collection or statistical analysis. I am self-taught. I also enlisted a science teacher to help me interpret the data.
  • Secondly, in both school years, the iPad class and the traditional class were very different. These are not apples-to-apples comparisons, because every class is made up of unique individuals with varying backgrounds, skills and abilities.
  • The curriculum varied from year to year as I responded to students' needs, learned from my own mistakes, and adjusted course when necessary. What students in the second semester 2013 class did was different from what students in the second semester 2012 class did.
  • The major assessments (midterm and final exams) are traditional tests. The speaking component is a recorded conversation on a curriculum-driven topic; the writing component is an expository essay on the same topic; and the listening, reading and grammar components have a multiple-choice format. 
  • The data sets are small. The 2012 groups had 17 (traditional) and 24 (iPad) students, and the 2013 groups had 24 and 22 students, respectively.
This is what I found:

© Katherine R. Matheson. All Rights Reserved.

My question was "Do iPads enhance language proficiency?" I decided to define "proficiency" as competence and confidence in a foreign language. Were my students able to communicate successfully? Competence. Did they recognize that they were able to do so? Confidence. I measured competence by using their raw scores on their midterm and final exams, as well as their yearlong averages. I measured confidence by surveying their attitudes about their Spanish skills. These were the questions I asked:


© Katherine R. Matheson. All Rights Reserved.
© Katherine R. Matheson. All Rights Reserved.
© Katherine R. Matheson. All Rights Reserved.
As you can see from the numbers, there is little difference in achievement between the iPad class and the traditional class. (Students in both classes met or exceeded standards on the speaking, writing, listening and reading parts of the major exams, and both sets of students bombed the grammar accuracy component.) The year-long average for both classes is also very similar.

But the real difference--in my opinion, the gains--can be found in the 21st century skills aspect of the study. The iPads allowed for many more experiences in which students could think critically, create, develop media and information literacies, communicate with others, and interact with authentic resources in Spanish. The available tools to measure their growth in these areas were traditional: paper tests, letter and number grades. And those traditional tools don't explicitly assess those "soft" skills. It's worth pointing out also that my "traditional" classes still used computers and cell phones to do some of the same things that the iPad class did; however, not all students had smartphones, and leaving class to go to the computer lab was often disruptive. In a 1:1 iPad classroom, I could count on students having the same access to the same technology, and we could integrate that technology seamlessly into the lesson and then put it away when we didn't need it.

What am I taking away from this experience? I've learned an incredible amount about pedagogy, educational technology and my own strengths and weaknesses as an educator. Most importantly, I've discovered that students develop real-world skills more readily when they use personalized mobile technology to engage in meaningful and authentic learning experiences. 

Do you agree with my analysis? What other conclusions can we draw from this (albeit imperfect) data?

10 comments:

  1. It's problematic from the start when we define "student learning" as test scores. As you discovered there was lots of learning, they just didn't show up on tests. That in itself is an issue. We have to stop defining learning and achievement as meaning test scores. It's inaccurate and perpetuates the lowest level of learning.

    ReplyDelete
    Replies
    1. I absolutely agree with you, shareski. Thank you for reading and commenting.

      Delete
  2. I totally agree. The final quest(ion) will then be about recognizing these "soft skills" in a systematic way. In reality, this reflects the difference between qualitative and quantitative research - in favour of the qualitative. In the present paradigm, qualitative research is in need of a facelift to regain trust in pedagogical contexts.

    Anyone who knows about existing tools, systems and/or descriptions for targeting and recognizing "soft skills"? It would be nice to see an alternative to the qualitative interview or questionnaire.

    ReplyDelete
    Replies
    1. Bjarne, this is definitely one of the Big Questions that education has to address right now. I'm working on using Evernote to compile student portfolios. I'll be posting on that soon. Thank you for reading and commenting.

      Delete
  3. There are a couple other things to keep in mind, I think, when looking at your results. One is the issue of student engagement. Was there any way to measure that? I am curious to see if you noticed students more engaged in their learning (both in terms of the content and the actual process of their own education) using the 1:1 structure versus the traditional model. I have found that in the earliest years of using more technology like this in the class, students were interested in the tech but still had the traditional model in mind when they thought about what they were trying to achieve (i.e. good grades) and so it impeded their learning. As they have become more used to this kind of learning, it is more natural to them and they are less engaged in traditional model classrooms.

    The second thing to remember with this is that the students in the iPad classes did not significantly lower their scores on the traditional assessments. This is important because it shows that they learned the same things the students in the traditional classroom learned while also learning many more things (viz. 21st century skills, as you mention). So, although they did not achieve higher than the others on the assessments, they managed to learn more...all the content of the traditional class plus all of the additional 21st century stuff.
    Keep up the good work. Thanks for sharing!

    ReplyDelete
    Replies
    1. Mark, thanks for your thoughtful comments. I can always count on you to see things from another perspective. My observations align with yours--I found most students to still be overly concerned with grades; I don't know if any of them ever really transcended that mindset, especially those in the second group. The first (2012) group seemed more open-minded to me. But maybe I was pushing them more.

      As for your interpretation of the test results--I love it! I had not been thinking of it that way, but now I definitely will. You're absolutely right.

      Thanks for reading! Your comments gave me a boost today.

      Delete
  4. Hi Catherine, Its a wonderful analysis. I just shared it. Could you please write a post for my blog (www.Blog.Qirtas.Org).

    ReplyDelete
    Replies
    1. Hi--Could you DM me on Twitter for more info? @katchiringa

      Delete
  5. This is very topical in my school at the moment Katherine and I have shared your research with our senior leadership team. I am an avid user and advocate of using technology to enhance the learning of my pupils. As with any resource it is ensuring its fitness for purpose that can also affect progress in learning. The thing I notice is the shift in pupil reaction to technology. No longer are they seduced to distraction by its novelty but more likely to have a tacit expectation of its functionality and a desire to use it creatively. Keep up the excellent work. Julian

    ReplyDelete
    Replies
    1. Julian, thanks for your comments! I hope my informal "research" can help inform technology implementation and practice at your school. Best of luck to you!

      Delete