Reflective Entry 8: Evaluate the outcomes of a digital and collaborative innovation in your practice from an educational research perspective.

Tiven, Fuchs, Bazari, and MacQuarrie (2018), state that there are three evaluation types important to understand what a program is accomplishing and how outcomes can be enhanced and strengthened: 

• needs assessments

• process evaluations

• outcome evaluations

I have considered this evaluation framework against the research I have undertaken and will make reference to the evaluation types throughout. 

Needs assessments

This 33-week journey has led me to the point of identifying and implementing a digital and collaborative innovation in my practice. This ultimately led me to explore the question:

If I use collaborative digital tools, how will it affect critical thinking (knowledge construction) and collaboration in my Year 1/2 priority learners? Key outcomes for me to evaluate from an educational research perspective concern whether there have been improvements in critical thinking (knowledge construction) and collaboration in my Year 1/2 priority learners as a result of them using collaborative digital tools.

Process evaluations

During the implementation of this innovation, there were a number of challenges that resulted in changes needed to be made to the implementation plan and for the innovation to pivot slightly, whilst still maintaining the integrity and working towards the intended outcomes. 

The student community within the classroom is made up of 14 students - 7 girls and 7 boys. I have identified 4 priority learners that I proposed to be the students that will most benefit from this innovation. I had planned for the implementation of my innovation to be staggered. This was based on my research into the Diffusion of Innovations Theory (Rodgers, 1962) and how innovations are communicated over time amongst participants in a social system. With knowledge of this I had identified 2 students within the classroom as early adopters to first trial the innovation with. It was then intended for these students to support other students by modelling their learning and collaboration on iPads and Explain Everything for the benefit of most of the remaining students - the early majority. Moore (1999), would define this stage of the innovation as “Crossing the Chasm” whereby the innovation could potentially fail due to its inability to reach the early majority. In reality, the innovation was unable to be implemented in a staggered way as a result of the classroom contact time that I had with the class, only working part-time. Instead, all students were exposed to the innovation at once and there was a surprisingly large number of students immediately feeling comfortable with this new way of learning. The students modelled a generally good level of the appropriate use of iPads and CyberSmart behaviours.

Outcome evaluations

The Collaboration and Knowledge Construction (Critical Thinking) 21st Century Learning Design Rubrics (ITL Research, 2012) were used three times throughout the implementation plan to measure changes in students against the rubric criteria. The data is set out on the two tables below:

Collaboration Rubric Score	Baseline	Midpoint	Summative
1
2	12
3	2	12	1
4		2	13
5

Knowledge Construction Rubric Score	Baseline	Midpoint	Summative
1
2	14
3		14	1
4			13
5

The first time the 21st Century Learning Design Rubrics (ITL Research, 2012) were implemented the findings were low scores for both collaboration and knowledge construction. The tasks the students were collaborating on required them to work together but the majority did not have shared responsibility. The students were also constructing knowledge, but that was not the main purpose of the tasks. 

The second time the 21st Century Learning Design Rubrics (ITL Research, 2012) were implemented the findings resulted in a shift for all students moving up a score in both collaboration and knowledge construction. Students had shared responsibility for the task but a majority were not required to make substantive decisions together during collaboration, however 2 students were. For knowledge construction all learners scored a 3 which meant the learning activity’s main requirement was knowledge construction but the learning activity did not require students to apply their knowledge in a new context. These findings informed changes to my implementation plan to see if I could elevate collaboration and knowledge construction further during the remainder of the innovation. 

The third and final time the 21st Century Learning Design Rubrics (ITL Research, 2012) were implemented, at the conclusion of the innovation, there had been significant shifts in the collaboration and knowledge construction rubric scores when compared against the start of the innovation. Most students, including all of my target students, were able to collaborate by having shared responsibility, and by making substantive decisions together about the content, process, or product of their work. In regards to knowledge construction, a majority of the students, including all of my identified target students reached a rubric score of 4. This means that the learning activity’s main requirement that they were engaged with was knowledge construction and it required students to apply their knowledge in a new context. Falloon (2015), suggests the mobility of an iPad supports effective collaboration by allowing students to work together in different spaces and places. However, simply having access to and using iPads didn't guarantee collaboration and critical thinking amongst students. The use of iPads and Explain Everything had to be embedded in learning that was purposely designed to develop collaboration and critical thinking skills (Falloon, 2015) and I believe the positive outcomes achieved in this innovation are as a result of that.

Student and Whānau Voice

A GoogleForm was administered to gather student and whānau voice in relation to collaboration and knowledge construction. It was clear from the initial form conducted part way through the innovation that students thought they were thinking critically and collaborating at a high level. This made me realise the need to further unpact these concepts with my learners and roleplay higher levels of collaboration and critical thinking. This change supported the findings of the final GoogleForm which indicated that as well as developing critical thinking and collaboration skills learners had further developed the Key Competencies (Ministry of Education, 2007) of ‘relating to others’ and ‘thinking’. 

As evidenced in the GoogleForm whānau have a much greater understanding of what critical thinking (knowledge construction) is and how they can support that both at home and school. This innovation changed the viewpoint of our whānau and further engaged them as an important learning stakeholder.

Future Focus 

Whilst there are many outcomes “specific changes in attitudes, behaviors, knowledge, skills, status” (W.K. Kellogg Foundation, 1998, p.8.) which changed as a result of my innovation there is still room to further develop the collaborative and knowledge constructing skills and dispositions in my class, as evidenced by qualitative and quantitative data gathered. In 2021, the next steps will be to provide for collaborative tasks that ensure learning is interdependent and the learning activities have learning goals in more than one subject. 

Conclusion 

Ultimately, despite its changes and the peculiarities of 2020, the innovation was a success as evidenced by whānau and student feedback and data. Therefore, it can be stated that the use of collaborative digital tools has positively affected critical thinking (knowledge construction) and collaboration amongst my Year 1/2 priority learners.

References

ITL Research. (2012). 21CLD Learning Activity Rubrics. Retrieved from https://www.academia.edu/34212053/21CLD_Learning_Activity_Rubrics

Ministry of Education. (2007). The New Zealand Curriculum. Wellington: Learning Media. Ministry of Education. 

Moore, G. (1999). Crossing the Chasm, Marketing and Selling High-Tech Products to Mainstream Customers. New York: HarperCollins.

Rogers, E. (1962). Diffusion of Innovations. Free Press of Glencoe.

Tiven, M. E., Fuchs, E., Bazari, A., & MacQuarrie, A. (2018). Evaluating global digital education: Student outcomes framework. New York, NY: Bloomberg Philanthropies and the Organisation for Economic Co-operation and Development. Retrieved from https://www.oecd.org/pisa/Evaluating-Global-Digital-Education-Student-Outcomes-Framework.pdf

W.K. Kellogg Foundation. (1998). Logic Model Development Guide. Retrieved from https://www.wkkf.org/resource-directory/resources/2004/01/logic-model-development-guide