Last week I was in France to meet research colleagues in entrepreneurial education from around Europe at the yearly 3E conference. One of the hot topics was assessing students through reflections. Of the 52 research papers presented, 39 touched upon reflection or assessment one way or another. Naturally, I spent the week reflecting much around how to assess students summatively or formatively through reflective assignments. What better way to end this week than to write down some of my reflections here on my blog?
I will first share some aspects of what was presented at the conference. Then I will give some of my own reflections based on a decade of working with reflective assessment with my own students and with apprenticeship educators around Sweden. These reflections are structured around a ‘stairway model’ of progression in how to assess students in value creation pedagogy.
What did scholars bring up in their papers?
Let’s first briefly summarize some key things written in the papers presented at 3E. I won’t share all the 52 papers here, but if any of the phrases below triggers your curiosity, send me an email and I will share that paper with you. I’ve not read them all, but I did a quick PDF search around reflection and assessment. Some illustrative phrases were:
- “…combining experiential, vicarious and reflective learning” (Aadland et al.)
- “…writing reflective essays” (Farrokhnia et al.)
- “In reflective coaching, the coach aims to trigger inner development” (Gabrielsson et al.)
- “Reflection setting: Weekly reflection logs” (Gössel)
- “…requires students to become reflective, critically aware” (Higgins et al)
- “…through reflective practice [students] can increase their understanding of their own weaknesses” (Lynch et al)
- “…encouraging reflective learning through a learning-by-doing approach (Martina et al)
- “…four interconnected stages: active experimentation, concrete experience, reflective observation, and abstract conceptualization” (Politis et al.)
- “The reflective educator must be prepared to re-design their teaching” (Robinson & Shumar)
- “…we ask students to write reflective journals” (Solbreux et al.)
- “Reflective Essays on what learning students gained” (Somià)
Reflection is a job for students, it seems. Only one paper treated the teachers’ own reflections. Some papers see reflection as something that happens implicitly as an effect of learning-by-doing, wheras others explicitly ask students to write weekly logs/journals or post-action reflective essays.
The session with Prof. Britta M. Gössel at Eberswalde University for Sustainable Development
The paper with the strongest focus on student reflections was written by Britta Gössel, a well-known and much appreciated scholar in our field. Naturally, I had to attend that one. And it was packed! Britta had planned to include an interactive workshop. But that was now impossible! So she first presented her paper that treats the follwing question: How can the development of key competencies in sustainability and entrepreneurship become visible through reflection logs? Then we discussed this for half an hour in plenary.
The engagement from people was substantial. It was obvious how much the topic engaged people. How to assess students through written reflections? What tools and methods can be used? What theories can underpin it all? How to make student reflections interesting and deep, instead of boring and annoying for both students and teachers? How can teachers make time for giving students feedback? And how to analyze the textual data?
In her article, Britta wrote about how she had used the university’s learning platform to collect student reflections. Students were asked to reflect weekly for around 15 weeks, and then to do a meta-reflection in the end of the semester. Afterwards, Britta had made word-clouds in an attempt to grasp the text and analyze which competencies students had developed. Here is a word cloud around entrepreneurial attitudes:
An a-ha moment for me – the value of apprenticeship education for entrepreneurial education
From the discussion it was apparent that most participants struggled with getting reflective assessment to work well in practice. How to vary the questions students reflect upon? How to collect the reflections? How to give good feedback? How to treat the textual data in terms of analysis? Some participants shared their experiences. Britta listened attentively. This was really a hot topic for the 3E community. Some of my closer colleagues remarked to the audience that I might be able to give some answers to Britta, since they know that I’ve worked extensively with digital student reflection.
There and then I realized something. My research on apprenticeship education could actually be quite useful for the 3E community. In parallell to my work with entrepreneurial education, I have spent the last 8 years working intensively with apprenticeship educators in Sweden. We have developed a digital tool for reflective assessment that is widely used by around 20.000 people in Sweden. I think we now have some 3.000 teachers and 17.000 students on secondary education level working with us specifically with reflective assessment. Last year, I summarized the learnings around assessment into a stairway model that I’ve written about in Swedish here and here. I mentioned the model in the plenary, and it triggered a lot of interest. So I thought I’d share a translated version of it here.
The stairway model of how to assess value creation pedagogy
I do a lot of research on ‘value creation pedagogy’ – letting students learn through creating value for others. The most extreme form of value creation pedagogy is apprenticeship education, where students spend 50% of their time at a workplace. Their teachers face some extraordinary demands on their assessment regimes. Therefore, they need to have a rather different assessment strategy. The teacher’s need for structure and control over the learning process needs to be combined with a large space for students to improvise and be creative in value creation. I therefore liken it all to jazz. If there is any phenomenon where many people together succeed in combining structure with creative improvisation, it is jazz. Value creation pedagogy in its most advanced form is like big band jazz, where teachers take the role of jazz orchestrator and distribute the initiative to students based on different pre-determined themes and “chords”.
Having successfully helped many apprenticeship educators around Sweden to manage their assessment in digital ways, we developed the stairway model to explain what we’ve seen. The stairway contains six steps, illustrating progression in assessment work through an increasing level of sophistication for each level up in the stairway. I will briefly go through the six levels below. To the right in the figure below, I relate to the jazz metaphor.
Level 1: Reflection
The most basic assessment strategy is to let students reflect in a digital logbook. It can be compared to loose jazz phrases by occasional jazz musicians. Free reflection gives a lot of room for improvisation but is not very structured or coordinated. Such an assessment strategy is easy to get started with but does not support the assessment of more sophisticated learning journeys. It also takes a lot of time for teachers to interpret large amounts of unstructured text when it is time to grade.
Level 2: Portfolio thinking
Assessing students’ competence by examining their creations has a long tradition in aesthetic-practical subjects and is called portfolio assessment. The creations are often linked to experiences and abilities through reflection, so-called portfolio thinking. This makes students active co-creators of formative assessment rather than passive recipients of grades. Combinations of text, image and video provide good opportunities for such assessment in digital form. We can see this strategy as a kind of pre-recorded solo jazz that is sent to the teacher. It allows for a great breadth in performance and also a clear structure for teachers. However, it does not provide as good support in what students need to do in order to learn how to create different works.
Level 3: Activity-based assessment
At the third level, the learning journey is more clearly described through series of action-oriented assignments. Here we begin to see the benefits of structure in combination with improvisation. With a set of different assignments, relatively sophisticated learning journeys can be textually described in action form. This is the first step where we see greater time savings in the assessment work. Here, the teacher follows students as a kind of jazz conductor with a pre-planned arrangement for the whole class. Each action-oriented assignment is a kind of chord the student can improvise to in the outside world and then reflect upon with a combination of text, tags, image and video. The chords are put together in arrangements (content packages) that students are expected to improvise upon during longer time periods, often a course, a semester or an entire year.
Level 4: Three-party collaboration
In the fourth step, a key person is added outside the school and becomes part of the digital assessment work. This step is common in vocational training with supervisors in the workplace who also read students’ reflections. Just like in jazz, the audience here gets an active role to play by giving inspiring feedback, what I here call assessment for motivation (AfM). Achieving a time-efficient tripartite collaboration in everyday life has proven to be almost impossible without digital support tailored to the purpose.
Level 5: Community of practice
In the fifth step, the teachers begin to exchange content packages with each other. This facilitates the dissemination and testing of more sophisticated learning journeys. It is more difficult to compose a unique learning journey yourself than to orchestrate a class based on an existing learning journey that someone else has composed. Therefore, the teachers benefit greatly from being able to share activity-based content with each other. Together with vocational teachers, we have been working with content packages since 2019, a way of working that has quickly become widespread. Today, there are about eighty different content packages developed for all national vocational programs. I guess that within a few years we could hope to see a spread of different content packages also for entrepreneurial education.
Level 6: The scientific teacher
The most sophisticated approach we have seen is when teachers not only exchange content packages with each other, but also analyze all students’ reflections and recordings collected with the digital reflection tool with scientific analysis. The purpose is to see which different activities give which effects on students’ learning. I’ve written an entire book about this approach, but it is in Swedish. It’s called ‘The Scientific Teacher’.
What next for the entrepreneurial education community?
I’ve experimented with digital reflective assessment for a decade now, both in my own teaching and with apprenticeship educators. But it has been a challenge to get entrepreneurial education scholars to join this intriguing work. A few early pioneers have joined – Mats Westerberg in Luleå, Sarah Robinson in Århus, Philip Clegg in the UK and of course my colleagues at Chalmers School of Entrepreneurship. But the large majority has not yet formed a “collective entrepreneurial intelligence” around this topic. Maybe the 3E conference in France can be a new starting point here?
Let me know if you want to work with me on action-reflective challenges!
If you email me, I can share with you some intriguing results from my latest cohort of entrepreneurship students. To trigger your curiosity, have a look at the figure below! It summarizes the quantitative self-coding of 350 reflections from my students having conducted 30 different action-reflection challenges centered around interaction with something I call “S-persons”. An S-person is defined as:
“A significant stakeholder relevant to your project but NOT part of your project or emotional owner to your project (i.e. NOT your idea partner, other close partner, funder, sponsor, internal coach, corporate coach, teammate, etc)”
These 30 challenges can be found in a content package I’ve made available here. You can easily try them out at your own program/course, and afterwards we can compare the data sets in a scientific way. The statistics shown in the figure below are rather intriguing, I think. But as interesting as statistics can be, it is in the qualitative reflections that the most interesting stuff resides. This year, over a period of 8 months, I received around 90.000 words of emotionally strong reflections. It’s around one book. So if you read one book a year or more, you will have time to read your students’ reflections too. And the students loved to reflect in this way! One student wrote to me:
“The module was a perfect way of thinking in new and more innovative approaches to reaching S-persons. The way it has been designed is almost like a video game where you are challenged to complete a specific set of tasks. Unfortunately, what we unlock by completing these tasks is not food, money, or tangible assets but rather invaluable knowledge and experience that might be taken for granted or overlooked.“
That heartwarming quote tells me that we might be onto something important here.