An Evaluation of Evaluation

appleIn the spirit of reflection and at the completion of yet another intense course experience, I find that I learned far more than I ever intended. While I can’t say I was giddy at the thought of taking EdTech 505 Evaluation for Educational Technologists, I did recognize that my learning curve would be huge. It was. I entered with not even a working definition of program evaluation and now have the skills to conduct one.

I began the course by creating a Gretel-at-a-glance word cloud explaining who I am and what I hope to gain from the course. My formal experience with program evaluation was nonexistent. My objectives for this course were to learn evaluation techniques that could help me evaluate some of the big picture programs in a school. I feel like I at least have the tools and background I need to begin.

Many of the course assignments were exercises found in the course textbook so I didn’t find the need post them on this learning log. One of the downsides of this course is that it is not really designed to increase my digital footprint.

Program evaluation “enables accountability” (Boulmetis & Dutwin, 2011, p. 38). I enjoyed reading about the various vantage points and considerations that make evaluations meaningful. Everyone, especially in today’s economy, wants to know “what did we get for our money [or time, or effort]? Did it work? Did it do what we hoped it would?” Those are fair and important questions. I appreciate the detailed explanations, both in the module and in the text, of programs, inputs, process, outputs, and outcomes.

Our final project was to conduct a small-scale but real program evaluation. The Explore Nepal program is an extensive school-wide program designed to help students reach out to the Nepali community and gain a deeper connection to their host country. I chose go evaluate the Grade 6 Explore Nepal week-long trip to a local Tibetan monastery to see whether the program objectives laid out for the trip were accomplished. The five objectives include: learning about Nepali culture through community interaction, environmental awareness, service learning, challenging physical activities, and team building. It was a major effort and here is a link to a generic copy of my final report:

Summative Evaluation: Grade 6 Explore Nepal Program

Summative Evaluation: Grade 6 Explore Nepal Program on Google Docs
Summative Evaluation Flipbook

One fun aspect of this course was the option on nearly every assignment to turn in an alternate submission format using some sort of tech tool. Some people created slideshows, videos, infographics, collaborative corkboards, flip magazines, and mind maps. It was fun to see the creativity and it sure was nice to have this option to shake things up. I wish every teacher would do that and allow students to complete an assignment while building their online presence, developing creativity, and taking ownership of their learning.

While I didn’t love everything about this course and would have liked more focus on education and technology, I certainly learned a lot that will serve me well, even if I don’t become a professional evaluator. What a ride it has been!

Other coursework:
Evaluation Design Format
Gap Analysis
Program Cycle
Goal-Based Method, Design, and Type
Top 3 Sites on Data Analysis
Review of Chapters 1-9 in course text
Request for Proposal (fictional)

References:
Boulmetis, J., & Dutwin, P. (2011). The ABCs of evaluation: Timeless techniques for     program and project Managers (3rd ed.). Jossey-Bass.

PBL Reflection and Assessment

I always loved the last night of girls’ camp. We sat around the fire, expressed feelings, supported each other, and reflected on the experiences we had shared together. After an intense week of crafts, hikes, food, skits, and certification skills, it was nice to gain some closure.

The same concepts apply to Project Based Learning. PBL units are intense. They take a great deal of time and energy on everyone’s part. Students invest much of themselves as they dive in and take responsibility for their learning. As a result, students grow in knowledge and collaborative skills. After all of this effort, it is nice to bring some closure to the unit as well as reinforce what the students learned in the process.

Teachers must plan time to debrief and reflect on the entire PBL experience. This involves both teacher and student reviews. Students need to be given an opportunity to discuss and reflect on what they have learned. This will help them realize what they have learned along their journey. Students should also have a chance to fill out a form or survey evaluating the unit. Teachers should take time while the unit is fresh in their minds to make adjustments that will help them the next time.

For Project: Me, I have answered three questions:

  • Who will be involved in the post project assessment process?
  • What will the process look like?
  • Is it just a one-time assessment?

Student Reviews:
For Project: Me, the last day will be spent in celebration. Students will watch the student-produced Day in the Life of Me photo project. They will be given “uniqueness” awards and participate in a class discussion on what they have learned and their observations. They will also complete a survey.

Teacher Review:
The teacher will read over student surveys, reflect on notes taken during the unit, and make any adjustments needed on an evaluation form. This will help them adjust timing, content, scaffolding, and assessment needs for the next time they teach the unit.

While the assessment and reflection process will likely take place during one class period, it is important that the teacher refers back to the project and ties it in to other units whenever possible. The technology tools introduced during the unit can be used in other units. Reminding students what they have already learned is an important way to help solidify their skills.

If done well, students will enjoy the chance to reflect and discuss the PBL unit. They will feel a sense of pride and authentic accomplishment.

Authentic Assessments

As a parent of four, I know that kids don’t always “get” it. Sometimes when I think they are listening they are not, or I think they are not paying attention and they remember specific details. It’s no different in the classroom. Teachers spend a great deal of time trying to help students learn and remember what is being taught, often with varying results.

How do we know they really get it? More importantly, how do we ensure that students will remember such content long-term and apply it in their lives? The answer lies, at least in part, in authentic assessments.

I’ve taken a great many tests in my lifetime. Most of them I performed very well on, especially ones that were detailed-oriented and involved memorization. I studied and crammed my brain with information, hoping to remember it all long enough to score well on the end-of-chapter/unit/year test. I usually did. But now that I have been out of school for years, most of what I learned I have long since forgotten.

Most students will not remember everything they learn in school for the rest of their lives. Does that mean we, as educators, give up and don’t try? Of course not. We try, we use different approaches, and we do our best.

Project Based Learning (PBL) involves students throughout the entire process, which can include having them help determine what is going to be assessed and how they will meet such criteria. The more students have a say in what they need to learn – and how they will demonstrate they have learned it – the more they will remember.

What Kids Can Do gives the following criteria for authentic assessments (which align very well with PBL essential elements):

Assessment is for students.

  • Has personal relevance for students (e.g., provides a tangible product they can use).
  • Students become more confident and articulate about what they know.
  • Students feel ownership over the process as well as the product of their work.

Assessment is faithful to the work students actually do.

  • Notebooks, works-in-progress, and routine presentations are basis of assessment.
  • Occasions for reflection and discussion are integrated into ongoing project work.
  • Students are assessed on what they know and do, not what they don’t.

Assessment is public.

  • Students’ goals are solicited and become part of those assessed.
  • Criteria for judgment remain visible and accessible to students from the beginning.
  • Performances are viewed and judged by a broad group of people.

Assessment promotes ongoing self-reflection and critical inquiry.

  • Teachers and students both speak of the qualities of good work, and how to attain it.
  • Standards used reflect those of adult practitioners in the field.
  • Categories and criteria of assessment remain open-ended, subject to challenge and revision.

If teachers would incorporate these four principles into the assessments they offer, even those not technically PBL, I believe students would perform better and retain more. Many teachers don’t actively involve students in designing the assessment, so this may take a little encouragement and practice. Teachers need to see the benefits of allowing more student input during assessment development and how it is a win/win.

For Project: Me, I have designed both formative and summative assessments that will help students “get it” and demonstrate that they “got it.” I will guide the students through a scaffolded brainstorming activity early on that will help them determine what it is they need to know and how they will demonstrate they have reached the standards. For formative assessments, they will self-evaluate their reflective journals and project work reports weekly, create a visual project organizer using a technology tool, and create a project prototype for evaluation. For summative assessments, students will complete a short essay test on what they have learned about themselves and uniqueness, self-evaluate their completed reflective journals, and self/peer-evaluate their presentation both for content knowledge and 21st Century skills. By using a variety of assessment tools, students will be encouraged to set goals and stay on task, track their progress and growth, and produce a culminating multimedia project. Such strategies will help students find relevancy in what they are doing, be assessed on what they are actually doing, communicate with an outside audience, and produce quality work. In the end, they will remember this project and will have truly learned something valuable about themselves.

Now, how can I implement such strategies at home…?

References:

Buck Institute for Education. (n.d.) What is PBL?. Retrieved from http://www.bie.org/about/what_is_pbl

What Kids Can Do. (n.d.). Retrieved from http://www.whatkidscando.org/

Tech Use Plan Presentation

It’s extremely gratifying to reach the end of my first semester in the EdTech program. I have worked hard to successfully juggle three courses, and I am looking forward to a break. However, I must admit I’m rather sad to end EdTech 501. It has been such an enjoyable course with an excellent instructor and it’s made me excited for the remainder of the EdTech program.

Our final project is the presentation below. For the last few weeks I have been analyzing technology use planning and the components of a well crafted technology plan. For this final presentation, I worked with three group members to create a Google doc presentation that we narrated in Slideshare.

Two AECT standards were met during this assignment:
3.1 Media Utilization: the systematic use of resources for learning
For this assignment I used various resources for collaborative learning and sharing, including Google apps (docs, presentation, survey) and Slideshare.
5.4 Long-Range Planning: focuses on the organization as a whole is strategic planning….Long-range is usually defined as a future period of about three to five years or longer. During strategic planning, managers are trying to decide in the present what must be done to ensure organizational success in the future.
Long-range planning was a key component of this assignment. I was required to proactively think through the next 3-5 years and develop a plan accordingly. Technology committees are tasked with continually assessing the present and planning for the future. This is a crucial role in the overall success of the school or institution.