An Evaluation of Evaluation

appleIn the spirit of reflection and at the completion of yet another intense course experience, I find that I learned far more than I ever intended. While I can’t say I was giddy at the thought of taking EdTech 505 Evaluation for Educational Technologists, I did recognize that my learning curve would be huge. It was. I entered with not even a working definition of program evaluation and now have the skills to conduct one.

I began the course by creating a Gretel-at-a-glance word cloud explaining who I am and what I hope to gain from the course. My formal experience with program evaluation was nonexistent. My objectives for this course were to learn evaluation techniques that could help me evaluate some of the big picture programs in a school. I feel like I at least have the tools and background I need to begin.

Many of the course assignments were exercises found in the course textbook so I didn’t find the need post them on this learning log. One of the downsides of this course is that it is not really designed to increase my digital footprint.

Program evaluation “enables accountability” (Boulmetis & Dutwin, 2011, p. 38). I enjoyed reading about the various vantage points and considerations that make evaluations meaningful. Everyone, especially in today’s economy, wants to know “what did we get for our money [or time, or effort]? Did it work? Did it do what we hoped it would?” Those are fair and important questions. I appreciate the detailed explanations, both in the module and in the text, of programs, inputs, process, outputs, and outcomes.

Our final project was to conduct a small-scale but real program evaluation. The Explore Nepal program is an extensive school-wide program designed to help students reach out to the Nepali community and gain a deeper connection to their host country. I chose go evaluate the Grade 6 Explore Nepal week-long trip to a local Tibetan monastery to see whether the program objectives laid out for the trip were accomplished. The five objectives include: learning about Nepali culture through community interaction, environmental awareness, service learning, challenging physical activities, and team building. It was a major effort and here is a link to a generic copy of my final report:

Summative Evaluation: Grade 6 Explore Nepal Program

Summative Evaluation: Grade 6 Explore Nepal Program on Google Docs
Summative Evaluation Flipbook

One fun aspect of this course was the option on nearly every assignment to turn in an alternate submission format using some sort of tech tool. Some people created slideshows, videos, infographics, collaborative corkboards, flip magazines, and mind maps. It was fun to see the creativity and it sure was nice to have this option to shake things up. I wish every teacher would do that and allow students to complete an assignment while building their online presence, developing creativity, and taking ownership of their learning.

While I didn’t love everything about this course and would have liked more focus on education and technology, I certainly learned a lot that will serve me well, even if I don’t become a professional evaluator. What a ride it has been!

Other coursework:
Evaluation Design Format
Gap Analysis
Program Cycle
Goal-Based Method, Design, and Type
Top 3 Sites on Data Analysis
Review of Chapters 1-9 in course text
Request for Proposal (fictional)

References:
Boulmetis, J., & Dutwin, P. (2011). The ABCs of evaluation: Timeless techniques for     program and project Managers (3rd ed.). Jossey-Bass.

Periodic Table of Connectivism

My EdTech543 assignment this week certainly stretched my creativity, but since I’m a wanna-be chemist at heart, I had to try it. The assignment was to non-linguistically represent the dense concepts of connectivism, personal learning networks, and communities of practice. Here is my attempt:

References

Authentic Assessments

As a parent of four, I know that kids don’t always “get” it. Sometimes when I think they are listening they are not, or I think they are not paying attention and they remember specific details. It’s no different in the classroom. Teachers spend a great deal of time trying to help students learn and remember what is being taught, often with varying results.

How do we know they really get it? More importantly, how do we ensure that students will remember such content long-term and apply it in their lives? The answer lies, at least in part, in authentic assessments.

I’ve taken a great many tests in my lifetime. Most of them I performed very well on, especially ones that were detailed-oriented and involved memorization. I studied and crammed my brain with information, hoping to remember it all long enough to score well on the end-of-chapter/unit/year test. I usually did. But now that I have been out of school for years, most of what I learned I have long since forgotten.

Most students will not remember everything they learn in school for the rest of their lives. Does that mean we, as educators, give up and don’t try? Of course not. We try, we use different approaches, and we do our best.

Project Based Learning (PBL) involves students throughout the entire process, which can include having them help determine what is going to be assessed and how they will meet such criteria. The more students have a say in what they need to learn – and how they will demonstrate they have learned it – the more they will remember.

What Kids Can Do gives the following criteria for authentic assessments (which align very well with PBL essential elements):

Assessment is for students.

  • Has personal relevance for students (e.g., provides a tangible product they can use).
  • Students become more confident and articulate about what they know.
  • Students feel ownership over the process as well as the product of their work.

Assessment is faithful to the work students actually do.

  • Notebooks, works-in-progress, and routine presentations are basis of assessment.
  • Occasions for reflection and discussion are integrated into ongoing project work.
  • Students are assessed on what they know and do, not what they don’t.

Assessment is public.

  • Students’ goals are solicited and become part of those assessed.
  • Criteria for judgment remain visible and accessible to students from the beginning.
  • Performances are viewed and judged by a broad group of people.

Assessment promotes ongoing self-reflection and critical inquiry.

  • Teachers and students both speak of the qualities of good work, and how to attain it.
  • Standards used reflect those of adult practitioners in the field.
  • Categories and criteria of assessment remain open-ended, subject to challenge and revision.

If teachers would incorporate these four principles into the assessments they offer, even those not technically PBL, I believe students would perform better and retain more. Many teachers don’t actively involve students in designing the assessment, so this may take a little encouragement and practice. Teachers need to see the benefits of allowing more student input during assessment development and how it is a win/win.

For Project: Me, I have designed both formative and summative assessments that will help students “get it” and demonstrate that they “got it.” I will guide the students through a scaffolded brainstorming activity early on that will help them determine what it is they need to know and how they will demonstrate they have reached the standards. For formative assessments, they will self-evaluate their reflective journals and project work reports weekly, create a visual project organizer using a technology tool, and create a project prototype for evaluation. For summative assessments, students will complete a short essay test on what they have learned about themselves and uniqueness, self-evaluate their completed reflective journals, and self/peer-evaluate their presentation both for content knowledge and 21st Century skills. By using a variety of assessment tools, students will be encouraged to set goals and stay on task, track their progress and growth, and produce a culminating multimedia project. Such strategies will help students find relevancy in what they are doing, be assessed on what they are actually doing, communicate with an outside audience, and produce quality work. In the end, they will remember this project and will have truly learned something valuable about themselves.

Now, how can I implement such strategies at home…?

References:

Buck Institute for Education. (n.d.) What is PBL?. Retrieved from http://www.bie.org/about/what_is_pbl

What Kids Can Do. (n.d.). Retrieved from http://www.whatkidscando.org/

Instructional Design Final Project

Wow.

When I started EdTech 503 I really had no idea what was in store. I do now. While I feel somewhat overwhelmed with all I have learned I also feel empowered. I certainly need and want to learn more.

My project for this course was to design three-hour instruction to help 8th and 9th grade students learn to use Google Docs presentations. It is geared for English Literature students who need to do a class presentation on a poet. They will not only gain content area knowledge but learn valuable 21st Century digital skills that they can apply in various real-world scenarios. It’s been fun but daunting.

Here is my 35-page project. There are many supporting documents linked throughout. It has been a grueling process but certainly a learning one.

Instructor’s Guide and ARCS Table

Piece by piece, my instructional design project is coming together. This week I have thought through what the actual instruction will look like. What materials will the instructor use? How will the lessons be organized? How will the instructor motivate the students? I enjoyed this assignment because it was less theoretical and more practical. There are two different models we used and I post them both here. Each image links to Google Docs:

Instructional Design Slideshow Overview

Instructional Design Slideshow

This assignment is actually a reading quiz on several chapters and articles. I am posting it here because I feel this is a good example of how a teacher can assess in a creative way that enables students to stretch in ways much broader than a standard assessment might do. I created a Google Docs presentation that included 21 slides with specific topics on each one. It was called a “post card” quiz because one slide would be a photo, and the next slide (like flipping over the post card) would include the content from our readings. To take it even a step further and to test my deeper understanding, the photo had to be a metaphor of the topic, not a 1:1 correlation. So, for example, when talking about systematic models, I showed a picture of a haphazardly wired telephone pole here in Nepal that demonstrated lack of order or planning.

I like Google Docs, even though the applications are rather basic. It provides students with an open-source option that can be accessed or edited from anywhere, while still providing sufficient practice with a software-specific platform. Since I like more design options than what is offered, I imported slides I created in Keynote for more visual appeal. The result is a product that took a long time but truly tested my reading knowledge in a much more interesting – and lasting – way than a standard written assessment would have.

Brain Fitness Grant Proposal

Here is my completed grant proposal for EdTech 551: Grant Writing as it currently stands. Now I just need to get it funded!

This grant seeks funding for a 5th grade class to obtain activity monitors to use in brain conditioning exercises prior to reading and math. The idea builds upon compelling research indicating a direct correlation between increased heart rate (and therefore more oxygen and blood pumping to the brain) and cognitive performance (the brain’s ability to function and learn). This effort strives to improve standardized test scores in English Language Arts and mathematics through increased cardiovascular activity. This grant would adopting similar fitness practices used by schools across the nation who have seen great success through their innovation and creativity.

As part of the course I also designed a simple Weebly website for my mom so that she has a place where parents, administrators, and potential funders can get more information on the grant. It is also a place where progress will also be recorded. While I am confident I could design my own site using my EdTech 502 skills, I chose to keep things easy for her sake.

Many AECT standards were met during this course. I have summarized these as follows:

2.1 Print Technologies: ways to produce or deliver materials, such as books and static visual materials, primarily through mechanical or photographic printing processes.
We used a variety of print technologies in this course, mostly through MS Word and Google docs. The aesthetic appeal of a document is important to me and I think a well written grant also needs to look good on a page.

2.2 Audiovisual Technologies: ways to produce or deliver materials by using mechanical devices or electronic machines to present auditory and visual messages.
A/V technologies are used for professional production the grant proposal and query letter. Copy machines and printers are used for this task. We also had to read and evaluate the writing and design of many external websites and critique specifics of what made them great or terrible.

2.3 Computer-Based Technologies Computer-based technologies are ways to produce or deliver materials using microprocessor-based resources.
This standard was met by devising a website to portray our grant proposal visually. The intent is not to publish our entire proposal but to glean the most important aspects that would be of interest to parents, administrators, and potential funders.

3.1 Media Utilization: the systematic use of resources for learning.
This standard was met throughout the course as I oversaw the development of my grant proposal. I began by assessing the need and determining a potential solution. I did online readings and research, collected supporting data, identified appropriate technologies and associated costs, and thought through specific plans to implement the project.

3.4 Policies and Regulations: the rules and actions of society (or its surrogates) that affect the diffusion and use of Instructional Technology.
In my writing and website creation, I adhered to copyright and fair use laws.I examined my writing language to make sure it was fair and unbiased and would appeal to a broad spectrum.

4.2 Resource Management: involves planning, monitoring, and controlling resource support systems and services.
The grant writing process is complex and requires a significant amount of time. Such an exercise requires a thorough examination of the project from beginning to end, with all of it’s costs, rationale, support, arguments, and practicalities addressed.

4.4 Information Management: involves planning, monitoring, and controlling the storage, transfer, or processing of information in order to provide resources for learning.
Serious organizational skills are needed in order to seek and obtain a grant. Such values as  planning, monitoring, and controlling are essential. The grant is based on the careful assumption the once the grant is funded, the educator will be responsible for careful accounting of the program. This is something that has forced careful consideration to details throughout the semester.

5.3 Formative and Summative Evaluation: involves gathering information on adequacy and using this information as a basis for further development. Summative evaluation involves gathering information on adequacy and using this information to make decisions about utilization.
For the grant I have chosen to pursue, there was a significant amount of background research and evaluation necessary to see where the students are at, what can help them, what other schools with similar schools are doing, and so forth. I’ve also had to research the various technologies available, their pros and cons, in meeting the goals outlined.

School Evaluation Summary

It’s easy to criticize schools, especially when it comes to the use of (or not enough use of) technology. Parents do it, teachers do it, even students do it. If you have never had to really evaluate your school based on a number of factors, you might want to look closer before voicing your opinion. It’s complicated.

This week in EdTech 501 I surveyed our school’s technology environment. As I became more familiar with the Technology Maturity Model and the Stages of Technology, I realized this was no easy task.

The Technology Maturity Benchmarks and Stages of Technology “link the technology resources to their use in every learning environment. In this way, the existing level of support for students, teachers and support staff can be determined. The premise is founded on the theory that both resource availability and behavioral changes are required to improve educational outcomes.”

The five organizational filters are: administrative, curricular, support, connectivity and innovation. Each of these levels of the organization are addressed with both behavioral and resource/infrastructure criteria.

I began by creating a Google survey form and sent it to a small mix of administrators, teachers, and technology committee members at my children’s school. I received enough responses to give me a good starting point. The results (using a pseudonym) are noted here:

 

As I analyzed these benchmarks, I realized that it takes a village to improve the technology environment in a school. There’s no easy solution, no magic device or gadget. Results are measured on a spectrum and continually adjust. Many intricate factors work together to create the overall picture. A detailed summary of the results are found here:

 

For this assignment, I polished my project management and collaboration skills, information-gathering and decision-making strategies, and formative evaluation technique. I think AECT would be pleased.