Evaluation and SME Review

The end is nearing for my semester-long instructional design project, and it is thrilling to see it come together. This week we focus on evaluation. I chose a Subject Matter Expert (SME) to review my materials and offer feedback. The other evaluation types are outside of the scope of this course, but I still had to design what types of questions I would ask in the various evaluation stages. It would be fun to be able to see a project all the way through.

ID Project Description: After three hours of classroom instruction, ninth and tenth grade English Literature students will be able to create, publish, and present a multimedia Google Docs presentation on an assigned poet.

SME Questions:

  • Do you feel this is a valuable and age-appropriate lesson for 9th and 10th grade English Literature students?
  • How valuable do you feel Google Docs is for this age group to learn?
  • Is the stated learning goal clear and attainable in the time allotted?
  • Do you feel you have an adequate picture of the target learners?
  • Do you feel the needs assessment questions provide helpful insights?
  • Are there any other questions you would ask?
  • Are the Learning Task Analysis flow charts clear and logical?
  • Are there any other Entry Behaviors that you would add?
  • Are the learning objectives clear? Realistic? Measurable? Attainable? Are any confusing?
  • Do you feel like the Part 3b Matrix accurately and appropriately assesses the students? Should anything be added or changed?
  • Do you feel that the PowerPoint presentation supports the instruction adequately? Are there any other materials you would include?
  • If you were to teach similar students this lesson, is there anything you would change?
  • Is the instructor guide clear and easy to understand?
  • Do you have any other comments or suggestions?

Expert Review Evaluation Form

One-to-One Evaluation
The purpose of this evaluation is to try out the instructional materials on a small scale before larger-scale implementation. This evaluation is geared towards a few members of the target audience and is used to fix any problems discovered upfront. This type of evaluation gives the designer an opportunity to fix typos, mistakes, unclear directions and vocabulary, and general ambiguity or confusion.

For my project, I intend to include 2-3 English Literature students and their teacher at Heritage Academy in Mesa, AZ. I have designed this project with this group of students in mind and will ask the teacher try it out with a couple of the more willing and bright students. After participating in the instruction’s discussion and presentation materials, some questions they will be asked are:

  • Do you understand the concepts explained in the presentation?
  • Have you learned anything new or did you already know the information presented? Please explain.
  • What is your level of confidence if you were to be tested right now on the information presented?
  • Did you understand what was expected of you during the multimedia quiz? If no, please explain.
  • Are there any graphics, text, or pictures that you did not understand? If yes, please explain.
  • Could you read and understand everything presented to you?
  • What do you think the relevancy, or real-life application, is of the subject matter presented?

Small Group Evaluation
This stage of evaluation takes into account the answers received during the one-to-one evaluation noted above. It involves a slightly larger, more varied group of students without the designer’s intervention. This evaluation notes the how well the instruction holds its own in a group of varying abilities.

For this evaluation, I will use the same school and teacher listed above but involve an entire Honors 9th grade English Literature class. This will allow the teacher to evaluate the instructional materials with a larger number and wider variety of students. Some of the questions the teacher will ask the students and teacher are:

  • Do you have the entry skills listed in the instructional materials? If any are missing, please list.
  • How confident are you that you could begin a Google Doc presentation right now? Please explain.
  • Are there any other skills not discussed that are important to creating and presenting a well-designed presentation? Please explain.
  • Did you feel enough time was given to accomplish what was expected? If not, please explain.
  • Do you feel the information presented is valuable to you personally? Did you enjoy it? Do you feel it is useful? Please explain.
  • What do you think could be improved about this instruction?

Field Trial
This stage of the evaluation takes into account the revisions made from the one-on-one and small-group evaluations. It determines the effectiveness of the revisions and looks at any problems that might arise in a real instructional setting. It also uses a large enough group to make a more accurate prediction of the effectiveness of the materials.

For this project, I will use the same teacher and school, and involve all of the 9th and 10th grade English Literature students. Any problems encountered and revisions made will be taken into account for the next year, and so on, in order to improve each year the instruction is given. If time and resources allow, it would be beneficial to try it out in a similar class in another school. At the conclusion of the instruction, the teacher and students will be asked some questions such as:

  • Can the instruction be used as designed or are revisions needed? Please explain.
  • Did the teacher present the information in a way you could understand? If no, please explain.
  • Do you feel you had enough entry-level skills to accomplish what was expected of you? If not, please explain.
  • Did you have enough time to complete the activities?
  • Did you enjoy the instruction? If not, what could make it more interesting?
  • Do you feel the things you learned have real-life application? Please explain.
  • (For the teacher) How do you feel about the instructional materials? Was it easy to implement as designed or are revisions needed? Did you make any changes or adaptations? Please explain.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s