Tag Archives: evaluation

Other Ways to Evaluate

This year, I decided I wanted more than numbers to evaluate my videoconference program. So I created two surveys, one for teachers who participated in a videoconference, and another for teachers who did not participate in a videoconference. So far, I have 54 responses and am hoping to get more before the end of the school year.

Here is what I asked my teachers:

If they participated in a videoconference this year:

  • What benefits do you see to your students in using videoconferencing?
  • Which videoconferences do you want to do again next year?
  • Which videoconference(s) were NOT good and not worth your time?
  • Are there any other topics that you wish there was a videoconference for?
  • Any other comments?

If they did NOT participate in a videoconference this year:

  • Have you ever done a videoconference in the past?
  • What videoconference(s) do you wish you could participate in next year?
  • What do you need to be able to participate in a videoconference next year?
  • Any other comments?

I also collected some demographic data: grade level, subject area, name & district.

So far I am getting interesting feedback, and hope that it will give me good data to plan programs and events for next year…

What questions are you asking YOUR teachers?

Evaluation: Comparing Against National Data

Yesterday we talked about comparing our end of year report numbers to last year’s data and between schools and districts.

What about comparing your data to national data?

Graph by nDevilTV

In the spring of 2008, many of you contributed to my dissertation study focusing on videoconference coordinators and the use of videoconference in K12 schools.

When you look at your end of year data, it might be useful to compare it to the data from this study.

Three measures

There are three ways to measure your videoconferences to compare against this data:

  • Total videoconference events (professional development, meetings, and student events, excluding daily courses) divided by the number of students times 100 to get a whole number.
    (Total events / #students * 100)
  • Total student events (projects, collaborations, content providers, etc; excluding daily courses) divided by the number of students times 100 to get a whole number.
    (Student events / #students * 100)
  • Percent of teachers using videoconferencing.
    (#Teachers who used VC / total#teachers)
  • You can add a fourth by adding these measures together.

Comparison Data

In my dissertation study (277 respondents from six countries and 31 U.S. states), the utilization statistics were:

  • Total events/# of students *100: ranged from 0 to 60 with a mean of 4.
  • Student events/# of students: ranged from 0 to 67 with a mean of 4.
  • Percentage of teachers using videoconferencing: ranged from 0 to 100% with a mean of 26%.
  • Total Utilization Score: The three added together: ranged from 0 to 180 with a mean of 35.

So, you can aim for the mean, or you can aim for the highest range. Either way it will give you a feel for how your program is doing in comparison with the respondents to my research study.

Sample Analysis

For the fun of it, I took the data from one of my high use elementary schools. This was a 2nd-3rd grade building with 16 regular classroom teachers. They did 63 VCs this year.

  • Total events: 63/404 students *100 = 15.6
  • Student events: the same. No PD or meetings this year. 15.6
  • Percent of teachers using VC: 14/16 of the regular classroom teachers used VC: 87.5%
  • Total Utilization Score: 118.7 (which is nicely higher than the mean of 35; but not quite as high as the top 180 score in my study).

On the other hand, I have a couple schools who didn’t use it at all this year. So don’t think that all is rosy and perfect in my corner of the world!

What do you think? Is this a fair comparison? Does it weight the percent of teachers using it too much? How does your school compare? How would you measure total utilization?

(and don’t forget, I’m not counting daily full length shared classes because they skew the data)….

Evaluation: Counting and Comparing Totals

One of the most common ways to evaluate our videoconference programs is to count!

What do we count?

  • Number of students impacted (which is sometimes hard or skewed if a teacher does more than one VC)
  • Number of videoconferences
  • Number of types of videoconferences (ASK programs, content providers, free programs, collaborations in all their varieties, meetings, professional development, experts hosted onsite, etc. etc.)
  • Grant funding
  • Total cost of programs
  • District contribution of the cost
  • What else? What do YOU count?

What do we compare to?

When you count the above items, what do you compare it to?

  • Last year’s data
  • Last few year’s data
  • Other similar organization’s data
  • Baseline data
  • Between schools/districts
  • What else? What do YOU compare to?

Some of my comparisons

In our year end reports, I always include some comparisons. These are two of them. Do you compare this way also?

This graph compares the total events among the districts I serve. This is mostly for their own comparison as they like to compare themselves against each other. For a casual observer such as you, the data is less useful because you don’t know the size of my districts. In addition, as seems to be true everywhere, there are certain districts that all the others like to measure themselves against.

Still, isn’t it interesting to see which districts grew their use of VC (when overall use is down about 100 events total)?

There are some drastic changes, some of which I have yet to explain. I need to do some phone calls and see what is going on and if there is a possible solution.

Another graph I keep an eye on is related to our funding trends.

The total program costs are what our whole program spent on content providers and ASK programs. The red grant funding columns show what we, Berrien RESA, funded for our districts, using funding from various sources. The difference between the two is what the districts contributed.

Note the sharp increase in grant funding and corresponding decrease in district contributions to the program. I totally expected this, with many districts cutting back to participate in only free or fully funded programs. This trend reflects the impact that Michigan’s economy is having on public schools.

I wonder also, if this trend is happening across the nation, and if so, how are the content providers faring? Are they doing ok or is it hitting them hard too?

What comparing are you doing as you end the year? What questions are raised from the information you’re collecting?

Informal Evaluation Strategies

by Horia Varian from Flickr Creative Commons

One of the ways to evaluate your videoconference program is to ask questions throughout the school year. Here are some of the questions I ask:

Questions to Ask Teachers

Often when I don’t get to actually watch a videoconference, I like to email the teacher afterwards. Particularly if the program is one I haven’t seen before, or if the teacher is new to VC. I ask:

  • How did it go?
  • Did the content meet your curriculum?
  • Was the quality of the VC ok?
  • Did you have any problems with it?

Often I can then resolve any issues positively so that the teacher will come back for another videoconference.

Questions to Ask VC Coordinators

I also like to ask questions of my VC coordinators whenever I get a chance. Often on the phone as we’re discussing an issue or problem, I ask open ended questions to learn more about how VC is going in their school.

  • How are your teachers doing this year? Are they busy and stressed?
  • How is it going? (often this question brings out barriers, which we can then discuss solutions together)
  • How is your principal supporting VC this year?

The trick is to really listen! Listen to what might seem to be “complaints” or “excuses” in your mind. Listen! Are there ways to address those issues to make it easier for your teachers & coordinators?

What ways do you informally evaluate your program throughout the school year? Please comment and share!

End of Year Evaluation Strategies

by kevinzhengli from Flickr Creative Commons

As we come to the end of the school year, it’s time to reflect on the year, evaluate how it went, and use that data to plan for next year. So this week, we’ll be focusing on evaluation of our videoconference programs.

What Do We Evaluate?

  • How many videoconferences were done by each school
  • Which teachers used VC and why
  • Which teachers that used VC in the past didn’t use it this year and why
  • Which programs that we offered were effective and which weren’t
  • Effect on student achievement (if possible)
  • Which schools need more assistance
  • What the training needs are for the summer and next year
  • What else can you think of?

Previous Thinking on Evaluation

Before thinking more this week, let’s review some of what has already been said about evaluating our videoconference programs.

For some additional reading, consider these research articles:

How are you evaluating your program from this school year? Please comment & share!

    How Do You Evaluate the Effectiveness of Your VC Program?

    In the past month, I’ve been thinking and reading a lot about evaluation, particularly in light of my videoconferencing program at Berrien RESA. Evaluation is more than just counting how many videoconferences occurred (which is my usual pattern); it is determining the effectiveness of the program. To determine the effectiveness, we need to have some expected results, a way to measure them, and a way to find out if the program had the desired results.

    What are the Expected Results?

    One of the books I read on evaluation (McNeil et al., 2005) suggested that to evaluate a program, you should first consider what you expect the results of the program to be. So, what do you expect as the results of your curriculum videoconferencing program? What benefits do we expect for the students involved? We know that teachers see a benefit to their students, but what benefit is it specifically?

    • Motivation (for what? Learning?)
    • Achievement? (in specific content areas?)
    • Expanded learning opportunities
    • Interactivity?
    • Cross cultural exposure?
    • Increased communication skills?

    How Can We Measure Those Results?

    Next, we consider how we might measure those results. This is where I get seriously confused and unsure. Most of my teachers do one or two videoconferences a year. The highest is 12 videoconferences, and that was a preschool teacher with an am and pm class. So that was really just 6 videoconferences per class. Can six videoconferences in a school year make any measurable change in students? Can one videoconference make a measurable change beyond an anecdote?

    • If we expect increased motivation, is that measurable after just one videoconference? Two?
    • If we are measuring achievement, how would it be measured across my 70 schools with VC carts who use VC in a myriad of ways in the curriculum?
    • If we are measuring expanded learning opportunities, maybe we just count how many they did and leave it at that?
    • Do we determine the effectiveness of our videoconferences by how interactive they are? Do students learn more from a one-on-one vs. a view only session? Maybe. But sometimes the teacher needs to see a view-only before they will attempt a live interactive session.
    • If we are measuring cross cultural exposure, does a Michigan-Texas videoconference count? (That’s for you, Rox!) What about connections to zoos & museums?
    • How would we measure students’ increased communication skills? Teacher perceptions? Student surveys?

    Measure Effectivness by Comparing Results

    The next step would be to compare results to baseline data (before the program) and after the program. McNeil suggests using last year’s data as the baseline for this year.

    But to do this, we have to resolve the issue of what is the benefit and can we measure it?

    I’m not asking these questions because I don’t think there is a benefit. I do!! But can we get clearer and more articulate about the benefits of using curriculum videoconferencing? Can we improve our annual evaluations beyond just counting participation? Is it possible?

    Reflection

    The problem with asking these questions is it starts to make you think that videoconferencing may not be worth doing. But are the things worth doing only those we can measure? Certainly the current NCLB climate leans that direction.

    How did schools justify in-person field trips? Should there be hard SBR data to prove that field trips to a zoo or museum are worth doing before we actually do them? Similarly, do short-term learning experiences like videoconferences need hard data to show it’s worth doing?

    I don’t know the answers to these questions. But I sure want to talk to you all about them!! Please comment and share your thoughts/reactions to these questions even if you don’t have an answer either! Help me think about this!!

    References

    McNeil, K. A., Newman, I., & Steinhauser, J. (2005). How to be involved in program evaluation: What every administrator needs to know. Lanham, MD: Scarecrow Education.

    Teacher Comments on VC

    This post is part of a series of posts with some of the results from a survey of my top VC-using teachers. Read more about it in the first post of this series. Remember, they are using videoconferencing to support curriculum instruction (not full length courses).

    The question featured in this post is the last question in the survey:

    Any Other Comments You Want to Share?

    This is a qualitative representation of the results using Wordle. Click the graphic for a larger version.

    Teacher Comments

    Students and learning are front and center. Are you surprised? Here are some more comments:

    This is a song we sang at our Read Across the Planet (to the tune of Twinkle Twinkle Little Star) Distance learning is a fabulous experience & I’m exceedingly thankful that I can give my kids the chance to be a part of it!

    We hope you’ve enjoyed our show
    Distance learning helps us grow
    We learn to speak and to perform
    We get to explore explore
    We hope you’ve enjoyed our show
    Distance learning helps us grow. – Heidi Clark, 1st grade teacher at Three Oaks Elementary, River Valley.

    We enjoy the videoconferences and I feel that my students learn about themselves, the curriculum, and the world around them while participating. –Wendy Zahrn, 4th grade, Sylvester Elementary, Berrien Springs

    This has been an exciting opportunity for our Preschoolers and created new methods of learning and teaching. –Esther Nixon, Preschool, Mars Elementary, Berrien Springs

    The students gain so much from a VC that they are worth the time and effort involved. -Karen Ennesser, 7th grade science, Dowagiac Middle School, Dowagiac

    My goal is to have videoconferencing right in my classroom. –Linda McConville, 4th grade, E. P. Clarke Elementary, St. Joseph

    Videoconferencing is much easier than many think it to be. It is an excellent tool to share with your students. Real world/life plays a big part in this process. We are becoming a technological world and students will need these kinds of experiences to succeed. -Dori Hughes, 2nd grade, Eagle Lake Elementary, Edwardsburg

    One of the 7th graders stated that the video conference was the most memorable activity for this school year. Students are “charged” to learn in this way. It’s a great tool to connect small-town students to the world around them. -Peggy Clore, 5-7th language arts, Coloma Middle School, Coloma

    I am a big fan of videoconferencing because of the interactions of host and classroom. -Barb Vegter, 3rd grade, Countryside Charter

    Videoconferencing has proven to be a great motivator of student interest. They enjoy the experience and invariably ask when we’ll do another. –Anonymous high school teacher

    Reflecting on the process:

    Did you get any ideas for your future program evaluations? I’m thinking that some of these questions could be easily made into a multiple choice to get more quantitative data to compare from year to year. I’m thinking I would also like to survey my teachers who have done a VC before, but didn’t do one this year. This could be very valuable data as well.

    Did you find this little series helpful? What did you learn? Click comment below to share your thoughts.

    Critical Supports for Teachers

    This post is part of a series of posts with some of the results from a survey of my top VC-using teachers. Read more about it in the first post of this series. Remember, they are using videoconferencing to support curriculum instruction (not full length courses).

    The question featured in this post is the following:

    What are the supports that are critical for you to keep on using videoconferencing?

    This is a qualitative representation of the results using Wordle. Click the graphic for a larger version. This time instead of including all the words, I renamed them similar topics to the same words so that the results were more understandable.

    Critical Supports to Sustain Use

    To me, these results are a mandate to continue the following components of our videoconferencing program:

    • Our online registration which provides scheduling for most of the programs our schools do
    • Supporting, training, sustaining VC Coordinators, making sure they are replaced and trained if they leave
    • Providing resources, both print and web-based, to help teachers see how VCs match their curriculum
    • Offering free programs to our schools (ASK and collaborative projects) that are tightly matched to their curriculum, including the “boxes” that come with the ASK programs
    • Offering mini-grants to help pay for programs
    • Increasing access to VC in every school
    • Assisting principals and tech coordinators in the districts with supporting VC

    How would your teachers answer this question? If you’re a teacher, do you agree with this list? What supports must continue to sustain your use of videoconferencing? If you support teachers, are you able to provide most of this support? Do you provide any other supports? What do you see as most important? Please comment!

    Selecting and Preparing for VCs

    This post is part of a series of posts with some of the results from a survey of my top VC-using teachers. Read more about it in the first post of this series. Remember, they are using videoconferencing to support curriculum instruction (not full length courses).

    The question featured in this post is the following:

    Comment on how you select and prepare for a videoconference.

    This is a qualitative representation of the results using Wordle. Click the graphic for a larger version.

    Selecting and Preparing for a VC

    Curriculum and questions sure stand out, don’t they? Let’s look at a list of how my teachers find out about programs:

    • Check websites
    • I use the ISD website to search for programs that are available.
    • I go to the teacher who helps us in our building for help. VC Coordinator.
    • Look is ASK directory.
    • Investigate messages listing available topics. (My emails to the teachers came up quite a bit.)
    • Jazz mini-sessions.
    • Librarian assistance. VC Coordinator.
    • I use some of the same ones each year because I know the expectations and the results. I watch for Janine’s emails to see what is new that I can use, and I look for FREE ones.
    • I use the ones I have experienced. Interesting on these repeat comments. What hooked them the first time?
    • There is a print out at the beginning of the year that I try to find things that go with my theme.

    What is their criteria for choosing?

    • Whatever is free!
    • I look for videoconferences that integrate with our learning.
    • Based on our GLCEs / curriculum. This came up again and again in various wordings.
    • Videoconferences that are at my students’ level.

    Preparing for a videoconference can include:

    • Making sure they understand the content before the VC
    • I usually provide the students with guidelines and we vote on what medium or type of presentation to give. The students then work in groups to complete the presentations.
    • We prepare according to the instructions, formulate questions, discuss the topics etc.
    • As for preparing, sometimes the ISD sends kits for us to use or I check with the place holding the VC.
    • I expose my students to the information necessary for them to become fully engaged when viewing the decided VC. WE discuss and then prepare questions that can be asked or answered by viewing the VC to make it a much more richer experience.
    • Prior to the LEST WE FORGET conferences I will spend a class period providing background info or reinforcing prior learning of the war involved. We then develop questions based on student ideas of what they would like to hear more about.

    Your Turn: How do you select and prepare for videoconferences? What resources and strategies do you use to help teachers select videoconferences? How do you help teachers prepare for videoconferences? Please comment!

    Videoconferencing is like…

    This post is part of a series of posts with some of the results from a survey of my top VC-using teachers. Read more about it in the first post of this series. Remember, they are using videoconferencing to support curriculum instruction (not full length courses).

    The question featured in this post is the following:

    If you asked your students to fill in the blank: Videoconferencing is like ___________ What would your students put in the blank?

    This is a qualitative representation of the results using Wordle. Click the graphic for a larger version.

    VC is Like...

    Some of the teachers actually had their students answer the question, and they came up with very creative answers. See what you think of these:

    … going on a cool fieldtrip without leaving our desks.

    … eating double deluxe chocolate pie.

    … playing the best video game ever.

    … real live movies that we participate in.

    … a rainbow on a Sunday afternoon.

    … a fieldtrip coming to our classroom.

    … talking to someone in the same room, even though they’re far away.

    … school… only way better! :)

    Do you get the sense that the students in these classes (where teachers have done 6-12 videoconferences in a year) found that videoconferencing enriched their learning and engaged them?

    So, how would you answer this question? How would your students answer this question? Do you get the same reaction from your students? Please comment!