This year, I decided I wanted more than numbers to evaluate my videoconference program. So I created two surveys, one for teachers who participated in a videoconference, and another for teachers who did not participate in a videoconference. So far, I have 54 responses and am hoping to get more before the end of the school year.

Here is what I asked my teachers:

If they participated in a videoconference this year:

• What benefits do you see to your students in using videoconferencing?
• Which videoconferences do you want to do again next year?
• Which videoconference(s) were NOT good and not worth your time?
• Are there any other topics that you wish there was a videoconference for?
• Any other comments?

If they did NOT participate in a videoconference this year:

• Have you ever done a videoconference in the past?
  
• What reason(s) prohibited you from participating in a videoconference this year?
• What videoconference(s) do you wish you could participate in next year?
• What do you need to be able to participate in a videoconference next year?
• Any other comments?

I also collected some demographic data: grade level, subject area, name & district.

So far I am getting interesting feedback, and hope that it will give me good data to plan programs and events for next year…

What questions are you asking YOUR teachers?

Yesterday we talked about comparing our end of year report numbers to last year’s data and between schools and districts.

What about comparing your data to national data?

In the spring of 2008, many of you contributed to my dissertation study focusing on videoconference coordinators and the use of videoconference in K12 schools.

When you look at your end of year data, it might be useful to compare it to the data from this study.

Three measures

There are three ways to measure your videoconferences to compare against this data:

• Total videoconference events (professional development, meetings, and student events, excluding daily courses) divided by the number of students times 100 to get a whole number.
  (Total events / #students * 100)
• Total student events (projects, collaborations, content providers, etc; excluding daily courses) divided by the number of students times 100 to get a whole number.
  (Student events / #students * 100)
• Percent of teachers using videoconferencing.
  (#Teachers who used VC / total#teachers)
• You can add a fourth by adding these measures together.

Comparison Data

In my dissertation study (277 respondents from six countries and 31 U.S. states), the utilization statistics were:

• Total events/# of students *100: ranged from 0 to 60 with a mean of 4.
• Student events/# of students: ranged from 0 to 67 with a mean of 4.
• Percentage of teachers using videoconferencing: ranged from 0 to 100% with a mean of 26%.
• Total Utilization Score: The three added together: ranged from 0 to 180 with a mean of 35.

So, you can aim for the mean, or you can aim for the highest range. Either way it will give you a feel for how your program is doing in comparison with the respondents to my research study.

Sample Analysis

For the fun of it, I took the data from one of my high use elementary schools. This was a 2nd-3rd grade building with 16 regular classroom teachers. They did 63 VCs this year.

• Total events: 63/404 students *100 = 15.6
• Student events: the same. No PD or meetings this year. 15.6
• Percent of teachers using VC: 14/16 of the regular classroom teachers used VC: 87.5%
• Total Utilization Score: 118.7 (which is nicely higher than the mean of 35; but not quite as high as the top 180 score in my study).

On the other hand, I have a couple schools who didn’t use it at all this year. So don’t think that all is rosy and perfect in my corner of the world!

What do you think? Is this a fair comparison? Does it weight the percent of teachers using it too much? How does your school compare? How would you measure total utilization?

(and don’t forget, I’m not counting daily full length shared classes because they skew the data)….

One of the most common ways to evaluate our videoconference programs is to count!

What do we count?

• Number of students impacted (which is sometimes hard or skewed if a teacher does more than one VC)
• Number of videoconferences
• Number of types of videoconferences (ASK programs, content providers, free programs, collaborations in all their varieties, meetings, professional development, experts hosted onsite, etc. etc.)
• Grant funding
• Total cost of programs
• District contribution of the cost
• What else? What do YOU count?

What do we compare to?

When you count the above items, what do you compare it to?

• Last year’s data
• Last few year’s data
• Other similar organization’s data
• Baseline data
• Between schools/districts
• What else? What do YOU compare to?

Some of my comparisons

In our year end reports, I always include some comparisons. These are two of them. Do you compare this way also?

This graph compares the total events among the districts I serve. This is mostly for their own comparison as they like to compare themselves against each other. For a casual observer such as you, the data is less useful because you don’t know the size of my districts. In addition, as seems to be true everywhere, there are certain districts that all the others like to measure themselves against.

Still, isn’t it interesting to see which districts grew their use of VC (when overall use is down about 100 events total)?

There are some drastic changes, some of which I have yet to explain. I need to do some phone calls and see what is going on and if there is a possible solution.

Another graph I keep an eye on is related to our funding trends.

The total program costs are what our whole program spent on content providers and ASK programs. The red grant funding columns show what we, Berrien RESA, funded for our districts, using funding from various sources. The difference between the two is what the districts contributed.

Note the sharp increase in grant funding and corresponding decrease in district contributions to the program. I totally expected this, with many districts cutting back to participate in only free or fully funded programs. This trend reflects the impact that Michigan’s economy is having on public schools.

I wonder also, if this trend is happening across the nation, and if so, how are the content providers faring? Are they doing ok or is it hitting them hard too?

What comparing are you doing as you end the year? What questions are raised from the information you’re collecting?

One of the ways to evaluate your videoconference program is to ask questions throughout the school year. Here are some of the questions I ask:

Questions to Ask Teachers

Often when I don’t get to actually watch a videoconference, I like to email the teacher afterwards. Particularly if the program is one I haven’t seen before, or if the teacher is new to VC. I ask:

• How did it go?
• Did the content meet your curriculum?
• Was the quality of the VC ok?
• Did you have any problems with it?

Often I can then resolve any issues positively so that the teacher will come back for another videoconference.

Questions to Ask VC Coordinators

I also like to ask questions of my VC coordinators whenever I get a chance. Often on the phone as we’re discussing an issue or problem, I ask open ended questions to learn more about how VC is going in their school.

• How are your teachers doing this year? Are they busy and stressed?
• How is it going? (often this question brings out barriers, which we can then discuss solutions together)
• How is your principal supporting VC this year?

The trick is to really listen! Listen to what might seem to be “complaints” or “excuses” in your mind. Listen! Are there ways to address those issues to make it easier for your teachers & coordinators?

What ways do you informally evaluate your program throughout the school year? Please comment and share!

As we come to the end of the school year, it’s time to reflect on the year, evaluate how it went, and use that data to plan for next year. So this week, we’ll be focusing on evaluation of our videoconference programs.

What Do We Evaluate?

• How many videoconferences were done by each school
• Which teachers used VC and why
• Which teachers that used VC in the past didn’t use it this year and why
• Which programs that we offered were effective and which weren’t
• Effect on student achievement (if possible)
• Which schools need more assistance
• What the training needs are for the summer and next year
• What else can you think of?

Previous Thinking on Evaluation

Before thinking more this week, let’s review some of what has already been said about evaluating our videoconference programs.

• How Do You Evaluate the Effectiveness of Your VC Program?
• Evaluation of Collaborative Projects
• Questions for Teachers to Ask when Evaluating an Event/Project/Collaboration
• Definition of Successful Implementation of Videoconferencing

For some additional reading, consider these research articles:

• It’s a Bit Like Star Trek: The effectiveness of video conferencing
• A Study Of The Factors That Impact Videoconferencing As A Learning Tool Within Three Regional Service Agencies In Michigan

How are you evaluating your program from this school year? Please comment & share!