ECUR 809 Assignment #2

If I were to conduct an evaluation of the prenatal fitness program targeted at Aboriginal women, I would use the Outcomes-Based Evaluation Approach.  The reason I chose to use this approach is because I don’t think that the organization that was running the program had a specific goal in mind, rather they had outcomes that they hoped their participants could meet.  So in this particular case, the stakeholders hoped that by participating in the prenatal fitness program, the participants would learn about living healthy physically and mentally, thereby decreasing their chances of getting Type 2 Diabetes or Gestational Diabetes Mellitus (GDM).  My job would be determining whether or not the organization was doing the right activities for the needs of the participant, thereby meeting the intended outcomes.  By following the steps to accomplish an outcomes-based evaluation, I will now explain why I believe this approach to be effective.

The first step is to identify the major outcomes of the program.  In this particular case, the outcomes are to optimize healthy pregnancies and lifestyles in Aboriginal women and to reduce the rates of Type 2 Diabetes and GDM in Aboriginal women.  Obviously as the client, they have no control over the health and well-being of another person; however, they can provide the services necessary for the participants to actively make the choice to take advantage of the programming being offered.  Just because the women participate in the program, doesn’t guarantee they won’t get Diabetes.   What they did was learn how to live healthy, and in this respect, I think the outcome is partially met.   Therefore, the participants will have been impacted in a positive way, even though the results may not be what they had hoped. 

The second step is to prioritize the outcomes that I would like to examine.  Because I have only identified two outcomes, I could examine them both.  However, I do see optimizing healthy pregnancies and lifestyles as being of most importance.  I think if this is accomplished then the participant has the skills and knowledge necessary to do everything in their power to meet the next outcome.

The third step involves identifying indicators to make sure the outcomes are being met.  For this step, I would need to interact with the participants in order to gain the information I needed.   Some participants may meet the outcomes, but in a different way than another participant.  Some indicators may be a comparison of their current lifestyle to their past lifestyle or past experiences being pregnant.  Another indicator could be whether or not the participants had begun to access additional services and community resources.  Just their participation alone in the program and coming every week could be an indicator.  I suppose as the evaluator I would be making a professional judgment call, similar to that of a teacher.

Specifying a “target” goal of the clients is the fourth step, and perhaps one that is easier to complete.  This would involve me asking the client what they hoped to accomplish with their program.  For example, “By the end of the program, 80% of the participants will have completed the program with healthy pregnancies and an increased knowledge of how to remain fit and live a healthy lifestyle.” 

The fifth step is determining what information needed to show whether or not the indicators have been met.  This step is more of the logistics of the evaluation.  Some of the information is provided from participants registering in the program, and the rest would have to be collected through whatever method I choose to use which leads to the next step.

Determining how to collect all of required information is the sixth step.  In this case I would choose to use questionnaires, interviews and/or case studies. I know it isn’t feasible to do all three, but in a case such as this, I think you would have to use at least two.  Conducting 67 interviews is not realistic, but I don’t think I would get the information I wanted from just giving questionnaires. I work with Aboriginal students on a daily basis, and from my experience, I get more information from my students through conversations.  I think a great time to gather information from them is after their post exercise time.  Conversations are a bit less formal, and I think the participants would be more willing to share and be honest about their experience.  A case study is also a possibility, but again, I wouldn’t use just this method as a means of gathering information. 

Finally, the last step is to analyse and report my findings.  This would be completed once I had compiled all of the information, and I would report on whether or not I thought the program was effective in meeting the needs of the participants. 

I think there are other ways that an evaluator could have used in evaluating this program.  I contemplated using Scriven’s summative and formative model, but the outcomes-based approach seemed to fit a bit better.  In the end, this program wasn’t something that was sought out; the need for it came about through another program.  The author’s seen a need that had to be addressed, and so this program was developed.  Having the knowledge and skills of how to live a healthy lifestyle is essential to decreasing anyone’s chance in getting diabetes. 

Assignment #1 ECUR 809- Looking at Program Evaluation

Mid-Term Evaluation- First Nations SchoolNet Program:

Final Report

http://www.ic.gc.ca/eic/site/ae-ve.nsf/vwapj/FirstNationsSchoolNetFinalReport.pdf/$file/FirstNationsSchoolNetFinalReport.pdf

The First Nations SchoolNet Program was an initiative created by the Federal government for the purpose of connecting federally funded First Nations schools through the Internet and providing the skills in Information and Communications Technologies (ICT). The program initiative began in 2000 by first connecting the school with Internet by providing the necessary infrastructure.  From 2004-2006, the program was granted $15 million a year to increase connectivity with high speed internet, as well as computers and ICT.  The program was facilitated by various organizations on a regional level who were to provide reports to Industry Canada (Federal Government). 

Malatest and Associates Ltd. was the firm hired by Industry Canada to conduct the evaluation of the First Nations SchoolNet Program.  The model they used in their evaluation was Scriven’s model.  The evaluation was conducted as a mid-evaluation to determine whether or not the initiative conducted by the government was going as planned and what modifications needed to be made so the necessary improvements could be made.  Therefore, the approach taken was formative. 

A strength in this evaluation is the simplicity of the discourse used.  Being an educator, I have some previous background knowledge in the field so this gives me some foundation in understanding the context of the initiative.  However, I found that reading the report was fairly easy to understand.  It didn’t contain a bunch of legal jargon or language that was too difficult to comprehend.  As a result, I was able to gain a good understanding of the initiative itself, as well as the process used in evaluating the program.

Another strength in the evaluation was the thoroughness of the development and pretesting of the questionnaire.  The consultant developed necessary questions that pertained to each of the stakeholders involved making the questionnaire relevant to them.  The questionnaires were then sent to a committee for finalization and then pretested. Once the questionnaires were passed in the pre-testing field, the Consultant translated the questionnaires for two stakeholders into French.  The Consultant also provided an introductory letter to explain the questionnaire as well as frequently asked questions.  To me this demonstrates that all factors are taken into account, so that the results can be obtained in a most convenient and efficient manner.   Evaluating a program is a big job, so it makes sense that the one tool that potentially could bring in the most valuable information requires much attention in the development stage.

Although I don’t have any experience in determining the strengths and weaknesses of programs that are evaluated, I do think that I identified a weakness in this particular evaluation.  I don’t think that the Consultant interviewed enough participants- particularly those who are most closely connected to the program.  The report indicated that there are 567 schools that are receiving the program, each of which are run by administrators.  In the evaluation, the Consultant was identified as interviewing a total of 19 administrators, 11 of whom were principals, four technical administrators and four directors of education.  I found that when reading the results, the Consultant was provided with the most valuable information, and yet of the 55 people interviewed, only 19 were directly connected to the schools.  If it were me conducting the interview, I would interview more people directly connected to the schools, such as teachers, students and community members.  The Consultant did look at two case studies, where 16 people including teachers, administrators and suppliers were interviewed.  Although lots of relevant information was collected, it was at two schools in similar regions (Manitoba and Ontario).  I believe that if an agency is looking to see whether or not the goals of the program they are implementing are effective, as many stakeholders possible should be interviewed.  Therefore additional case studies should have been examined.  I also think that they are missing the component of student voice.  They are the reason that the program is being implemented in the first place, so their opinions about the effectiveness should be included.  Because Industry Canada was looking to gain a better understanding of the progress and challenges of the program, more people should have been interviewed. 

Overall, I feel that the Consultant did a good job in providing the information that Industry Canada was looking for.  The format and language used was clear and could be understood by a variety of audiences.  The Consultant demonstrated professionalism in making sure the questionnaires were relevant and could provide as much information as possible.  Although I believe more stakeholders could have been interviewed, the Consultant was still able to provide many relevant recommendations to improve the management, delivery, progress and challenges of the program.