The journey into program evaluation continues for me as a student in ECUR 809 by completing Assignment #5. The assignment requires me to design and test a short survey. I needed to include a variety of question types such as scale rating, short answer, and open-ended. I needed to share the original version, feedback from the survey and re-post the final adapted version of the survey.
The purpose of my survey is to gather information which will bring insight and opinions from potential participants (teachers, students, parent/community members) as to the effectiveness and appropriateness of a potential curriculum being considered for future implementation, in meeting the priority of offering music instruction and experiences in schools that currently do not have any opportunities.
This program evaluation is aligning around a new potential curriculum being considered for implementation in the fall of 2013 in a rural school division to enhance and expand music instruction in schools that currently do not have any opportunities or formal curriculum offered. The program is called Musical Futures.
Coincidentally, while I was concurrently working on this evaluation survey, I got involved with a collaborative research project between the Gwenna Moss Centre for Teaching Effectiveness, University Learning Centre and the Vice-Provost of the University. This project is taking a look at gathering evidence and examples of Experiential Learning opportunities for undergrad students in all colleges and within every department available.
My role, as a team leader on the project, is to working collaboratively with the other team leaders and undergrad assistants to engage in executing 1 on 1 summative survey interviews. These interviews will help in gathering data to formulate some insight into the existence and shape of experiential learning today and the directions that it will take in the future at the University. I took the lead in the survey design for the team leaders and then allowed for group input in shaping and refining the questions.
I have chosen to share both surveys as part of my learning in this area of program evaluation.
SURVEY #1 - MUSICAL FUTURES PROGRAM
(Original Version - comments from survey participants are inserted underneath each question)
Musical Futures Evaluation Survey
Purpose:
The Musical Futures Program is an approach and philosophy to music instruction which is being considered as a potential program for a local school division in the near future to provide music instruction and opportunities to all students. This evaluation survey is designed to gain insight and opinions from potential participants (teachers, students, parent/community member) as to the effectiveness and appropriateness in meeting the priority of offering authentic music instruction and experiences in schools that currently do not have any opportunities.
“Authentic” is a value-ladden term. What do you mean by authentic? It would be better to use the actual criteria than to assume all people filling out the survey have the same definition of what “authentic music instruction” looks like.
About You
Please check the appropriate response
1. Do you consider Arts Education valuable to students’ formative
schooling? Yes_____ No _____
2. What is your role in the school system?
* Teacher ___* Student ____ * Parent/Community member ____
Consider having more options here, like admin or board office, or what have you… And an "Other (please specify)" type deal, just for those outliers
3. What is your gender? Male ___ Female ____
4. If you are a student, what grade are you currently enrolled in?
(Circle one)
a) 5th grade
a) 5th grade
b) 6th grade
c) 7th grade
d) 8th grade
e) 9th grade
f) 10th grade
g) 11th grade
h) 12th grade
About Your Musical Experience
5. Have you at any time participated in formal music instruction?
Yes ____ No _____
6. Indicate all areas you have participated in formal music instruction.
* Private Piano Lessons ____
* Private Guitar Lessons ____
* Private String Lessons ____
* School Music Class (K-4) ____
* School Instrumental (band, orchestra) Programs (5-12) ____
* School Choral Music Classes (5-12) ____
* Community Organization Involvement
(Youth Orchestra, Children's Choir, Church Choir) ____
* Other (specify) _________________________
* None _____
7. What instruments do you currently or previously played ?
(Specify) ________________________________________.
8. To what extent did your life music experience come from informal
instruction (self-taught, learning on-line, shared learning from
family)? (Specify)_________________________________.
(Circle best answer)
Not at all Some Most All
Why is there a line to write on if we are circling? Just a thought
9. If you have no experience in music instruction or performance, what
were the obstacles for you to gaining access to it?
_______________________________________________________________________________.
You might want to ask a broader question to catch those on the fringe who had some but not much, like: “What were the obstacles, if any, for you to gain access to music instruction or performance?
10. If you have previous experience but chose to not continue, what were the reasons?
(Check all those that fit your opinion)
* Too challenging _____
* Did not meet my interests ____
* Not engaging enough ____
* Not financially an option ____
* No access ____
* I did not self-identify as a musician ____
* Engaged with other interests ____
Also, have an "other (please explain)" option for people
Consider providing an “Other” category here. For instance, none of the options listed really fit why I did not continue. My answer in an “Other” option would be:
About Musical Futures Program
Prior to answering the final questions, please take a moment to view the following on-line videos which introduce you to the Musical Futures Program.
This video was glitch and did not work well enough to watch.
The sound quality was really poor on this video and I wasn’t able to watch it to the end.
11. Does the Musical Futures Program seem interesting to you as a potential participant?
Yes _____ No _____
12. In what ways does this program intrigue you as a potential participant?
Check those that best fit your opinion
* Informal Instruction _____
* Informal Learning _____
* Popular/Rock Band Format _____
* Individual Choice of Learning _____
* Transfer of Skills Outside of School ____
* None ____
13. Do you foresee this type of program meeting the needs of potential
participants in engaging in authentic music instruction and experience?
Yes ______ No ______
I can’t answer this question because I don’t know what you mean by authentic. If authentic means deep rich learning that engages students through music then I would say “yes”. If authentic means learning a particular instrument and learning musical performance in an ensemble deeply then I would say “no”. I don’t think this program has the same purpose as a formal music program.
If YES….why? _______________________________________________________
If NO…why?
_______________________________________________________
Other comments given:
- The survey looks good!
- I like that most of it is short answer or checklists.
- Much more likely to get people to answer it that way.
- The survey piques my interest in the subject matter and to me that is the most significant piece of a survey. - In a narrative way… the survey opens possibilities for reflection of past, present, and future. Nice work.
SURVEY #1 - MUSICAL FUTURES PROGRAM (Revised Version)
Musical Futures Evaluation Survey
Purpose:
The Musical Futures Program is an approach and philosophy to music instruction which is being considered as a potential program for our school division in the near future to provide music instruction and opportunities to all students.
This evaluation survey is designed to gain insight and opinions, from potential participants (teachers, students, parent, community member, school and division administration) as to the effectiveness and suitability in meeting the priority goal of offering music curriculum and instruction which targets essential theoretical, skill, appreciation and experience components.
About You
Please check the appropriate response
1. Do you consider Arts Education valuable to students’ formative schooling?
Yes_____ No _____
2. What is your role in the school division?
* Teacher ____ * Student ____ * Parent/Community member ____
* School/Division Administration ____ * Other (specify) _______
3. What is your gender? Male ___ Female ____
4. If you are a student, what grade are you currently enrolled in? (Circle one)
a) 5th grade
b) 6th grade
c) 7th grade
d) 8th grade
e) 9th grade
f) 10th grade
g) 11th grade
h) 12th grade
About Your Musical Experience
5. Have you at any time participated in formal music instruction?
Yes ____ No _____
6. Indicate all areas you have participated in formal music instruction.
* Private Piano Lessons ____
* Private Guitar Lessons ____
* Private String Lessons ____
* School Music Class (K-4) ____
* School Instrumental (band, orchestra)
Programs (5-12) ____
* School Choral Music Classes (5-12) ____
* Community Organization Involvement
(Youth Orchestra, Children’s Choir, Church Choir) ____
* Other (specify) _________________________
* None _____
7. What instruments do you currently or previously played?
(Specify)_________________________________________
(Specify)_________________________________________
8. a) To what extent did your life music experience come from informal instruction - self-taught, learning
on-line, shared learning from family?
Circle best answer
Not at all Some Most All
b) If you circled (some, most or all) please circle which forms of learning
you benefited from.
Self-taught Learning on-line Shared learning from family
9. What were the obstacles, if any, for you to gain access to music instruction or
performance? ___________________________________________________
10. If you have previous experience but chose to not continue, what were the reasons?
(Check all those that fit your opinion)
* Too challenging _____
* Did not meet my interests ____
* Not engaging enough ____
* Not financially an option ____
* No access ____
* I did not self-identify as a musician ___
* Engaged with other interests ____
* Other (explain) _________________________________
_______________________________________________
About Musical Futures Program
Prior to answering the final questions, please take a moment to view the following on-line videos which introduce you to the Musical Futures Program.
11. Does the Musical Futures Program seem interesting to you as a potential participant?
Yes _____ No _____
12. In what ways does this program intrigue you as a potential participant?
Check those that best fit your opinion
* Informal Instruction _____
* Informal Learning _____
* Popular/Rock Band Format _____
* Individual Choice of Learning _____
* Transfer of Skills Outside of School ____
* None ____
13. Do you foresee this type of program meeting the needs of potential
participants in engaging in music curriculum and instruction which
targets essential theoretical, skill, appreciation and experience
components? Yes ______ No ______
If YES….why? ____________________________________________________________
If NO…why? ____________________________________________________________
Assessment of Survey Process:
I selected to have this survey previewed by my research group at the University, who happen to all be art specialists in some form (visual art, literary, music). I appreciated the feedback and suggestions I received for editing the questions or adding information. This helped to refine the questions and make them more understandable. The comments about the "video links" and the technical problems were from the same person. No other participant mentioned it and I did not experience any troubles with the links, so I can only reason that it might have been that persons computer or internet connections. My intent with using this survey would be to provide an on-line survey option that would be more user friendly than a paper form. The questions which ask to "circle" would be drop down box choices to highlight the appropriate answer for the respondent.
SURVEY #2 GMCTE Project – Experiential Learning Survey
(Original Version - team leaders and project leader comments below)
Summative Interview Questions
1) Describe your understanding of opportunities for students in your department to engage in experiential learning.
Perhaps, in addition to this, ask them, “How is it working?” or “Are you satisfied with the amount of experiential learning opportunities that your College/department offers?
Perhaps “are you satisfied with the current experiential learning opportunities…” might be better, but as a question asked after asking for them to think about/discuss the types of e.l.o. that they see in the department. They might not be thinking at the beginning of the interview of the scope of e.l.o. they have.
2) What are your perceptions/insight about the department’s offerings in the areas of;
This is a bit open-ended. Are you asking them if their college/department’s offerings in these areas are working well, if they exist or how they offer them?
*Community engagement
* Study abroad programming
* Field experience
* Practical placements
* Undergrad research
This could be a prompt for the interviewer - Do these categories accurately reflect on the experiential learning opportunities your College/department offers? Do you offer courses that don’t really fit these categories?
3)Describe the process that through which these offerings are generated and implemented.
Are they conceived through individual instructors? Or at the program level?
This could be a prompt for the interviewer
Move Q7 here
4) To what extent does your department engage in community partnerships/relationships to enhance experiential learning ?
Love this question
relationships between the community partners and your college/department mutually beneficial?
5) What resources (facility, equipment, programs) does your department use to enhance experiential learning opportunities?
Fabulous question
6) What, if any, extensions to the department offerings are there off-campus?
- Or, rather, does your department have any experiential learning opportunities for students outside of current course offerings?
7) What could be done to enhance and support experiential learning in your department?
Move to follow q3 - what else would you like to be doing in your program…don’t make it about what resources they need but rather what can they imagine as valuable new activity.
- Gauge quantity and quality of the experiential learning opportunities their college/department offer?
I’m not sure if department heads are the right people to ask, but I’d like to know how they feel students have responded to these programs (do they feel that there is demand for e.l.o in their department from students) and whether their departments are currently looking/planning to increase e.l.o. or just maintain the ones they have. Definitely ask this in the dept head UG chair interviews could be worked into the conversation as aspect of Q 1 or Q2
SURVEY #2 - Experiential Learning Survey (Revised Version)
GMCTE Project – Experiential Learning Survey
Interview Questions
1) Describe your understanding of opportunities for students in your department to engage in experiential learning. How is it working?
2) Are you satisfied with the amount of experiential learning opportunities that your College/department offers?
3) Can you address the department’s offerings in the areas of;
*Community engagement
* Study abroad programming
* Field experience
* Practical placements
* Undergrad research
Are the college/department’s offerings in these areas working well? (if they exist) How are they offered? Which are involved?
4) Do these categories accurately reflect on the experiential learning opportunities your College/department offers? Do you offer courses that don’t really fit these categories?
5) Describe the process through which these offerings are generated and implemented. **Are they conceived through individual instructors? Or, at the program level?**
6) What could be done to enhance and support experiential learning in your department?
7) What else would you like to be doing in your program? What can you imagine as valuable new activity?
8) To what extent does your department engage in community partnerships/relationships to enhance experiential learning?
9) In your opinion, are the relationships between the community partners and your college/department mutually beneficial?
10) What resources (facility, equipment, programs) does your department use to enhance experiential learning opportunities?
11) What, if any, extensions to the department offerings are offered? Does your department have any experiential learning opportunities for students outside of current course offerings?
12) From your perspective, how do feel students have responded to these programs?
Do you feel that there is demand for e.l.o in their department from students?
Is the department currently looking/planning to increase e.l.o. or just maintain the ones they have?
Survey Implementation and Project Plan
Since completing the survey development, the project leader and supportive staff have set up interviews for each of the team leaders to initiate with all the College department heads. The interviews began on March 6, 2013 and as of March 22nd we will have conducted close to 60 interviews. The interview process consists of the team leader orally guiding the department head through the questions while an undergrad student is recording and documenting answers on a laptop directly into the survey. This approach is really useful in fostering a comfortable dialogue atmosphere between interviewer and interviewee. The interviews last between 30-45 minutes.
The next stage of the project is to contact each instructor in the departments that are associated with the relevant experiential learning opportunities and discuss specifics regarding the class/course offerings. The data will be compiled by the project leader and then finally a report will be shared with the University community.
The most significant experience related to this project for me is having an opportunity to witness the nervous/apprehensive responses related to program evaluation. The participants interpret that with evaluation comes assessment. This puts them in a vulnerable place of having to justify their programs. The unfortunate timing of this project is distorting the project aim as most University participants are aligning the data collection with Transform US, where in fact, it has little to do with that at all. The interview is about finessing a trusting relationship to help draw out critical data that will be useful for the project.
