Executive Summary of Results

This project was designed to provide a means of evaluating the writing and speaking skills of our Bachelor of General Studies program to determine a) what their skill level is, and further, to establish benchmarks to be reached in order to be considered at a level appropriate to a graduate of this program.  Rubrics were developed (rubrics attached) for evaluation of writing and speaking competency, and the Oncourse ePortfolio was used to as the vehicle for students to upload writing and speaking samples for evaluation.  While there were some difficulties in developing the ePortfolio tool so that the evaluators could determine skill levels, the project was successful in developing rubrics that were reliable across evaluators.  Many of the problems encountered with the ePortfolio tool have also been worked through and a workable procedure has been developed.

Need for the Project

Briefly explain why you believed there was a need for your project and what teaching approach was used to address this need.

Indiana University Kokomo has a large population of General Studies students. During the five-year period of 1999 – 2004, 204 Bachelor of General Studies degrees were awarded. B.G.S. degrees comprised approximately 19% of the baccalaureate degrees awarded at Indiana University Kokomo and accounted for more baccalaureate degrees than any degree program with the exception of degrees in Education.

The B.G.S. degree is designed for returning adult students with previous college credit who, for personal or professional reasons, are seeking to finish college. This adult degree completion program is vital to the mission of Indiana University Kokomo, which is “to enhance the educational and professional attainment of the residents of North Central Indiana by providing a wide range of bachelor’s degrees...and to [strengthen] the economic and cultural vitality of the region.”

The School of Continuing Studies does not have its own faculty. B.G.S. students obtain their degrees by taking general education courses and courses in the various disciplines. They work with their advisor to set their learning goals, and these goals are considered “met” when the student completes particular courses. There has been no formal assessment mechanism for ascertaining whether students actually gained the knowledge or acquired the skills they were seeking. Nor has there been a good way to track whether the program itself is successful in helping students achieve their desired learning outcomes.  Thus, the program has been unable to collect data to help it improve student learning. In addition, the lack of data on student learning outcomes for this large group of baccalaureate degree students will be problematic when the Kokomo campus seeks re-accreditation by the Higher Learning Commission in Spring, 2009.

Furthermore, this appears to be an issue across IU campuses. An informal email survey of faculty and directors responsible for B.G.S. students at other campuses revealed a broad concern about how to assess learning outcomes in their students. The Interim Dean of the School of Continuing Studies has said that since she began in the School 25 years ago, they have wrestled with the assessment question.

This project was primarily directed toward developing a way in which students could be evaluated in the basic skills of writing and speaking effectively.  No specific pedagogical techniques were used, but a method of evaluating these basic skills reliably across students was determined to be an important goal to ensure that the graduates of this program did meet a specific level of competence in these two basic areas of their education.

Use of Technology

Briefly explain how your project used instructional technology in a new or different way.

• How did the project take advantage of IU’s learning environment?

We are fortunate that Indiana University has a wide range of technology available to faculty and students.  For this project, we had Oncourse and were developing an ePortfolio tool that would enable us to be able to have students upload writing and speaking samples, and then to have a selected group of faculty to be able to access theses samples and evaluate them online.  Although there were a few difficulties in developing the process, which delayed the onset of the evaluation process, these difficulties were resolved and the evaluation of materials uploaded to the ePortfolio tool was finally a reality.

• Did the project lead to collaboration that will help IU improve or expand classroom   and online learning environment? 

This project required the collaboration of individuals from different disciplines (particularly from our writing and speech faculty) on our campus to work on the development of rubrics that would enable us to reliably assess the skill levels of our students.  We also are fortunate to have a very skilled staff at IUPUI who worked with us to develop the format to allow us to use the rubrics developed online.  This process was very complicated, yet those responsible for helping us do this at the same time that some significant changes were being made in the “new” Oncourse gave a great deal of time and effort in getting the project working
 

• How did the project make use of instructional technology and design?

The use of instructional technology in this project was to use some of the existing and developing Oncourse technologies and the ePortfolio tool that was being developed to allow the evaluation of student writing and speech samples within the student’s ePortfolio.  This provides the student with examples and evaluations of their work in these areas which may be used when applying for new positions or promotion within a company with which they are already employed.

Instructional Design Plan

Describe how the use of technology used supported your teaching approach:

• What student learning outcomes were expected?

The primary purpose of this project was to develop a way of providing students with feedback in regard to the basic skills of writing and speaking effectively, since these are considered the basic tools of any educated person.  Although the plan is in the early stages of development, we do expect to have this become a fundamental tool to provide students objective evaluations of these skills.  This will enable them to have a better understanding of what they need to do in order to reach the level of communications skills that the faculty believe are required of a college graduate.
As a result of this project, rubrics are being developed to assess the student’s progress in other areas of competence. With the development of IU’s ePortfolio, we now have a useful way for (1) B.G.S. students and their advisor to track their progress in their learning goals, (2) the program to assess its effectiveness in helping students attain their learning goals, and (3) faculty to assess whether the students actually are meeting their learning goals. The School has formed a faculty advisory group, chaired by Dr. Robert Wildblood, with representation from several academic departments on campus. This group, along with the Director (Fred Hakes) and the Advisor (Lori Collins) is developing a formal assessment plan. The plan lists student learning outcomes common to all B.G.S. students, and will ultimately also contain a menu of student learning outcomes which students may select for themselves to meet their own educational goals.  The assessment plan will require all B.G.S. students to use ePortfolio to track their progress in the degree. Together with their advisor, B.G.S. students will select the learning outcomes appropriate to their goals and the requirements of the program. These student learning outcomes will be entered into the matrix of the student’s ePortfolio, giving the student and his or her advisor a visual plan for what the student needs to achieve in order to complete the degree. As the student takes courses, he or she will collect artifacts that serve as evidence for having achieved the various learning outcomes. The student will upload these artifacts, and then provide, within the ePortfolio, an explanation of why this piece of evidence reflects achievement of the outcome.

• How did your teaching plan encourage active learning, provide  feedback, address respective learning styles, and build collaboration? 

The project was developed with the cooperation of members of the English, Communications, and Psychology faculty.  The feedback was accomplished in two ways.  First, the faculty members involved were able to test the rubric to ensure that it was truly representative of the skills that are the foundation of writing and speaking, and second, it has provided feedback to the students who were “volunteers” in this project.  The process also allowed for the editing of the original rubrics to make them more inter-rater reliable and somewhat easier to use.  Since this was an evaluation process, which was to provide feedback to students throughout their college career, it was planned that by the time the student was ready to graduate, that he or she would know exactly what needed to be done to reach the expected level of competence.

• What resources were used? (campus technology support services, etc.)

This project requires the efforts and cooperation of faculty from various disciplines, the staff of the General Studies program, the campus Center for Teaching, Learning, and Assessment, and the staff of IUITS.  Without the high level of cooperation of all of these people the project could not have been as successful as it was.

• Would your project be easily understood and executed by other faculty?

Given the work that has gone into developing the methods used in this project, it would be something that, given the appropriate level of training, other faculty should have few difficulties in implementing a similar program.  Training should not be difficult for anyone with a familiarity with the use of Oncourse.

• What resources were used? (campus technology support services, etc.)

This project requires the efforts and cooperation of faculty from various disciplines, the staff of the General Studies program, the campus Center for Teaching, Learning, and Assessment, and the staff of IUITS.  Without the high level of cooperation of all of these people the project could not have been as successful as it was.

• Would your project be easily understood and executed by other faculty?
• Given the work that has gone into developing the methods used in this project, it would be something that, given the appropriate level of training, other faculty should have few difficulties in implementing a similar program.  Training should not be difficult for anyone with a familiarity with the use of Oncourse.

Potential to Impact Student Learning

Clearly define how your project improved student learning - include specific examples of how your project:

• Fostered depth of learning

Because use of the methods developed in this project were specifically designed to give feedback to students as to how they were, or were not, meeting the specific requirements on which they were being evaluated, it would give them the opportunity to understand the evaluation process better and to apply themselves to developing the necessary knowledge and skills to make their educational experience more successful. 

• Supported efforts to improve retention

Although this project was not specifically developed as a retention tool, it is obvious that a student who clearly understands the evaluation process and who is willing and motivated to apply this knowledge would likely be a more successful student and would be more likely to continue to the completion of their educational goals.

• Fostered  the learning of large numbers of students

Although it remains to be seen how widely the results of this project will be implemented, it is clear that this is a process that can be applied to any student in any discipline for which rubrics are developed, particularly rubrics in those courses which are designed to provide basic skills required to be successful in their field of study.  The only limitation would appear to be the willingness of the faculty of various disciplines to adopt the model to provide evaluation to which students would have easy access, and which could be used to assist them in their studies while enrolled in their program of choice and to be used as a tool for employment and/or advancement in their professional life.

• Has broader applications that will affect learning K-12 or community environments?

While this project could have some application to K-12 and/or community environments, it has not been designed to do this specifically.

Assessment Plan

Briefly explain the effectiveness of your assessment plan:

• What measure of assessment was used?

During the development of this method, students who volunteered to have their writing or speeches assessed.  For each of the papers or videos submitted, two faculty members independently evaluated the work using the rubrics that had been developed.  The results of the evaluations were checked for inter-rater reliability to determine if the rubrics were such that we could be confident that faculty who chose to use them could have confidence that they did reflect the skills that are important in writing and speaking.

• How did your approach obtain measures of performance?

Students in two classes in the Spring and Summer sessions were asked to volunteer to submit materials to the ePortfolio for evaluation.  Eleven students who were juniors and seniors enrolled in a 300 level Psychology course submitted papers and twenty-one students in the required speech course submitted a speech sample.  Two faculty members for writing and two faculty Members for speech using the rubrics that had been developed independently evaluated the submissions.

• Did you feel your measurement was accurate and effective?

The evaluations were checked for inter-rater reliability and the following results were obtained.
Writing:  Inter-rater reliability of + .83
Speech:   Inter-rater reliability of + .87
These findings are considered to be very good indications that the rubrics are easy to apply, the results of independent evaluation are consistent with good measurement procedures, and that the process is one that should be easy to learn by other faculty.

• What were the student learning outcomes?

Feedback from students who volunteered for this project suggest that they see this as a useful tool in understanding the criteria for evaluation and giving them specific information as to the areas that they needed to emphasize in further assignments.

Plan for Colleague Development

Describe your role and activities as a mentor:

• What mentoring activities were implemented to assist colleagues?

Because of the nature of this project we are still at the beginnings of assisting other faculty to use the process.  Rubrics have been revised and are currently being tested to see if they are easier to use and will produce even better inter-rater reliability.  We have yet to make any off-campus presentations in regard to this project; however, we are currently exploring appropriate venues.

• How did you help to develop colleague's instructional technology skills?

Faculty, using the rubrics developed, should have a greater understanding of the skills that have determined are important to assess and to give adequate feedback to students.  This should result in more consistent teaching in those courses that are considered to provide the basic skills within a discipline.

• How do you think your contributions to the Knowledge Base will help colleagues planning similar projects? 

Since the groundwork has been laid, any faculty who would like to have an evaluation process that will allow them to give more specific feedback to students in regard to their basic skills should be able to implement a project like this with little difficulty.  It would seem that one of the easiest ways to do this would be to ensure that the staff of Teaching and Learning Centers were trained adequately and that they would then be able to assist faculty in setting up their projects.

• Is your project approach potentially applicable to other contexts or disciplines?

Although this project was started with providing the ability to accurately assess basic writing and speaking skills, it can be applied to any course or disciple that is able to agree on the basic skills required for that program.  Rubrics can be developed for any discipline and online evaluation so that students can have a permanent file of their progress in developing these essential skills and competencies.

• What tips would you offer?

Any faculty member who wants to use this process can learn how to do it.  It would be much better if they were people who believe that technology can be a useful tool to provide a better level of feedback and if they were committed to the process.  It isn’t difficult, but it can take a bit more time for the evaluation process.

Final Comments on Project Results

• Do you think you met the goals of your project?

I believe that the goals of the project have been met.  We were committed to developing rubrics that could be used in the evaluation of basic writing and speaking skills and we have done that.  We were also committed to being able to develop a process that would utilize the ePortfolio and that would provide the student with a permanent file that they could use during their time as a student and would also be able to use in their professional life and I believe that we have done that as well.

• What, if anything, would you do differently?

I don’t think that there was anything that we would have done differently.  We did have some difficulties with implementing the technology; however, that was, to a large degree, a problem of having the new ePortfolio being implemented at the same time as some major revisions of Oncourse being implemented at the same time.  The dedication of the technical staff was outstanding.

• Did you have any unexpected outcomes? (positive or negative)

I was pleased that the project turned out as well as it did.  We did learn a lot about how the process works and were able to take the information that we got from the testing of the rubrics to begin the process of revision.  We also have had a commitment from the Dean of the School of Arts and Sciences to use this process in the evaluation of the W131 and S121 courses. 

• Would you recommend this teaching approach to colleagues?

Yes.  This approach to evaluation is a very good one.  It allows for using the same items for evaluation of the basic skills across all faculty members who teach these courses and it can now be easily adapted to work with other disciplines.

• What other comments or suggestions would you offer to complete this report?

I think that this project has given us enough information to consider it a useful method of evaluation.  Although it was originally conceived as a tool for the General Studies program, it is also a method that can be adapted to any disciple that has a set of basic skills that all students in that discipline are required to know.  This method of evaluation of student work also seems to fit in well with the current emphasis of evaluation for accreditation. 

Return to Final Reports page

AT&T Fellows Home Page

Last updated: 20 February, 2008

Comments and Questions | (317) 278-4833
Copyright © 1999-2003, The Trustees of Indiana University