AT&T Fellows Final Report: Eugenia Fernandez and Sally Catlin

AT&T Fellows Final Reports

June 2006

Name	Eugenia Fernandez and Sally Catlin
Title	Associate Professor and Trustees Lecturer
Department	CIT
Campus	IUPUI
Project Title	Exploring the Use of ePortfolio for Assessment
Project Goal	The goals of this project are to (1) explore the use of ePortfolio to support our assessment process, and (2) increase student and faculty awareness of, and participation in, assessment.
Type of Technology Used in the Project	ePortfolio, narrated PowerPoint presentation, Oncourse surveys

Executive Summary of Results

To explore the use of ePortfolio for the assessment process, we created a customized matrix in ePortfolio which maps the course objectives for CIT 212 to our ABET IT assessment outcomes. We developed instructions for using the customized matrix, and developed an online module on reflective writing. We conducted an empirical study to assess student awareness of these outcomes and the use of ePortfolio. Surveys were created; participants solicited; and the pre- and post- surveys administered. Students were directed to upload assignments to the matrix. However, due to technical difficulties, few students successfully uploaded to ePortfolio or completed the post-survey, making any statistical analysis impossible.

Need for the Project

Our department definitely needs help with assessment. Many faculty do not completely understand our assessment guidelines and how to collect artifacts. Currently, for each course they teach, faculty members are asked to pick an outcome that is covered in the course and to collect student work to assess it. Ideally, faculty should be gathering the samples throughout the semester, creating the necessary rubrics and assessing work based on the ABET IT outcomes. In practice the process is left to the end of the semester and, instead of gathering the samples, faculty members have to hunt for them. Rubrics are often created after the fact.This project involved the creation of a matrix in ePortfolio for students to upload appropriate documents. These documents will be used for our department's assessment needs. In addition, this project helps students learn about CIT's assessment process and what, as a department, we feel that they should be learn.

Use of Technology

We used Oncourse to distribute surveys to students willing to participate in the study. We developed a customized matrix in ePortfolio for students to upload their work. We also created directions for students and faculty in how to upload student work into ePortfolio. We worked with a colleague in the Technical Communications program and the recording studio at the Center for Teaching and Learning to create to develop an online audio presentation to help students with the reflective writing portion of the matrix.

We are one of the first departments in Engineering & Technology to create a customized matrix in ePortfolio based on our department's ABET IT Outcomes that are used to assess our students. We ran into a variety of problems with setting up permissions for students to be able to upload their work and complete their reflections so that only the student, reviewer and instructor able to view their work. The knowledge that we gain from using the customized matrix will help future faculty who want to use a matrix other than the PUL matrix.

Instructional Design Plan

We expected that student awareness of the ABET IT Outcomes and the assessment process would increase. More specifically, we expected students to learn about assessment, what the CIT department expects them to learn by the time they graduate; and how to reflect on their learning in writing.

None for the students. We collaborated with Becky Fitterling, a Lecturer, from the Technical Communications program to develop our online Reflective Writing module. In addition, we will be encouraging collaboration among the faculty who teach CIT 212 in the development of the grading rubrics.

We used ePortfolio, Oncourse, the Center for Teaching and Learning and faculty in the Technical Communications program.

Yes, it will be very easy for other faculty to understand our project and to implement a similar project based on their needs. The only difficulty may be in faculty learning ePortfolio and creating their own matrix. However, we have developed some materials to help with this area. The project will also be helpful for faculty within our department to use ePortfolio as a way of assessing the teaching and learning that is taking place in their classes.

Potential to Impact Student Learning

Our project involved students in the assessment process so they could gain a greater understanding of the broad principles that they should be learning. In addition, the use of ePortfolio helped them improve their ability to write reflectively on what they have learned.

Assessment Plan

Student knowledge about assessment and the ABET IT outcomes was tested using a pretest -post test survey experimental design. IRB approval was obtained. The post test also solicited student feedback on using ePortfolio. We had good student response in taking the pre-survey, but lacked sufficient response to the post survey for any statistical analysis. We can’t effectively access the new tools because of ePortfolio's unforeseen problems, but we can assess that the tool was much more difficult to use than we had anticipated.

We worked with members of the Center for Teaching and Learning to validate the customized matrix, and will do the same for our assessment rubrics. Faculty feedback will be solicited as well.

Plan for Colleague Development

Eugenia Fernandez has been mentoring Sally Catlin throughout the development and execution of this project. In turn, Ms. Catlin mentored other faculty who were involved and also mentored a CIT student assistant who was helping the study participants with any ePortfolio problems. In addition, we will be training assessors on the use of ePortfolio using rubrics and when the project is complete, we will present a CIT Tech Talk on the use of ePortfolio.

Our contributions are to provide detailed directions (and suggestions) for creating and publishing a customized matrix in ePortfolio, to provide an online narrated module on reflective writing, and lessons learned from our experience training other faculty in assessment.

Yes, we believe that departments in need of help using ePortfolio for the collection of artifacts for assessment and review of these artifacts could greatly benefit from this project.

Test thoroughly, and expect set-backs, and take advantage of all the help you can get from the Center for Teaching and Learning.

All specific tips we offer deal with setting up the customized matrix. We initially thought we had worked out all the problems with ePortfolio permissions so student work would be secure from others being able to view their work. This was not the case. We ran into a variety of problems with the set up of the permissions for students to upload and complete reflections so that only the student, reviewer and instructor would be able to view their work. We finally found the solution. If we had been using the preset PUL matrix we would have had fewer problems. Also initially we started with a project site instead of a class site for ePortfolio which led to additional problems with permissions and the setup of ePortfolio. Lorie Shuck helped us immensely on the ePortfolio problems that we encountered. She is a great resource.

Final Comments on Project Results

We met some of the goals of our project, but in limited scope. We ran into many obstacles in the creation of the matrix and directions for using ePortfolio and uploading to the customized matrix. We had to go through many iterations of creating matrices, testing the uploading of documents, testing the reflective writing portions and the assessment sections before we felt we could have students upload files, reflect on their assignments with security and without errors. The end of the semester approached and very few students who had initially signed up for the study completed the final tasks.

We plan to continue this project and run another iteration of our plan during the Fall 2006 semester. We will work more diligently to encourage student participation, and the matrix will be correctly setup and available during the first week of classes.

We would have the customized matrix ready at the beginning of the semester. Visiting classes helped get more students involved in the study. Emailing via Oncourse did not work very well. It took awhile to get students to take the pre-survey. Next time, we would have the pre-survey ready at the beginning of the semester when students have more time and have them upload their assignment/artifacts as they turn their assignments. This should all be completed within the first 6 weeks of the course so students have time for their reflections and time to take the post survey before the semester draws to a close. We would then have time to have faculty assess the artifacts in ePortfolio.

We had to change from an ePortfolio project site to an ePortfolio class site. This change resulted in recreating much of the work we had already done in the creation of the matrix and the testing of the various tasks needed to be done using ePortfolio.

Using ePortfolio was much more confusing to students then we had expected. Even the few students who uploaded had problems following the directions.

Yes. Creating the customized matrix forced us to explicitly tie course assignments to assessment outcomes.

Comment from Sally Catlin:

I have enjoyed working with ePortfolio and being mentored by Professor Fernandez. I have never organized a study in the past. It has been an enriching experience. I have learned a lot about obtaining participants, selecting student workers, working in a new field area and working with faculty from other disciplines

Comment from Eugenia Fernandez:

It has been a pleasure to guide Sally through the planning and execution of an empirical study. This project has helped me in my professional development as I progress from conducting research on my own to mentoring and working with other faculty on joint research projects.

It has also been a great learning experience to work with ePortfolio, especially as it continued to evolve as we worked with it.

Return to Final Reports page

AT&T Fellows Home Page