AT&T Fellows Final Report: Sara Anne Hook

AT&T Fellows Final Reports

March 2006

Name	Sara Anne Hook
Title	Professor of Informatics and Associate Dean for Academic Affairs and Undergraduate Studies
Department	IU School of Informatics
Campus	IUPUI
Project Title	Managing, Extracting and Assessing Data from Discussion Forum Postings in Online Courses
Project Goal	The proposed project will employ the services of SIFTER to develop a method and algorithms for counting, sorting, categorizing and rank ordering individual postings for discussion forums. The SIFTER will include a front-end management template that will allow courses and instructors to be added. It will be written in JAVA so that it can be implemented at any institution. In order to maximize the capabilities of SIFTER, the investigator will need to standardize her questions across modules. The investigator will design, develop, implement, test and evaluate the SIFTER in her N480 Technology and the Law course in spring 2006. This course was taught in spring 2005, providing an excellent opportunity for comparison. Data gathering for the project will be automatic through the SIFTER system and can be compared with the paper-based and labor-intensive process that was used to count, read, assess and respond to discussion forum postings from the spring 2005 version of the course.
Type of Technology Used in the Project	The broad aim of SIFTER is to develop information agents based on advanced concepts and techniques from the domains of artificial intelligence, information retrieval and distributed object computing. The basic functions supported by an agent include culling information from complex resources that may reside in diverse locations and conducting analysis, synthesis and customization according to dynamic requirements of the user. A primary objective of SIFTER is to create a robust architecture for reactive, proactive, and collaborative filtering. (For more information about the SIFTER project, please see http://sifter.indiana.edu/about.shtml.) In order to maximize the capabilities of SIFTER, the investigator needed to standardize her discussion forum questions across modules. The program was written in JAVA so that it is compatible with OnCourse and OnCourse CL and thus can be offered to others teaching in the Indiana University system and beyond.

Executive Summary of Results

This project evaluated the use of SIFTER to develop a methodology and algorithms for counting, sorting, indexing and rank ordering individual student postings to weekly discussion forums for one of the investigator’s online courses that was taught in spring 2006. The purpose of the project was to analyze whether SIFTER could be designed to extract, manage and assess the content of student postings to a weekly discussion forum in a way that maintains student-to-instructor interaction, streamlines the management of the process for faculty members and assists in the measurement of student learning.

Need for the Project

Faculty members are encouraged to develop courses that can be delivered online or through other distance education technologies. Although faculty members are mindful that this will attract more students to their courses and reduce the demand for classroom space, they remain fearful of the considerably increased time commitment required for teaching online. Faculty members are particularly wary of the avalanche of emails and discussion forum postings that must be monitored and answered in a timely, organized and regular fashion. Research has shown that one major predictor of student success and satisfaction in online and distance education courses is student-to-instructor interaction. In other words, what faculty fear most about teaching online – reading and responding to student emails and weekly discussion forum postings – is actually one of the most significant ways to ensure a positive outcome for students. Reading and responding to these postings is time-consuming, as is keeping track of which students have submitted postings for each week, yet this is an essential part of measuring student learning, particularly in an online course where there is no face-to-face interaction. The literature attests to the discussion forum as a major feature of most online courses and the importance of preserving student-to-instructor interaction. However, the literature does not reveal widespread efforts to use the power of data mining, text mining or natural language processing in the management of this important part of online courses.

Use of Technology

This project evaluated whether SIFTER could be developed to help faculty members manage the discussion forum feature within OnCourse.SIFTER is collaboration between faculty and students in the Computer & Information Science Department, the School of Informatics and the School of Library & Information Science at Indiana University. Some of the objectives of SIFTER include dealing with heterogeneous information sources that may change over time, handling shifting requirements and interest of users with minimal user involvement, creating robust agent architecture for reactive, proactive and collaborative filtering and supporting agent collaboration based on natural or artificial economic frameworks for multi-agent tasks. Many of SIFTER’s applications to date have been in information representation and classification, filtering and modeling with an emphasis on biomedical information. This project looked at SIFTER’s capabilities as a course management tool.

Instructional Design Plan

Participation in weekly discussion forums is not only a characteristic of most online courses, but the investigator has found thestudents who posted regularly and provided the most thorough and thoughtful responses to the discussion forum were the ones who achieved the highest grades in her courses. The investigator asks four types of questions in her weekly discussion forums. The first few questions deal with content and the next questions ask students to reflect on how a particular topic or issue will have an impact on their career. These questions help the investigator assess whether students understand the material presented in the module and the reading assignment and whether they can apply it to a real-world situation. She then asks whether the students had any myths or preconceived notions that were dispelled by the module and the reading assignment. Finally, the investigator asks students if the module included any issues or topics that were particularly interesting or helpful. These questions allow the investigator to determine whether the course is making a difference for students. Comments such as “I never realized that” or “before this module, I thought that” help identify specific topics that were especially valuable and which could be explored more fully in subsequent semesters. However, reading and responding to these postings is time-consuming, as is keeping track of which students have submitted postings for individual discussion forums. A method of sorting and rank-ordering these postings would not only be extremely helpful for the investigator’s courses, but could be replicated across the university. The project will also yield information on what kinds of questions are compatible with a SIFTER system. There is considerable interest in the development of this kind of a tool because it could be deployed in online courses and even in face-to-face courses that make extensive use of OnCourse’s email and discussion forum features.

The resources used for the project were SIFTER and a graduate student from the Computer & Information Science Department.

The project is easily understood by other faculty members and could be replicated for nearly any course.

Potential to Impact Student Learning

The focus of the project was on the management of discussion forum postings, a component of nearly every online course and essential to providing the kind of instructor-to-student interaction that students require for success and satisfaction in their courses and in overall retention. Discussion forums can be used with any size class, which the investigator demonstrated by teaching an online course with 77 students with more than 90% retention. With discussion forums as a common feature of online and even face-to-face courses at all educational levels, the opportunity to use SIFTER to manage postings has broad applications in universities as well as K-12 and community environments.

Assessment Plan

The same course, N480 Technology and the Law, was taught in spring 2005, making it easy to compare the overload of discussion forum postings in printed form with the opportunity to search and sort responses in spring 2006. Also, the project offered an opportunity to standardize discussion forum questions over several modules and to evaluate which kinds of questions worked best with SIFTER.

Plan for Colleague Development

Hook, S.A. and Salandri, M. Are They REALLY Getting It? Extracting and Assessing Data from Discussion Forum Postings in Online Courses as a Measure of Student Learning: Interim Report [poster]. E.C. Moore Symposium, IUPUI campus, February 24, 2006.

Hook, S.A. and Salandri, M. Are They REALLY Getting It? Extracting and Assessing Data from Discussion Forum Postings in Online Courses as a Measure of Student Learning: Final Report [poster]. The Seventh Annual Midwest Conference on the Scholarship of Teaching and Learning, Indiana University South Bend, April 7, 2006.

Salandri, M., Palakal, M., Hook, S.A. Managing and Extracting Data from Discussion Forums in Online Courses (in preparation).

As Associate Dean for Academic Affairs and Undergraduate Studies at the Indiana University School of Informatics, IUPUI, the investigator has encouraged faculty members at her school to develop courses that can be delivered online or through other distance education technologies. She finds that faculty members remain fearful of the greater time commitment required for teaching online. Their concerns are not trivial, since it has been estimated that teaching online takes as much as three times more time than teaching face-to-face. The investigator is sympathetic to these concerns. During the spring 2005, she taught two courses online, one with 13 students and the other with 77 students. Overall, the investigator teaches five online courses per year and she is scheduled to develop more online courses in the future. Using SIFTER can provide one way for faculty members to organize and have some measure of control over the management and evaluation of discussion forum postings.

There is considerable interest in the development of this kind of a tool because it could be deployed in online courses and even in face-to-face courses that make extensive use of OnCourse’s email and discussion forum features. The investigator has shared her ideas with Professor Anastasia Morrone, Executive Director of the IUPUI Center for Teaching & Learning, and there is great enthusiasm for the project because of its potential to be implemented throughout the university and beyond.

Final Comments on Project Results

The main goal for the project was met, proving that SIFTER could be used to ease the burden of managing and responding to discussion forum postings. The front-end management templates have been completed and can be used with other courses. Postings to the discussion forum can be analyzed by question or by keyword. It is also possible to limit retrieval to a certain amount of text around each keyword. A series of standardized questions have been developed and tested for their effectiveness with SIFTER. Data sets from several weeks of course modules are available for comparison.

More refinement and functionality is needed in the ability of SIFTER to sort and categorize by keywords. It would be most helpful to be able to combine a series of keywords into a more advanced search strategy and to be able to sort the postings even further. The value of SIFTER as an assessment tool should also be examined more fully. Currently, the investigator assigns grades for participation in the discussion forum based on whether a student has posted to all or only some of the forums throughout the semester. Ideally, she would like to be able to manipulate SIFTER in a way that would help her to determine a grade for discussion forum participation based on the quality of the postings rather than the quantity.

One negative aspect of the project was the difficulty of finding and keeping a graduate student to work on this project. Because of the nature of the project, the graduate student had to be familiar with SIFTER and the technical part of the project was under the purview of the Computer & Information Science Department. It was an awkward arrangement and many weeks passed when there was no graduate student to work on the project. Fortunately, the overall goals of the project were met.

In spite of the extra workload entailed by teaching online, the investigator remains committed to this approach towards distance education and is enthusiastic about ways that the process and established methodologies can be enhanced using technology tools such as SIFTER.

Return to Final Reports page

AT&T Fellows Home Page