Zhizhuo Yang

a) attend each class and help students do in-class activities
b) meet with Dr. Liu at least once per week to discuss the TA work
c) hold office hours to assist students with homework
d) grade homework assignments for 60 students (around 15 hours each assignment, 7 total assignments during semester. )
e) help proctor exams in case the instructor has a schedule conflict and provide additional grading support for course project/poster session if needed

I worked with Rakshit Kothari with this project. We used eye-tracking data such as gaze velocity and head velocity to classify Gaze-In-World (GIW) events. In the phase 1 of the project, we use scalar features as inputs, with the help of Forward-Backward window parallel LSTM deep neural network, we achieved very good performance. In the phase 2 of the project, we tried to use CNN model to extract features directly from eye images and then feed those features to the parallel LSTM model we used in the phase 1. This method is still under adjustment and fine tuning.

Passive haptics in an alternative way to provide the user with realistic haptic feedback with a low cost comparing to device-based active haptics. This paper presents a mixed reality system with passive haptics integration that supports bare-hand interaction for users to see and touch virtual objects in immersive virtual environment (IVE). With the educational potential, the purposed system is applied to geography teaching. The prototype combines feature-based visual tracking for user's head and objects in the real environment with free-hand motion detection via a depth camera, thus user can see both their hands and objects in the virtual environment. A study of virtual teaching of geography of Germany was conducted, 18 subjects participated in the formal experiments which include three different learning environments. The results showed that the participants learning in the virtual environment with passive haptics consume more time than the virtual environment with free-hand gesture in finishing geography teaching tasks which is contrary to the expected result. The reasons are discussed in detail. Furthermore, quality of memorizing geographical knowledge in the group with passive haptic feedback exceeds that in the group without.

This paper introduces a unique type of mixed reality system based on passive-haptic feedback, which enables its user to manipulate real objects with bare hands while seeing its counterpart in the HMD with more information attached. The system provides users in MR systems with a more realistic sense of touch and costs less money comparing to the active force feedback devices. In this paper, accuracy of tactile and visual perception in user interaction with real objects in close range MR environment is studied. A calibration method of the user's head and hands is proposed and the calibration accuracy of head and hands tracking has been evaluated respectively with different experiments. An application of chemical experiment is also implemented to prove the reliability of the proposed system.