Final Treatise

After two rounds of revision, I finally handed in the treatise, which is the biggest assessment for this project, on Wednesday.

The main body consists of 6 chapters -- the Introduction, Literature Review, Design, Analysis, Discussion and Conclusion. It also uses some graphs and tables to help illustrate the problem. The treatise shows what I have done and what I have learnt through the whole project from March to October.

Now, I will start to prepare the presentation on November 12th or 13th.

Draft Treatise

This week, I have been working on the draft version of the treatise, and submitted the soft copy online this afternoon.

Although I got several separated documents which I edited during the process, it still took me a great efforts to write the treatise. Despite of the compulsory description of experiment design, implementation and analysis, I would like to focus on the learning process and trade-off decisions I made during the whole project. Thanks to all the documentations, notes and online blogs, it was not too hard to recall all details in each phase.

Up to now, the draft treatise includes the introduction, literature review, experiment design, implementation work and part of the analysis. Next week, I will continue to work on the draft and want to finish the main body before Thursday evening.

Data Analysis 2.2

This week I finished the further analysis of symmetry of m by adjusting the height of the line, to move it up and down a bit. I also test the feature on external dataset as well. The result is similar to what I got from the middle line.

Then I organised the results and start to edit the draft treatise. I discussed with the supervisor about the outline, and got a clear idea of what to write in each part.

Data analysis 2.1

This week, I finished the first letter specific feature's analysis -- symmetry for letter m.

I took these steps to extract the feature

Step 1: scan the file, and if the gesture is m, find the bounding box and compute the middle line function y = b

Step 2: pass the b value (from step 1) and the corresponding list of points to find all crossing points, and store them in another list - cross point list. If two adjacent points Point 1 and Point 2 are on different sides of the line y = b, I consider the second point as a crossing point.

Step 3: filter some of the gestures according to the number of crossing points.

            Only 1 crossing point: maybe a gesture that was not logged properly

Too many crossing points (e.g. 12 crossing points in Gesture 31 from Subject 10): the subject wrote two m…

Step 4: compute length of segments, and check which is on the left and which is on the right.

Step 5: get the ratio between two segments. Here, I use right_segment_length / left_segment_length, and compute average value for each level

The result shows that there are only significant differences in limited conditions. I think this is a potential feature, but still need adjustment to investigate in more details.

Data Analysis 1.3

I finished statistical tests for basic features like dimensions of bounding box, pen pressure and length of the gesture. I mainly use statistical methods including t-test and ANOVA. The results are not as good as expected, but I still found something. For example, the gesture length in the session 1. However, how to combine my result with external dataset and gave a generalised explanation is not easy to do.

In the next phase, I will focus on the letter-specific geometrical features for letter a, g and m.

Data Analysis 1.2

Today, I added the what subjects wrote during the experiment to the logging files. As it was almost impossible to collect what they wrote on-fly without recognition package, I took letters down by observing them writing. However, the problem is that I have to add the letter tag manually in the logging to facilitate further analysis.

It took me longer time than I expected, and I also encountered several formatting problems. Finally, all the gesture has the corresponding letter tag at the end of the gesture. It helps me to understand the importance of constructing logging files. If the format is not right, it may bring a lot of problems in the future stages.

Data Analysis 1

I have finished the user experiment on Tuesday, and started the first phase of data analysis.

First of all, I computed the accuracy of all questions to get an overview of the task performance. Then, I did the Friedman ANOVA on the subjective feedback to validate my experiment design (i.e. difficulty for each level). 

In the next step, I'll continue to analyse the features I've tested on the external dataset, and focus on the letter specific features.

Conduct the Experiment

This week, I conduct my user experiment in the lab. Up to now, I have finished 7 out of 12, and all experiments are expected to be finished before next Tuesday. Since I've got some experience from the summer project, this time everything goes smoothly.

Next week, I will start the analysis of my dataset, and compare some of the result from the external dataset. I think it would be good to get some inspiration from my dataset and in turn do further analysis to the external dataset.

Experiment Implementation 5

This week I improved my implementation according to the result in the pilot study and finalised the experiment on Wednesday. I also finished booking subjects, and the experiment will start on next Monday. It will take about 1 hour and 10 minutes per subject, including 10 minutes instruction explanation and training time.

As the logging format is similar to the external dataset, so I think after the experiment, I could modify previous data analysis scripts to get some basic feature results. However, I still need explore more "letter specific" features before the end of the analysis.

Experiment Implementation 4

Today, I do a formal pilot study to tune the parameter of the experiment, such as time arrangement, task difficulty and settings of the tablet.

First of all, I increased the task difficulty a bit, which I think could make the task harder to complete than before. Then, I also adjust some time arrangement, like resting time between two sections and between two figures as well. Finally, the angle of the tablet is tested by three separate pilot runs. Taking two factors into consideration -- the average pen pressure and the proportion of points which reached the pressure limit, I adjust the angle to be 30 degrees from the vertical plane.

Additionally, I also finish the consent form and the subjective ratings.

Experiment Implementation 3

I think the experiment implementation has been finished, except for the pictures will be used in the experiment. I've drawn draft ones, but the size and layout of them needed to be fixed. Additionally, I edited subjective questions and summary table to be filled in during the experiment.

Tomorrow, I will do further pilot on task difficulty and time arrangement, and hopefully I can start booking subjects. According to the timeline, the experiment will take place on next Wednesday. 

Experiment Implementation 2

During the implementation, the most challenging part for me is the user interface. In my project, I use Java Swing to realise the interaction between the user and the machine, and special logging package to collect the experiment data.

Currently, the main problem for me is how to display the gesture feedback on the screen in real time. I have to get familiar with GlassPane as soon as possible, and then apply it in front of other panels to show the trace to the subject.

Next week, the time arrangement for each part will be tested in the pilot study and the experiment implementation is hopefully to be finalised before the end of next week.

Experiment Implementation 1

There are several parts in the implementation, including program for main tasks, pictures used in the task and the subjective rating. As I have finished the experiment description, it will be relatively easy to follow those steps to finish the implementation.

Currently, I have some issues in choosing the platform, so I need to test the compatibility of two different WACOM tablets first.

Experiment Desgin

I finished the design and documentation of my experiment v1.0. Before the end of this week, I will finish the pictures used in the experiment and the main implementation.

Data Analysis 1.3

I did further data analysis about the average gesture pressure and gesture length. For these two features, I also used the normalisation to to eliminate the effect on the value from the target letter, e.g. the shape or the way to write that letter.

1. Pressure: The mean value indicates that the pressure tended to be larger in the high load session than the normal session, but the p-value is greater than 0.05, which means the differences between two sessions are not significant.

2. Gesture Length: in Lisa's paper, she noticed the significant effect of cognitive load on gesture length. In my analysis, I added the normalisation step, and the p-value is still less than 0.05. It means there is a significant differences between average gesture length in the high load session and normal session.

In the next stage, I'll do more analysis related to the gesture length, and try to come up with more features.

Data Analysis - 1.2

I finished the width/height ratio normalisation, and then I did the t-test to compare the new normal and stress results. There are no significant differences between them, with p-value (two-tail) > 0.05. Although this feature fails to be a good feature, I learnt a lot from the normalisation process. What is more, I think this idea can be used in the future analysis as well, because the differences between letters may also have other impacts on the gestures.

Data Analysis-1.1

After discussing with my supervisor, he suggest me to normalise the width/height ratio for different letters, so that it can eliminate the effect of the generic shape of the letter. Then, I will explore more about the bounding box, to understand it in more details and compare my results with the experimenter's results.

In the further analysis, I need to use the actual written letter to group gestures, instead of the automatic recognition, so I need another field in the Stroke called "written". Since I have already summarised all the recognition errors, I will use the list to correct recognition result for each gesture. 

Data Analysis - 1

I would like to start the analysis from an easy and straightforward feature -- bounding box. First of all, this week I wrote some scripts to abstract bounding box information from processed data (one file for each subject), like the width, height, and width over height ratios. Then, I computed average values for different sessions (Normal and High). I will use t-test to compare two groups of results.

However, there are some issues I want to discuss with the supervisor. The most important one is the letter's shape has significant effect on the bounding box, and I am not sure whether I can compare the result. Additionally, I am not sure how to deal with slant, which the stoke is not vertical to the plane, because slant stroke will change the dimension of bounding box.

Progress Report - 2

I have finished the progress report, and it helped me to summarise what I have done in this semester, especially the papers I have read. 

At the end of the report, I attached a new Gantt Chart, adjusting the time allocation of the rest tasks. There are really a lot of things to do in the next stage, as my data analysis and experiment implementation are a bit behind the schedule.  As to the literature review, I still need more articles about the pen-input features.

Progress Report

Today, I started editing the progress report. It will cover the project introduction, literature review, current progress, future work and revised project plan.

For the most important part -- literature review, I would like to summarise the notes I took in the previous two months, and introduce the fundamental theory under my project, including working memory model, cognitive load theory, measurement and research on the pen gesture. The current progress will cover the data analysis of the external dataset, and my experiment design version 1.0.

My aim is to finish this report before the end of next week, and discuss with the supervisor for any issues in it.

Some ideas about the experiment design

Today, I discussed experiment design with my supervisor. Currently, I designed three different tasks, which are graph transforming task, counting task and math operation task.The tasks involve recalling, applying rules and processing information. They use number of rules, time limit and amount of information to manipulate the cognitive load levels.

As my hypothesis for this experiment is related to the cognitive recovery time, so an important point in the experiment is how to control the task order and rest time between them. Now, I haven't come up with the detailed plan, but I would do more research on this topic and conduct some pilot study on it in June or July.

The supervisor also reminded me that I need to think about how to make the task adaptive to different people according to their real-time performance. 

Features for data analysis

About plotting, I added labels, including the gesture number, cognitive load level and recognise result, on each gesture to show the attribute clearer. 

Also, I reviewed two papers discussing the gesture features. They are all about the geometrical features of a gesture, ranging from simple ones like length, bounding box, and some complicated ones curvature and sharpness. In the papers, these features were used in handwritten recognition or input mode detection. They gave me some ideas about the feature selection, and I will search for more papers in the topic "online input recognition" (--> general features) and "handwriting verification" (--> subjective features). For my study, the features should be sensitive to the cognitive load and easy to test. I think the sharpness and other features related to angles between certain points might be potential good features, and I will summarise and test them in the next stage.

Preview of the Dataset - 2

Today, I finished the plot of the gesture, from which I got a clearer overview of the dataset. The python scripts were used to plot all the gestures according to the gesture position (x, y) and cognitive levels. The original size of the letter (about 4000 * 4000) is too big to display on the screen, so I zoomed them out to fit all the gestures from one subject into one screen (2056 * 1024). The reason why I arranged them in one screen is to make it easier to find potential features by comparing all the gestures at the same time. Currently, the gestures are ordered by time.

The next step for me is to look into those gestures and try to find interesting and valuable features to test. And another thing to be finished before the end of the next week is to summarise the frequently used features for further reference. 

Preview of the Dataset

Today, I did some preprocessing of the gesture data.

With the help from the supervisor, I removed all the blank line in the log, and I also checked the integrity of the data from three files by the userId and timestamp.

Currently, I have some questions to think about. Firstly, the priority of the research, which feature should I start from? Then, every gesture has about 50 points collected, how to consider those points as a whole, and looking for the trends of them.

I need to write data parsing scripts these days and before the end of the May, the analysis of some basic features should be finished.

Literature Review

Today, I continued the literature review. The focus is on the experiment design.

The paper Reducing Cognitive Load by Mixing Auditory and Visual Presentation Modes (Mousavi, Low & Sweller, 1995) introduced 6 experiments, which tested the hypotheses in levels.Experiments are designed according to the results from the previous experiments, to test some new hypotheses, so that at last, they could explain the whole thing more thoroughly and clearly. Although, for the honours project, the time may not allow me to do more than one experiment, it is a good idea to apply in the further study.

Another paper, Free-form Pen-Input as Evidence of Cognitive Load and Expertise (Ruiz, Taib & Chen, 2011), explored the use of the scratchpad and the cognitive load. I think the reason for why pen input needs to be researched is quite important for me. It provides background information for my project.

Preparation of data analysis

Today, I installed SPSS, the software for data analysis on my machine. I think I should follow the steps I went through in the summer project.

To begin with, I will have an overview study of the dataset. Understand what is stored in each column and then write some Python scripts to do preprocessing, e.g. revise data formatting, compute some intermediatory results. I think this first phase of data analysis could be finished before the end of next week. Then, I need to select what tests to do, and import data to SPSS to do them..

Meeting

This morning, my supervisor and I had a meeting with the owner of the external dataset, which I am going to use. She introduced the background of their user experiment and the basic information about the dataset. She also explained some questions about the design of the experiment, especially the different tasks they used.

I will start the analysis of pen gesture data from this dataset this week, and this task will be paralleled with my experiment design. 

I also had a meeting with the supervisor, discussing the idea about the experiment design. Last week, I read several papers about n-back recall task. Today, we briefly analysed the advantages and disadvantages about it.

Proposal 3

I revised the proposal according to the suggestion from the supervisor and finalised it today. 

In the proposal, I clarified the scope and made a detailed plan. I think it could guide me to finish the whole process.

This is my Gantt chart, which has more tasks than my previous plan.

Proposal 2

Today, I wrote the proposal and finished most of the parts, except part of the procedures, the schedule and reference list. Tomorrow, I will continue to work on the proposal and I plan to finish drawing the gantt chart, which can clearly show my weekly tasks.

However, for what I have written today, I still have some questions which need to be discussed with the supervisor:

1. Are all the assumptions I have made reasonable? Are there any more assumptions I should make to define the project scope more clearly?
2. The description of the procedure.
3. Expand current plan to show weekly tasks.

Despite the proposal, I will also read more about the extraction of pen gesture features.

Proposal

Today I had a meeting with the supervisor about writing the proposal.

Before the meeting, I went through the evaluation criteria from the School 

http://ee.usyd.edu.au/project/UserFiles/File/Evaluation_of_a_Treatise.pdf and internal supervisor Rafael's requirements. I had some questions about the criteria from the uni -- how to write sampling and data gathering methods... at this stage?

In the meeting, we followed my proposal outline and he clarified what is expected for each section. We also exchanged some basic thoughts about the experiment design, and the scope of my research project. About my questions mentioned above, the suggestion from the supervisor was to make it clear what issues I would consider in the sampling and data gathering, so that it can prove I understand them.

After the meeting, I got a clearer idea and direction for the next two weeks. I planned to finish the draft proposal by next Tuesday.

Literature Review - 4

Today, I reviewed some papers on pen input in details. I want to get some inspiration for my experiment design from those papers. I summarised their experiments in the table.

From my opinion, the second one which asks participants to compose sentences from the given words is too complicated, especially for the recognition part. As to the math problems, it also has too many variables in the experiment. What is more, the aim of this experiment is to help to improve the student learning process, so it is not very close to my purpose. Therefore, I prefer the map-task and target-finding.

My issue is that it is still unsure whether I could get access to the external dataset on time. Therefore, I am preparing to design some simple experiment.

Literature Review - 3

Today I read through parts of an online tutorial  named Federal Aviation Administration (FAA) Human Factors Awareness Course, which covers some branches of Human-Computer Interaction. Specifically, I read Human Factors Model and Cognition sections in details. If I can get access to the research dataset to analyse the cognitive states of the team working, I will also go through the last section -- Team Performance.

In the sub-section introducing working memory, it used the Wicken's Model, which stresses a different aspect than Baddeley's Model. It focused on the human information processing, starting from outside stimuli to the perception and response mechanism. In this model, working memory plays a critical role in the selection of decision and response. I think this model can be used to explain some performance in the actual experiments.

* The picture is from http://www.hf.faa.gov/Webtraining/Cognition/CogFinal008.htm

Then, I also read a paper "Using Pen Input Features as Indices of Cognitive Load" from our research group, which is very close to my topic. It helped me to get a brief idea of how to collect pen input and use the Malahanobis distance (MDIST), a weighted Euclidean distance, to measure the degeneration of shapes, which could be an indicator of cognitive overload.

The supervisor reminded me to think about my experiment design within these two weeks, so I will find some papers on the experiment collecting pen input.

Reference
http://www.hf.faa.gov/Webtraining/Cognition/CogFinal008.htm

Ruiz, N., Taib, R., Shi, Y., Choi, E. & Fang, C., 2007, Using Pen Input Features as Indices of Cognitive Load, 9th international conference on Multimodal interfaces, USA

Literature Review - 2

Today, I had a meeting with the supervisor discussing the direction of the literature searching.

The supervisor recommended some important conferences to me, like ICMI (International Conference on Multimodal Interfaces), CHI (ACM Conference on Human Factors in Computing Systems), some workshops and a local forum OzCHI. Additionally, it would also be useful to browse some pages for the leading teams working in the same research areas.

We also talked about some concerns of the published time of the papers, because some papers are written about ten years ago. I was not sure whether I should read them. From the Research Method Lecture in the uni, it recommended the papers published within about three years. Of course, for the basic theories, the published time does not matter very much,  but if I do want to know the current condition of the research area, I have to find more up to date ones. Finally, I decided to pay more attention to the papers from conferences held in 2011.

Literature Review - 1

Spending three months on a summer research project, which is also about the measurement of human cognitive state, I've collected some famous, important and basic papers in this area. 

In order to lay a more concrete foundation, I decide to reread some very important papers in this week, and for those which are also relevant but I didn't have enough time to read in details before, I would spend more time on them this time.

The basic theories behind this research project is Baddeley's Working Memory Model and the Cognitive Load Theory (CLT). First of all, I reread Alan Baddeley's Working Memory (1992), which discussed the central executive and the two subsidiary systems -- phonological loop for auditory information and the visuospatial sketch pad for the visual information in the working memory. I also searched the later paper of Baddeley published about ten years after the previous one. It introduced another component in the working memory called episodic buffer, which holds the temporary multimodal code.

Then, I also reviewed another paper talking about the cognitive load measurement (Fred Paas, et al., 2003). This paper explained the definition of cognitive load theory and three main measurements. If I am going to design my own experiment, I think the task performance and subjective rating are the top choices.

Reference

Baddeley, A., 1992, Working Memory, Science, 1992 255: 556-559

Baddeley, A., 2000, The episodic buffer: a new component of working memory, Trends in Cognitive Sciences, Vol. 4., No. 11

Paas, F., Tuovinen, J., Tabbers, H. & Van Gerven, P., 2003, Cognitive Load Measurement as a Means to Advance Cognitive Load Theory, Educational Psychologist, 38(1), 63-71

Project Plan

Today I had a meeting with the supervisor about the project plan. Considering the due date of some assessments of the honours thesis course and the uncertainty of whether I can get the access to the dataset from the external collaborator, we made two versions of draft plans as below.

With the external dataset

    1. Literature Review and Feature Selection

      Mar 12th -- April 13 th (Topic Proposal due)

    2. Data Analysis (Statistics) and Progress Report 

      April 16th -- May 30th (Progress Report due)

    3. Data Analysis (Machine Learning) and Small-Scale Experiment 

      June 1st -- August 31st (Inc. the break)

    4. Final Analysis (Machine Learning)

      September 3rd -- September 26th

    5. Conclusion and Documentation

      September 27th -- October 22nd (Treatise due on October 24th)

Design and Implement my own Experiment

    1. Literature Review and Feature Selection

      Mar 12th -- April 13 th (Topic Proposal due)

    2. Experiment Design, Implementation and Progress Report 

      April 16th -- May 30th (Progress Report due)

    3. Pilot Study and Actual Experiment

      June 1st -- August 31st (Inc. the break)

    4. Data Analysis (Statistics)

      September 3rd -- September 26th

    5. Conclusion and Documentation

      September 27th -- October 22nd (Treatise due on October 24th)

This plan with all the milestones inside can help me to do the time management more efficiently. I also confirmed with the supervisor that I would work on the project in NICTA two days per week and have at least one meeting every week with him.

A New Start

This is the start of my honours project -- Multimodal Learning Analytics, under the supervisions from Ronnie Taib (NICTA) and Rafael Calvo (EIE).

My project is to research into ways to estimate human cognitive state and capability, unobtrusively and in real-time. The result can help to optimise the human decision support systems, and adaptive user interface as well. During my honours, I will work with one or more data sets offering a range of features such as speech, task performance or digital pen input. Among them, I want to focus on the digital pen input and task performance. I may also be required to design, implement and carry out a user experiment targeting specific aspects of pen interaction under varying levels of cognitive load.

There are also several research goals:

• Identify changes in task performance and digital pen input patterns in individual and team behaviour to distinguish different cognitive load levels.

• Explore most predictive behavioural patterns, within and across modalities, and within and between subjects for cognitive load assessment.

• Improve the ability to plan and undertake a research project