Image and video analysis theory and applications

MULTIMODAL PROBABILISTIC LEARNING OF HUMAN COMMUNICATION

Time: Tuesday & Thursday 3:30-5:20

TA:

Office: ICT 338 (by appointment only)

Introduction and Purposes

Human face-to-face communication is a little like a dance, in that participants continuously adjust their behaviors based on verbal and nonverbal displays and signals. Human
interpersonal behaviors have long been studied in linguistic, communication, sociology and psychology. The recent advances in machine learning, pattern recognition and signal processing enabled a new generation of computational tools to analyze, recognize and predict human communication behaviors during social interactions. This new research direction has broad applicability, including the improvement of human behavior recognition, the synthesis of natural animations for robots and virtual humans, the development of intelligent tutoring systems, and the diagnoses of social disorders (e.g., autism spectrum disorder).

computational modeling;

4. To learn about recent advances in machine learning and pattern recognition to analyze, recognize and predict human social and communicative behaviors;

Required:

Reading material will be based on published technical papers available via the ACM/IEEE/Springer digital libraries or freely available online. All USC students have automatic access to these digital archives.

● Nonverbal Communication in Human Interaction (7thedition), Mark Knapp and Judith Hall, Wadsworth, 2010

● Speech and Language Processing (2ndedition), Daniel Jurafsky and James Martin, Pearson, 2008

Introduction and communication models

Week 2 (17, 19 January)
Machine Learning reminder/recap: basic concepts

Week 3 (24, 26 January)
Study Design, Evaluation and Analysis
User studies (guest lecture by Gale Lucas - January 25) please complete this survey before this class
● Coder agreement, kappa
● Statistical analysis
● Student t-test, effect-size

Reading material
● McDuff, et al. Longitudinal Observational Evidence of the Impact of Emotion Regulation Strategies on Affective Expression. 2019.

Reading material
● Bachorowiski et al. Sounds of emotion: production and perception of affect-related vocal acoustics 2003

● (Optional) Schuller et al., (2011), Recognising realistic emotions and affect in speech: State of the art and lessons ..

Reading material

Week 8 (28 February, 2 March)

Verbal and Conversational messages

Reading material - bias and ethics
● McDuff et al., 2019
● Raghavan et al., 2019
● Yan et al., 2020

● Pang et al. EMNLP 2002: Thumbs up? Sentiment classification using machine learning techniques
● Soleymani et al. 2017, definition
● Tavabi et al, 2019
Week 15 (18, 20 April) - Assignment 3 is due
Health
● Multimodal behavior analysis in Health
● Mental health assessment 19 April
● Neurodevelopmental disorders
●
Neurodegenerative disorders Reading material - health
● Tavabi et al., 2020
● Cummings et al, 2015
Dyadic and Multiparty Interactions

Week 16 (25, 27 April)

Final project presentations

2. Krauss, R.M. (2002). The psychology of verbal communication. In, N. Smelser & P. Baltes (eds.), International Encyclopedia of the Social and Behavioral Sciences. London: Elsevier.

3. (optional) Morency, L.-P., Modeling Human Communication Dynamics, IEEE Signal Processing Magazine, September 2010

Study Design, Evaluation and Analysis

8. Gale M. Lucas, Jonathan Gratch, Aisha King, Louis-Philippe Morency, It’s only a computer: Virtual humans increase willingness to disclose, Computers in Human Behavior, Volume 37, August 2014, Pages 94-100.

Affective messages and personality traits

13. McDuff, Daniel, Eunice Jun, Kael Rowan, and Mary Czerwinski. "Longitudinal Observational Evidence of the Impact of Emotion Regulation Strategies on Affective Expression." IEEE Transactions on Affective Computing (2019).

17. (optional) Mr Barrick, Mk Mount (1991) The Big Five Personality Dimensions And Job Performance: A Meta-Analysis - Personnel Psychology

Vocal messages

21. (Optional) Ladefoged (2004), A course in phonetics

Visual messages

25. (optional) Adam Kendon, An Agenda for Gesture Studies, This article appeared in Volume 7 (3) of the Semiotic Review of Books.

26. (optional) Michael Argyle and Janet Dean, Eye-contact, distance and Affiliation, Sociometry, Vol. 28, No. 3, pp. 289-304, 1965

29. Ahuja, Chaitanya, Dong Won Lee, Ryo Ishii, and Louis-Philippe Morency. "No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures." In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 1884-1895. 2020.

Verbal messages

34. (optional) Jurafsky and Martin (2008), Speech and Language Processing, Sections 4.1-4.4, 5.1-5.3 and 12.1-12.2

35. (optional) Soo-Min Kim and Eduard Hovy (2004) , Proceedings of the COLING conference, Geneva

39. Bohus, D., Horvitz, E., (2010) - , Microsoft Technical Report MSR-TR-2010-115

40. (optional) Jurafsky and Martin (2008), Speech and Language Processing, Sections 17.1-17.4 and 21.1-21.4

43. I. McCowan, D. Gatica-Perez, S. Bengio, G. Lathoud, M. Barnard, M., D. Zhang,“”, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 27, pp. 305–317, 2005

44. (optional) Gros, Potamianos and Maragos (2008) Multimodal Processing and Interaction, SpringerLink, Chapter 1 [SpringerLink or USC blackboard]

47. Celiktutan, Oya, Evangelos Sariyanidi, and Hatice Gunes. "Computational Analysis of Affect, Personality, and Engagement in Human–Robot Interactions." Computer Vision for Assistive Healthcare. Academic Press, 2018. 283-318.

48. (optional) Deng, Eric, Bilge Mutlu, and Maja J. Mataric. "Embodiment in socially interactive robots." Foundations and Trends® in Robotics 7.4 (2019): 251-356.

52. (optional) K. Stefanov, J. Beskow and G. Salvi, "Self-supervised vision-based detection of the active speaker as support for socially-aware language acquisition," in IEEE Transactions on Cognitive and Developmental Systems, 2019.

53. (optional) X. Anguera, S. Bozonnet, N. Evans, C. Fredouille, G. Friedland and O. Vinyals, "Speaker diarization: a review of recent research," in IEEE Transactions on Audio, Speech, and Language Processing, 2012.

57. Poria, Soujanya, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. "Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research." IEEE Trans. Affective Computing 2020.

Dyadic and team processes

Cohn. "Dyadic behavior analysis in depression severity assessment interviews." In Proceedings of the 16th International Conference on Multimodal Interaction, pp. 112-119. 2014.

Bias and ethics

58. Tavabi, Leili, Anna Poon, Albert Skip Rizzo, and Mohammad Soleymani. "Computer-Based PTSD Assessment in VR Exposure Therapy." In International Conference on Human-Computer Interaction, pp. 440-449. Springer, 2020.

59. Cohen, A. S., Mitchell, K. R., & Elvevåg, B. (2014). What do we really know about blunted vocal affect and alogia? A meta-analysis of objective assessments. In

Attendance

Assignments

o These three assignments will be designed to give hands-on experience with multimodal data analysis and machine learning (e.g., SVM, neural networks) for multimodal behavior recognition.

o Students can perform the project individually or in a team of up to three. The mid-term and final report will need to outline the tasks of each participant. Team projects will be expected to include a deeper analysis than individual projects.

o Mid-term report: The mid-term report will present a qualitative analysis of the selected dataset and communicative behaviors. The report should include correct transcription and annotations of the language, vocal and nonverbal behaviors. Using standard statistical tools and qualitative observations, the students should highlight the challenges with this dataset (and communicative behaviors) and suggest an approach to solve them.

Statement on Academic Integrity

USC seeks to maintain an optimal learning environment. General principles of academic honesty include the concept of respect for the intellectual property of others, the expectation that individual work will be submitted unless otherwise allowed by an instructor, and the obligations both to protect one’s own academic work from misuse by others as well as to avoid using another’s work as one’s own. All students are expected to understand and abide by these principles. Scampus, the Student Guidebook, contains the Student Conduct Code in Section 11.00, while the recommended sanctions are located in Appendix A:
. Students will be referred to the Office of Student Judicial Affairs and Community Standards for further review, should there be any suspicion of academic dishonesty. The Review process can be found at:
.