Ph.D Position AI and Neuroscience

ED 269: Mathématiques, Sciences de l’Information et de l’Ingénieur [ED269]

Laboratoire des sciences de l’ingénieur, de l’informatique et de l’imagerie [UMR 7357]

Understanding human behavior is important for various applications, including healthcare, education, and human-computer interaction. Traditional methods for analyzing human behavior often rely on subjective observations or limited data sources. However, recent advancements in technologies producing diverse data streams offer promising opportunities for more accurate and comprehensive analysis.

This research project aims to develop an advanced framework capable of analyzing and learning human behavior by leveraging data from multiple sources such as eye tracking, inertial measurement units (IMU), blinks, fixations, video recordings, and neuroscience-based annotations. The proposed framework seeks to provide deeper insights into behavioral patterns and underlying cognitive processes.

The primary objectives of this research project include:

Development of a comprehensive framework for human behavior analysis by integrating data from various sources, including eye tracking, IMUs, blinks, fixations, video recordings, and neuroscience labeling.
Exploration of methodologies to process and analyze diverse data streams effectively.
Application of the developed framework to real-world scenarios to evaluate its effectiveness and potential impact.

In collaboration with neuroscientists at Fribourg University, access to datasets collected from diverse scenarios using advanced equipment—such as eye-tracking devices, IMUs, and video recordings—will enable a holistic perspective on human behavior and allow for more precise analysis.

During the initial phase of the PhD project, the candidate will explore various techniques for processing and analyzing data from multiple sources, including approaches based on pattern recognition, sequence modeling, and graph-based representations. Each technique will be adapted to handle the complexity of heterogeneous data and optimize performance.

The developed framework will be tested in real-world settings, considering factors such as scalability, reliability, and ethical implications. Potential applications include personalized healthcare solutions, adaptive learning systems, and intelligent human-computer interfaces.

Research on this topic has already begun with multiple M2 internships to establish foundational tools for analyzing various behavioral data sources.

The anticipated outcomes of this research project include: (1) A robust framework for analyzing and understanding human behavior from multiple data sources, (2) Insights into the effectiveness of different data analysis techniques, (3) Potential applications across various fields such as healthcare, industry, education, and human-computer interaction.

The PhD candidate will focus on:

Processing and representing data while investigating existing analysis techniques.
Developing new methods for data representation and analysis.
Discovering behavioral patterns by analyzing diverse input data sources.

The PhD candidate will work on three sites, ICube, Icam Strasbourg-Europe in Schiltigheim, and Fribourg University.

References:

Summaira Jabeen, Xi Li, Muhammad Shoib Amin, Omar Bourahla, Songyuan Li, and Abdul Jabbar. 2023. A Review on Methods and Applications in Multimodal Deep Learning. ACM Trans. Multimedia Comput. Commun. Appl. 19, 2s, Article 76 (April 2023), 41 pages. https://doi.org/10.1145/3545572
Tang, Qin & Liang, Jing & Zhu, Fangqi. (2023). A Comparative Review on Multi-modal Sensors Fusion based on Deep Learning. Signal Processing. 213. https://doi.org/10.1016/j.sigpro.2023.109165
Guang Yang, Qinghao Ye, Jun Xia, Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond, Information Fusion, Volume 77, 2022 https://doi.org/10.1016/j.inffus.2021.07.016
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives, 2023 https://doi.org/10.48550/arXiv.2311.18259
Deng, R., Gao, Y. A review of eye tracking research on video-based learning. Educ Inf Technol 28, 7671–7702 (2023). https://doi.org/10.1007/s10639-022-11486-7
Yuhong Z., Shilai Y., From (2024) Word Embedding to Reading Embedding Using Large Language Model, EEG and Eye-tracking https://doi.org/10.48550/arXiv.2401.15681
Yuhong Z., et al. (2023) Integrating LLM, EEG, and Eye-Tracking Biomarker Analysis for Word-Level Neural State Classification in Semantic Inference Reading Comprehension https://doi.org/10.48550/arXiv.2309.15714
Santosh Kumar Yadav, Kamlesh Tiwari, Hari Mohan Pandey, Shaik Ali Akbar, (2021) A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions, Knowledge-Based Systems, https://doi.org/10.1016/j.knosys.2021.106970
Shaoyue Wen et al. (2025) AdaptiveCoPilot: Design and Testing of a NeuroAdaptive LLM Cockpit Guidance System in both Novice and Expert Pilots https://doi.org/10.48550/arXiv.2501.04156