1. Objective Hearing Aid Fitting Based on Ear-Near Cortical Signals
This project aims to develop an objective method for fitting and incidental automated adaptation of hearing aid systems by analyzing cortical responses to speech. Incoming speech signals evoke measurable cortical representations, which are necessary for understanding but not sufficient on their own. In this project, the cortical responses to speech will be captured by Electroencephalography (EEG), and the team will compare two alternative representations of speech in EEG – the traditional speech envelope and a Mel-scaled spectrogram – for their efficacy in predicting individual speech comprehension. The Mel-spectrogram, which captures features like fundamental frequency and formants, has shown advantages in distinguishing speech stimuli. The study will involve both normal-hearing participants and hearing aid users, correlating subjective speech understanding under various fitting conditions with objective EEG measurements. Crucially, the project will compare standard 32-64 electrode EEG caps with a reduced, ear-near electrode array (e.g., cEEGrid) to assess the feasibility of a practical, wearable system.
2. From Signal to Communication: VLA-Based Communication with a Humanoid Robot under Simulated Hearing Impairments
This project addresses the challenge of training healthcare professionals and caregivers to communicate effectively with people who have hearing impairments. Currently, systematic training is only possible with actual hearing-impaired individuals. This project will develop a simulator using the humanoid robot Ameca and a virtual avatar to represent a person with varying degrees of hearing loss. The system will process a speaker’s voice by (1) applying parametrizable, hearing-loss-specific filters, (2) converting the degraded signal to text via Automatic Speech Recognition (ASR), and (3) using a Large Language Model (LLM) to generate a response. A novel aspect is the integration of the visual channel (lip movements, gestures) via Vision-Language-Action (VLA) models to compensate for the reduced acoustic bandwidth. The resulting demonstrator will serve as a training tool for medical, nursing, and audiology professionals, as well as for public education.
3. Hearable Tracking of Everyday Communication: A System for Assessing Spoken Communication Effort in Real-Life Contexts
This project aims to develop a hearable-based system to continuously track and analyze daily communication, quantifying both speaking and listening effort in real-world settings. Using biosignal sensors, the system will capture the acoustic scene to segment speech, noise, and silence, and identify recurring conversation partners to estimate social network diversity. To assess listening effort, the project will use neural sensing (e.g., mobile EEG) to predict when a person is actively attending to a speaker. Key methodological questions include sensor selection, using context to improve accuracy, on-device data processing for privacy, and developing metrics to detect “communication deprivation” (i.e., social isolation or withdrawal). The system will integrate multimodal biosignal and context data (audio, motion, environmental noise) based on machine learning methods to create individual communication models and profiles.
4. Value-Sensitive Design of Hearables
This project addresses the ethical dimensions of increasingly multifunctional hearables, which go beyond hearing compensation to include health monitoring, care coordination, and communication. While these functions offer benefits, they raise concerns about privacy, care, and justice. Using a value-sensitive design approach, the project will systematically identify and integrate ethical values into the technology development process. The work will proceed in three steps: (1) a systematic review of moral reasons in hearable technology literature to create a theoretical framework; (2) qualitative interviews and ethnography with stakeholders (users, caregivers, providers) to understand their values and concerns; and (3) value-oriented co-design and prototyping workshops to translate findings into concrete design recommendations. A particular focus will be on the relational aspects of care in outpatient settings.
5. Listening Effort and Physiologic Stress Monitoring for Hearables in Real Life (Rennies-Hochmuth /Kollmeier/Schultz)
This project aims to monitor the overall stress level of hearable users, with a specific focus on stress induced by listening effort in daily life. The goal is to differentiate between modifiable stress from poor sound perception (which can be improved by adjusting the hearable’s algorithms) and non-modifiable general stress from other factors. The project will combine hearable-based physiological sensing (e.g., EEG biomarkers of stress) with real-time listening-effort prediction models. Studies will progress from controlled virtual-reality conditions to supervised soundwalks and finally to unsupervised real-life use. The system will log acoustic features (e.g., speaker count, noise type) without recording audio, while the listening-effort model will be validated and refined.