About the project

Objective
The project objectives:

Background
Micromobility refers to lightweight, typically electric, transportation modes like bicycles, e-scooters, e-bikes, and similar small vehicles. To simulate riding experience in micromobility, researchers typically design and construct indoor simulators to advance research, development, and understanding of the behaviour of micromobility riders.

While many bicycle simulators have been previously developed and investigated in terms of safety, realism, and motion sickness, other micromobility simulators still need to be explored. Particularly, four new types of micromobility are slowly growing within urban spaces: e-scooters, segways, electric unicycles, and one-wheeled skateboards. This opens new research opportunities focused on improving the safety of riders and understanding their behaviour in urban environments. This demonstrator project is focused on addressing how a VR micromobilty simulator can be designed to accommodate the current and future micromobility vehicle innovations and what the implications are for research and innovation in the domain of VR simulators.

To investigate this research question and understand riders’ behaviour on four types of micromobility vehicles safely, one indoor stationary Virtual Reality Micromobility Simulator (MicroVRide) will be designed and built. Different riding modes concerning four types of vehicles will be accommodated under safe and controlled conditions by this simulator. The design and construction of this simulator will involve the utilization of two main elements: (1) a fixed hardware platform (with a dismountable handlebar) with tracking of body weight distribution and (2) Virtual Reality (VR) simulation (presented in a wireless VR headset) to experience a virtual world. Following the construction of MicroVRide, experiences with users of different levels of riding proficiency will be investigated to understand their riding behaviour, performance, and experience.

The development of MicroVRide is motivated by a commitment to advancing VR simulator technology, establishing a secure and realistic environment for safe vehicle practice.

Crossdisciplinary collaboration
The researchers in the team represent the KTH School of Electrical Engineering and Computer Science and RISE Research Institutes of Sweden, Digital Systems Division.

About the project

Objective
The project aims to model and predict the dynamic urban road traffic noise by integrating the data-driven microscopic traffic simulation and instantaneous noise emission and propagation models. It will use passive traffic and publicly available built environment data to demonstrate a high-fidelity dynamic traffic noise simulator with geographic information system (GIS) tools to predict and visualize noise levels at a time scale and geographic granularity previously unattained.

The DIRAC model and tool will pave the way for complex, dynamic urban road traffic noise modeling using passive traffic data, bridge the digital and physical urban world, and support ongoing efforts and collaborative decision-making of noise mitigation measures for a livable and healthy city, particularly for the growing demand of urban mobility for both people and goods in cities.

Background
Noise pollution is increasingly considered a major environmental issue in urban areas, with rapid urbanization (projected to reach 68% of the world’s population by 2050) in the context of the growing demand for mobility for people and goods in cities. It is recognized as a major cause of public health concerns, e.g., annoyance, sleep disturbance, other health effects (e.g., depression, anxiety, and mood swings), and decreased productivity. In particular, road traffic is deemed the major noise source in urban areas. Some 125 million people in the EU (32% of the total population) are estimated to be exposed to harmful traffic noise levels.

Several initiatives have, therefore, followed the European Noise Directive, mandating the development of urban noise maps. However, strategic noise maps exhibit strong limitations in view of noise exposure mitigation measures: long-time (yearly) averages based on traffic flow, static representations, source-rather than receiver-centric, and non-representative of transitioning vehicle fleet. They are limited in modeling and predicting fluctuations of noise levels over time under planning and management interventions, yet they are a fundamental tool to address specific dimensions of human health effects.

Built on the multidisciplinary team’s expertise and project in traffic simulation and traffic noise modeling, the DIRAC project aims to develop and demonstrate a high-fidelity road traffic noise simulation model in urban areas empowered by ubiquitous passive traffic and open-source data and Digital Twin models. For this, data-driven models work in parallel with real-life measurements to reproduce findings and predict the results of response actions. The models are agent-based (ABM) and open-source, enabling city stakeholders to recognize their roles and management models for informative decision-making of noise mitigation measures toward more livable and healthy cities.

Crossdisciplinary collaboration
The researchers in the team represent the KTH School of Architecture and Built Environment (ABE), Civil and Architectural Engineering Department, Transport Planning Division and KTH School of Engineering Science (SCI), Engineering Mechanics Department, The Marcus Wallenberg Laboratory for Sound and Vibration Research. The project is supported by strategic research partners at the KTH Center of Traffic Research (CTR), VTI (Swedish National Road and Transport Research Institute), Linköping University, and the University of Tartu.

About the project

“Advancing real-time exoskeleton control for human-in-the-loop optimization” is a continuation of the completed project Real-time exoskeleton control for human-in-the-loop optimization.

Objective
Our proposal is to build a physical prototype of a modular lower-limb exoskeleton system with a digital interface to a real-time variable controller. The prototype will be equipped with motors that provide variable control to different joints and joint ranges of motion while being capable of supporting real-time control of its kinetic properties. By varying the assistive strategies in the exoskeleton system via a digital interface, we will enable human-in-the-loop (HILO) optimization in order to find optimal control strategies for different users and different goals.

Background
Persons with physical disabilities are the largest minority group in the world, and global trends in ageing populations indicate an expected increase in the population affected by disability. In Sweden, musculoskeletal disorders are one of the most common causes of long-term disability.

Wearable robotic assistive exoskeletons have undergone rapid developments in the past decades, yet only a handful of products are used frequently, either within or outside of research environments. A major reason for this is that a device must be adjusted for user compliance and efficacy for optimal effect and comfort. Using methods for automatically discovering, customizing, and continuously adapting assistance could overcome these challenges, allowing exoskeletons and prostheses to achieve their potential.

Crossdisciplinary collaboration
The researchers in the team represent the KTH School of Engineering Science, Department of Engineering Mechanics and KTH School of Industrial engineering and management, department of Machine Design.

About the project

Objective
The main goal of the dBrain collaborative project is to develop an interdisciplinary approach to improve early diagnosis and prognosis for brain diseases and to facilitate the development of therapies by combining computational modelling of the brain and advanced artificial intelligence (AI) based data analyses.
The computational approaches will initially focus on Parkinson’s (PD) and Alzheimer’s diseases (AD), the two most common neurodegenerative diseases. These disease cases are strategically selected in collaboration with the Karolinska Hospital (KS) and Karolinska Institute (KI).

This research is important because the incidence of brain diseases will only increase with increasing human longevity. It is estimated that by 2030, mental disorders will be the largest contributor to the total disease burden.

Background
With increased longevity, the number of patients with brain-related diseases will grow. In 2030, mental illness is estimated to be the main imposition on health care. New advances in data analysis and computational modelling of the brain can use the potential of patient data to prevent and treat brain diseases.

Cross-disciplinary collaboration
The research team represents the School of Electrical Engineering and Computer Science (EECS), the School of Engineering Sciences in Chemistry, Biotechnology and Health (CBH) and the School of Engineering Sciences (SCI).

Activities & Results

Find out what’s going on!

Activities, awards, and other outputs

Results

dBrain addresses two major challenges in our understanding of the brain in health and disease: a) what is the relationship between the structural connectivity and functional interactions between different brain regions and within local neuronal networks?; and b) how do disease-related slow changes in the chemical balance, gene expression and neuronal degeneration shape the brain activity dynamics? We address these challenges via four work packages (WPs), using digital approaches comprising AI-based data analysis, math, computational modelling and simulations.

With the help of Digital Futures, we created a vibrant environment involving three KTH schools and collaborators at KS and KI. In addition to regular weekly meetings at the Digital Futures for the whole consortium (i.e. PIs, the recruited postdocs, and other students), several subgroups also meet more frequently in one of the labs or via zoom. Also, several Master’s Student projects have been completed in addition to the work of PhD students and postdocs. In the weekly meetings, postdocs have particular roles – regularly presenting updates on their work and journal articles selected for general interest. These activities provide the postdocs with training in presenting their work and leading discussions. The postdocs also have arranged team-building activities.

Publications

We like to inspire and share interesting knowledge!

Publications

  1. Sanna Persson, Rodrigo Moreno, Bounding tractogram redundancy, Frontiers in Neuroscience. 18:1403804, 2024, https://doi.org/10.3389/fnins.2024.1403804
  2. Malin Siegbahn, Daniel Jörgens, Filip Asp, Malou Hultcrantz, Rodrigo Moreno, Cecilia Engmér Berglin. Asymmetry in cortical thickness of the Heschl’s gyrus in unilateral ear canal atresia. Otology & Neurotology. 45(4),e342-e350, 2024. https://doi.org/10.1097/MAO.0000000000004137
  3. Caroline Dartora, Anna Marseglia, Gustav Mårtensson, Gull Rukh, Junhua Dang, J- Sebastian Muehlboeck, Lars-Olof Wahlund, Rodrigo Moreno, José Barroso, Daniel Ferreira, Helgi Schiöth, Eric Westman. A deep learning model for brain age prediction using minimally preprocessed T1w-images as input. Frontiers in Aging Neuroscience 15, 1303036, 2024. https://doi.org/10.3389/fnagi.2023.1303036
  4. Antonia Hain, Daniel Jörgens, Rodrigo Moreno. Randomized iterative spherical-deconvolution informed tractogram filtering. Neuroimage 278, 120248, 2023. https://doi.org/10.1016/j.neuroimage.2023.120248
  5. Jingru Fu, Antonios Tzortzakakis, José Barroso, Eric Westman, Daniel Ferreira, Rodrigo Moreno. Fast three-dimensional image generation for healthy brain aging using diffeomorphic registration. Human Brain Mapping 44 (4), 1289-1308, 2023. https://doi.org/10.1002/hbm.26165
  6. Kajsa Ahlgren, Christoffer Olsson, Inna Ermilova, Jan Swenson. New insights into the protein stabilizing effects of trehalose by comparing with sucrose. Physical Chemistry Chemical Physics 25 (32), 21215-21226, 2023. https://doi.org/10.1039/D3CP02639F
  7. Zhou Zhou, Chirstoffer Olsson, T Christian Gasser, Xiaogai Li, Svein Kleiven. The White Matter Fiber Tract Deforms Most in the Perpendicular Direction During In Vivo Volunteer Impacts. J Neurotrauma. 2024 41(23-24):2554-2570, 2024. https://doi.org/10.1089/neu.2024.0183
  8. Jingru Fu, Daniel Ferreira, Örjan Smedby, Rodrigo Moreno. A deformation-based morphometry framework for disentangling Alzheimer’s disease from normal aging using learned normal aging templates. arXiv.2311.08176, 2023. https://doi.org/10.48550/arXiv.2311.08176
  9. Daniel Jörgens, Pierre Marc Jodoin, Maxime Descoteaux, Rodrigo Moreno. Merging multiple input descriptors and supervisors in a deep neural network for tractogram filtering. arXiv:2307.05786, 2023. https://doi.org/10.48550/arXiv.2307.05786
  10. Jingru Fu, Simone Bendazzoli, Örjan Smedby, Rodrigo Moreno. Unsupervised Domain Adaptation for Pediatric Brain Tumor Segmentation. MICCAI 2024 Workshop on Advancing Data Solutions in Medical Imaging AI (ADSMI’24) (in press). https://doi.org/10.48550/arXiv.2406.16848
  11. Sanna Persson, Xinyi Wan, and Rodrigo Moreno. Randomly COMMITting: Iterative Convex Optimization for Microstructure-Informed Tractography. MICCAI 2024 Workshop on Computational Diffusion MRI (CDMRI’24) (in press)
  12. Yu Zhou, Jingru Fu, Örjan Smedby, Rodrigo Moreno. Synthesis of Pediatric Brain Tumor Images with Mass Effect. SPIE – Medical Imaging: Image Processing 12464, 699-707, 2023. San Diego, USA, February 2023. https://doi.org/10.1117/12.2654366
  13. I Carannante, Y Johansson, G Silberberg, J Hellgren Kotaleski. Data-Driven Model of Postsynaptic Currents Mediated by NMDA or AMPA Receptors in Striatal Neurons. Front Comput Neurosci. 2022; 16: 806086.
  14. J Fu, A Tzortzakakis, J Barroso, E Westman, D Ferreira, R Moreno. Generative Aging of Brain Images with Diffeomorphic Registration. arXiv preprint. 2022; arXiv:2205.15607. https://doi.org/10.48550/arXiv.2205.15607
  15. A Hain, D Jörgens, R Moreno. Assessing Streamline Plausibility Through Randomized Iterative Spherical-Deconvolution Informed Tractogram Filtering. arXiv preprint 2022; arXiv:2205.04843. https://doi.org/10.48550/arXiv.2205.04843
  16. M Siegbahn, C Engmér Berglin, R Moreno. Automatic segmentation of the core of the acoustic radiation in humans. Front Neurol. 2022 : 934650. https://doi.org/10.3389/fneur.2022.934650
  17. C Dartora, A Marseglia, G Mårtensson, G Rukh, J Dang, JS Muehlboeck, LO Wahlund, R Moreno, J Barroso, D Ferreira, HB Schiöth, E Westman. Predicting the Age of the Brain with Minimally Processed T1-weighted MRI Data. medRxiv preprint. 2022; medRxiv:2022.09.06.22279594. https://doi.org/10.1101/2022.09.06.22279594
  18. Chakravarty K, Roy S, Sinha A, Nambu A, Chiken S, Hellgren Kotaleski J, Kumar A. Transient Response of Basal Ganglia Network in Healthy and Low-Dopamine State. eNeuro. 2022 Mar 18;9(2):ENEURO.0376-21.2022. doi: 10.1523/ENEURO.0376-21.2022.
  19. Hjorth, J.J.J., Hellgren Kotaleski, J. & Kozlov, A. Predicting Synaptic Connectivity for Large-Scale Microcircuit Simulations Using Snudda. Neuroinform 19, 685–701 (2021). https://doi.org/10.1007/s12021-021-09531-w
  20. G Colombo, R Cuber, L Kanari, A Venturino, R Schulz, M Scolamiero, J Agerberg, H Mathys, L Tsai, W Chachólski,, K Hess, S Siegert. Microglial morphOMICs, a tool for mapping microglial morphology, reveals brain-region-and sex-dependent phenotypes. Nature neuroscience, 2022.
  21. W Chachólski, A Guidolin, I Ren, M Scolamiero, F Tombari. Effective computation of relative homological invariants for functors over posets. arXiv preprint 2022; arXiv:2209.05923
  22. W Chachólski, A Jin, F Tombari. Realisations of posets and tameness. arXiv preprint 2022; arXiv:2112.12209
  23. W Chachólski, René Corbet, Anna-Laura Sattelberger. The Shift-Dimension of Multipersistence Modules. arXiv preprint 2022; arXiv:2112.06509.
  24. W Chachólski, B Gunti, C Landi, Decomposing filtered chain complexes: Geometry behind barcoding algorithms. Computational Geometry, Volume 109, 2023.
  25. Georgios Moschovis. NeuralDynamicsLab at ImageCLEF Medical 2022. ImageCLEF, 2022.
  26. Helson Pascal, Lundqvist Daniel, Svenningsson Per, Vinding Mikkel, Kumar Arvind (2023). Cortex-wide topography of 1/f-exponent in Parkinson’s disease. Nature Parkinson’s Disease, 9, Article number: 109.
  27. Zang, Jie, Liu Shenquan, Helson Pascal, Kumar Arvind (2024). Structural constraints on the emergence of oscillations in multi-population neural networks. eLIFE, 12(Feb 5).
  28. Wärnberg Emil, Kumar Arvind (2023). Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia. PNAS 32(120):e2221994120

Master School Theses

  1. Yuqi Zheng. Predictive MR Image Generation for Alzheimer’s Disease and Normal Aging Using Diffeomorphic Registration. MS thesis KTH 2023. Supervisors Rodrigo Moreno, Jingru Fu. Diva diva2:1833589
  2. Xinyi Wan. Assessing the Streamline Plausibility Through Convex Optimization for Microstructure Informed Tractography(COMMIT) with Deep Learning. MS thesis KTH 2023. Supervisor Rodrigo Moreno. Diva diva2:1754277
  3. Teodor Pstrusiński. Impact of the autoencoder-based FINTA tractogram filtering method on brain networks in subjects with Mild Cognitive Impairment. MS thesis KTH 2023. Supervisors Rodrigo Moreno, Fabian Sinzinger. Diva diva2:1799683
  4. Erika Bengtsdotter. Towards Anatomically Plausible Streamline Tractography with Deep Reinforcement Learning. MS thesis KTH 2022. Supervisors Rodrigo Moreno, Fabian Sinzinger. Diva diva2:1695872
  5. Filippo Maschio. Influence of tractogram filtering in the analysis of tractography data in rat brains. MS thesis University of Padua, Italy 2022. Supervisor Rodrigo Moreno.
  6. Yu Zhou. Synthesis of Pediatric Brain Tumor Image With Mass Effect. MS thesis KTH 2022. Supervisors Rodrigo Moreno, Jingru Fu. Diva diva2:1686799
  7. Antonia Hain. Assessing individual streamline plausibility through randomized spherical deconvolution-informed tractogram filtering. MS thesis University of Saarland, Germany 2021. Supervisor Rodrigo Moreno.
  8. N Hulst, Exploring persistent homology as a method for capturing functional connectivity differences in Parkinson’s Disease. Master Thesis. DiVA, id: diva2:1687257.
  9. Marvin Köpff. Impact of tractogram filtering and graph creation for structural connectomics in subjects with mild cognitive impairment. MS thesis KTH 2020. Supervisors Rodrigo Moreno, Joana Pereira. Diva: diva2:1463915
  10. Johannes Wennberg. Longitudinal assessment of functional connectivity impairment in rat brains. MS thesis KTH 2019. Supervisor Rodrigo Moreno, diva2:1327919
  11. Cristina Zanin. Graph Convolutional Networks for static and dynamic functional connectivity: an application to the Autism Spectrum Disorder. MS thesis University of Padua, Italy 2020. Supervisor Rodrigo Moreno.
  12. Lucas Höglund. Analysis of Eye Tracking Data from Parkinson’s Patients using Machine Learning. MS thesis, KTH.
  13. Leo Bergman. Feature extraction with self-supervised learning on eye-tracking data from Parkinson’s patients and healthy individuals. MS thesis, KTH.
  14. Emma Lind. Analysis of Brain Signals from Patients with Parkinson’s Disease using Self-Supervised Learning. MS thesis, KTH.
  15. Wilhelm Ågren. Feature extraction from MEG data using self-supervised learning. MS thesis, KTH.
  16. Giulia Tuccio. Parameter estimation in a cardiovascular computational model using numerical optimization. MS thesis, KTH.
  17. Paolo Calderaro. Patient simulation. Generation of a machine learning “inverse” digital twin. MS thesis, KTH.

About the project

Objective
This project aims to develop adaptive social robots that can understand humans’ communicative behaviour and task-related physical actions and adapt their interaction to suit. We aim to investigate and demonstrate fluid and seamless adaptation of intelligent systems to users’ contexts, needs or preferences. To achieve fluidity, such adaptation needs to happen with minimal interruption to the users’ ongoing interaction with the system, without requiring user intervention, while providing accountability and control of the adaption in a task-appropriate, timely, and understandable manner. This will be explored in multiple embodiments: smart speakers, back-projected robotic heads, and dual-arm robots.

Our use case scenario is an adaptive intelligent kitchen assistant that helps humans prepare food and other kitchen-centric tasks, focusing on supporting ageing in place. Our systems will engage in face-to-face spoken and physical collaboration with humans, track the users’ affective states and task-related actions in real-time, adjust performance based on previous interactions, adapt to user preferences, and show intention using a self-regulation perception-production loop. The project will use the Intelligence Augmentation Lab that TMH and RPL plan to set up.

Background
Intelligent systems built around big datasets and machine learning techniques are becoming ubiquitous in people’s lives – smart appliances, wearables, and, increasingly, robots. As these systems are intended to assist an ever wider range of users in their homes, workplaces or public spaces, a typical one-fits-all approach becomes insufficient. Instead, these systems will need to take advantage of the machine learning techniques upon which they are built to adapt to the specific task, user constellation continually, and shared environment in which they are operating. In long-term deployments, the state of the environment, user preferences, skills, and abilities change and must be adapted. This is relevant for socially assistive robots in people’s homes, education or healthcare settings, and robots working alongside workers in small-scale manufacturing environments.

Crossdisciplinary collaboration
The research team represents the School of Electrical Engineering and Computer Science (EECS, KTH), the School of Engineering Sciences in Chemistry, Biotechnology and Health (CBH, KTH) and the Department of Computer and System Science (DSV) at Stockholm University.

Articles

Avancerade adaptiva intelligenta system

Activities & Results

Find out what’s going on!

Results

The main results achieved in the first half of the project towards the 3 main objectives enumerated in the proposal:

  1. Understanding and Design for the Smart Kitchen

Deeper understanding of the cooking process (Kuoppamäki et al, 2021) led to a number of studies on interaction in and around cooking, including list advancement (Jaber & McMillan, 2022), content navigation (Zhao et al., 2022), designing conversational interaction (Kuoppamäki et al., 2023), the impact of contextual awareness on command construction and delivery (Jaber et al., in submission) and an ongoing study comparing proactive organisational support between young adults and those over 65 (Kuoppamäki et al.).

  1. Perception and Representation of Long Term Human Action

Research showing that thermal imaging is a modality relevant for detecting frustration in human-robot interaction (Mohamed et al., 2022). The models to predict frustration based on a dataset of 18 participants interacting with a Nao social robot in our lab were tested using features from several modalities: thermal, RGB, Electrodermal Activity (EDA), and all three combined. The models reached an accuracy of 89% with just RGB features, 87% using only thermal features, 84% using EDA, and 86% when using all modalities. We are also investigating the accuracy of frutration prediction models using data collected at the KTH Library where students ask directions to a Furhat robot.

  1. Input, Output, and Interaction for Smart Assistive Technologies

In order to fulfil the goal of adaptation of virtual agent and robot behaviour to different contexts, we have researched interlocutor-aware facial expressions in dyadic interaction (Jonell et al, 2020) as well as adaptive facial expressions in controllable generation of speech and gesture: we have developed techniques to generate conversational speech, with control over speaking style to signal e.g. certainty or uncertainty (Wang et al, 2022, Kirkland et al 2022) as well as models that are able to generate coherent speech and gesture from a common representation (Wang et al 2021). Another direction concerns adaptation to spatial contexts and environments, where we have used imitation learning and physical simulation to produce referential gestures (Deichler et al, 2022).

Publications

We like to inspire and share interesting knowledge!

  1. Parag Khanna, Mårten Björkman, Christian Smith. “Human Inspired Grip-Release Technique for Robot-Human Handovers”, 2022 IEEE-RAS International Conference on Humanoid Robots, Ginowan, Japan. (Accepted for publication)
  2. Sanna Kuoppamäki, Mikaela Hellstrand, and Donald McMillan (2023, forthcoming). Designing conversational scenarios with older adults digital repertoires: Graphic transcript as a design method. In: Hänninen, R., Taipale, S., Haapio-Kirk, L. (eds). Embedded and everyday technology: Digital repertoires in an ageing society. UCL Press.
  3. Youssef Mohamed, Giulia Ballardini, Maria Teresa Parreira, Séverin Lemaignan, and Iolanda Leite. 2022. Automatic Frustration Detection Using Thermal Imaging. In Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’22). IEEE Press, 451–460
  4. Razan Jaber and Donald McMillan. 2022. Cross-Modal Repair: Gaze and Speech Interaction for List Advancement. In Proceedings of the 4th Conference on Conversational User Interfaces (CUI ’22). Association for Computing Machinery, New York, NY, USA, Article 25, 1–11.
  5. Yaxi Zhao, Razan Jaber, Donald McMillan, and Cosmin Munteanu. 2022. “Rewind to the Jiggling Meat Part”: Understanding Voice Control of Instructional Videos in Everyday Tasks. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 58, 1–11.
  6. Kirkland, A., Lameris, H., Gustafson, J., Székely, É. (2022) “Where’s the uh, hesitation? The interplay between filled pause location, speech rate and fundamental frequency”, Interspeech 2022, Incheon, Korea.
  7. Wang, S., Gustafson, J., Székely, É. (2022) “Evaluating Sampling-based Filler Insertion with Spontaneous TTS”, 13th Edition of the Language Resources and Evaluation Conference (LREC 2022), Marseille.
  8. Deichler, A., Wang, S., Alexanderson, S., & Beskow, J. (2022). Towards Context-Aware Human-like Pointing Gestures with RL Motion Imitation. In Context-Awareness in Human-Robot Interaction: Approaches and Challenges, workshop at 2022 ACM/IEEE International Conference on Human-Robot Interaction.
  9. Donald McMillan and Razan Jaber. 2021. Leaving the Butler Behind: The Future of Role Reproduction in CUI. In Proceedings of the 3rd Conference on Conversational User Interfaces (CUI ’21). Association for Computing Machinery, New York, NY, USA, Article 11, 1–4.
  10. Sanna Kuoppamäki, Sylvaine Tuncer, Sara Eriksson, and Donald McMillan. 2021. Designing Kitchen Technologies for Ageing in Place: A Video Study of Older Adults’ Cooking at Home. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 2, Article 69 (June 2021), 19 pages.
  11. Katie Winkle, Gaspar Isaac Melsión, Donald McMillan, and Iolanda Leite. 2021. Boosting Robot Credibility and Challenging Gender Norms in Responding to Abusive Behaviour: A Case for Feminist Robots. In Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’21 Companion). Association for Computing Machinery, New York, NY, USA, 29–37.
  12. Wang, S., Alexanderson, A., Gustafson, J., Beskow, J., Henter, G. and Székely, É. (2021) “Integrated Speech and Gesture Synthesis”, 23rd ACM International Conference on Multimodal Interaction (ICMI 2021), Montreal
  13. Jonell, P., Kucherenko, T., Henter, G. E., & Beskow, J. (2020, October). Let’s Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (pp. 1-8).

About the project

Objective
Adaptive Intelligent Homes (AIH) aims to develop a suite of demonstrators, combining the cutting-edge advances made within the parent collaborative research project (AAIS) on user state recognition, human-robot handovers, multimodal referring expression generation, controllable speech synthesis, and proactively supporting users in complex tasks. AIH will extend the context-awareness and adaptiveness of those conversational robots from a laboratory environment to a real-life environment and from individual to collaborative settings, where multiple users interact with the system simultaneously. Through engaging with societal and municipal actors, the project contributes to more inclusive and accessible robotic systems with potential applications to promote healthy lifestyles and provide home care services for older adults.

Background
AI-powered agents provide an opportunity to manage, coordinate and initiate activities within and across households, contributing to healthy lifestyles and improved quality of life for users of all ages. To achieve this goal, these robotic systems must be aligned with the user’s needs, preferences, and interests, and they must adapt themselves to complex environments involving multiple users performing overlapping and interwoven household tasks, such as cooking.

Cooking in the home is not simply instruction-giving; it is a multi-party, parallelized activity requiring timing, action relevance, and adaptability from both the human and any AI assistance. The project has envisioned an adaptive intelligent kitchen assistant that could help humans prepare food and other kitchen-centric tasks to enhance healthy ageing and improve quality of life.

Cross-disciplinary collaboration
The interdisciplinary approach will allow us to synthesize the human-centric understanding of the home and kitchen environment to the technical capabilities of dialogue systems and robotic agents. This could indicate, for instance, new modalities for personalising the intelligent kitchen assistant for different user groups or linguistic styles that could be adapted to different family members’ skill sets and preferences. An important societal aspect is to explore and design for improved inclusivity of dialogue systems by recognizing diverse ways of interacting with the system based on user’s age, gender, ethnicity, or disability and investigate the long-term impact of deployment of such systems in the home environment to the quality of life and self-efficacy among end-users. Designing for improved accessibility of dialogue systems has the potential to result in more equal access to intelligent and assistive home environments.

About the project

Objective
The Explainable and Ethical Machine Learning for Knowledge Discovery from Medical Data Sources (EXTREMUM) project aims to develop a novel platform for learning from complex medical data sources with a focus on two healthcare application areas: adverse drug event detection and early detection and treatment of cardiovascular diseases.

The team will present a new framework for data management and analysis of data integration, methods for machine learning, and ethical issues related to predictive models. This project’s fundamental breakthrough is establishing a novel knowledge management and discovery framework for medical data sources. The outcome will be a set of methods and tools for integrating complex medical data sources, a set of predictive models for learning from these sources emphasising interpretability and explanatory features, and simultaneously focusing on maintaining ethical integrity in the underlying decision mechanisms that rule machine learning.

Background
One of the biggest challenges that research and business entities have been facing recently is that the data provided by today’s technologies originate from multiple data sources in massive quantities and at rapid rates. In addition to their volume, such data sources are complex and heterogeneous. In the presence of these complex and continuously growing information sources, domain scientists, data analysts, and novice users have been struggling to manage this complexity and arrive from abundant data to usable and interpretable models and exploitable domain knowledge. Towards this end, these data sources need to be monitored in real-time. Hence, data integration indexing and predictive modelling have become a major challenge. Consider, for example, the healthcare domain, where numerous data sources in the form of Electronic Health Records (EHRs), such as billing codes, registry data, and pharmaceutical data, are used for developing predictive models of, e.g., heart failure prevention and treatment progression, or adverse drug effect (ADE) detection.

An example workflow of the EXTREMUM framework starts from radiography exams, ranking them based on their severity and urgency, then moving on to their classification and tagging while eventually producing clinically relevant explanatory text.

Cross-disciplinary collaboration
The project is a collaborative effort between four research institutions: the Department of Computer and Systems Sciences at Stockholm University, the Department of Law at Stockholm University, RISE Research Institutes Sweden, Division ICT, and the Department of Automatic Control, School of Electrical Engineering and Computer Science (EECS, KTH).

Activities & Results

Find out what’s going on!

Activities, awards, and other outputs

Results

The main goal of this project is to develop and establish a novel data representation, integration, and knowledge discovery framework for medical data sources, focusing on explainable machine learning governed by legal and ethical principles. Particular emphasis will be given to healthcare data sources available in EHRs, emphasising two particular healthcare problems: (a) early prediction and treatment of cardiovascular diseases and (b) adverse drug event identification and prevention.

Project objectives, along with the current achievements and results:

Objective 1: Unified representation and integration of complex data spaces. We have explored and defined novel space representations, similarity measures, and methods for searching and indexing large and complex data spaces for complex data sources. The basic challenge is the temporal nature of the data spaces and the inherent temporal dependencies that may exist within the same and across different data sources in these spaces. Our main solutions include the adoption and employment of temporal abstractions for temporal event sequences (both time series and discrete event sequences) of univariate and multivariate events.

Objective 2: Explainable predictive models for complex data sources. Regarding the second objective, our intention is to develop novel predictive modelling algorithms that can support and exploit data sources of complex nature and heterogeneity while at the same time providing explainability of their predictions in the form of interpretable features or rule sets. Towards this end, we have developed new methods for explainable machine learning with an emphasis on example-based explanations, such as counterfactuals for time series counterfactuals, as well as for event sequences). Moreover, we have explored model-based explanations and more concretely local explanations (such as LIME and SHAP), their applicability to the medical domain, and to what extent they are trusted and can be adopted by medical practitioners. In the case of time series data, interpretability can be achieved by focusing on white-box models, such as those described by differential equations, for which we have developed and analyzed several estimation algorithms.

We have also developed novel methods to reverse engineer predictors/filters for Hidden Markov Models and Linear Dynamical Systems, rendering them explainable by “opening the box”; furthermore, by modifying the samples arriving at a learning algorithm, we are developing methods that can improve learning while preserving privacy, enforcing fairness and arriving at more interpretable models. Our future plan is to develop similar methods for counterfactual generation for forecasting (e.g., for predicting critical events in the ICU and preventing them by suggesting critical actions. Furthermore, we intend to conduct a more extensive qualitative study on the generated explanations and counterfactuals involving a larger group of medical practitioners. This will also involve a qualitative assessment and deployment of our current demonstrator tool (published recently in the hospital setting on actual use cases.

Objective 3: Adherence to legal and ethical frameworks. We have focused on the legal and ethical implications associated with the development and use of predictive modelling in relation to healthcare data analysis. To this end, the General Data Protection Regulation (GDPR) is of utmost relevance. We have identified the predominant social values promoted in the GDPR and transposed these legal rules into mathematical equivalents. The finding from this process has been interesting to the extent that the identified social values are mutually exclusive and that promoting one of them occurs at the expense of the others, necessitating a balancing act. The social values studied thus far were privacy, accuracy and explainability. The results of this examination are relevant from a legislative techniques perspective, culminating in these interdisciplinary findings being published in a legal publication venue.

Publications

We like to inspire and share interesting knowledge!

Journal articles (J)

  1. R.A.González,C.R.Rojas,S.Pan,andJ.Welsh.“Theoreticalandpracticalaspects of the convergence of the SRIVC estimator for over-parameterized models”. Automatica, 142:110355, 2022
  2. S. Pan, J. Welsh, R. A. González, and C. R. Rojas. “Consistency analysis and bias elimination of the instrumental variable based state variable filter method”. Automatica, 144:110511, 2022
  3. P. Wachel and C. R. Rojas. “An adversarial approach to adaptive model predictive control”. Journal of Advances in Applied & Computational Mathematics, 9, 135–146, 2022
  4. R. A. González, C. R. Rojas, S. Pan, and J. Welsh. “Refined instrumental variable methods for unstable continuous-time systems in closed-loop”. International Journal of Control (accepted for publication), 2022
  5. C.R.RojasandP.Wachel.“Onstate-spacerepresentationsofgeneraldiscrete-time dynamical systems”. IEEE Transactions on Automatic Control (accepted for publication), 2022
  6. R.Mochaourab,A.Venkitaraman,I.Samsten,P.Papapetrou,andC.R.Rojas,“Post- hoc Explainability for Time Series Classification: Toward a Signal Processing Perspective,” IEEE Signal Processing Magazine, Special Issue on Explainability in Data Science: Interpretability, Reproducibility, and Replicability, vol. 39, no. 4, Jul. 2022
  7. S. Greenstein, P. Papapetrou, and R. Mochaourab, “Embedding Human Values into Artificial Intelligence,” in De Vries, Katja (ed.), De Lege, Iustus förlag, Uppsala University, 2022
  8. J. Rebane, I. Samsten, L. Bornemann, and P. Papapetrou, “SMILE: A feature-based temporal abstraction framework for event-interval sequence classification”. In Data Mining and Knowledge Discovery (DAMI) 35(1): 372-399, 2021
  9. B. Djehiche, O. Mazhar, and C. R. Rojas. “Finite impulse response models: A nonasymptotic analysis of the least squares estimator”. Bernoulli, 27(2):976–1000, 2021
  10. R. A. González, C. R. Rojas, S. Pan, and J. Welsh. “Consistent identification of continuous-time systems under multisine input signal excitation”. Automatica, 133:109859, 2021
  11. I. Lourenço, R. Mattila, C. R. Rojas, X. Hu, and B.Wahlberg. “Hidden Markov models: Inverse filtering, belief estimation and privacy protection”. Journal of Systems Science and Complexity, 34:1801–1820, 2021
  12. J. Rebane, I. Samsten, and P. Papapetrou, “Exploiting Complex Medical Data with Interpretable Deep Learning for Adverse Drug Event Prediction”. In Artificial Intelligence in Medicine (AIM), 28(8): 1651-1659, 2021
  13. Locally and globally explainable time series tweaking KAIS), 62 (5): 1671- 1700, 2020
  14. R. Mattila, C. R. Rojas, V. Krishnamurthy, and B. Wahlberg. “Inverse filtering for hidden Markov models with applications to counter-adversarial autonomous systems”. IEEE Transactions on Signal Processing, 68:4987–5002, 2020

Conference papers (C)

  1. Z. Lee, M. Trincavelli, and P. Papapetrou, “Finding Local Groupings of Time Series“, In the European Conference on Machine Learning and Principles and Practices of Knowledge Discovery in Databases (ECML/PKDD), to appear
  2. L. Mondrejevski, I. Miliou, A. Montanino, D. Pitts, J. Hollmén, and P. Papapetrou, “FLICU: A Federated Learning Workflow for Intensive Care Unit Mortality Prediction”. In Computer-Based Medical Systems (CBMS), to appear
  3. M. Bampa, T. Fasth, S. Magnusson, and P. Papapetrou, “EpidRLearn: Learning intervention strategies for epidemics with Reinforcement Learning“. In the International Conference on Artificial Intelligence in Medicine (AIME), 2022
  4. S. Guarnizo, I. Miliou, and P. Papapetrou, “Impact of Dimensionality on Nowcasting Seasonal Influenza with Environmental Factors“. In the International Symposium on Intelligent Data Analysis (IDA), 128-142, 2022
  5. J. Parsa, C. R. Rojas, and H. Hjalmarsson. “Optimal input design for sparse system identification”. In Proceedings of the 2022 European Control Conference (ECC), 2022
  6. B. Lakshminarayanan and C. R. Rojas. “A statistical decision-theoretical perspective on the two-stage approach to parameter estimation”. In Proceedings of the 61th IEEE Conference on Decision and Control (CDC 2022) (accepted for publication), 2022
  7. I. Lourenço, R. Winqvist, C. R. Rojas, and B. Wahlberg. “A teacher-student Markov decision process-based framework for online correctional learning”. In Proceedings of the 61th IEEE Conference on Decision and Control (CDC 2022) (accepted for publication), 2022
  8. Z. Wang, I. Samsten, and P. Papapetrou, Counterfactual Explanations for Survival Prediction of Cardiovascular ICU Patients. In Artificial Intelligence in Medicine (AIME), 338-348, 2021 [best student paper award]
  9. J.PavlopoulosandP.Papapetrou,CustomizedNeuralPredictiveMedicalText:AUse- Case on Caregivers. In Artificial Intelligence in Medicine (AIME), 438-443, 2021
  10. J. Rebane, I. Samsten, P. Pantelidis, and P. Papapetrou, Assessing the Clinical Validity of Attention-based and SHAP Temporal Explanations for Adverse Drug Event Predictions. In Computer-based Medical Systems (CBMS), 235-240, 2021
  11. Z. Wang, I. Samsten, R. Mochaourab, and P. Papapetrou, Learning Time Series Counterfactuals via Latent Space Representations. In Discovery Science (DS), 369- 384, 2021
  12. I. Lourenço, R. Mattila, C. R. Rojas, and B. Wahlberg. “Cooperative system identification via correctional learning”. In Proceedings of the 19th IFAC Symposium on System Identification (SYSID 2021), 2021
  13. R. A. González, C. R. Rojas, and H. Hjalmarsson. “Non-causal regularized least- squares for continuous-time system identification with band-limited input excitations”. In Proceedings of the 60th IEEE Conference on Decision and Control (CDC 2021), 2021
  14. R. A. González, C. R. Rojas, S. Pan, and J. S. Welsh. “The SRIVC algorithm for continuous-time system identification with arbitrary input excitation in open and closed loop”. In Proceedings of the 60th IEEE Conference on Decision and Control (CDC 2021), 2021
  15. Z. Lee, T. Lindgren, and P. Papapetrou, “Z-Miner: an efficient method for mining frequent arrangements of event intervals“. In ACM Knowledge Discovery and Data Mining (KDD), 524-534, 2020
  16. Z. Lee, S. Girdzijauskas, and P. Papapetrou, “Z-Embedding: A spectral representation of event intervals for efficient clustering and classification“. In the European Conference on Machine Learning and Principles and Practices of Knowledge Discovery in Databases (ECML/PKDD), 710-726, 2020
  17. M. Bampa, P. Papapetrou, and J. Hollmen, “A clustering framework for patient phenotyping with application to adverse drug events“. In IEEE Computer-Based Medical Systems (CBMS), 177-182, 2020
  18. J. Pavlopoulos and P. Papapetrou, “Clinical predictive keyboard using statistical and neural language modeling“. In IEEE Computer-Based Medical Systems (CBMS), 293- 296, 2020
  19. Z. Lee, J. Rebane, and P. Papapetrou, “Mining disproportional frequent arrangements of event intervals for investigating adverse drug events“. In IEEE Computer-Based Medical Systems (CBMS), 293-296, 2020
  20. N. B. Kumarakulasinghe, T. Blomberg, J. Liu, A. S. Leao and P. Papapetrou, “Evaluating local interpretable model-agnostic explanations on clinical machine learning classification models“. In IEEE Computer-Based Medical Systems (CBMS), 7-12, 2020
  21. R. Mattila, I. Lourenço, V. Krishnamurthy, C. R. Rojas, and B. Wahlberg. “What did your adversary believe? Optimal smoothing in counter-autonomous systems”. In Proceedings of the 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2020), Barcelona, Spain, 2020
  22. R. A. González and C. R. Rojas. “Finite sample deviation and variance bounds for first order autoregressive processes”. In Proceedings of the 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2020), Barcelona, Spain, 2020
  23. R. A. González and C. R. Rojas. “A finite-sample deviation bound for stable autoregressive processes”. In Proceedings of the 2nd Conference on Learning for Decision and Control (L4DC), Berkeley, USA, 2020
  24. R. A. González, J. S. Welsh, and C. R. Rojas. “Enforcing stability through ellipsoidal inner approximations in the indirect approach for continuous-time system identification”. In Proceedings of the 21st IFAC World Congress (IFAC 2020), Berlin, Germany, 2020
  25. R. Mattila, C. R. Rojas, E. Moulines, V. Krishnamurthy, and B.Wahlberg. “Fast and consistent learning of hidden Markov models by incorporating non-consecutive correlations”. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020
  26. I. Lourenço, R. Mattila, C. R. Rojas, and B. Wahlberg. “How to protect your privacy? A framework for counter-adversarial decision making”. In Proceedings of the 59th IEEE Conference on Decision and Control (CDC 2020), 2020
  27. R. Mochaourab, S. Sinha, S. Greenstein, and P. Papapetrou, “Robust Counterfactual Explanations for Privacy-Preserving SVMs,” International Conference on Machine Learning (ICML 2021), Workshop on Socially Responsible Machine Learning, Jul. 2021

Demo paper (D)

  1. R. Mochaourab, S. Sinha, S. Greenstein, and P. Papapetrou, “Demonstrator on Counterfactual Explanations for Differentially Private Support Vector Machines,” European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2022), Demo Track, Sept. 2022