About the project

Objective
In recent years, Sweden has emerged as one of the most welcoming European countries for migrants from diverse backgrounds and cultures. However, the challenge lies in effectively integrating these individuals into Swedish society, given the nation’s relatively small population and the rapid influx of migrants. Language learning is critical to this integration process, posing significant challenges for learners and educators. We aim to develop a language learning system focusing specifically on conversational skills utilizing social robots and conversational AI. This will enable learners to practice language through conversation with diverse personas in real-life situations.

Background
Recent advancements in Large Language Models (LLMs) have transformed our approach to complex challenges such as language learning, opening up for new opportunities that were not available a few years ago. Alongside LLMs, advancements in speech technology — specifically in the fields of speech recognition and synthesis — enable the creation of much more natural, lifelike conversational experiences with AI. This progress empowers us to develop a system that not only comprehensively understands Swedish but can also understand the user’s native language. This multilingual competence facilitates a more effective, personalized, and user-centric language learning platform.

Crossdisciplinary collaboration
Our project benefits from the expertise of a diverse team drawn from the Division of Speech, Music, and Hearing at KTH and the Department of Education at Stockholm University. This interdisciplinary collaboration ensures that we have the necessary theoretical and practical background to develop an innovative solution that addresses the challenges of language development and cultural integration in the context of digital transformation.

About the project

Objective
We propose energy-efficient, high-speed, real-time monitoring of complex traffic scenarios in urban areas. Reliable real-time information about urban traffic supports emergency responders and city planners and reduces carbon emissions, thereby increasing the urban citizens’ quality of life. Using decentralized neuromorphic sensing and processing, we obtain four benefits compared to state-of-the-art: (1) low latency, (2) minimal transmission bandwidth, (3) low power, and (4) enhanced privacy.

As a result of the project, we expect to provide a Live Demonstrator which, in real-time, identifies objects in traffic (e.g., cars, bicycles, and scooters) while continuously running on a negligible energy budget (e.g., a solar cell).

Background
In the upcoming decades, urban areas will become “much larger, more complex, and interconnected, ” requiring optimized infrastructure with increasing automatization at increasingly larger scales. More broadly, real-time information about traffic and infrastructure utilization will become an increasingly valuable commodity for companies and officials to adapt and update the infrastructure as needed. Such data is generated by video cameras in cities today.

Embedding low-power event cameras with low-power neuromorphic local computation provides a uniquely scalable solution for the growing need to monitor urban traffic and resource flows in real time. Our project estimates a 100x reduction in power and a 20x reduction in installation cost.

Crossdisciplinary collaboration
The teams contribute expertise in neuromorphic hardware and algorithms for system development and expertise in traffic observation and urban traffic management. Only both teams and their complementary expertise can develop real-time low-power computing systems for the societal relevant task of urban traffic monitoring.

About the project

Objective
The research team’s ambition is to develop a new research area in urban development (studies). Well-being in smart cities is the defined research area, focusing on interactions of human-machine-computers or “cyber-physical-human systems” based on human decision-making on an institutional, individual and neurological abstraction level. The smart city of the future is our main application area, as these are complex cyber-physical-human systems. The project will develop a framework for capturing interactions and dynamics in these systems and demonstrate the applications in user case studies.

Background
The health condition of a human being is the basis of individual and social well-being. The driving force for human-social behaviour and many choices individuals make is the desire for well-being, which will manifest in the future of smart cities. Networks, human agents, cyber agents, and physical infrastructure perform feedback and interactions in smart cities. Smart cities can efficiently and sustainably increase human well-being.

Cross-disciplinary collaboration
The research team represents the School of Electrical Engineering and Computer Science (EECS, KTH), the School of Industrial Engineering and Management (ITM, KTH) and the School of Architecture and the Built Environment (ABE, KTH).

Watch the recorded presentation at the Digitalize in Stockholm 2023 event:

Find out more on HiSS webpage

Activities & Results

Find out what’s going on!

Activities, awards, and other outputs

Results

A general objective of the project is to link dominant mechanisms of decision-making and choice between the micro, meso and macro scales that are most relevant for advancing the sustainability agenda in smart cities. The specific objectives are related to theoretical and experimental studies of different aspects of decision-making at micro, meso and macro scales that help answer the following questions:

Publications

We like to inspire and share interesting knowledge!

  1. M. Lenninger, M. Skoglund, P. Herman & A. Kumar. Are single-peaked tuning curves tuned for speed rather than accuracy? Nature Communications (in review).
  2. M. Lundqvist, S.L. Brincat, M.R. Warden, T.J. Buschman, E.K. Miller & P. Herman. Working memory control dynamics follow principles of spatial computing. Nature Communications (in review).
  3. M. Molinari, J. Anund Vogel, D. Rolando. Using Living Labs to tackle innovation bottlenecks: the KTH Live-In Lab case study,Applied energy(Extension under review).
  4. N. Chrysanthidis, F. Fiebig, A. Lansner & P. Herman. “Traces of semantization-from episodic to semantic memory in a spiking cortical network model”, eNeuro, July 2022, 9 (4). https://doi.org/10.1523/ENEURO.0062-22.2022.
  5. Fontan, V. Cvetkovic, K. H. Johansson. On behavioral changes towards sustainability for connected individuals: a dynamic decision-making approach, in 4th IFAC Workshop on Cyber-Physical Human Systems, Houston, Texas, December 1-2, 2022.
  6. Taras Kucherenko, Rajmund Nagy, Michael Neff, Hedvig Kjellström, and Gustav Eje Henter. Multimodal analysis of the predictability of hand-gesture properties. In
  7. International Conference on Autonomous Agents and Multi-Agent Systems, 2022.
  8. M. Lundqvist, J. Rose, S.L. Brincat, M.R. Warden, T.J. Buschman, P. Herman, & E.K. Miller. “Reduced variability of bursting activity during working memory.” Scientific Reports 12, no. 1 (2022): 1-10.
  9. N.B. Ravichandran, A. Lansner & P. Herman. “Brain-like combination of feedforward and recurrent network components achieves prototype extraction and robust pattern recognition”. In: Machine Learning, Optimization, and Data Science. LOD 2022. Lecture Notes in Computer Science, Springer, Cham
  10. D. Rolando, W. Mazzotti, M. Molinari. Long-Term Evaluation of Comfort, Indoor Air Quality and Energy Performance in Buildings: The Case of the KTH Live-In Lab Testbeds, Energies, vol. 15, no. 14, pp. 4955, 2022.
  11. N. Chrysanthidis, F. Fiebig, A. Lansner & P. Herman. “Semantization of episodic memory in a spiking cortical attractor network model”, Journal of Computational Neuroscience, vol. 49, no. SUPPL 1, pp. S86–S87, 2021.
  12. A. Karvonen, V. Cvetkovic, P. Herman, K.H. Johansson, H. Kjellström, M. Molinari & M. Skoglund. “The ‘New Urban Science’: towards the interdisciplinary and transdisciplinary pursuit of sustainable transformations.” Urban Transformations 3, no. 1 (2021): 1-13.
  13. M. Molinari, J. Anund Vogel, D. Rolando. Using Living Labs to tackle innovation bottlenecks: the KTH Live-In Lab case study, in Energy Proceedings – Applied Energy Symposium: MIT A+B, 2021.
  14. N.B. Ravichandran, A. Lansner & P. Herman. “Semi-supervised learning with Bayesian Confidence Propagation Neural Network”, in Proc. European Symposium on Artificial Neural Networks (ESANN) 2021. doi.org/10.14428/esann/2021.es2021-156.
  15. Ruibo Tu, Kun Zhang, Hedvig Kjellström, and Cheng Zhang. Optimal transport for causal discovery. In International Conference on Learning Representations, 2022.
  16. Carles Balsells Rodas, Ruibo Tu, and Hedvig Kjellström. Causal discovery from conditionally stationary time-series, arXiv:2110.06257, 2021.
  17. M. Lenninger, M. Skoglund, P. Herman and A. Kumar. Bandwidth expansion in the brain: Optimal encoding manifolds for population coding. In Cosyne, 2021.
  18. S. Molavipour, G. Bassi, and M. Skoglund. On neural estimators for conditional mutual information using nearest neighbors sampling. IEEE Transactions on Signal Processing 69:766-780, 2021.
  19. M. Sorkhei, G. Eje Henter, and H. Kjellström. Full-Glow: Fully conditional Glow for more realistic image generation. In DAGM German Conference on Pattern Recognition, 2021.
  20. Chenda Zhang and Hedvig Kjellström. A subjective model of human decision making
  21. based on Quantum Decision Theory, arXiv:2101.05851, 2021.
  22. M. Molinari and D. Rolando. Digital twin of the Live-In Lab Testbed KTH: Development and calibration. In Buildsim Nordic, 2020.
  23. D. Rolando and M. Molinari. Development of a comfort platform for user feedback: The experience of the KTH Live-In Lab. In International Conference on Applied Energy, 2020.
  24. E. Stefansson, F. J. Jiang, E. Nekouei, H. Nilsson, and K. H. Johansson. Modeling the decision-making in human driver overtaking. In IFAC World Congress, 2020.
  25. Y. Yi, L. Shan, P. E. Paré, and K. H. Johansson. Edge deletion algorithms for minimizing spread in SIR epidemic models. arXiv preprint arXiv:2011.11087, 2020.

About the project

Objective
The main goal of the dBrain collaborative project is to develop an interdisciplinary approach to improve early diagnosis and prognosis for brain diseases and to facilitate the development of therapies by combining computational modelling of the brain and advanced artificial intelligence (AI) based data analyses.
The computational approaches will initially focus on Parkinson’s (PD) and Alzheimer’s diseases (AD), the two most common neurodegenerative diseases. These disease cases are strategically selected in collaboration with the Karolinska Hospital (KS) and Karolinska Institute (KI).

This research is important because the incidence of brain diseases will only increase with increasing human longevity. It is estimated that by 2030, mental disorders will be the largest contributor to the total disease burden.

Background
With increased longevity, the number of patients with brain-related diseases will grow. In 2030, mental illness is estimated to be the main imposition on health care. New advances in data analysis and computational modelling of the brain can use the potential of patient data to prevent and treat brain diseases.

Cross-disciplinary collaboration
The research team represents the School of Electrical Engineering and Computer Science (EECS), the School of Engineering Sciences in Chemistry, Biotechnology and Health (CBH) and the School of Engineering Sciences (SCI).

Watch the recorded presentation at the Digitalize in Stockholm 2023 event:

Activities & Results

Find out what’s going on!

Activities, awards, and other outputs

Results

dBrain addresses two major challenges in our understanding of the brain in health and disease: a) what is the relationship between the structural connectivity and functional interactions between different brain regions and within local neuronal networks?; and b) how do disease-related slow changes in the chemical balance, gene expression and neuronal degeneration shape the brain activity dynamics? We address these challenges via four work packages (WPs), using digital approaches comprising AI-based data analysis, math, computational modelling and simulations.

With the help of Digital Futures, we created a vibrant environment involving three KTH schools and collaborators at KS and KI. In addition to regular weekly meetings at the Digital Futures for the whole consortium (i.e. PIs, the recruited postdocs, and other students), several subgroups also meet more frequently in one of the labs or via zoom. Also, several Master’s Student projects have been completed in addition to the work of PhD students and postdocs. In the weekly meetings, postdocs have particular roles – regularly presenting updates on their work and journal articles selected for general interest. These activities provide the postdocs with training in presenting their work and leading discussions. The postdocs also have arranged team-building activities.

Publications

We like to inspire and share interesting knowledge!

Publications

  1. Sanna Persson, Rodrigo Moreno, Bounding tractogram redundancy, Frontiers in Neuroscience. 18:1403804, 2024, https://doi.org/10.3389/fnins.2024.1403804
  2. Malin Siegbahn, Daniel Jörgens, Filip Asp, Malou Hultcrantz, Rodrigo Moreno, Cecilia Engmér Berglin. Asymmetry in cortical thickness of the Heschl’s gyrus in unilateral ear canal atresia. Otology & Neurotology. 45(4),e342-e350, 2024. https://doi.org/10.1097/MAO.0000000000004137
  3. Caroline Dartora, Anna Marseglia, Gustav Mårtensson, Gull Rukh, Junhua Dang, J- Sebastian Muehlboeck, Lars-Olof Wahlund, Rodrigo Moreno, José Barroso, Daniel Ferreira, Helgi Schiöth, Eric Westman. A deep learning model for brain age prediction using minimally preprocessed T1w-images as input. Frontiers in Aging Neuroscience 15, 1303036, 2024. https://doi.org/10.3389/fnagi.2023.1303036
  4. Antonia Hain, Daniel Jörgens, Rodrigo Moreno. Randomized iterative spherical-deconvolution informed tractogram filtering. Neuroimage 278, 120248, 2023. https://doi.org/10.1016/j.neuroimage.2023.120248
  5. Jingru Fu, Antonios Tzortzakakis, José Barroso, Eric Westman, Daniel Ferreira, Rodrigo Moreno. Fast three-dimensional image generation for healthy brain aging using diffeomorphic registration. Human Brain Mapping 44 (4), 1289-1308, 2023. https://doi.org/10.1002/hbm.26165
  6. Kajsa Ahlgren, Christoffer Olsson, Inna Ermilova, Jan Swenson. New insights into the protein stabilizing effects of trehalose by comparing with sucrose. Physical Chemistry Chemical Physics 25 (32), 21215-21226, 2023. https://doi.org/10.1039/D3CP02639F
  7. Zhou Zhou, Chirstoffer Olsson, T Christian Gasser, Xiaogai Li, Svein Kleiven. The White Matter Fiber Tract Deforms Most in the Perpendicular Direction During In Vivo Volunteer Impacts. J Neurotrauma. 2024 41(23-24):2554-2570, 2024. https://doi.org/10.1089/neu.2024.0183
  8. Jingru Fu, Daniel Ferreira, Örjan Smedby, Rodrigo Moreno. A deformation-based morphometry framework for disentangling Alzheimer’s disease from normal aging using learned normal aging templates. arXiv.2311.08176, 2023. https://doi.org/10.48550/arXiv.2311.08176
  9. Daniel Jörgens, Pierre Marc Jodoin, Maxime Descoteaux, Rodrigo Moreno. Merging multiple input descriptors and supervisors in a deep neural network for tractogram filtering. arXiv:2307.05786, 2023. https://doi.org/10.48550/arXiv.2307.05786
  10. Jingru Fu, Simone Bendazzoli, Örjan Smedby, Rodrigo Moreno. Unsupervised Domain Adaptation for Pediatric Brain Tumor Segmentation. MICCAI 2024 Workshop on Advancing Data Solutions in Medical Imaging AI (ADSMI’24) (in press). https://doi.org/10.48550/arXiv.2406.16848
  11. Sanna Persson, Xinyi Wan, and Rodrigo Moreno. Randomly COMMITting: Iterative Convex Optimization for Microstructure-Informed Tractography. MICCAI 2024 Workshop on Computational Diffusion MRI (CDMRI’24) (in press)
  12. Yu Zhou, Jingru Fu, Örjan Smedby, Rodrigo Moreno. Synthesis of Pediatric Brain Tumor Images with Mass Effect. SPIE – Medical Imaging: Image Processing 12464, 699-707, 2023. San Diego, USA, February 2023. https://doi.org/10.1117/12.2654366
  13. I Carannante, Y Johansson, G Silberberg, J Hellgren Kotaleski. Data-Driven Model of Postsynaptic Currents Mediated by NMDA or AMPA Receptors in Striatal Neurons. Front Comput Neurosci. 2022; 16: 806086.
  14. J Fu, A Tzortzakakis, J Barroso, E Westman, D Ferreira, R Moreno. Generative Aging of Brain Images with Diffeomorphic Registration. arXiv preprint. 2022; arXiv:2205.15607. https://doi.org/10.48550/arXiv.2205.15607
  15. A Hain, D Jörgens, R Moreno. Assessing Streamline Plausibility Through Randomized Iterative Spherical-Deconvolution Informed Tractogram Filtering. arXiv preprint 2022; arXiv:2205.04843. https://doi.org/10.48550/arXiv.2205.04843
  16. M Siegbahn, C Engmér Berglin, R Moreno. Automatic segmentation of the core of the acoustic radiation in humans. Front Neurol. 2022 : 934650. https://doi.org/10.3389/fneur.2022.934650
  17. C Dartora, A Marseglia, G Mårtensson, G Rukh, J Dang, JS Muehlboeck, LO Wahlund, R Moreno, J Barroso, D Ferreira, HB Schiöth, E Westman. Predicting the Age of the Brain with Minimally Processed T1-weighted MRI Data. medRxiv preprint. 2022; medRxiv:2022.09.06.22279594. https://doi.org/10.1101/2022.09.06.22279594
  18. Chakravarty K, Roy S, Sinha A, Nambu A, Chiken S, Hellgren Kotaleski J, Kumar A. Transient Response of Basal Ganglia Network in Healthy and Low-Dopamine State. eNeuro. 2022 Mar 18;9(2):ENEURO.0376-21.2022. doi: 10.1523/ENEURO.0376-21.2022.
  19. Hjorth, J.J.J., Hellgren Kotaleski, J. & Kozlov, A. Predicting Synaptic Connectivity for Large-Scale Microcircuit Simulations Using Snudda. Neuroinform 19, 685–701 (2021). https://doi.org/10.1007/s12021-021-09531-w
  20. G Colombo, R Cuber, L Kanari, A Venturino, R Schulz, M Scolamiero, J Agerberg, H Mathys, L Tsai, W Chachólski,, K Hess, S Siegert. Microglial morphOMICs, a tool for mapping microglial morphology, reveals brain-region-and sex-dependent phenotypes. Nature neuroscience, 2022.
  21. W Chachólski, A Guidolin, I Ren, M Scolamiero, F Tombari. Effective computation of relative homological invariants for functors over posets. arXiv preprint 2022; arXiv:2209.05923
  22. W Chachólski, A Jin, F Tombari. Realisations of posets and tameness. arXiv preprint 2022; arXiv:2112.12209
  23. W Chachólski, René Corbet, Anna-Laura Sattelberger. The Shift-Dimension of Multipersistence Modules. arXiv preprint 2022; arXiv:2112.06509.
  24. W Chachólski, B Gunti, C Landi, Decomposing filtered chain complexes: Geometry behind barcoding algorithms. Computational Geometry, Volume 109, 2023.
  25. Georgios Moschovis. NeuralDynamicsLab at ImageCLEF Medical 2022. ImageCLEF, 2022.
  26. Helson Pascal, Lundqvist Daniel, Svenningsson Per, Vinding Mikkel, Kumar Arvind (2023). Cortex-wide topography of 1/f-exponent in Parkinson’s disease. Nature Parkinson’s Disease, 9, Article number: 109.
  27. Zang, Jie, Liu Shenquan, Helson Pascal, Kumar Arvind (2024). Structural constraints on the emergence of oscillations in multi-population neural networks. eLIFE, 12(Feb 5).
  28. Wärnberg Emil, Kumar Arvind (2023). Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia. PNAS 32(120):e2221994120

Master School Theses

  1. Yuqi Zheng. Predictive MR Image Generation for Alzheimer’s Disease and Normal Aging Using Diffeomorphic Registration. MS thesis KTH 2023. Supervisors Rodrigo Moreno, Jingru Fu. Diva diva2:1833589
  2. Xinyi Wan. Assessing the Streamline Plausibility Through Convex Optimization for Microstructure Informed Tractography(COMMIT) with Deep Learning. MS thesis KTH 2023. Supervisor Rodrigo Moreno. Diva diva2:1754277
  3. Teodor Pstrusiński. Impact of the autoencoder-based FINTA tractogram filtering method on brain networks in subjects with Mild Cognitive Impairment. MS thesis KTH 2023. Supervisors Rodrigo Moreno, Fabian Sinzinger. Diva diva2:1799683
  4. Erika Bengtsdotter. Towards Anatomically Plausible Streamline Tractography with Deep Reinforcement Learning. MS thesis KTH 2022. Supervisors Rodrigo Moreno, Fabian Sinzinger. Diva diva2:1695872
  5. Filippo Maschio. Influence of tractogram filtering in the analysis of tractography data in rat brains. MS thesis University of Padua, Italy 2022. Supervisor Rodrigo Moreno.
  6. Yu Zhou. Synthesis of Pediatric Brain Tumor Image With Mass Effect. MS thesis KTH 2022. Supervisors Rodrigo Moreno, Jingru Fu. Diva diva2:1686799
  7. Antonia Hain. Assessing individual streamline plausibility through randomized spherical deconvolution-informed tractogram filtering. MS thesis University of Saarland, Germany 2021. Supervisor Rodrigo Moreno.
  8. N Hulst, Exploring persistent homology as a method for capturing functional connectivity differences in Parkinson’s Disease. Master Thesis. DiVA, id: diva2:1687257.
  9. Marvin Köpff. Impact of tractogram filtering and graph creation for structural connectomics in subjects with mild cognitive impairment. MS thesis KTH 2020. Supervisors Rodrigo Moreno, Joana Pereira. Diva: diva2:1463915
  10. Johannes Wennberg. Longitudinal assessment of functional connectivity impairment in rat brains. MS thesis KTH 2019. Supervisor Rodrigo Moreno, diva2:1327919
  11. Cristina Zanin. Graph Convolutional Networks for static and dynamic functional connectivity: an application to the Autism Spectrum Disorder. MS thesis University of Padua, Italy 2020. Supervisor Rodrigo Moreno.
  12. Lucas Höglund. Analysis of Eye Tracking Data from Parkinson’s Patients using Machine Learning. MS thesis, KTH.
  13. Leo Bergman. Feature extraction with self-supervised learning on eye-tracking data from Parkinson’s patients and healthy individuals. MS thesis, KTH.
  14. Emma Lind. Analysis of Brain Signals from Patients with Parkinson’s Disease using Self-Supervised Learning. MS thesis, KTH.
  15. Wilhelm Ågren. Feature extraction from MEG data using self-supervised learning. MS thesis, KTH.
  16. Giulia Tuccio. Parameter estimation in a cardiovascular computational model using numerical optimization. MS thesis, KTH.
  17. Paolo Calderaro. Patient simulation. Generation of a machine learning “inverse” digital twin. MS thesis, KTH.

About the project

Objective
In the Data-Limited Learning of Complex Dynamical Systems (DLL) project, we develop methods and tools to learn to control complex dynamic systems using limited data samples. In contrast to traditional machine learning techniques that require large amounts of data for training, this project aims to utilize a priori knowledge of the system and combine such structural knowledge reliably with a limited number of data samples.

The project focuses on two application domains: (i) continuous bioprocessing for safer and more efficient production of biopharmaceuticals and (ii) reinforcement learning of cyber-physical systems in general and robotics in particular. In these domains, probing large amounts of data from the system can be expensive or even impossible without wearing out the system.

Background
Recent years have witnessed spectacular successes in applying machine learning tools to the decision-making and control of complex dynamical systems. These techniques typically combine reinforcement learning with large neural networks, thus requiring a tremendous amount of training data, even to learn simple tasks. Their applications have been mainly limited to specific scenarios, such as board and video games, where generating and gathering data is inexpensive. However, in many biological or physical application domains, data is limited, and probing the system for more data can be expensive or even impossible without destroying the system.

Cross-disciplinary collaboration
This project involves co-PIs and researchers from different disciplines, including computer science, automatic control, machine learning, and biotechnology. To make the interaction as fruitful as possible, the project is divided into three sub-projects: continuous bioprocessing, (ii) reinforcement learning in cyber-physical systems, and (iii) theory. The research involves fundamental theory and practical applications, including involvement from industrial partners. The collaboration between the research team at Digital Futures and the Competence Centre for Advanced BioProduction (AdBIOPRO) will establish a strong, visible, and sustainable research environment that will overarch digitalization and life science research at KTH.

Watch the recorded presentation at the Digitalize in Stockholm 2023 event:

Activities & Results

Find out what’s going on!

Activities, awards, and other outputs

Results

This project has so far resulted in several fundamental research results. The results include but are not limited to: a new biological modelling approach that includes transcriptional information, new system identification techniques for differential-algebraic equations subject to disturbances, fundamental limits for sample complexity and lower bounds for the regret of deterministic discrete dynamical systems.

The results are published in top-tier journals and conferences such as ICML, NeurIPS, and CDC.

Publications

  1. Filippo Vannella, Jaeseong Jeong, and Alexandre Proutiere. Off-Policy Learning in Contextual Bandits for Remote Electrical Tilt Optimization. IEEE transactions on Vehicular Technology, 2022.
  2. Filippo Vannella, Alexandre Proutiere, Yassir Jedra, and Jaeseong Jeong. Remote Electrical Tilt Optimization: a contextual bandit approach. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM) 2022.
  3. Daniel Lundén, Joey Öhman, Jan Kudlicka, Viktor Senderov, Fredrik Ronquist, and David Broman. Compiling Universal Probabilistic Programming Languages with Efficient Parallel Sequential Monte Carlo Inference. In the Proceedings of 31st European Symposium on Programming (ESOP), 2022.
  4. Kévin Colin, Mina Férizbegovic, and Håkan Hjalmarsson, Regret Minimization for Linear Quadratic Adaptive Controllers using Fisher Feedback Exploration, IEEE Control Systems Letters and In Proceedings of the 61th IEEE Conference on Decision and Control (CDC), 2022.
  5. Yassir Jedra and Alexandre Proutiere. Minimal expected regret in online LQR. In Proceedings of the International Conference on Artificial Intelligence and Statistics(AISTATS), 2022.
  6. Robert Bereza, Oscar Eriksson, Mohamed R.-H. Abdalmoaty, David Broman andHåkan Hjalmarsson. Stochastic Approximation for Identification of Non-Linear Differential-Algebraic Equations with Process Disturbances, In Proceedings of the IEEE Conference on Decision and Control (CDC), 2022.
  7. A Ghosh, A.E. Fontcurbeta, Mohamed R. Abdalmoaty, Saikat Chatterjee, Time-varying Normalizing Flow for Generative Modeling of Dynamical Signals, European Signal Processing Conference (EUSIPCO), 2022.
  8. S.Das, A.M. Javid, P.B. Gohain, Y.C. Eldar, S. Chatterjee. Neural Greedy Pursuit for Feature Selection, In the Proceedings of the International Joint Conference on Neural Networks (IJCNN), IEEE WCCI 2022.
  9. X Liang, AM Javid, M Skoglund, S Chatterjee, Decentralized learning of randomization-based neural networks with centralized equivalence, AppliedSoft Computing, 2022.
  10. David Broman. Interactive Programmatic Modeling. In ACM Transactions on Embedded Computing Systems (TECS), Volume 20, Issue 4, Article No 33, Pages 1-26, ACM, 2021.
  11. Mohamed R.-H. Abdalmoaty, Oscar Eriksson, Robert Bereza, David Broman and Håkan Hjalmarsson. Identification of Non-Linear Differential-Algebraic Equation Models with Process Disturbances, In Proceedings of the IEEE Conference on Decision and Control (CDC), 2021.
  12. Mina Ferizbegovic, Per Mattsson, Thomas Schön, and H. Hjalmarsson. Bayes Control of Hammerstein Systems, 19th IFAC Symposium on System Identification, 2021.
  13. Daniel Lundén, Johannes Borgström, and David Broman. Correctness of Sequential Monte Carlo Inference for Probabilistic Programming Languages. In Proceedings of 30th European Symposium on Programming (ESOP), LNCS vol. 12648, Springer, 2021.
  14. Viktor Palmkvist, Elias Castegren, Philipp Haller, and David Broman. Resolvable Ambiguity: Principled Resolution of Syntactically Ambiguous Programs. In Proceedings of the 30th ACM SIGPLAN International Conference on Compiler Construction (CC), ACM, 2021.
  15. A.M. Javid, S. Das, M. Skoglund and S. Chatterjee, A ReLU Dense Layer to Improve the Performance of Neural Networks, in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
  16. X. Liang, M. Skoglund, and S. Chatterjee, Feature reuse for a randomization based neural network, in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
  17. P.G. Jurado X. Liang, A.M. Javid, and S. Chatterjee, Use of deterministic transforms to design weight matrices of a neural network, in European Signal Processing Conference (EUSIPCO), 2021.
  18. Díogo Rodrigues, Mohamed R. Abdalmoaty, Elling W. Jacobsen, Véronqiue Chotteau, and Håkan Hjalmarsson. An Integrated Approach for Modeling and Identification of Perfusion Bioreactors via Basis Flux Modes, Computers & Chemical Engineering, 2021.
  19. Aymen Al Marjani and Alexandre Proutiere. Adaptive Sampling for Best Policy Identification in Markov Decision Processes. In Proceedings of International Conference on Machine Learning (ICML), 2021.
  20. Aymen Al Marjani, Aurélien Garivier, and Alexandre Proutiere: Navigating to the Best Policy in Markov Decision Processes. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2021.
  21. Damianos Tranos and Alexandre Proutiere: Regret Analysis in Deterministic Reinforcement Learning. In Proceedings of the IEEE Conference on Decision and Control (CDC), 2021.
  22. Zhang, L., Wang, M., Castan, A., Hjalmarsson, H. and Chotteau, V., Probabilistic model by Bayesian network for the prediction of antibody glycosylation in perfusion and fed‐batch cell cultures. Biotechnology and Bioengineering, 118(9), pp.3447-3459, 2021.
  23. Chotteau, V., Hagrot, E., Zhang, L., Mäkinen, M. Mathematical modelling of cell culture processes. In Cell Culture Engineering and Technology (pp. 467-484). Springer, Cham, 2021.
  24. Mingliang Wang, Riccardo S. Risuleo, Elling W. Jacobsen, Véronique Chotteau and Håkan Hjalmarsson. Identification of Nonlinear Kinetics of Macroscopic Bio-reactions Using Multilinear Gaussian Processes, Computers and Chemical Engineering, 123(2), 2020.
  25. Stefanie Fonken, Mina Ferizbegovic, and H. Hjalmarsson. Consistent Identification of Dynamic Networks Subject to White Noise Using Weighted Null-Space Fitting, In Proceedings of the IFAC 2020 World Congress, 2020.
  26. Riccardo S. Risuleo and Håkan Hjalmarsson. Nonparametric Models for Hammerstein-Wiener and Wiener-Hammerstein System Identification, In Proceedings of the IFAC 2020 World Congress, 2020.
  27. Saranya Natarajan and David Broman. Temporal Property-Based Testing of a Timed C Compiler using Time-Flow Graph Semantics. In the Proceedings of the Forum on specification & Design Languages (FDL 2020), IEEE, 2020.
  28. Mohamed R. Abdalmoaty and Håkan Hjalmarsson. Identification of Stochastic Nonlinear Models Using Optimal Estimating Functions, Automatica, 119, 2020. Mina Ferizbegovic, Jack Umenberger, Håkan Hjalmarsson and Thomas Schön. Learning robust LQ-controllers using application oriented exploration, IEEE Control Systems Letters, 4(4):19-24 and jointly published in the Proceedings of the IEEE Conference on Decision and Control (CDC), 2020.
  29. A.M. Javid, A. Venkitaraman, M. Skoglund, S. Chatterjee, High-dimensional neural feature design for layer-wise reduction of training cost, EURASIP Journal on Advances in Signal Processing, 2020.
  30. Yassir Jedra and Alexandre Proutiere. Finite-time Identification of Stable Linear Systems Optimality of the Least-Squares Estimator. In Proceedings of the IEEE Conference on Decision and Control (CDC), 2020.
  31. Filippo Vannella, Jaeseong Jeong, and Alexandre Proutiere. Off-policy Learning for Remote Electrical Tilt Optimization. VTC-Fall, 2020.
  32. Jack Umenberger, Mina Ferizbegovic, Thomas Schön and Håkan Hjalmarsson. Robust exploration in linear quadratic reinforcement learning, In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2019 (Spotlight paper).
  33. Yassir Jedra, Alexandre Proutiere. Sample Complexity Lower Bounds for Linear System Identification. In Proceedings of the IEEE Conference on Decision and Control (CDC), 2019.

About the project

Objective
This project aims to develop adaptive social robots that can understand humans’ communicative behaviour and task-related physical actions and adapt their interaction to suit. We aim to investigate and demonstrate fluid and seamless adaptation of intelligent systems to users’ contexts, needs or preferences. To achieve fluidity, such adaptation needs to happen with minimal interruption to the users’ ongoing interaction with the system, without requiring user intervention, while providing accountability and control of the adaption in a task-appropriate, timely, and understandable manner. This will be explored in multiple embodiments: smart speakers, back-projected robotic heads, and dual-arm robots.

Our use case scenario is an adaptive intelligent kitchen assistant that helps humans prepare food and other kitchen-centric tasks, focusing on supporting ageing in place. Our systems will engage in face-to-face spoken and physical collaboration with humans, track the users’ affective states and task-related actions in real-time, adjust performance based on previous interactions, adapt to user preferences, and show intention using a self-regulation perception-production loop. The project will use the Intelligence Augmentation Lab that TMH and RPL plan to set up.

Background
Intelligent systems built around big datasets and machine learning techniques are becoming ubiquitous in people’s lives – smart appliances, wearables, and, increasingly, robots. As these systems are intended to assist an ever wider range of users in their homes, workplaces or public spaces, a typical one-fits-all approach becomes insufficient. Instead, these systems will need to take advantage of the machine learning techniques upon which they are built to adapt to the specific task, user constellation continually, and shared environment in which they are operating. In long-term deployments, the state of the environment, user preferences, skills, and abilities change and must be adapted. This is relevant for socially assistive robots in people’s homes, education or healthcare settings, and robots working alongside workers in small-scale manufacturing environments.

Crossdisciplinary collaboration
The research team represents the School of Electrical Engineering and Computer Science (EECS, KTH), the School of Engineering Sciences in Chemistry, Biotechnology and Health (CBH, KTH) and the Department of Computer and System Science (DSV) at Stockholm University.

Watch the recorded presentation at the Digitalize in Stockholm 2023 event:

Articles

Avancerade adaptiva intelligenta system

Activities & Results

Find out what’s going on!

Results

The main results achieved in the first half of the project towards the 3 main objectives enumerated in the proposal:

  1. Understanding and Design for the Smart Kitchen

Deeper understanding of the cooking process (Kuoppamäki et al, 2021) led to a number of studies on interaction in and around cooking, including list advancement (Jaber & McMillan, 2022), content navigation (Zhao et al., 2022), designing conversational interaction (Kuoppamäki et al., 2023), the impact of contextual awareness on command construction and delivery (Jaber et al., in submission) and an ongoing study comparing proactive organisational support between young adults and those over 65 (Kuoppamäki et al.).

  1. Perception and Representation of Long Term Human Action

Research showing that thermal imaging is a modality relevant for detecting frustration in human-robot interaction (Mohamed et al., 2022). The models to predict frustration based on a dataset of 18 participants interacting with a Nao social robot in our lab were tested using features from several modalities: thermal, RGB, Electrodermal Activity (EDA), and all three combined. The models reached an accuracy of 89% with just RGB features, 87% using only thermal features, 84% using EDA, and 86% when using all modalities. We are also investigating the accuracy of frutration prediction models using data collected at the KTH Library where students ask directions to a Furhat robot.

  1. Input, Output, and Interaction for Smart Assistive Technologies

In order to fulfil the goal of adaptation of virtual agent and robot behaviour to different contexts, we have researched interlocutor-aware facial expressions in dyadic interaction (Jonell et al, 2020) as well as adaptive facial expressions in controllable generation of speech and gesture: we have developed techniques to generate conversational speech, with control over speaking style to signal e.g. certainty or uncertainty (Wang et al, 2022, Kirkland et al 2022) as well as models that are able to generate coherent speech and gesture from a common representation (Wang et al 2021). Another direction concerns adaptation to spatial contexts and environments, where we have used imitation learning and physical simulation to produce referential gestures (Deichler et al, 2022).

Publications

We like to inspire and share interesting knowledge!

  1. Parag Khanna, Mårten Björkman, Christian Smith. “Human Inspired Grip-Release Technique for Robot-Human Handovers”, 2022 IEEE-RAS International Conference on Humanoid Robots, Ginowan, Japan. (Accepted for publication)
  2. Sanna Kuoppamäki, Mikaela Hellstrand, and Donald McMillan (2023, forthcoming). Designing conversational scenarios with older adults digital repertoires: Graphic transcript as a design method. In: Hänninen, R., Taipale, S., Haapio-Kirk, L. (eds). Embedded and everyday technology: Digital repertoires in an ageing society. UCL Press.
  3. Youssef Mohamed, Giulia Ballardini, Maria Teresa Parreira, Séverin Lemaignan, and Iolanda Leite. 2022. Automatic Frustration Detection Using Thermal Imaging. In Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’22). IEEE Press, 451–460
  4. Razan Jaber and Donald McMillan. 2022. Cross-Modal Repair: Gaze and Speech Interaction for List Advancement. In Proceedings of the 4th Conference on Conversational User Interfaces (CUI ’22). Association for Computing Machinery, New York, NY, USA, Article 25, 1–11.
  5. Yaxi Zhao, Razan Jaber, Donald McMillan, and Cosmin Munteanu. 2022. “Rewind to the Jiggling Meat Part”: Understanding Voice Control of Instructional Videos in Everyday Tasks. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 58, 1–11.
  6. Kirkland, A., Lameris, H., Gustafson, J., Székely, É. (2022) “Where’s the uh, hesitation? The interplay between filled pause location, speech rate and fundamental frequency”, Interspeech 2022, Incheon, Korea.
  7. Wang, S., Gustafson, J., Székely, É. (2022) “Evaluating Sampling-based Filler Insertion with Spontaneous TTS”, 13th Edition of the Language Resources and Evaluation Conference (LREC 2022), Marseille.
  8. Deichler, A., Wang, S., Alexanderson, S., & Beskow, J. (2022). Towards Context-Aware Human-like Pointing Gestures with RL Motion Imitation. In Context-Awareness in Human-Robot Interaction: Approaches and Challenges, workshop at 2022 ACM/IEEE International Conference on Human-Robot Interaction.
  9. Donald McMillan and Razan Jaber. 2021. Leaving the Butler Behind: The Future of Role Reproduction in CUI. In Proceedings of the 3rd Conference on Conversational User Interfaces (CUI ’21). Association for Computing Machinery, New York, NY, USA, Article 11, 1–4.
  10. Sanna Kuoppamäki, Sylvaine Tuncer, Sara Eriksson, and Donald McMillan. 2021. Designing Kitchen Technologies for Ageing in Place: A Video Study of Older Adults’ Cooking at Home. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 2, Article 69 (June 2021), 19 pages.
  11. Katie Winkle, Gaspar Isaac Melsión, Donald McMillan, and Iolanda Leite. 2021. Boosting Robot Credibility and Challenging Gender Norms in Responding to Abusive Behaviour: A Case for Feminist Robots. In Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’21 Companion). Association for Computing Machinery, New York, NY, USA, 29–37.
  12. Wang, S., Alexanderson, A., Gustafson, J., Beskow, J., Henter, G. and Székely, É. (2021) “Integrated Speech and Gesture Synthesis”, 23rd ACM International Conference on Multimodal Interaction (ICMI 2021), Montreal
  13. Jonell, P., Kucherenko, T., Henter, G. E., & Beskow, J. (2020, October). Let’s Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (pp. 1-8).

About the project

Objective
The main goal of the extension HiSSx is to plan for long-term scientific and societal impact by establishing a new transdisciplinary research centre on a human-centric smart built environment. Three specific tasks have been identified to reach this goal:

Background
The building sector accounts for over one-third of global energy consumption and emissions; therefore, transforming the built environment is urgent to achieve the societal climate and sustainability goals by 2030. Traditional solutions for smart and sustainable buildings and cities have focused and prioritized the technological perspective, overlooking the human dimension and leading to sub-optimization. The project HiSS tackled this complex, unresolved, and cross-disciplinary challenge addressing human decision-making and behaviour in complex environments like smart buildings. Using a multi-layered system perspective approach, the project HiSS aimed to integrate knowledge, tools, and methodologies from cognitive and behavioural sciences, control, building, and energy technology.

Among multiple success factors, HiSS developed a novel model for choice probability; the project also delved into and made notable contributions to human behavioural modelling, learning and modelling social networks and addressing key limitations in behavioural modelling and control in buildings. These advancements in the study of human behaviour further inspired us to investigate cognitive brain processes and simulate human decision-making in neuroeconomic game scenarios using computational brain models. Joint interest in brain function also sparked a collaborative effort towards a deeper theoretical understanding of how the brain processes input information at the level of neural circuits.

These modelling advancements, together with an innovative data-driven control architecture and the development of a digital twin for smart building testbeds, provided the proof-of-concept of the feasibility of scalable human-centric controls in buildings. These concepts have proven and demonstrated the technical feasibility of the exploitation of human-centric networks and controls in the KTH Live-In Lab, a platform of smart building testbeds and have been integrated into the analysis of policy expectations of households’ role in the smart grid, resulting in identified gaps, where social issues are not accounted for.

Cross-disciplinary collaboration
The cross-disciplinary research team is formed by seven PIs representing three KTH Schools: Electrical Engineering and Computer Science (EECS), Architecture and the Build Environment (ABE), and Industrial Engineering and Management (ITM). The project team will also involve key industrial and societal stakeholders and our external academic collaborators from Technion, Imperial College London and Uppsala University.