About the project

Objective
In the Data-Limited Learning of Complex Dynamical Systems (DLL) project, we develop methods and tools to learn to control complex dynamic systems using limited data samples. In contrast to traditional machine learning techniques that require large amounts of data for training, this project aims to utilize a priori knowledge of the system and combine such structural knowledge reliably with a limited number of data samples.

The project focuses on two application domains: (i) continuous bioprocessing for safer and more efficient production of biopharmaceuticals and (ii) reinforcement learning of cyber-physical systems in general and robotics in particular. In these domains, probing large amounts of data from the system can be expensive or even impossible without wearing out the system.

Background
Recent years have witnessed spectacular successes in applying machine learning tools to the decision-making and control of complex dynamical systems. These techniques typically combine reinforcement learning with large neural networks, thus requiring a tremendous amount of training data, even to learn simple tasks. Their applications have been mainly limited to specific scenarios, such as board and video games, where generating and gathering data is inexpensive. However, in many biological or physical application domains, data is limited, and probing the system for more data can be expensive or even impossible without destroying the system.

Cross-disciplinary collaboration
This project involves co-PIs and researchers from different disciplines, including computer science, automatic control, machine learning, and biotechnology. To make the interaction as fruitful as possible, the project is divided into three sub-projects: continuous bioprocessing, (ii) reinforcement learning in cyber-physical systems, and (iii) theory. The research involves fundamental theory and practical applications, including involvement from industrial partners. The collaboration between the research team at Digital Futures and the Competence Centre for Advanced BioProduction (AdBIOPRO) will establish a strong, visible, and sustainable research environment that will overarch digitalization and life science research at KTH.

Watch the recorded presentation at the Digitalize in Stockholm 2023 event:

Activities & Results

Find out what’s going on!

Activities, awards, and other outputs

Results

This project has so far resulted in several fundamental research results. The results include but are not limited to: a new biological modelling approach that includes transcriptional information, new system identification techniques for differential-algebraic equations subject to disturbances, fundamental limits for sample complexity and lower bounds for the regret of deterministic discrete dynamical systems.

The results are published in top-tier journals and conferences such as ICML, NeurIPS, and CDC.

Publications

  1. Filippo Vannella, Jaeseong Jeong, and Alexandre Proutiere. Off-Policy Learning in Contextual Bandits for Remote Electrical Tilt Optimization. IEEE transactions on Vehicular Technology, 2022.
  2. Filippo Vannella, Alexandre Proutiere, Yassir Jedra, and Jaeseong Jeong. Remote Electrical Tilt Optimization: a contextual bandit approach. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM) 2022.
  3. Daniel Lundén, Joey Öhman, Jan Kudlicka, Viktor Senderov, Fredrik Ronquist, and David Broman. Compiling Universal Probabilistic Programming Languages with Efficient Parallel Sequential Monte Carlo Inference. In the Proceedings of 31st European Symposium on Programming (ESOP), 2022.
  4. Kévin Colin, Mina Férizbegovic, and Håkan Hjalmarsson, Regret Minimization for Linear Quadratic Adaptive Controllers using Fisher Feedback Exploration, IEEE Control Systems Letters and In Proceedings of the 61th IEEE Conference on Decision and Control (CDC), 2022.
  5. Yassir Jedra and Alexandre Proutiere. Minimal expected regret in online LQR. In Proceedings of the International Conference on Artificial Intelligence and Statistics(AISTATS), 2022.
  6. Robert Bereza, Oscar Eriksson, Mohamed R.-H. Abdalmoaty, David Broman andHåkan Hjalmarsson. Stochastic Approximation for Identification of Non-Linear Differential-Algebraic Equations with Process Disturbances, In Proceedings of the IEEE Conference on Decision and Control (CDC), 2022.
  7. A Ghosh, A.E. Fontcurbeta, Mohamed R. Abdalmoaty, Saikat Chatterjee, Time-varying Normalizing Flow for Generative Modeling of Dynamical Signals, European Signal Processing Conference (EUSIPCO), 2022.
  8. S.Das, A.M. Javid, P.B. Gohain, Y.C. Eldar, S. Chatterjee. Neural Greedy Pursuit for Feature Selection, In the Proceedings of the International Joint Conference on Neural Networks (IJCNN), IEEE WCCI 2022.
  9. X Liang, AM Javid, M Skoglund, S Chatterjee, Decentralized learning of randomization-based neural networks with centralized equivalence, AppliedSoft Computing, 2022.
  10. David Broman. Interactive Programmatic Modeling. In ACM Transactions on Embedded Computing Systems (TECS), Volume 20, Issue 4, Article No 33, Pages 1-26, ACM, 2021.
  11. Mohamed R.-H. Abdalmoaty, Oscar Eriksson, Robert Bereza, David Broman and Håkan Hjalmarsson. Identification of Non-Linear Differential-Algebraic Equation Models with Process Disturbances, In Proceedings of the IEEE Conference on Decision and Control (CDC), 2021.
  12. Mina Ferizbegovic, Per Mattsson, Thomas Schön, and H. Hjalmarsson. Bayes Control of Hammerstein Systems, 19th IFAC Symposium on System Identification, 2021.
  13. Daniel Lundén, Johannes Borgström, and David Broman. Correctness of Sequential Monte Carlo Inference for Probabilistic Programming Languages. In Proceedings of 30th European Symposium on Programming (ESOP), LNCS vol. 12648, Springer, 2021.
  14. Viktor Palmkvist, Elias Castegren, Philipp Haller, and David Broman. Resolvable Ambiguity: Principled Resolution of Syntactically Ambiguous Programs. In Proceedings of the 30th ACM SIGPLAN International Conference on Compiler Construction (CC), ACM, 2021.
  15. A.M. Javid, S. Das, M. Skoglund and S. Chatterjee, A ReLU Dense Layer to Improve the Performance of Neural Networks, in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
  16. X. Liang, M. Skoglund, and S. Chatterjee, Feature reuse for a randomization based neural network, in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
  17. P.G. Jurado X. Liang, A.M. Javid, and S. Chatterjee, Use of deterministic transforms to design weight matrices of a neural network, in European Signal Processing Conference (EUSIPCO), 2021.
  18. Díogo Rodrigues, Mohamed R. Abdalmoaty, Elling W. Jacobsen, Véronqiue Chotteau, and Håkan Hjalmarsson. An Integrated Approach for Modeling and Identification of Perfusion Bioreactors via Basis Flux Modes, Computers & Chemical Engineering, 2021.
  19. Aymen Al Marjani and Alexandre Proutiere. Adaptive Sampling for Best Policy Identification in Markov Decision Processes. In Proceedings of International Conference on Machine Learning (ICML), 2021.
  20. Aymen Al Marjani, Aurélien Garivier, and Alexandre Proutiere: Navigating to the Best Policy in Markov Decision Processes. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2021.
  21. Damianos Tranos and Alexandre Proutiere: Regret Analysis in Deterministic Reinforcement Learning. In Proceedings of the IEEE Conference on Decision and Control (CDC), 2021.
  22. Zhang, L., Wang, M., Castan, A., Hjalmarsson, H. and Chotteau, V., Probabilistic model by Bayesian network for the prediction of antibody glycosylation in perfusion and fed‐batch cell cultures. Biotechnology and Bioengineering, 118(9), pp.3447-3459, 2021.
  23. Chotteau, V., Hagrot, E., Zhang, L., Mäkinen, M. Mathematical modelling of cell culture processes. In Cell Culture Engineering and Technology (pp. 467-484). Springer, Cham, 2021.
  24. Mingliang Wang, Riccardo S. Risuleo, Elling W. Jacobsen, Véronique Chotteau and Håkan Hjalmarsson. Identification of Nonlinear Kinetics of Macroscopic Bio-reactions Using Multilinear Gaussian Processes, Computers and Chemical Engineering, 123(2), 2020.
  25. Stefanie Fonken, Mina Ferizbegovic, and H. Hjalmarsson. Consistent Identification of Dynamic Networks Subject to White Noise Using Weighted Null-Space Fitting, In Proceedings of the IFAC 2020 World Congress, 2020.
  26. Riccardo S. Risuleo and Håkan Hjalmarsson. Nonparametric Models for Hammerstein-Wiener and Wiener-Hammerstein System Identification, In Proceedings of the IFAC 2020 World Congress, 2020.
  27. Saranya Natarajan and David Broman. Temporal Property-Based Testing of a Timed C Compiler using Time-Flow Graph Semantics. In the Proceedings of the Forum on specification & Design Languages (FDL 2020), IEEE, 2020.
  28. Mohamed R. Abdalmoaty and Håkan Hjalmarsson. Identification of Stochastic Nonlinear Models Using Optimal Estimating Functions, Automatica, 119, 2020. Mina Ferizbegovic, Jack Umenberger, Håkan Hjalmarsson and Thomas Schön. Learning robust LQ-controllers using application oriented exploration, IEEE Control Systems Letters, 4(4):19-24 and jointly published in the Proceedings of the IEEE Conference on Decision and Control (CDC), 2020.
  29. A.M. Javid, A. Venkitaraman, M. Skoglund, S. Chatterjee, High-dimensional neural feature design for layer-wise reduction of training cost, EURASIP Journal on Advances in Signal Processing, 2020.
  30. Yassir Jedra and Alexandre Proutiere. Finite-time Identification of Stable Linear Systems Optimality of the Least-Squares Estimator. In Proceedings of the IEEE Conference on Decision and Control (CDC), 2020.
  31. Filippo Vannella, Jaeseong Jeong, and Alexandre Proutiere. Off-policy Learning for Remote Electrical Tilt Optimization. VTC-Fall, 2020.
  32. Jack Umenberger, Mina Ferizbegovic, Thomas Schön and Håkan Hjalmarsson. Robust exploration in linear quadratic reinforcement learning, In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2019 (Spotlight paper).
  33. Yassir Jedra, Alexandre Proutiere. Sample Complexity Lower Bounds for Linear System Identification. In Proceedings of the IEEE Conference on Decision and Control (CDC), 2019.

About the project

Objective
This project aims to develop adaptive social robots that can understand humans’ communicative behaviour and task-related physical actions and adapt their interaction to suit. We aim to investigate and demonstrate fluid and seamless adaptation of intelligent systems to users’ contexts, needs or preferences. To achieve fluidity, such adaptation needs to happen with minimal interruption to the users’ ongoing interaction with the system, without requiring user intervention, while providing accountability and control of the adaption in a task-appropriate, timely, and understandable manner. This will be explored in multiple embodiments: smart speakers, back-projected robotic heads, and dual-arm robots.

Our use case scenario is an adaptive intelligent kitchen assistant that helps humans prepare food and other kitchen-centric tasks, focusing on supporting ageing in place. Our systems will engage in face-to-face spoken and physical collaboration with humans, track the users’ affective states and task-related actions in real-time, adjust performance based on previous interactions, adapt to user preferences, and show intention using a self-regulation perception-production loop. The project will use the Intelligence Augmentation Lab that TMH and RPL plan to set up.

Background
Intelligent systems built around big datasets and machine learning techniques are becoming ubiquitous in people’s lives – smart appliances, wearables, and, increasingly, robots. As these systems are intended to assist an ever wider range of users in their homes, workplaces or public spaces, a typical one-fits-all approach becomes insufficient. Instead, these systems will need to take advantage of the machine learning techniques upon which they are built to adapt to the specific task, user constellation continually, and shared environment in which they are operating. In long-term deployments, the state of the environment, user preferences, skills, and abilities change and must be adapted. This is relevant for socially assistive robots in people’s homes, education or healthcare settings, and robots working alongside workers in small-scale manufacturing environments.

Crossdisciplinary collaboration
The research team represents the School of Electrical Engineering and Computer Science (EECS, KTH), the School of Engineering Sciences in Chemistry, Biotechnology and Health (CBH, KTH) and the Department of Computer and System Science (DSV) at Stockholm University.

Watch the recorded presentation at the Digitalize in Stockholm 2023 event:

Articles

Avancerade adaptiva intelligenta system

Activities & Results

Find out what’s going on!

Results

The main results achieved in the first half of the project towards the 3 main objectives enumerated in the proposal:

  1. Understanding and Design for the Smart Kitchen

Deeper understanding of the cooking process (Kuoppamäki et al, 2021) led to a number of studies on interaction in and around cooking, including list advancement (Jaber & McMillan, 2022), content navigation (Zhao et al., 2022), designing conversational interaction (Kuoppamäki et al., 2023), the impact of contextual awareness on command construction and delivery (Jaber et al., in submission) and an ongoing study comparing proactive organisational support between young adults and those over 65 (Kuoppamäki et al.).

  1. Perception and Representation of Long Term Human Action

Research showing that thermal imaging is a modality relevant for detecting frustration in human-robot interaction (Mohamed et al., 2022). The models to predict frustration based on a dataset of 18 participants interacting with a Nao social robot in our lab were tested using features from several modalities: thermal, RGB, Electrodermal Activity (EDA), and all three combined. The models reached an accuracy of 89% with just RGB features, 87% using only thermal features, 84% using EDA, and 86% when using all modalities. We are also investigating the accuracy of frutration prediction models using data collected at the KTH Library where students ask directions to a Furhat robot.

  1. Input, Output, and Interaction for Smart Assistive Technologies

In order to fulfil the goal of adaptation of virtual agent and robot behaviour to different contexts, we have researched interlocutor-aware facial expressions in dyadic interaction (Jonell et al, 2020) as well as adaptive facial expressions in controllable generation of speech and gesture: we have developed techniques to generate conversational speech, with control over speaking style to signal e.g. certainty or uncertainty (Wang et al, 2022, Kirkland et al 2022) as well as models that are able to generate coherent speech and gesture from a common representation (Wang et al 2021). Another direction concerns adaptation to spatial contexts and environments, where we have used imitation learning and physical simulation to produce referential gestures (Deichler et al, 2022).

Publications

We like to inspire and share interesting knowledge!

  1. Parag Khanna, Mårten Björkman, Christian Smith. “Human Inspired Grip-Release Technique for Robot-Human Handovers”, 2022 IEEE-RAS International Conference on Humanoid Robots, Ginowan, Japan. (Accepted for publication)
  2. Sanna Kuoppamäki, Mikaela Hellstrand, and Donald McMillan (2023, forthcoming). Designing conversational scenarios with older adults digital repertoires: Graphic transcript as a design method. In: Hänninen, R., Taipale, S., Haapio-Kirk, L. (eds). Embedded and everyday technology: Digital repertoires in an ageing society. UCL Press.
  3. Youssef Mohamed, Giulia Ballardini, Maria Teresa Parreira, Séverin Lemaignan, and Iolanda Leite. 2022. Automatic Frustration Detection Using Thermal Imaging. In Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’22). IEEE Press, 451–460
  4. Razan Jaber and Donald McMillan. 2022. Cross-Modal Repair: Gaze and Speech Interaction for List Advancement. In Proceedings of the 4th Conference on Conversational User Interfaces (CUI ’22). Association for Computing Machinery, New York, NY, USA, Article 25, 1–11.
  5. Yaxi Zhao, Razan Jaber, Donald McMillan, and Cosmin Munteanu. 2022. “Rewind to the Jiggling Meat Part”: Understanding Voice Control of Instructional Videos in Everyday Tasks. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 58, 1–11.
  6. Kirkland, A., Lameris, H., Gustafson, J., Székely, É. (2022) “Where’s the uh, hesitation? The interplay between filled pause location, speech rate and fundamental frequency”, Interspeech 2022, Incheon, Korea.
  7. Wang, S., Gustafson, J., Székely, É. (2022) “Evaluating Sampling-based Filler Insertion with Spontaneous TTS”, 13th Edition of the Language Resources and Evaluation Conference (LREC 2022), Marseille.
  8. Deichler, A., Wang, S., Alexanderson, S., & Beskow, J. (2022). Towards Context-Aware Human-like Pointing Gestures with RL Motion Imitation. In Context-Awareness in Human-Robot Interaction: Approaches and Challenges, workshop at 2022 ACM/IEEE International Conference on Human-Robot Interaction.
  9. Donald McMillan and Razan Jaber. 2021. Leaving the Butler Behind: The Future of Role Reproduction in CUI. In Proceedings of the 3rd Conference on Conversational User Interfaces (CUI ’21). Association for Computing Machinery, New York, NY, USA, Article 11, 1–4.
  10. Sanna Kuoppamäki, Sylvaine Tuncer, Sara Eriksson, and Donald McMillan. 2021. Designing Kitchen Technologies for Ageing in Place: A Video Study of Older Adults’ Cooking at Home. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 2, Article 69 (June 2021), 19 pages.
  11. Katie Winkle, Gaspar Isaac Melsión, Donald McMillan, and Iolanda Leite. 2021. Boosting Robot Credibility and Challenging Gender Norms in Responding to Abusive Behaviour: A Case for Feminist Robots. In Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’21 Companion). Association for Computing Machinery, New York, NY, USA, 29–37.
  12. Wang, S., Alexanderson, A., Gustafson, J., Beskow, J., Henter, G. and Székely, É. (2021) “Integrated Speech and Gesture Synthesis”, 23rd ACM International Conference on Multimodal Interaction (ICMI 2021), Montreal
  13. Jonell, P., Kucherenko, T., Henter, G. E., & Beskow, J. (2020, October). Let’s Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (pp. 1-8).

About the project

Objective
The main goal of the extension HiSSx is to plan for long-term scientific and societal impact by establishing a new transdisciplinary research centre on a human-centric smart built environment. Three specific tasks have been identified to reach this goal:

Background
The building sector accounts for over one-third of global energy consumption and emissions; therefore, transforming the built environment is urgent to achieve the societal climate and sustainability goals by 2030. Traditional solutions for smart and sustainable buildings and cities have focused and prioritized the technological perspective, overlooking the human dimension and leading to sub-optimization. The project HiSS tackled this complex, unresolved, and cross-disciplinary challenge addressing human decision-making and behaviour in complex environments like smart buildings. Using a multi-layered system perspective approach, the project HiSS aimed to integrate knowledge, tools, and methodologies from cognitive and behavioural sciences, control, building, and energy technology.

Among multiple success factors, HiSS developed a novel model for choice probability; the project also delved into and made notable contributions to human behavioural modelling, learning and modelling social networks and addressing key limitations in behavioural modelling and control in buildings. These advancements in the study of human behaviour further inspired us to investigate cognitive brain processes and simulate human decision-making in neuroeconomic game scenarios using computational brain models. Joint interest in brain function also sparked a collaborative effort towards a deeper theoretical understanding of how the brain processes input information at the level of neural circuits.

These modelling advancements, together with an innovative data-driven control architecture and the development of a digital twin for smart building testbeds, provided the proof-of-concept of the feasibility of scalable human-centric controls in buildings. These concepts have proven and demonstrated the technical feasibility of the exploitation of human-centric networks and controls in the KTH Live-In Lab, a platform of smart building testbeds and have been integrated into the analysis of policy expectations of households’ role in the smart grid, resulting in identified gaps, where social issues are not accounted for.

Cross-disciplinary collaboration
The cross-disciplinary research team is formed by seven PIs representing three KTH Schools: Electrical Engineering and Computer Science (EECS), Architecture and the Build Environment (ABE), and Industrial Engineering and Management (ITM). The project team will also involve key industrial and societal stakeholders and our external academic collaborators from Technion, Imperial College London and Uppsala University.

About the project

Objective
The goal of the collaborative impact project is to leverage the novel technical and theoretical research results from the data-limited learning project to create added value and technical transfer to the Swedish industry (Saab and AstraZeneca) and to create interactive physical demonstrators for the Digital Futures Hub and the general public.

Background
This impact project is an extension to the 4-year project Data-Limited Learning of Complex Dynamical Systems (DLL), where the new project focuses on impact activities, software prototypes, and demonstrators. See the following for the previous DLL project.

The previous DLL project consisted of three connected sub-projects (i) bioprocessing, (ii) reinforcement learning for cyber-physical systems, and (iii) theoretical foundation. Within these sub-projects, our project has resulted in significant research results. In this new collaborative impact project, we focus on a subset of these results within the three tasks. The key aspect of the usefulness of this new project is to take the results from a theoretical or pure academic setting, to create demonstrators for the public, and to enable technical transfer to the Swedish industry.

Cross-disciplinary collaboration
This project involves co-PIs and researchers from different disciplines, including computer science, automatic control, machine learning, and biotechnology. The project is divided into three main project tasks, each aiming for separate impact activities:

About the project

Objective
Adaptive Intelligent Homes (AIH) aims to develop a suite of demonstrators, combining the cutting-edge advances made within the parent collaborative research project (AAIS) on user state recognition, human-robot handovers, multimodal referring expression generation, controllable speech synthesis, and proactively supporting users in complex tasks. AIH will extend the context-awareness and adaptiveness of those conversational robots from a laboratory environment to a real-life environment and from individual to collaborative settings, where multiple users interact with the system simultaneously. Through engaging with societal and municipal actors, the project contributes to more inclusive and accessible robotic systems with potential applications to promote healthy lifestyles and provide home care services for older adults.

Background
AI-powered agents provide an opportunity to manage, coordinate and initiate activities within and across households, contributing to healthy lifestyles and improved quality of life for users of all ages. To achieve this goal, these robotic systems must be aligned with the user’s needs, preferences, and interests, and they must adapt themselves to complex environments involving multiple users performing overlapping and interwoven household tasks, such as cooking.

Cooking in the home is not simply instruction-giving; it is a multi-party, parallelized activity requiring timing, action relevance, and adaptability from both the human and any AI assistance. The project has envisioned an adaptive intelligent kitchen assistant that could help humans prepare food and other kitchen-centric tasks to enhance healthy ageing and improve quality of life.

Cross-disciplinary collaboration
The interdisciplinary approach will allow us to synthesize the human-centric understanding of the home and kitchen environment to the technical capabilities of dialogue systems and robotic agents. This could indicate, for instance, new modalities for personalising the intelligent kitchen assistant for different user groups or linguistic styles that could be adapted to different family members’ skill sets and preferences. An important societal aspect is to explore and design for improved inclusivity of dialogue systems by recognizing diverse ways of interacting with the system based on user’s age, gender, ethnicity, or disability and investigate the long-term impact of deployment of such systems in the home environment to the quality of life and self-efficacy among end-users. Designing for improved accessibility of dialogue systems has the potential to result in more equal access to intelligent and assistive home environments.

About the project

Objective
The Explainable and Ethical Machine Learning for Knowledge Discovery from Medical Data Sources (EXTREMUM) project aims to develop a novel platform for learning from complex medical data sources with a focus on two healthcare application areas: adverse drug event detection and early detection and treatment of cardiovascular diseases.

The team will present a new framework for data management and analysis of data integration, methods for machine learning, and ethical issues related to predictive models. This project’s fundamental breakthrough is establishing a novel knowledge management and discovery framework for medical data sources. The outcome will be a set of methods and tools for integrating complex medical data sources, a set of predictive models for learning from these sources emphasising interpretability and explanatory features, and simultaneously focusing on maintaining ethical integrity in the underlying decision mechanisms that rule machine learning.

Background
One of the biggest challenges that research and business entities have been facing recently is that the data provided by today’s technologies originate from multiple data sources in massive quantities and at rapid rates. In addition to their volume, such data sources are complex and heterogeneous. In the presence of these complex and continuously growing information sources, domain scientists, data analysts, and novice users have been struggling to manage this complexity and arrive from abundant data to usable and interpretable models and exploitable domain knowledge. Towards this end, these data sources need to be monitored in real-time. Hence, data integration indexing and predictive modelling have become a major challenge. Consider, for example, the healthcare domain, where numerous data sources in the form of Electronic Health Records (EHRs), such as billing codes, registry data, and pharmaceutical data, are used for developing predictive models of, e.g., heart failure prevention and treatment progression, or adverse drug effect (ADE) detection.

An example workflow of the EXTREMUM framework starts from radiography exams, ranking them based on their severity and urgency, then moving on to their classification and tagging while eventually producing clinically relevant explanatory text.

Cross-disciplinary collaboration
The project is a collaborative effort between four research institutions: the Department of Computer and Systems Sciences at Stockholm University, the Department of Law at Stockholm University, RISE Research Institutes Sweden, Division ICT, and the Department of Automatic Control, School of Electrical Engineering and Computer Science (EECS, KTH).

Watch the recorded presentation at the Digitalize in Stockholm 2023 event:

Activities & Results

Find out what’s going on!

Activities, awards, and other outputs

Results

The main goal of this project is to develop and establish a novel data representation, integration, and knowledge discovery framework for medical data sources, focusing on explainable machine learning governed by legal and ethical principles. Particular emphasis will be given to healthcare data sources available in EHRs, emphasising two particular healthcare problems: (a) early prediction and treatment of cardiovascular diseases and (b) adverse drug event identification and prevention.

Project objectives, along with the current achievements and results:

Objective 1: Unified representation and integration of complex data spaces. We have explored and defined novel space representations, similarity measures, and methods for searching and indexing large and complex data spaces for complex data sources. The basic challenge is the temporal nature of the data spaces and the inherent temporal dependencies that may exist within the same and across different data sources in these spaces. Our main solutions include the adoption and employment of temporal abstractions for temporal event sequences (both time series and discrete event sequences) of univariate and multivariate events.

Objective 2: Explainable predictive models for complex data sources. Regarding the second objective, our intention is to develop novel predictive modelling algorithms that can support and exploit data sources of complex nature and heterogeneity while at the same time providing explainability of their predictions in the form of interpretable features or rule sets. Towards this end, we have developed new methods for explainable machine learning with an emphasis on example-based explanations, such as counterfactuals for time series counterfactuals, as well as for event sequences). Moreover, we have explored model-based explanations and more concretely local explanations (such as LIME and SHAP), their applicability to the medical domain, and to what extent they are trusted and can be adopted by medical practitioners. In the case of time series data, interpretability can be achieved by focusing on white-box models, such as those described by differential equations, for which we have developed and analyzed several estimation algorithms.

We have also developed novel methods to reverse engineer predictors/filters for Hidden Markov Models and Linear Dynamical Systems, rendering them explainable by “opening the box”; furthermore, by modifying the samples arriving at a learning algorithm, we are developing methods that can improve learning while preserving privacy, enforcing fairness and arriving at more interpretable models. Our future plan is to develop similar methods for counterfactual generation for forecasting (e.g., for predicting critical events in the ICU and preventing them by suggesting critical actions. Furthermore, we intend to conduct a more extensive qualitative study on the generated explanations and counterfactuals involving a larger group of medical practitioners. This will also involve a qualitative assessment and deployment of our current demonstrator tool (published recently in the hospital setting on actual use cases.

Objective 3: Adherence to legal and ethical frameworks. We have focused on the legal and ethical implications associated with the development and use of predictive modelling in relation to healthcare data analysis. To this end, the General Data Protection Regulation (GDPR) is of utmost relevance. We have identified the predominant social values promoted in the GDPR and transposed these legal rules into mathematical equivalents. The finding from this process has been interesting to the extent that the identified social values are mutually exclusive and that promoting one of them occurs at the expense of the others, necessitating a balancing act. The social values studied thus far were privacy, accuracy and explainability. The results of this examination are relevant from a legislative techniques perspective, culminating in these interdisciplinary findings being published in a legal publication venue.

Publications

We like to inspire and share interesting knowledge!

Journal articles (J)

  1. R.A.González,C.R.Rojas,S.Pan,andJ.Welsh.“Theoreticalandpracticalaspects of the convergence of the SRIVC estimator for over-parameterized models”. Automatica, 142:110355, 2022
  2. S. Pan, J. Welsh, R. A. González, and C. R. Rojas. “Consistency analysis and bias elimination of the instrumental variable based state variable filter method”. Automatica, 144:110511, 2022
  3. P. Wachel and C. R. Rojas. “An adversarial approach to adaptive model predictive control”. Journal of Advances in Applied & Computational Mathematics, 9, 135–146, 2022
  4. R. A. González, C. R. Rojas, S. Pan, and J. Welsh. “Refined instrumental variable methods for unstable continuous-time systems in closed-loop”. International Journal of Control (accepted for publication), 2022
  5. C.R.RojasandP.Wachel.“Onstate-spacerepresentationsofgeneraldiscrete-time dynamical systems”. IEEE Transactions on Automatic Control (accepted for publication), 2022
  6. R.Mochaourab,A.Venkitaraman,I.Samsten,P.Papapetrou,andC.R.Rojas,“Post- hoc Explainability for Time Series Classification: Toward a Signal Processing Perspective,” IEEE Signal Processing Magazine, Special Issue on Explainability in Data Science: Interpretability, Reproducibility, and Replicability, vol. 39, no. 4, Jul. 2022
  7. S. Greenstein, P. Papapetrou, and R. Mochaourab, “Embedding Human Values into Artificial Intelligence,” in De Vries, Katja (ed.), De Lege, Iustus förlag, Uppsala University, 2022
  8. J. Rebane, I. Samsten, L. Bornemann, and P. Papapetrou, “SMILE: A feature-based temporal abstraction framework for event-interval sequence classification”. In Data Mining and Knowledge Discovery (DAMI) 35(1): 372-399, 2021
  9. B. Djehiche, O. Mazhar, and C. R. Rojas. “Finite impulse response models: A nonasymptotic analysis of the least squares estimator”. Bernoulli, 27(2):976–1000, 2021
  10. R. A. González, C. R. Rojas, S. Pan, and J. Welsh. “Consistent identification of continuous-time systems under multisine input signal excitation”. Automatica, 133:109859, 2021
  11. I. Lourenço, R. Mattila, C. R. Rojas, X. Hu, and B.Wahlberg. “Hidden Markov models: Inverse filtering, belief estimation and privacy protection”. Journal of Systems Science and Complexity, 34:1801–1820, 2021
  12. J. Rebane, I. Samsten, and P. Papapetrou, “Exploiting Complex Medical Data with Interpretable Deep Learning for Adverse Drug Event Prediction”. In Artificial Intelligence in Medicine (AIM), 28(8): 1651-1659, 2021
  13. Locally and globally explainable time series tweaking KAIS), 62 (5): 1671- 1700, 2020
  14. R. Mattila, C. R. Rojas, V. Krishnamurthy, and B. Wahlberg. “Inverse filtering for hidden Markov models with applications to counter-adversarial autonomous systems”. IEEE Transactions on Signal Processing, 68:4987–5002, 2020

Conference papers (C)

  1. Z. Lee, M. Trincavelli, and P. Papapetrou, “Finding Local Groupings of Time Series“, In the European Conference on Machine Learning and Principles and Practices of Knowledge Discovery in Databases (ECML/PKDD), to appear
  2. L. Mondrejevski, I. Miliou, A. Montanino, D. Pitts, J. Hollmén, and P. Papapetrou, “FLICU: A Federated Learning Workflow for Intensive Care Unit Mortality Prediction”. In Computer-Based Medical Systems (CBMS), to appear
  3. M. Bampa, T. Fasth, S. Magnusson, and P. Papapetrou, “EpidRLearn: Learning intervention strategies for epidemics with Reinforcement Learning“. In the International Conference on Artificial Intelligence in Medicine (AIME), 2022
  4. S. Guarnizo, I. Miliou, and P. Papapetrou, “Impact of Dimensionality on Nowcasting Seasonal Influenza with Environmental Factors“. In the International Symposium on Intelligent Data Analysis (IDA), 128-142, 2022
  5. J. Parsa, C. R. Rojas, and H. Hjalmarsson. “Optimal input design for sparse system identification”. In Proceedings of the 2022 European Control Conference (ECC), 2022
  6. B. Lakshminarayanan and C. R. Rojas. “A statistical decision-theoretical perspective on the two-stage approach to parameter estimation”. In Proceedings of the 61th IEEE Conference on Decision and Control (CDC 2022) (accepted for publication), 2022
  7. I. Lourenço, R. Winqvist, C. R. Rojas, and B. Wahlberg. “A teacher-student Markov decision process-based framework for online correctional learning”. In Proceedings of the 61th IEEE Conference on Decision and Control (CDC 2022) (accepted for publication), 2022
  8. Z. Wang, I. Samsten, and P. Papapetrou, Counterfactual Explanations for Survival Prediction of Cardiovascular ICU Patients. In Artificial Intelligence in Medicine (AIME), 338-348, 2021 [best student paper award]
  9. J.PavlopoulosandP.Papapetrou,CustomizedNeuralPredictiveMedicalText:AUse- Case on Caregivers. In Artificial Intelligence in Medicine (AIME), 438-443, 2021
  10. J. Rebane, I. Samsten, P. Pantelidis, and P. Papapetrou, Assessing the Clinical Validity of Attention-based and SHAP Temporal Explanations for Adverse Drug Event Predictions. In Computer-based Medical Systems (CBMS), 235-240, 2021
  11. Z. Wang, I. Samsten, R. Mochaourab, and P. Papapetrou, Learning Time Series Counterfactuals via Latent Space Representations. In Discovery Science (DS), 369- 384, 2021
  12. I. Lourenço, R. Mattila, C. R. Rojas, and B. Wahlberg. “Cooperative system identification via correctional learning”. In Proceedings of the 19th IFAC Symposium on System Identification (SYSID 2021), 2021
  13. R. A. González, C. R. Rojas, and H. Hjalmarsson. “Non-causal regularized least- squares for continuous-time system identification with band-limited input excitations”. In Proceedings of the 60th IEEE Conference on Decision and Control (CDC 2021), 2021
  14. R. A. González, C. R. Rojas, S. Pan, and J. S. Welsh. “The SRIVC algorithm for continuous-time system identification with arbitrary input excitation in open and closed loop”. In Proceedings of the 60th IEEE Conference on Decision and Control (CDC 2021), 2021
  15. Z. Lee, T. Lindgren, and P. Papapetrou, “Z-Miner: an efficient method for mining frequent arrangements of event intervals“. In ACM Knowledge Discovery and Data Mining (KDD), 524-534, 2020
  16. Z. Lee, S. Girdzijauskas, and P. Papapetrou, “Z-Embedding: A spectral representation of event intervals for efficient clustering and classification“. In the European Conference on Machine Learning and Principles and Practices of Knowledge Discovery in Databases (ECML/PKDD), 710-726, 2020
  17. M. Bampa, P. Papapetrou, and J. Hollmen, “A clustering framework for patient phenotyping with application to adverse drug events“. In IEEE Computer-Based Medical Systems (CBMS), 177-182, 2020
  18. J. Pavlopoulos and P. Papapetrou, “Clinical predictive keyboard using statistical and neural language modeling“. In IEEE Computer-Based Medical Systems (CBMS), 293- 296, 2020
  19. Z. Lee, J. Rebane, and P. Papapetrou, “Mining disproportional frequent arrangements of event intervals for investigating adverse drug events“. In IEEE Computer-Based Medical Systems (CBMS), 293-296, 2020
  20. N. B. Kumarakulasinghe, T. Blomberg, J. Liu, A. S. Leao and P. Papapetrou, “Evaluating local interpretable model-agnostic explanations on clinical machine learning classification models“. In IEEE Computer-Based Medical Systems (CBMS), 7-12, 2020
  21. R. Mattila, I. Lourenço, V. Krishnamurthy, C. R. Rojas, and B. Wahlberg. “What did your adversary believe? Optimal smoothing in counter-autonomous systems”. In Proceedings of the 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2020), Barcelona, Spain, 2020
  22. R. A. González and C. R. Rojas. “Finite sample deviation and variance bounds for first order autoregressive processes”. In Proceedings of the 45th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2020), Barcelona, Spain, 2020
  23. R. A. González and C. R. Rojas. “A finite-sample deviation bound for stable autoregressive processes”. In Proceedings of the 2nd Conference on Learning for Decision and Control (L4DC), Berkeley, USA, 2020
  24. R. A. González, J. S. Welsh, and C. R. Rojas. “Enforcing stability through ellipsoidal inner approximations in the indirect approach for continuous-time system identification”. In Proceedings of the 21st IFAC World Congress (IFAC 2020), Berlin, Germany, 2020
  25. R. Mattila, C. R. Rojas, E. Moulines, V. Krishnamurthy, and B.Wahlberg. “Fast and consistent learning of hidden Markov models by incorporating non-consecutive correlations”. In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), 2020
  26. I. Lourenço, R. Mattila, C. R. Rojas, and B. Wahlberg. “How to protect your privacy? A framework for counter-adversarial decision making”. In Proceedings of the 59th IEEE Conference on Decision and Control (CDC 2020), 2020
  27. R. Mochaourab, S. Sinha, S. Greenstein, and P. Papapetrou, “Robust Counterfactual Explanations for Privacy-Preserving SVMs,” International Conference on Machine Learning (ICML 2021), Workshop on Socially Responsible Machine Learning, Jul. 2021

Demo paper (D)

  1. R. Mochaourab, S. Sinha, S. Greenstein, and P. Papapetrou, “Demonstrator on Counterfactual Explanations for Differentially Private Support Vector Machines,” European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2022), Demo Track, Sept. 2022

About the project

Objective
Our planet faces unprecedented environmental challenges, including rapid urbanization, deforestation, pollution, loss of biodiversity, melting glacier, rising sea-level, and climate change. In recent years, the world also witnessed numerous natural disasters, from droughts, heat waves and wildfires to flooding and hurricanes, killing thousands and causing billions in property and infrastructural damages. With its synoptic view and large area coverage at regular revisits, satellite remote sensing has played a crucial role in monitoring our changing planet.

The overall objective of the EO-AI4GlobalChange (EO-AI4GlobalChange: Earth Observation Big Data and Deep Learning for Change Detection and Environmental Impact Assessment: Urbanization and Forest Fire Monitoring as Examples) project is to develop innovative and robust methods for monitoring global environmental changes using Earth Observation big data and deep learning. This research will focus on three major global environmental challenges: urbanization, wildfires and flooding.

Open and free Earth observation big data such as Sentionel-1 SAR and Sentinel-2 MSI data have been used to demonstrate the novel deep learning-based methods in selected cities worldwide and various wildfire and flooding sites across the globe. For urban mapping, a novel Domain Adaptation (DA) approach using semi-supervised learning has been developed for built-up area extraction. Several novel methods have been developed for urban change detection, including a dual-stream U-Net and a Siamese Difference Dual-Task network. For early detection of active fires, Gated Recurrent Units and transformer networks have been used to improve GOES-R and VIIRS dense time series detections.

Figure 1. The Dixie Fire burned 963,309 acres (389,837 ha), was the largest single (i.e. non-complex) wildfire in recorded California history and the second-largest wildfire overall. The burned areas were clearly detected by the Sentinel-2 MSI (Left) and Sentinel-1 SAR (Right) images.

Figure 2. Started on July 13, 2021, and contained on October 25, 2021, the progression of the Dixie Fire was captured by Sentinel-1 and Sentinel-2 images.

For wildfire progression monitoring, transfer learning-based models have been evaluated to exploit Sentinel-1 SAR and Sentinel-2 MSI data. Civil contingencies agencies can use the timely and reliable information the project generates to support effective emergency management and decision-making during and after wildfires and flooding. Automatic and continuous mapping of urban areas and their changes can support sustainable and resilient city planning and contribute to monitoring the UN 2030 Urban Sustainable Development Goal (SDG 11).

Background
In recent years, the world has experienced many devastating wildfires due to human-induced climate change, most recently in Australia around the turn of 2019/2020. Wildfires kill and displace people, damage property and infrastructure, burn vegetation and harm wildlife, and cost billions of euros to fight. Up-to-date and reliable information on fire risk, active fires, fire extent, progression and damage assessment is critical for effective emergency management and decision support.

The pace of urbanization has been unprecedented. Rapid urbanization poses significant social and environmental challenges, including sprawling informal settlements, increased pollution, urban heat island, loss of biodiversity and ecosystem services, and making cities more vulnerable to disasters. Therefore, accurate and consistent information on urban changing patterns is essential to support sustainable urban development and the UN’s New Urban Agenda.

Activities & Results

Find out what’s going on!

Activities, awards, and other outputs

Notable Presentations

Conference Presentations

Results

The overall objective of the EO-AI4GlobalChange project is to develop innovative and robust methods for monitoring environmental changes using Earth Observation big data and deep learning. This research will focus on three major global environmental challenges: urbanization, wildfires and flooding.

The specific objectives are:

  1. to develop novel, automatic and globally applicable methods for effective change detection using deep learning and big data analytics to exploit all available Earth Observation data;
  2. to adapt the developed methods for continuous urban change detection, for flood mapping, and for wildfire monitoring including early detection of active fires, near real-time monitoring of wildfire progression and rapid damage estimation;
  3. to assess the environmental impacts of urbanization and wildfires on biodiversity and ecosystem services.

The research achievements are:

List of open-data repositories and developed software

Publications

We like to inspire and share interesting knowledge!

Peer-reviewed journal publications and book chapter

  1. Hafner, S. Y. Ban and A. Nascetti, 2022. Unsupervised Domain Adaptation for Global Urban Extraction Using Sentinel-1 and Sentinel-2 Data. Remote Sensing of Environment. Vol. 280,113192, https://doi.org/10.1016/j.rse.2022.113192.
  2. Hafner, S., A. Nascetti, H. Azizpour and Y. Ban, 2022. Sentinel-1 and Sentinel- 2 Data Fusion for Urban Change Detection Using a Dual Stream U-Net. IEEE Geoscience and Remote Sensing Letters, Vol. 19, pp. 1-5. doi: 10.1109/LGRS.2021.3119856.
  3. Mugiraneza, T., S. Hafner, J. Haas, Y. Ban. 2022. Monitoring urbanization and environmental impact in Kigali, Rwanda using Sentinel-2 MSI data and ecosystem service bundles. International Journal of Applied Earth Observation and Geoinformation, Vol. 109, 102775. DOI: https://doi.org/10.1016/j.jag.2022.102775.
  4. Yadav, R., A. Nascetti, Y. Ban. 2022. Deep Attentive Fusion network for Flood Detection on Uni-temporal Sentinel-1 Data. Frontiers in Remote Sensing, section Microwave Remote Sensing (Accepted).
  5. Hu, X., Y. Ban, and A, Nascetti. 2021. Uni-Temporal Multispectral Imagery for Burned Area Mapping with Deep Learning. Remote Sensing, 13, no. 8: 1509.
  6. Hu, X., Y. Ban, and A, Nascetti. 2021. Sentinel-2 MSI data for active fire detection in major fire-prone biomes: A multi-criteria approach. International Journal of Applied Earth Observation and Geoinformation, 101, 102347.
  7. Zhang, P., Y. Ban, and A. Nascetti. 2021. Learning U-Net without Forgetting for Near Real-Time Wildfire Monitoring by the Fusion of SAR and Optical Time Series. Remote Sensing of Environment, 1-12. https://doi.org/10.1016/j.rse.2021.112467.
  8. Zhao Y, and Y. Ban. 2022. GOES-R Time Series for Early Detection of Wildfires with Deep GRU-Network. Remote Sensing. 2022; 14(17):4347. https://doi.org/10.3390/rs14174347.
  9. Furberg, D., and Y. Ban. 2021. Satellite monitoring of urbanization and environmental impacts in Stockholm, Sweden, through a multiscale approach. Urban Remote Sensing, 2nd Edition, Ed: X. Yang, Wiley.
  10. Furberg, D., Ban, Y. & Mörtberg, U., 2020. Monitoring urban green infrastructure changes and impact on habitat connectivity using high-resolution satellite data. Remote Sensing, 12(18), 3072. https://doi.org/10.3390/rs12183072.
  11. Ban, Y., Zhang, P., Nascetti, A., Bevington, A. R., Wulder, M. A., 2020. Near Real-Time Wildfire Progression Monitoring with Sentinel-1 SAR Time Series and Deep Learning. Scientific Reports, 10(1), 1–15. https://www.nature.com/articles/s41598-019-56967-x.

Peer-Reviewed Conference Papers

  1. Hafner, S., Y. Ban and A. Nascetti, 2022. Urban Change Detection Using a Dual-Task Siamese Network and Semi-Supervised Learning. Proceedings of IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 2022, Kuala Lumpur, Malaysia.
  2. Hu, X., P. Zhang and Y. Ban, 2022. Gan-Based SAR to Optical Image Translation in Fire- Disturbed Regions. Proceedings of IGARSS’2022, Kuala Lumpur, Malaysia.
  3. Zhang, P., X. Hu, and Y. Ban, 2022. Wildfire-S1S2-Canada: A Large-Scale Sentinel-1/2 Wildfire Burned Area Mapping Dataset Based on the 2017-2019 Wildfires in Canada. Proceedings of IGARSS’2022, Kuala Lumpur, Malaysia.
  4. Zhao, Y., Y. Ban, 2022. Global Scale Burned Area Mapping Using Bi-Temporal ALOS-2 PALSAR-2 L-Band Data. Proceedings of IGARSS’2022, Kuala Lumpur, Malaysia.
  5. Yadav, R., A. Nascetti and Y. Ban, 2022. “Attentive Dual Stream Siamese U-net for Flood Detection on Multi-temporal Sentinel-1 Data,” Proceedings, IGARSS’2022.
  6. Yadav, R., Y. Ban and A. Nascetti, 2022. “Building Change Detection using Multi-Temporal Airborne LiDAR Data,” Proceedings, XXIV ISPRS Congress (2022 edition), 6–11 June 2022, Nice, France.
  7. Hafner, S., Y. Ban and A. Nascetti, 2021. Exploring the Fusion of Sentinel-1 SAR and Sentinel-2 MSI Data for Built-Up Area Mapping Using Deep Learning. Proceedings of IGARSS’2021, Brussels, Belgium.
  8. Zhao, Y., Y. Ban and A. Nascetti, 2021. “Early Detection of Wildfires with GOES-R Time- Series and Deep GRU Network,” Proceedings of IGARSS’2021, Brussels, Belgium.