Exploring the depths with João de Sousa, Digital Futures Scholar in residence

Hi João, you have an extensive background in autonomous vehicles and cyber-physical systems. Could you share some insights into your research interests and how they have evolved over the years?

– My main research is about systems of autonomous maritime vehicle systems, including underwater, surface, and air vehicles, with applications to the ocean sciences, oil and gas and wind farms, security, and defence. The vision is to design and deploy systems of systems (SoS) that will deliver novel capabilities that are well beyond what can be delivered by isolated vehicles or even by robotic swarms (as recognized by some of the proponents of robotic swarming approaches). Important concepts are teams, intra and inter-team coordination, team formation and maintenance, dynamic allocation of vehicles to teams, and high-level planning and execution control at the system of systems level. Most of the initial insights about this research area were developed when I was studying and working at the University of California at Berkeley (UCB) in the early 2000s. I was a member of the SHIFT language development group, focused on designing and implementing a modelling language for dynamic networks of hybrid automata – dynamic networks are an important component of SoS. The development of the language was partly inspired by UCB’s pioneering work in layered control architectures for automated highway systems (AHS) – the layering principles still inform our current designs. AHS work also highlighted the merits and limitations of formal verification, strongly influenced by Josepf Sifakis, Turing Award in 2007 along with Edmund M. Clarke and E. Allen Emerson, for their work on model checking. The compositional inspiration for SoS came from the ONR-funded Mobile Offshore Base, in which we developed a control architecture and implementation for scaled models to enable independent semi-submersible scaled platforms to form a runway, as it should happen in the middle of the ocean. Yet, in another project funded by DARPA, I developed teaming concepts to solve a complex coordination problem under adversarial behaviour. These concepts provided the inspiration for the design of a layered architecture of controllers in which vehicles, teams, sub-teams, and task controllers were dynamically created and linked for teams to perform complex missions.

Some of these insights and developments were later transitioned to developments in maritime robotics at the Laboratório de Sistemas e Tecnologias Subaquáticas (LSTS). This was at a time when the focus in maritime robotics was mainly on single-vehicle operations in the presence of the challenges posed by the underwater environment (e.g., no GPS or refuelling stations underwater, fast attenuation of radio waves, slow and unreliable acoustic communications, etc.). While the underwater environment presented major challenges for deploying isolated vehicles, it also presented opportunities to innovate in cooperative robotics to overcome the localization, power consumption, and communication challenges.

It was then that we understood the need to develop low-cost vehicles to test our novel developments in multi-vehicle systems – commercial-of-the-shelf vehicles were just too expensive and were mostly based on closed software systems, thus precluding their use for research and development. Until then, our developments were mainly about a technological push. The first application’s pull came from the Portuguese Navy. This started a unique partnership, initially focused on developing an autonomous underwater vehicles program. In the process, we learned a lot from the operational know-how of the Portuguese Navy and strengthened our partnership.

Our operational know-how has been key to successful LSTS campaigns in the Atlantic, Arctic, Pacific Oceans, Mediterranean, and Adriatic seas. These campaigns added another dimension to our research: interdisciplinary ocean studies with robotic vehicles and ship-robotic surveys. This new dimension prompted the development of novel interdisciplinary approaches to find, track, and sample dynamic features of the ocean with unprecedented spatial and temporal resolution. We are especially interested in closing the observation, modelling, assimilation, and sampling cycle with networked unmanned vehicles and ship-robotic surveys. This is not a simple endeavour. In fact, the problem of modelling and assimilation is especially challenging because the spatial and temporal resolutions of ocean models are typically not consistent with the high-resolution data provided by unmanned vehicles. Moreover, in some cases, lower-resolution models fail to capture, in an aggregate fashion, behaviours exhibited in higher-resolution models. Recent work in the field highlights the role of high-resolution modelling, sampling, and machine learning as one promising approach to address this problem. A reference should be made here to the fact that Syukuro Manabe and Klaus Hasselmann, two of the three 2021 Nobel laureates in Physics, were awarded “for the physical modelling of Earth’s climate, quantifying variability and reliably predicting global warming”.

In 2010, we started, in cooperation with the Portuguese Navy, what is now known as the Robotic Experimentation and Prototyping with Maritime Unmanned Systems (REPMUS) exercise. Today, this is the world’s largest exercise of its kind, co-organized by the Portuguese Navy, Porto University, the European Defense Agency, the Maritime Unmanned Systems initiative and the Center for Maritime Research and Experimentation from NATO. The success of the exercise, developed in the framework of the triple helix (academia, industry, and armed forces), is predicated on the evaluation and testing of new developments in operational environments. REPMUS is also pushing the envelope in interoperability into interchangeability (I2I), with tens of networked heterogeneous underwater, surface, and aerial vehicles coordinated with the help of a hierarchy of human operators, tactical units, and commanders. In doing so, the exercise provides a glimpse of future SoS operations and additional motivation and challenges to our developments.

Finally, we observe that SoS are cyber-physical systems in which physical and computational dynamics are tightly coupled, while physical entities (e.g., vehicles) and computational entities (e.g., processes) dynamically interact over time – e.g., computational processes can migrate between vehicles and vehicles may interact with other vehicles, that may come and go. This is why research in SoS is highly interdisciplinary, at the intersection of control and computation. For example, models of SoS will have to incorporate, in addition to vehicle dynamic models, other types of dynamics, including mobile linkage and mobile locality, and related concepts of mobile computing and mobile computation. In this respect, the pioneering work of Robin Milner, Turing Award winner in 1991, was truly inspirational, mainly for his development of the Pi calculus, which models systems with a dynamically changing communication topology, and Bigraphs, which models dynamic linkage and mobility. This is done with the help of a place graph and a link graph (actually, a hypergraph), together with reaction rules governing the evolution of the bigraph.

Now, it is the time for analysis, synthesis, evaluation and testing of SoS.

Your leadership at the Laboratório de Sistemas e Tecnologias Subaquáticas (LSTS) has led to significant advancements in underwater vehicle technology. Can you discuss some of the major accomplishments of the LSTS, including the design of the Light Autonomous Underwater Vehicle and the LSTS open-source software toolchain?

– We have been working for over 20 years in the field. I will briefly allude to some key accomplishments. One of the most significant accomplishments is our trajectory as a laboratory. Starting from scratch, we designed, built, and deployed multiple vehicles and developed a reputation for excellence in networked vehicle systems.

Another significant accomplishment was the development of the award-winning Light Autonomous Underwater Vehicle (LAUV), which started almost two decades ago when mainstream developments focused on bigger and very expensive vehicles. In fact, we demonstrated that small vehicles could deliver comparable performance for some applications. The LAUV is commercialized through our spinoff, Oceanscan, Marine Systems and Technologies – there are over 100 units in operation in Europe, North America, and Asia. In addition, we built a frugal fleet of vehicles to evaluate and test novel multi-vehicle systems capabilities. This was at a time when the operation of fleets was prohibitive for most research institutions.

The LSTS open-source software toolchain (https://lsts.fe.up.pt/toolchain) was another major accomplishment. It supports underwater, surface, and aerial vehicles, as well as inter-operated communications networks and interoperability. One decade later, our software still provides some unmatched functionalities, and it is evolving to become a powerful cloud-based system.

Our unique cooperation with the Portuguese Navy is still a case study and an example to follow. In addition to co-developing an underwater vehicle program for the Portuguese Navy, we jointly founded the REP, now REPMUS exercise, the world’s largest exercise of its kind.

Another relevant accomplishment was coordinating the H2020 EU Marine robotics research infrastructure network project. The project mobilized a comprehensive consortium of most of the key European marine robotics players to open up marine robotics research infrastructures to European and worldwide researchers and establish a world-class marine robotics integrated infrastructure. Access to infrastructures was granted through open calls. Close to 60 projects worldwide were granted access to our infrastructures, thus providing a glimpse of future best practices in cooperation and utilization of robotic infrastructures. Projects ranged from exploring the Blanes canyon with an HROV deployed from a ship from Ifremer to cooperative control of multiple LAUVs from LSTS.

In 2018, we led an international team on board the R/V Falkor during the highly successful Schmidt Ocean Institute cruise entitled Exploring Fronts with Multiple Robots. The goal of the cruise was to demonstrate a novel approach to observing the ocean with multiple autonomous underwater, surface, and aerial vehicles operated by the R/V Falkor. These vehicles were powered by our software toolchain to study the Northern Pacific Subtropical front, located approximately 1000 nautical miles west of San Diego, United States of America. The cruise duration was 3 weeks, with two weeks on-site working non-stop in 4 daily shifts.

Currently, we are developing a Command-and-Control Center for Atlantic Operations to develop, test, and demonstrate long-endurance operations in the Atlantic, with special emphasis on the triangle joining Continental Portugal and the Madeira and Azores islands. Operations will involve our expanding fleet, including long-endurance vehicles and vehicles from partners and associates. The goal is also to connect our centre to similar centres in Europe and the United States for cooperative missions to take place in the Atlantic.

You’ve been instrumental in fostering a global research community through conferences and workshops on hybrid systems and autonomous vehicles. How do you see the future of this field evolving, and what opportunities or challenges do you anticipate?

– Maritime unmanned vehicle systems are already impacting the ocean sciences, security and defence, offshore oil, gas, and wind projects. But this is just the beginning. In fact, most of the current deployments are still based on single-vehicle operations, closed software systems, limited interactions with other systems, and significant support from human operators. However, as ocean robotic operations become more complex, with increasing numbers of platforms, operators, manned vessels, and interactions, novel forms of organization, well beyond what is done today, will have to be developed. This will require a paradigm shift from operations with a few vehicle systems to systems of systems and underlying organizing principles. However, this paradigm shift presents another challenge summarized as Martec’s law: “Technology changes exponentially fast, yet organizations change logarithmically slow”. Development and organisational resets may be needed as the exponential/logarithmic gap widens over time. This will have profound impacts on models of engineering development, as well as on systems’ life cycles.

Complexity cannot be addressed in isolation. SoS are evolving into complex network structures powered by Artificial Intelligence, targeting varied applications and requiring unprecedented forms of interdisciplinary work. Similarly, research and development should become more networked and interdisciplinary, with agile development models closed around evaluation and testing, first in Digital Twins, before transitioning to operational environments. However, the promise of Digital Twins should not be overstated (e.g., ocean models with varying degrees of spatial and temporal resolution may not be consistent), and more research into solving these challenges is needed. We can only speculate at this stage, but it seems that networked research and development could be one of the ways for organizations to match exponential technological growth and complexity.

Finally, we will be facing significant challenges in cyber-security, ethics, interoperability, interchangeability, standardization, legal frameworks, and frugality, to name just a few.

As a member of various committees and advisory boards, including the NATO MUS Innovation Advisory Board and the Advisory Board of the Swedish Marine Robotics Center, how do you envision collaboration and innovation within the international research community in the coming years?

In line with my previous answer, international collaboration and innovation networks will be key to addressing the challenges of exponential technological growth and market demand. Moreover, some collaboration efforts should be tentatively aligned with developments in other areas (e.g., space). This is why some of the space agencies are considering underwater environments as space analogues, closer to home, to test space systems and technologies.

The effectiveness of collaboration and innovation networks poses significant challenges regarding coordination mechanisms and the integrity of intellectual property, to name just a few. Examples coming from projects like the H2020 EU Marine robotics research infrastructure network or large-scale exercises like REPMUS may provide templates and insights into effective work practices.

Last but not least, you’re a Digital Futures Scholar in residence from February 2024 to September 2024 – can you elaborate on your experience so far? What are your main objectives during this period? What do you expect to be the key outcomes or achievements?

– I have just finished my first weeks with Digital Futures. I really like the informal and friendly atmosphere, the open space, the design of the building, and, of course, the interdisciplinary work being developed here. This is simply fantastic and intellectually stimulating, as I would expect from Digital Futures and KTH.

I had excellent discussions about the future of marine robotics with my host, Professor Ivan Stenius, and other colleagues from SMARC. I had fantastic discussions about models of systems with dynamic structure, machine learning, and control of networked vehicles with Professor Karl Johansson and other colleagues from the division of decision and control systems. I am very grateful to all of them. But this is just the beginning of my Scholar in Residence program. Now, I will be away for several weeks, but I will stay in touch and work on concrete plans for future collaborative efforts, aiming at addressing fundamental problems in models of computation and layered control architectures for multi-vehicle systems and nested ocean models. Equally important will be the co-development of novel robotic systems for sustained and economically sustainable ocean exploration with colleagues in the ocean sciences.

I am ambitious and tend to be very optimistic, but I will do my best to develop and strengthen an international network of collaborations at the intersection of science and technology with the goal of starting innovative coursework development and collaborative projects that will help us to understand and monitor how key issues such as climate change, ocean acidification, unsustainable fishing, pollution, waste, loss of habitats and biodiversity, shipping, security, and mining are affecting global ocean sustainability and stewardship.

Of course, I must finish in Portuguese by saying “Muito obrigado”.

Text: Johanna Gavefalk/João de Sousa