Abstract
Keywords
1. Introduction
Ross Ashby, in Conant (1981)
Swarm robotics is one of the grand frontiers in modern robotics (Yang et al., 2018). Robot swarms promise utility in multiple application domains such as search and rescue, military operations, pollution control and space exploration. In contrast to other multi-robot systems (Parker, 2007), robot swarms are designed to solve tasks by local robot interactions only, resulting in robustness against node failures and scalability of swarms (Balch & Parker, 2002; Barca & Sekercioglu, 2013; Bayındır, 2016; Brambilla, Ferrante, Birattari, & Dorigo, 2013; Dudek, Jenkin, Milios, & Wilkes, 1996; Hamann, 2018; Şahin, 2004; Schranz, Umlauft, Sende, & Elmenreich, 2020). Fascinatingly, simple interaction rules between agents can be sufficient to produce complex collective behaviour (Bonabeau, Dorigo, Marco, Theraulaz, & Théraulaz, 1999; Camazine et al., 2003; Couzin, 2009). Swarm intelligence may, therefore, be implemented in simple robots as is the current standard in swarm robotics research. However, collective algorithms may also be useful in multi-robot systems consisting of capable robot agents.
One of the challenges to deploy robot swarms in the real world is the coupling of human operators with robot swarms. The need for human systems integration can be grouped into three categories. First, the swarm may not be capable of achieving its goals autonomously, resulting in the need to incorporate a human controller to compensate for the mismatch between reality and the robot’s control model (Woods & Hollnagel, 2006). Second, increased autonomy may promote human-out-of-loop effects that may even result in reduced performance of the overall system (Bainbridge, 1983; Hollnagel & Woods, 2005; Woods & Hollnagel, 2006). Third, some applications require human supervision because of legal and ethical concerns (Verbruggen, 2019).
Human–swarm interaction (HSI; Kolling, Walker, Chakraborty, Sycara, & Lewis, 2015) as a research field combines human-factors engineering and swarm robotics to address the challenge how the distributed nature of robot swarms and the centralized control and feedback demands of humans can be combined into one loop. A human operator must be able to exert control over the swarm, which stands in opposition to the traditional concept of swarm intelligence that strives for solutions without leadership (Barca & Sekercioglu, 2013). In addition, flexibility and scalability of swarms pose high demands on operator capabilities (Kolling et al., 2015). In general, HSI can be subdivided into remote interaction, where operator and swarm are not located in the same environment, and proximal interaction, where operator and swarm are in a line-of-sight relationship (Goodrich & Schultz, 2008; Kolling et al., 2015). Most real-world applications that include proximal interaction, for example, search and rescue, will probably also be in need of a remote component for overall command and control.
Around a decade ago, Penders et al. (2011) presented a prototype for HSI in the context of a fire-fighter application. In line with Penders et al., we often observe that humans are fully occupied with their tasks in real-world applications, such as in search and rescue, without the capacity to allocate mental workload to additional machine-supervision tasks. This has been formulated by Woods and Hollnagel (2006) as the law of stretched systems, which states that systems including humans tend to exploit their full capacity. Therefore, we envision a future human–swarm implementation where human operators are not required to deliberately control a robot swarm but where the swarm acts together with the human operator as one unity.
Imagine a future fire-fighter, who we shall call Tom, arriving at a warehouse emergency scene with a search and rescue assignment (Carrillo-Zapata et al., 2020; Payton, Estowski, & Howard, 2004; Penders et al., 2011). The warehouse is filled with smoke, resulting in poor visibility. Even worse, the deployed fire-responder drone could not be of much help because its sensors could not penetrate the specialized building material of the warehouse. Tom’s order is to locate trapped humans, there may be four according to warehouse security. Being in the warehouse, he must continuously assess the situation and balance trade-offs in this high-risk scenario. Fortunately, Tom is supported by a swarm of small robots that have already been deployed. Tom activates his capability amplifier which, besides communicating with human team members, amplifies his natural human sensory–motor range. The swarm robots spread into the whole warehouse (Payton, Daily, Estowski, Howard, & Lee, 2001; Penders et al., 2011), each individual being unaware of higher goals and solely driven by local stimulus–response rules (Arkin, 1998). After a short while, the sensor fusion related to life detection of one lucky robot activates, which triggers a recruiting message to other neighbouring robots that excitingly storm into the area and start exploring (Bashyal & Venayagamoorthy, 2008). Indeed, the subswarm concludes by collective estimation (Hamann, 2018; Valentini, Ferrante, & Dorigo, 2017) that it is likely for a human survivor to be present. The collective triggers a new message that is spread across the swarm ad hoc network (Payton et al., 2004) to Tom and the other fire-fighters. Tom reports to control that he is on his way and starts following his ‘artificial sixth sense’ that guides him into the direction of the possible victim (Payton et al., 2004; Penders et al., 2011). At the control terminal that is located in a fire truck standing outside the warehouse, the command and control operator Mary supervises possible paths for Tom as provided by the swarm estimation (Lee, Fekete, & McLurkin, 2016; Penders et al., 2011) while coordinating Tom’s movement with other deployed forces. As Tom approaches a closed door, he suddenly feels being repelled by it, seemingly dragged back, induced by an increased stiffness of his body armour. His visual interface flags the door in a red threatening colour (Figure 1) and his whole front armour starts to vibrate. His head-up display shows that several of his cheap swarm companions have been destroyed only recently while going underneath the door, now forming a cemetery of melted robots. In a last act, the faithful robots propagated a distress signal (Hölldobler & Wilson, 1990) to other neighbouring robots that by their programmed instincts formed a safety zone in front of the door. Tom starts following an alternative route to the possible victim. His heartbeat and respiration accelerate–even the advanced homeostatic temperature control of his body armour cannot make Tom’s physical and psychological stress go all away as he enters increasingly hazardous zones. With increasing stress (Schwarz & Fuchs, 2017), some robots that have been exploring the warehouse allocate to Tom (Villani, Capelli, Secchi, Fantuzzi, & Sabattini, 2020). It appears as if the robots sensed that Tom requires increased protection, as if they are part of Tom. The allocated robots form a safety mesh around him (Alboul, Saez-Pons, & Penders, 2008; Figure 2, left side), closely monitoring the local area and always ready to protect Tom from explosions or falling objects–even if the act would disable themselves. Finally, Tom reaches the survivor. To his distress, the victim is trapped under a heavy rack. Tom engages his hand-controlled swarm manipulator (Tsykunov et al., 2019); after a few seconds, different tiny robots start to form what looks like growing tentacles originating from Tom (Nouyan, Campo, & Dorigo, 2008) with handlike structures at their tips (Dorigo et al., 2013; Mathews, Christensen, O’Grady, Mondada, & Dorigo, 2017; Oh, Shirazi, Sun, & Jin, 2017; Rubenstein, Cornejo, & Nagpal, 2014; Slavkov et al., 2018), slowly lifting and holding the rack in a secure position (Figure 2, right side). In the meantime, Sarah, a fellow fire-fighter, has arrived and in close coordination both of them expertly provide first aid and transport the victim out of the building aided by the robot swarm (Dorigo et al., 2013; Gross & Dorigo, 2009). Outside, the patient is handed over to the medical staff on scene. Later that day, Tom is being told that the patient survived.
Sketch of Tom’s head-up display showing the swarm forming a safety perimeter to warn Tom of a hazardous area.
Sketch of Tom being protected by the robot swarm while he manipulates an object with the swarm.
The above story highlights the ecological interdependence of task objectives, humans, robots, interfaces and environment in a real-world application. If we strive towards such a future exploitation of technology, it is essential that HSI combines the human and swarm perspectives in such a way that the overall system behaviour is tuned to the real-world tasks. This highlights the importance of a holistic-system view that can formulate how components of the human–swarm loop should be combined. The objective of the present work is to develop the theoretical foundations of such a systems view for HSI in terms of what we call a design theory, that is, a theory that formulates predictions about how human–swarm components should be combined so that desired joint-system dynamics emerge. In particular, we combine perspectives from swarm robotics, human-factors engineering and theoretical neuroscience to develop a design theory that utilizes the concept of self-organization. Emphasis is given to how work in swarm robotics relates to well-known concepts in neuroscience. We hope that this approach can serve as a path for integrating fragmented HSI solutions into one framework and may trigger the development of diverse HSI design theories.
The structure of this article and its embedment in the research landscape can be seen in Figure 3. Table 1 provides a glossary of central terms. In Section 2, joint human–swarm loops (JHSLs) are introduced as a holistic perspective on HSI. Section 3 defines the challenge for HSI as the design of human–swarm intelligence. In Section 4, the swarm-amplified human (SAH) is discussed as a possible design theory that builds on the concept of self-organization to join human and swarm. The feasibility of a self-organizing SAH architecture is demonstrated in Section 5 by means of a simple computational experiment. Section 6 provides advanced implications that follow from the SAH. Finally, Section 7 summarizes central insights and empathizes the benefit of developing design theories.
Structure of this article and its embedment in the research landscape.
Glossary of central terms (in order of appearance).
2. Joint human–swarm loop
2.1. Holistic perspective on HSI
JHSL refers to a cybernetic perspective on HSI where human and swarm are seen as interdependent parts of the same goal-oriented system (Ashby, 1961a; Clark & Chalmers, 1998; Hasbach, Witte, & Bennewitz, 2020; Hollnagel & Woods, 2005; Hutchins, 1995; Rasmussen, Pejtersen, & Goodstein, 1994; Woods & Hollnagel, 2006). The focus is on how the components of the JHSL can be combined together so that the whole system displays desired behaviours in given situations. We choose the term JHSL to stress that HSI as a research discipline has to consider the whole human–swarm loop.
In its simplest form, a JHSL is conceptualized as made of three interdependent components (i.e. human, swarm and interface) placed in a certain environment, while each component relates to a given perspective for design (Figure 4). From the human perspective, which may be called human-factors engineering or human-centred design, one must investigate the following questions:
What human needs must be satisfied to allow them to achieve their goals efficiently when part of the socio-technical system (Carrillo-Zapata et al., 2020; Chen & Barnes, 2014; Lewis, Sycara, & Walker, 2018; Nam, Walker, Li, Lewis, & Sycara, 2019; Penders et al., 2011; Podevijn et al., 2016; Roundtree, Goodrich, & Adams, 2019)?
How do the operators tend to use the swarm to align it with their goals (Walker et al., 2012; Woods & Hollnagel, 2006)?
How do operators coordinate goals and actions with other agents of the socio-technical system (Coucke, Heinrich, Cleeremans, & Dorigo, 2020; Ellwart, 2011; Ellwart, Happ, Gurtner, & Rack, 2015; Klein, Woods, Bradshaw, Hoffman, & Feltovich, 2004; Tavakoli, Nalbandian, & Ayanian, 2016; Woods & Hollnagel, 2006)?
How does the human operator’s mental model about swarm control tend to change over time, and how should the operator be trained in swarm control (Hasbach et al., 2020)?
A human–swarm loop can be approached from different perspectives. HSI should consider the whole JHSL.
From the swarm perspective, which may be called swarm engineering, one must investigate the questions:
What socio-technical requirements must be satisfied so that given swarm implementations can achieve their goals efficiently (Bashyal & Venayagamoorthy, 2008; Brambilla et al., 2013; Hamann, 2018; Kolling et al., 2015; Schranz et al., 2020)?
How can the swarm robots coordinate goals and actions with other agents of the socio-technical system (Coucke et al., 2020; Hamann, 2018)?
How can the robots’ control models be updated during development (Birattari et al., 2019; Brambilla et al., 2013; Hamann, 2018; Nolfi & Floreano, 2000) and deployment?
Finally, HSI joins human and swarm into one goal-oriented system. The design objective is to find an interaction solution that combines the strengths of humans and swarms so that the joint-system capabilities that cope with the application scenario are maximized while reducing inner complexities that do not contribute to goal achievement. Thus, the JHSL behaviour is the desired level of performance analysis (Hollnagel & Woods, 2005). A failure to produce a joint-system relates to a violation of the law of requisite variety (Ashby, 1961a); a disjointed system produces variety in the internal human–swarm loops that is in conflict with producing variety that copes with the task objectives the system was actually built for (Hollnagel & Woods, 2005). Put simply, the swarm should not become unreasonably complex for the human operator, as well as the other way around, wasting resources that are needed to deal with actual tasks. From the HSI perspective, one must investigate the additional questions:
What is the overall JHSL objective and what is required to solve it efficiently (Carrillo-Zapata et al., 2020; Cooke, Gorman, Myers, & Duran, 2013; Hollnagel & Woods, 2005; Hutchins, 1995; Penders et al., 2011; Rasmussen et al., 1994)?
How can the capabilities of humans and swarm be combined to solve the JHSL task including task sharing and task allocation (Brown, Jung, & Goodrich, 2014; Coppin & Legras, 2011; Hutchins, 1995; Kolling et al., 2015; Musić & Hirche, 2017; Nam et al., 2019; Parasuraman, Sheridan, & Wickens, 2000; Walker et al., 2013)?
How can the JHSL coordinate with external systems (Rasmussen et al., 1994)?
How can the communication link between human and swarm be established (Balch & Parker, 2002; Barca & Sekercioglu, 2013; Kolling et al., 2015)?
How can humans issue control commands (Kolling et al., 2015)?
How can swarms be controlled by external inputs (Kolling et al., 2015)?
How can information about the environment and the swarm be fused to produce global information (Hamann, 2018; Kolling et al., 2015)?
How can this information be presented to humans (Kolling et al., 2015)?
2.2. Centralized–decentralized trade-off
Each facet of the JHSL must be designed in relation to the joint-system goals as well as the capabilities and demands of the other components. Probably most importantly, a centralized control and communication approach may be a reasonable design solution from the human’s perspective but ignores the benefit of distributed control and communication provided by the swarm intelligence approach (Barca & Sekercioglu, 2013; Crandall et al., 2017). After all, integrating a human operator in a fully distributed system makes him a potential single point of failure. On the contrary, there is a need for the human to be part of the system loop after all. Therefore, balancing between centralized and distributed computation may be defined as the main challenge for combining human and swarm into one efficient joint-system. From a human-factors engineering perspective, this challenge can be seen as a special case of human–autonomy teaming via designing (flexible) levels of automation (Brown et al., 2014; Chen & Barnes, 2014; Coppin & Legras, 2011; Musić & Hirche, 2017; Nam et al., 2019; Parasuraman et al., 2000; Walker et al., 2013). For example, a shared control between human operator and swarm has been proposed to dynamically moderate the impact of human agents on the swarm (Ashcraft, Goodrich, & Crandall, 2019; Crandall et al., 2017). The challenge also relates to investigations on hierarchical control in swarm robotics, where a central robot with superior information is part of the control architecture (Dorigo, Theraulaz, & Trianni, 2020; Mathews et al., 2017; Zhu et al., 2020). From the literature, an underlying theme emerges where both a central unit with broad capabilities (i.e. humans) and decentralized units with narrow capabilities (i.e. swarm robots) have, sometimes partly, a stake in decision-making processes to balance the centralized–decentralized control trade-off.
3. Human–swarm intelligence
3.1. Precondition for a design theory
Given the interdependence of design solutions for human, robot swarm and interface, in the following a holistic approach on how to best join human and swarm into one goal-directed loop is developed. Such a design theory as part of the empirical cycle must formulate testable predictions about what combination of design solutions are beneficial (Figure 5). While classical scientific theories model natural phenomena, a design theory models what would be a good design. In this framework, design prototypes are seen as hypotheses deducted of the underlying design theory (Woods & Hollnagel, 2006).
Design as part of the empirical cycle requires the development of a theory about what would be a good design. The theory should be subjected to empirical probing by evaluating prototypes that implement the theory.
A precondition for the construction of a design theory is a definition of desired JHSL behaviour. While most joint-system goals depend on the application scenario, there are three system properties of HSI that seem desirable independent of the application; adaptation, robustness and scalability. We selected these variables guided by Beer’s aphorism: ‘
There are no globally accepted definitions of robustness, flexibility, scalability and adaptivity across different scientific fields. According to the seminal work of Şahin, (2004), the following definitions can be given from the perspective of swarm engineering. Robustness is the ability of a swarm to continue operation when disturbed, while particular focus is given to the loss of robot agents. Flexibility is the property of a swarm to deal with different tasks. Scalability is the capability to solve tasks over different swarm sizes. The term ‘adaptation’ is not used by Şahin and seems to have different meanings in the swarm robotics literature. In behaviour-based robotics, which is closely related to swarm robotics, adaptation may refer to behavioural changes, evolution, sensory tuning and learning (Arkin, 1998). As we investigate the JHSL rather than the swarm as an isolated system, we will reassess the desired properties in the following to fit the joint-systems view.
3.2. HSI as human–swarm intelligence
In general, one wants any intelligent system to choose the correct decision given the present situation set (Ashby, 1960, 1961b). Let a system be called robust if it is capable of achieving its system purpose, that is, choosing correct decisions out of a set of possible decisions, over a variety of situations given a particular internal organization (Ashby, 1960; Woods, 2015). The internal organization is defined as the interdependence of the system parts (Ashby, 1947; Conant, 1976) that provide the algorithms or input–output laws to cope with a set of situations. It may also be referred to as the agent’s controller. By applying Beer’s aphorism (Section 3.1), it follows that the internal organization encodes the system’s goals in terms of possible attractor states, or rather the corresponding selected parameter set, of the system (Ashby, 1960). In terms of stability, a robust system can absorb particular shocks from the environment (Woods, 2015) without driving essential variables out of their tolerance values (Ashby, 1960). However, a particular internal organization can only deal with a limited set of situations when faced with complex or dynamic environmental demands (i.e. when environmental variety is high; Ashby, 1961a; Woods, 2018). To keep demands in check that are not modelled by the robust system, while modelling is required as stated by the good regulator theorem (Conant & Ashby, 1970; Seth, 2014), the system must be adaptive, that is, it must be able to change its internal organization to contentiously match the demands of the environment (Ashby, 1960; Friston, 2010; Nikolić, 2015; Powers, 1973; Russell & Norvig, 2016; Woods, 2018). If a system can continuously adjust its internal organization to fit new opportunities and demands, it is referred to as resilient (Woods, 2015).
Figure 6 shows the desired intelligent system as two abstract controllers (Ashby, 1960; Seth, 2014) placed in the world. By abstract, we mean that the two loops may not be observed as distinct matters and functions in the real system. The first-order robust controller can autonomously cope with the inputs of the world to produce desired internal and external states. In general, by connecting the robust controller to the world, the system behaviour ‘flows’ as defined by the robust controller logic and environment (Nikolić, 2015). However, the robust controller alone will ultimately fall short when faced with the variety of the world. Therefore, the second-order adaptation controller implements rules that adjusts the robust controller when the fit between the robust controller and the situation set is not optimal. While the robust controller may switch between encoded system goals (i.e. switch between attractors in the state space as given by the chosen robust controller and environment), the adaptation controller may also generate new system goals by changing the robust controller (i.e. change the robust controller’s state space). The adaptation controller may be said to be robust itself if it correctly adjusts the robust controller over a variety of situations.
An intelligent system is abstracted as an internal organization that includes two types of controllers. The robust controller has the capability to cope with possible situations while the adaptation controller improves the robust controller’s capabilities.
By imagining a kitten approaching a fire, Ross Ashby (1960) has provided us with an illustrative example for intelligent behaviour. If a kitten gets too close to a fire for the first time, the unpleasant stimuli may trigger a reorganization of the internal neural machinery leading to a behavioural avoidance of this situation in the future. Thus, the kitten’s adaptation controller, here in the form of operant conditioning, has changed the robust controller that deals with the behavioural response to the situation ‘
In HSI, a JHSL learning that swarm agents have a high probability to be destroyed in a given area and, therefore, updates its internal organization to cope with the situation implements true adaptivity. The update of the internal organization can either refer to the human’s control model of the world (Friston, 2010; Hollnagel & Woods, 2005; Johnson-Laird, 1983), or to the robots’ control models (Arkin, 1998). In the former, the human changes his control behaviour while making use of the robot swarm’s capabilities, for example, by manually steering the swarm away from the area. In the latter, the robots’ capabilities are updated, which enable the robot swarm to autonomously avoid the area. It should be noted that both forms of adaptation lead to the desired change in JHSL behaviour. When the JHSL is observed at a later time, we may call it robust because it is behaviourally avoiding the loss of robots while being tolerant to certain changes in the environment. If robots are lost in that area and the JHSL can accomplish its goals nonetheless, the JHSL displays another strategy leading to robustness that corresponds to the view of robustness in swarm robotics.
Some situations that the JHSL may have to cope with are highly demanding, resulting in the need that goals of the systems need to be balanced. For example, the JHSL could be faced with an unmodelled exploration–exploitation trade-off. An intelligent JHSL will not keep this trade-off constant but apply a search strategy o find an acceptable exploration–exploitation balance. If observed, the JHSL may look like switching between different behavioural states in its attempt to narrow down plateaus in the solution landscape. It is said that the adaptation controller searches for an acceptable solution for the robust controller. The adaptation controller may even ignore one goal for a time period by, for example, arguing that attempting to reach both goals simultaneously will result in both goals being out of reach. Note how adaptation can relate to cognitive reasoning (Hollnagel & Woods, 2005).
Finally, the third desired system property is scalability that can be seen as a special case of robustness as defined above. In swarm robotics, it is the capability to adjust the number of robots during deployment, such as two separated bird flocks forming interactions to become one flock. In HSI, swarm scalability is challenging because a human operator has a finite capacity to supervise independent robots (Kolling et al., 2015; Lewis, 2013). Scalability can also refer to the possibility to control the swarm with a different number of human operators working as a team (Coucke et al., 2020; Grosh & Goodrich, 2020). This makes the combination of proximal and remote interaction a special case of scalability, as it essentially combines human operators with different ranges of skills, situational awareness and control possibilities.
From the discussed point of view, flexibility underlies all three discussed properties (robustness, adaptation and scalability) and is, therefore, not explicitly formulated as a desired joint-system behaviour. A robust controller is flexible in the sense that it applies different solutions to deal with the situation set. An adaptation controller results in system flexibility by increasing the range of the robust controller. In addition, the robust controller itself must be flexible so it can be adjusted by adaptation. Finally, scalability as a special form of robustness results in the flexibility over different swarm size situations.
To conclude, HSI may also be read as human–swarm intelligence referring to learning JHSL as a capable controller of environmental demands. Note that the term also relates to human swarm intelligence as human collectives that display swarming behaviour (Sumpter, 2006), thereby stressing the importance of integrating the human into the swarm (Kolling et al., 2015) as well as human teamwork (Coucke et al., 2020; Tavakoli et al., 2016). Now the overall challenge of HSI can be defined as combining the centralized control demands of human operators with the decentralized control demands of swarms into one goal-seeking system while promoting sustained adaptive, robust and scalable behaviour. A candidate design theory of JHSL must provide a solution to this challenge, that is, formulate predictions about how the desired properties can be designed.
4. Design theory of the SAH
4.1. Neural control architecture
The main challenge was formulated as how centralized and decentralized control can be brought together while preserving the benefits of both. As humans are biological systems and swarm robots are heavily bio-inspired, we searched for bio-inspiration that can inform how to cope with the centralized–decentralized control trade-off. We found such bio-inspiration in the abstract neural control architecture. This is a good start, because the nervous system is assumed to be a capable controller of environmental demands, that is, it implements the good regulator theorem (Friston, 2010; Seth, 2014).
In biological nervous systems, stereotypical locomotion is generated by a special form of neural networks that are located at the low-level, or at the ‘front end’, of the nervous system. These neural networks are called central pattern generators (CPGs) (Arbib, 1989; Ijspeert, 2008; Marder & Bucher, 2001; Pearson & Gordon, 2013). CPGs are examples of semi-autonomous distributed controllers that are robust to a set of situations. While the correct CPG function depends on sensory feedback, higher-order (‘back end’) vertically descending brain signals may also modulate the activity of the lower-order CPGs (Figure 7). Thus, at least some neural circuits of the biological sensory–motor loop seem to be capable of producing robust stereotypical sensory–motor behaviour while sensitive to higher-order neural modulatory signals. Imagine walking down the street. When walking, you rarely think about walking. Instead, the sterotypical locomotion is taken care of by your low-level sensory–-motor loop. Yet, you can easily choose to shift from walking to running when you desire to do so; your higher-order system modulates the operation of the hidden low-level sensory–motor loop.
Central pattern generators (CPGs) illustrate the abstract neural control architecture. CPGs are low-level neural networks that produce rhythmic output for stereotypical locomotion while making use of sensory feedback. At the same time, CPGs are sensitive to modulation signals from higher-order systems. This architecture balances between centralized and decentralized control.
This control architecture with semi-autonomous subsystems that are yet responsive to higher-order modulation signals may be a fundamental principle in the control of artificial and natural complexity. To the roboticist, it is known as the hybrid deliberative/reactive architecture (Arkin, 1998; Murphy, 2019). It is a principle known in human–robot interaction as well. For example, a space telerobot must implement sufficient semi-autonomy while at the same time responsive to delayed inputs from human operators on earth (Sheridan, 1992). The creation of semi-autonomous subsystems is also relevant in the steering of human organizations (Senge, 2006).
4.2. Neural architecture applied to HSI
We think this abstract neural control architecture may be also mapped onto the design challenge of HSI (Figure 8), because a higher-order controller (
Mapping from the abstract neural control architecture to HSI gives the SAH. In the SAH, the swarm is treated as a neural extension of the human’s low-level sensory–motor loop. The swarm organizes into semi-autonomous subswarms that are sensitive to modulation signals from the human operator. For the human, these subswarms take the form of artificial body parts.
The main advantages of this control architecture can be observed in biological systems. CPGs represent a fast controller that will, however, fail when faced with the complexities of the world, while brain cognition 1 represents a slow controller that is much more capable of dealing with world complexities. Imagine again walking down the street. A slow feedback controller may not be able to efficiently stabilize your locomotion. The CPGs, on the contrary, which are located in proximity to the body’s sensors and effectors, can react much faster. The CPGs are ‘outsourced’ from the brain’s perspective to establish fast (short) feedback loops, which decreases time synchronous information bandwidth needs between the brain and the body’s local sensor–effector systems (Ijspeert, 2008). Importantly, the brain as a controller of body states is confronted with less internal variety it needs to cope with, that is, the architecture minimizes internal control complexity. However, when you continue walking down the street you see an ice-cream truck that induces the response of excitingly running towards the potential food source. Classifying the ice-cream truck, adjusting your goal and updating the locomotion behaviour requires much more sensory integration and processing than producing stable stereotypical locomotion and, therefore, requires more resources and time. In addition, advanced anticipatory control may be needed to plan how to accomplish your goal of finally getting to enjoy your ice-cream. Importantly, it is the interplay of the fast but heuristic and the slow and sophisticated system that makes you a capable controller to deal with the world (Kahneman, 2011). Optimally, the slow-sophisticated controller implements learning mechanisms, that is, it is also an adaptation controller. Similarly, the swarm is part of the heuristic controller that can react fast to local changes (similar to walking on an uneven pavement) but requires cognitive modulation for complex and unmodelled scenarios. By implementing short feedback loops between subswarms and local environmental features, the required coordination between human operator and subswarm is reduced.
Based on the defined mapping
This swarm perspective can be viewed as an application of the mergeable nervous system concept in swarm robotics (Mathews et al., 2017) where a central unit, here the human operator instead of a robot unit, exerts high-level control over its ‘mergeable swarm body’. The SAH defines the vital extension that the subswarms can directly react to stimuli of the environment without the need that the signals are processed first at the central unit. One may say that the SAH includes inborn reactive pathways (i.e. stimulus–response mappings) in addition to modulation by higher-order units. From an evolutionary point of view the higher-order neural correlates (i.e. the neocortex) are younger than the lower-order neural correlates (e.g. reflex pathways). This highlights how higher-order units build on lower-order capabilities to make a capable controller of environmental complexities (Braitenberg, 1986; Brooks, 1986; Dawkins, 1976; Minsky, 1988; Nikolić, 2015; Pfeifer & Bongard, 2006; Powers, 1973).
4.3. SAH as human–swarm intelligence
Human–swarm intelligence was defined in Section 3.2. In the SAH, the swarm’s distributiveness is joined with the central control of the human after the blueprint of CPG control in the biological nervous system. Similar to CPGs, the swarm is capable of reacting quickly to the environment in a closed-loop (i.e. cope with some situations autonomously), while being biased by the human component. The swarm therefore is influenced by the human without fully being centralized to it. This property is demonstrated by taking the human out of the loop: the subparts of the swarm will, though with degraded performance, still show purposeful behaviour. A morbid but known example is the headless chicken running around, although neuroscientific investigations in that area focus more on other animals like cats (Pearson & Gordon, 2013).
The SAH architecture is expected to result in adaptive, robust and scalable JHSL behaviour. Sustained adaptivity is the consequence of equipping the human operator with native high-level control over his swarm ‘body’, as the human component is the most capable adaptive system known to us. Thus, the biological brain is supported as joint-system adaptation controller by design, a solution that is applied by cognitive systems engineering (Hollnagel & Woods, 2005; Rasmussen et al., 1994; Vicente & Rasmussen, 1992; Woods & Hollnagel, 2006). Integrating the swarm at the low-level nervous system amplifies cognition as the latter builds on lower-order capabilities (Section 4.2). In other words, the cognitive system is provided with more task-relevant variety it can use for model update and goal-achievement (Ashby, 1961a; Friston, 2010); see outer loop in Figure 9. Given that neural systems feature high adaptive capability, implementing the swarm based on neural principles aims at further strengthening system adaptivity. Robustness and scalability is created by treating the swarm as a distributed neural machinery that excursively relies on local interactions. In neural terms, robustness comes in the form of plasticity (i.e. flexibility by adaptation) and graceful degradation to neuron failures. Neural scalability is vividly observed when comparing nervous systems across species.
In the SAH, the swarm is part of the human’s perception–action cycle, which supports the update of the human’s control model about the world (outer loop). Treating the swarm as part of the body relates to utilizing high-level human state estimations as swarm modulation signals and to providing sensory feedback consistent with the feeling of body stimulation.
4.4. The human operator’s perspective
From the human’s perspective, treating the swarm as part of its body could provide a feasible design solution for controlling a complex system without the need for the swarm to explain itself. Humans are, assuming robustness of the neural implementation, not aware of the complexity of their own low-level neural machinery (Doyle & Csete, 2011; Kahneman, 2011). Yet it feels controllable and often effortless to us to produce complex behavioural patterns like riding a bike, dancing or making music. Therefore, functionally integrating the swarm into the nervous system may allow operators to develop an intuition of swarm control similar to their own body control.
Interfacing at the low-level nervous system means essentially two things (Figure 9). First, as said before, low-level stereotypic sensory–motor control often feels automatic to the conscious mind. When walking down the street, you rarely think about walking. Therefore, the (cognitive) state of humans should be translated by the interface into commands (Alboul et al., 2008; Bales & Kong, 2017; Karavas, Larsson, & Artemiadis, 2017; L Mondada, Karim, & F Mondada, 2016; Podevijn et al., 2016; Pourmehr, Monajjemi, Vaughan, & Mori, 2013; Schwarz & Fuchs, 2017; Villani et al., 2020) for the swarm clusters. This is referred to as passive (Kolling et al., 2015) or implicit (Gildert, Millard, Pomfret, & Timmis, 2018) interaction, that is, unconscious control, which can be contrasted to active/explicit control (e.g. deliberate gesture control; Alonso-Mora, Lohaus, Leemann, Siegwart, & Beardsley, 2015; Gromov, Gambardella, & Di Caro, 2016; Nagi, Ngo, Gambardella, & Di Caro, 2015). The swarm forming a protective layer based on Tom’s distress is an example of passive interaction (Section 1). A further design restriction is that the human operator perceives the passive translation as deterministic (Braun et al., 2018; Woods & Hollnagel, 2006). After all, you can deliberately control leg movement if you desire to do so. This feeling of being in control over the swarm clusters (alias artificial body parts) relates to human-factor issues like trust in automation (Lewis et al., 2018; Nam et al., 2019). Note that passive interaction may be used deliberately by a human if he knows the mapping between his own states and swarm states. The interface may additionally implement deliberative control of the artificial body parts, in line with how body control feels (Tsykunov et al., 2019), that can manually be triggered by the human operator. One may think of the swarm forming handlike structures where the individual swarm-fingers can be controlled by Tom.
Second, feedback should be provided via the bottom–up route similar to how sensor stimulation of the body feels. Take again the previous example of approaching a fire. While blindfolded, you are able to locate the source of the fire by moving around and observing the gradient of the perceived heat (Bashyal & Venayagamoorthy, 2008; Braitenberg, 1986; Lilienthal & Duckett, 2004; Nouyan et al., 2008; Schmickl & Crailsheim, 2008). The example illustrates the active nature of exploration in the perception–action cycle where sensory inputs regulate action outputs that, in turn, alter sensory inputs (Neisser, 1976). A similar loop may be implemented by a wearable interface that adjusts its feedback based on the estimated stimulus strength and location (Penders et al., 2011). The cognitive capability, that is, updating a model of fire location, is amplified by providing an increased sensory–motor range at the low level. Importantly, this fits to the swarm robotics’ behaviour-based approach where stimulus class and strength is used to adjust behaviour (Arkin, 1998). Note also how this example relates to the real-world scenario of fire-fighter Tom.
By closing the loop as shown in Figure 9, the swarm potentially becomes part of the human’s perception–action cycle. The SAH, therefore, relates to the notion of a cyborg that ‘[…]
4.5. Self-organization as the glue between humans and swarm
Based on the engineering goal as put forward by the SAH to functionally integrate the swarm at the low-level nervous system, a concept is needed that allows to translate computational principles between nervous system and artificial swarm. Self-organization is a concept well-known to both neuroscience and swarm intelligence and has been proposed as an underlying computational mechanism for both systems (Couzin, 2009; Trianni, Tuci, Passino, & Marshall, 2011). It is defined as the observable shift from unordered to ordered spatio-temporal patterns on the group level that are the result of local interactions on the individual level (Ashby, 1947, 1962; Bonabeau et al., 1999; Camazine et al., 2003; Couzin, 2009; Hamann, 2018; Sumpter, 2006; Trianni et al., 2011). The individual level and their interactions refer to neurons as well as swarm agents, for example, ants, bees or termites, that form global networks based on local rules. Capabilities at the group level are observed that cannot be explained by neurons or ants in isolation. This system property is often paraphrased as ‘more than the sum of its parts’, stressing the importance of interactions between components.
Intriguingly, neural networks and biological swarms seem to resemble not only on this abstract level. On the microlevel, for example, both individual neurons and ants may have a refectory period, making activation unlikely shortly after being active (Couzin, 2009). On the macrolevel, swarms may follow psychophysical laws previously thought to be an exclusive property of cognitive systems (Passino, Seeley, & Visscher, 2008; Reina, Bose, Trianni, & Marshall, 2018;Sasaki & Pratt, 2018). It is an attractive hypothesis that neural and swarm decision-making share the same computational principles that have been manifested by evolutionary selection in different connected systems to cope with environmental complexity.
In the following, we will use self-organization in swarms and neural networks as common computational ground, as the glue, to join human and swarm into one goal-directed system. Rather than claiming that neural networks and biological swarms share the actual same computational basis, we argue that both can be modelled as self-organizing systems. Thus, both humans and swarms are abstracted into one common computational reference frame (Marr, 1982) to integrate the swarm at the low-level of neural sensory–motor loops. On this abstracted level, one may think of the human operator himself as a distributed self-organizing neural system that is extended by a neural–swarm machine. The benefit for swarm engineering by transferring neural principles to swarm networks that goes beyond the usage of neural robot controllers has been demonstrated recently by the Bruessel’s group (Mathews et al., 2017; Zhu et al., 2020), Monaco, Hwang, Schultz, & Zhang (2020), and Otte (2018).
4.6. SAH as self-organizing HSI
When the swarm is the interface between the variety of human cognition and the world, it becomes a potential bottleneck. The swarm must, therefore, be designed to allow for the sufficient and reliable conductance of information between human and situation set (Ashby, 1961a), while the required bandwidth for signal forwarding is minimized by implementing local sensory–motor loops. A possible solution lies in the application of self-organizing principles for pathway formation that, based on local interactions, allows for scalable parallel processing.
The SAH loop
The SAH loop consists of humans
Treating the swarm as a functional part of the body first means that the robot collective
Figure 11 shows the three self-organizing interaction networks of the SAH. On the individual layer, called node layer, each robot
The SAH loop consists of three self-organizing interaction networks. On the node layer, the robots’ controllers specify the microlevel rules that define the macrolevel behaviours for the connectome layer and the swarm layer. The connectome layer is the robot interaction network
When observed, the swarm will show diverse behaviour if the environment
Before we turn to a computational experiment that shows how to create a self-organizing graph
5. Computational experiment
5.1. Problem statement
The problem is defined as finding a robot controller (node layer) that allows for the formation of the connectome graph
Based on this objective, we were in particular interested if
5.2. Experimental setup
The simulation was implemented in MATLAB 2019b. The robot model is assumed to be equipped with eight direction-sensitive infrared (IR) transmitter/receiver pairs (Figure 12, right bottom corner). Let the robot state be defined by
with
Schematic for the formation of the connectome graph
The state of the human is defined by
with
Each simulation run was repeated over 1000 samples over a range of different swarm sizes
While the position of the human agent
5.3. Connectome layer
The human agent blindly transmits his cognitive state
FSA for local signal forwarding.
The emergence of a stable connectome graph
FSA for pathway formation.
5.4. Swarm layer
Based on the forwarded signals, the robot controller for the emergence of the local subswarms
FSA for local sensory–motor loop formation.
5.5. Results
Figure 16 shows the performance of the algorithm in terms of required iterations until a pathway between
Mean number of simulation loop iterations
An additional property of the algorithm is the shortening of the path with simulation loop iterations. As the robots regularly reset to form a new
Simulation run examples with
The algorithm can be evaluated in terms of human–swarm intelligence (Section 3.2). The performance is robust in that it will find a pathway (a subgoal of the SAH) independently of the spatial arrangement of
Based on the formation of
6. Advanced implications
6.1. On neural computation in SAH
The robot controllers described above resulted in the generation of a pathway between human
The established connections are then tuned to fit the dynamic environment (Sanes & Jessell, 2013a). From the outside, this tuning is observed as learning. Donald Hebb (1949) has described a process for the tuning of connections on the neural level. Hebbian learning is often summarized as ‘neurons that fire together, wire together’, although it should be remarked that this statement alone is too simplified. We may implement Hebbian learning in the
In the SAH, synaptic tuning may be implemented by restricting robot movement based on the importance of the pathway.
We now turn to the topic of providing feedback via the swarm. In Section 4.4, it was discussed that feedback should be provided at the low-level sensory system. The localization of a feature source, a global information, may then be achieved by an actively exploring human agent moving through a swarm gradient. This swarm gradient may be seen as the functional equivalent of neural population coding (Rolls, 2016). It may also be asked how neural networks process local sensory information to produce global knowledge. Known to practitioners familiar with neural networks is the notion of convergence, where the receptive field of neurons are broadened over neural layers (Hubel & Wiesel, 1962; Rolls, 2016). Figure 19 shows convergence in the SAH. Information from two different features
In the SAH, neural convergence may be implemented by robots that are connected to multiple features.
6.2. On human scalability in SAH
In Section 3.2, scalability was defined as a desired capability for HSI. In addition to swarm scalability as described, one may adjust the controllers above to include multiple human agents. Figure 20 shows an example of two pathways between one feature
In the SAH, updating the behaviour of local sensory–motor loops can be based on the cognitive states of multiple humans.
The cognitive states of humans may also influence each other via the swarm pathway, resulting in a ‘socio-technical group mind’ established by the self-organizing swarm. Note how the ‘socio-technical group mind’ relates to human swarm intelligence (Tavakoli et al., 2016) as humans displaying swarm intelligence. This ‘group mind’ may be used to combine proximal and remote interaction where the states of humans with local situational awareness (narrow, detailed information) are pooled with the states of humans that have global situational awareness (broad, abstract information). The ‘group mind’ highlights the view of human scalability from the neural perspective; human operators can be seen as different cognitive modules that collectively vote for JHSL output (Minsky, 1988; Richards, 2015).
6.3. On artificial body parts in SAH
In the SAH, the swarm is integrated into the human’s nervous system function. We have said that this fundamentally relates to passive interaction and to providing bottom–up feedback (Section 4.4). In the following paragraphs, additional considerations are discussed.
Joining nervous system and swarm into one system means that the state of one depends, at least in part, on the other. There are two routes to establish a communication between human and nervous system with the swarm: neural and behavioural. A method is categorized as neural if it is directly targeted at the biological nervous system to implement a translation between neural state and swarm state. Neural methods, therefore, include both invasive and non-invasive brain–computer interfaces (Karavas et al., 2017; Lebedev & Nicolelis, 2017; Wolpaw, Birbaumer, McFarland, Pfurtscheller, & Vaughan, 2002). On the contrary, behavioural methods refer to interfacing with an additional transformation between nervous system and swarm. Thus, one infers and manipulates the neural states based on psychological and psychophysiological states (Alboul et al., 2008; Schwarz & Fuchs, 2017) that are, at least partly, a result of neural computation. For example, transcranial magnetic stimulation (TMS) application at the visual cortex would directly target neural states, while providing simple visual cues ‘hitch-hike’ a biological sensor that is connected to the nervous system and by this route indirectly influence the neural state.
It will, of course, not be sufficient to establish a mere state dependence of nervous system and swarm to produce a real SAH. After all, traditional human–machine interfaces establish connections via the behavioural routes (e.g. clicking a mouse button is a behavioural state that is an output of nervous system control). To include the swarm in the human’s nervous system, the brain must be tricked into integrating the swarm into its body representation. This is achieved if the human operator has the illusion that the swarm feels like part of himself, if it feels like an artificial body part. This requires first, and most fundamentally, that the subswarms form operational unities in the sense that they create, recreate and hold into an observable form separated from the environment (Varela, Maturana, & Uribe, 1974). The experiment above has shown how this can be in principle done with minimal effort. Second, additional requirements must be formulated that satisfy how the brain integrates and controls the body. This renders contemporary work on the brain’s body representation, tool use and neural engineering relevant for the SAH (Abbott, 2006; Botvinick & Cohen, 1998; Braun et al., 2018; Cardinali et al., 2009; Cheng, Ehrlich, Lebedev, & Nicolelis, 2020; Kieliba et al., 2021; Kwakkel, Kollen, & Krebs, 2008; Lebedev & Nicolelis, 2017; Maravita & Iriki, 2004; Miller et al., 2018; Schettler, Raja, & Anderson, 2019; Seth, 2013). Work on body ownership and agency suggest that bottom–up multi-modal feedback should be spatio-temporally correlating and that the subswarms probably need to look like part of the body, for example, a hand (Braun et al., 2018). Robot swarms, especially shape-changing swarms (Mathews et al., 2017; Oh et al., 2017; Rubenstein et al., 2014; Slavkov et al., 2018) that are observed through augmented reality displays (Payton et al., 2004; Reina et al., 2017), can therefore be advanced engineering testbeds for theories on how the brain controls the body.
In sum, the SAH requires the translation of neural signals to control inputs for self-organized body parts in accordance with body control. The indirect route may already be a feasible solution for real-world applications, for example, by utilizing the human’s position (Alboul et al., 2008) and physiological measurements (Schwarz & Fuchs, 2017) as control inputs. For example, fire-fighter Tom (Section 1) rapidly changing into a defensive body posture can be used as a control signal for the swarm body parts engaging in a reflex-like protection of Tom. However, this behavioural classification seems generally limited to low-resolution and context-dependent interpretations of the neural state. Brain–computer interface technology has the potential of high-resolution classification, although the technology is not yet mature. In the future, Tom may be equipped with an advanced brain–computer interface technology that provides high-resolution classifications of cognitive states and adapted sensory–motor signals. This could allow him, for example, to feel the location (i.e. the direction and the distance) of a victim, alike to having a sixth sense, while allocating more robots to the estimated location of the victim, similar to closing your hands around an object.
7. Conclusion
This work started with a description of JHSL components, that is, human, swarm and interface. The design of a good JHSL must inevitably take into account the properties of the components in relation to each other, in particular to consider the centralized–decentralized control and communication trade-off. It was stated that a systems theory in terms of a design theory, that is, a theory that predicts what would be a good combination of human, swarm and interface, can be beneficial in this endeavour. Based on the discussion of desired JHSL properties, it was concluded that a design theory must formulate solutions for how the centralized control demands of human operators and the decentralized control demands of swarms can be joined into one goal-seeking system while promoting sustained adaptive, robust and scalable behaviour. This was referred to as the challenge of human–swarm intelligence.
The design theory that we called the SAH, the swarm-amplified human, predicts that integrating semi-autonomous robot swarm clusters into the human’s low-level nervous system is a good design solution. It was argued that doing so establishes a capable joint-systems controller that can react fast to local changes but can also deal with more complex scenarios while balancing between central and distributed control. JHSL adaptivity is mainly supported by amplifying the human operator’s natural cognitive capabilities by extending his sensory–motor possibilities while robustness and scalability are preserved by treating the swarm as a neural machinery.
Subsequent discussion focused on relating work in swarm robotics and theoretical neuroscience when both systems are modelled as self-organizing systems. The SAH defines three levels of self-organization: the node layer, the connectome layer and the swarm layer. The node layer defines the microlevel interactions between robot–neurons that result in the formation of the swarm as interface between human operators and environmental features as well as swarm clusters that locally act on the environmental features and constitute the artificial body parts.
In sum, the SAH defines the following set of four central interdependent requirements to join the perspectives of human, swarm and interface. These requirements should be seen as hypotheses that must be subjected to empirical probing through testing prototypes that preferably satisfy all four requirements.
Rather than controlling the whole swarm, the swarm should self-organize into semi-autonomous subswarms that locally interact with relevant features of the environment while being modulated by the human’s cognitive state. This mimics the abstract neural control architecture.
Rather than being the target of the interaction, the swarm should self-organize as the interface between the human operator and relevant features of the environment. This mimics the formation of neural pathways in the nervous system that are tuned to relevant environmental stimuli.
Rather than consciously supervising a robot swarm, the human state should be translated into swarm commands in a passive yet controllable manner with the option to switch to manual control of subswarms. This mimics the daily life phenomenology of body control.
Rather than providing artificially constructed feedback, environmental feedback should be provided in a bottom–up manner featuring raw stimulation of biological sensory systems. This mimics the dependence of cognition on lower-order sensory–motor capabilities.
The implemented computational experiment, which focused on the first two of the requirements above, showed how the human–feature pathways and local subswarms can self-organize with minimal logic. It was further discussed how the SAH theory provides a range of additional solutions for HSI design that blur the distinction between swarm intelligence and neural computation. This is to be desired because a good theory should encompass, or relate to, a high number of (possible) observations.
We may say that the described robot controllers in the computational experiment in itself are rather simple and, therefore, perhaps of little value. Such a proposition, however, would miss the points we want to make tremendously. First, self-organization is formalized as equipping a system with complex behaviours even if the mechanism is simple. To quote Valentino Braitenberg discussing his law of uphill analysis and downhill invention: ‘
Starting at the top of our argument, the SAH as design theory allows predictions that we deduced as a testable prototype. We have tested the algorithm for robustness and scalability in the context of the SAH and concluded that it fits demands. One may, therefore, proceed with caution for we have only tested an isolated part of the SAH theory, in accepting the SAH as a suitable candidate to inform design.
The application of the empirical cycle that includes global theories for the big picture is essential in the scientific design of artificial systems such as HSI. Unfortunately, the benefit of understanding the design space is often overlooked in the pursuit of producing artificial systems fast and cheap. Based on a good design theory, design patterns can be constructed that fit to particular environmental demands and mission objectives (Alexander, 1977; Woods & Hollnagel, 2006) such as Tom searching for survivors. In the long run, formulating design theories is, therefore, cheaper because the impact of design decisions on the overall system can be predicted more reliable.
In general, the scientifically oriented JHSL-engineer takes the role of an external adaptation controller who updates the internal organization of the JHSL to fit expected situations (Woods & Hollnagel, 2006). Design is a process of intelligence after all, thus the selection of appropriate actions (Simon, 1982). We hope that our work shows that treating robot swarms as neural machines can inform the selection of design actions in HSI. From the view of natural science, the construction of artificial distributed intelligence that merges neural and swarm computation provides a method for comparing and integrating theories about nervous systems and swarms including whether evolutionary pressure could indeed have led to the invention of similar computational underpinnings across distributed systems observed in nature.
Footnotes
Declaration of conflicting interests
Funding
ORCID iD
