Killer Robots and the Concept of Meaningful Human Control

Introduction

Machines have long served as instruments of war, but historically humans have always dictated how they are used. The evolution of technology has the potential to change that reality, and the implications are profoundly disturbing. According to experts in artificial intelligence, fully autonomous weapons, which would select and engage targets without meaningful human control, could be developed for use within years, not decades.^[1] Also known as “killer robots,” these weapons would have the power to make life-and-death determinations, a power previously reserved for humans. The prospect raises a host of moral, legal, and other concerns.^[2]

States parties to the Convention on Conventional Weapons (CCW) have discussed these concerns at two meetings devoted to “lethal autonomous weapons systems,” the CCW term for fully autonomous weapons. A third informal meeting of experts is scheduled for April 2016. While states are still considering how to deal with the problems posed by these weapons, there is emerging agreement that the issue of meaningful human control should be a central point of discussion.

In the arms arena, the term “meaningful human control” signifies control over the selection and engagement of targets, that is, the “critical functions” of a weapon.^[3] “This means when, where and how weapons are used; what or whom they are used against; and the effects of their use,” according to Article 36, a UK nongovernmental organization.^[4] Humans should exercise control over individual attacks, not simply overall operations. Only by prohibiting the use of fully autonomous weapons can such control be guaranteed.

It is both important and feasible to require human control over the use of weapons. Mandating human control would resolve many of the moral and legal concerns that fully autonomous weapons raise. For this reason, the concept is gaining currency among CCW states parties, many of whom see it as central to avoiding the threats presented by this new class of weapons. International law offers models for making control a legal requirement. Disarmament law prohibits several weapons that lack human control. Other legal frameworks treat control as a threshold for liability or an obligation to prevent harm. Such precedents could inform application of the concept of control to law governing new weapons.

Recognizing meaningful human control as a viable means to address the problems posed by emerging weapons, Human Rights Watch and Harvard Law School’s International Human Rights Clinic call on states to ban weapons without such control.

In particular, states should:

• Adopt an international, legally binding instrument that prohibits the development, production, and use of fully autonomous weapons; and
• Adopt national laws or policies that establish similar prohibitions on fully autonomous weapons.

The Importance of Meaningful Human Control

A requirement to maintain human control over the use of weapons would eliminate many of the problems associated with fully autonomous weapons. Such a requirement would protect the dignity of human life, facilitate compliance with international humanitarian and human rights law, and promote accountability for unlawful acts.

A Moral Imperative

Mandating meaningful human control over weapons would help obviate threats to fundamental moral principles. Any decision to use force should be made with great care and respect for the value of human life. From a moral perspective, the power to come to such a decision should rest with humans because they are endowed with reason and possess “prudential judgment,” the ability to apply broad principles and past experience to particular situations.^[5] Because the exercise of prudential judgment depends on more than numeric analysis of data on lawful and unlawful attacks, it would be very difficult for a fully autonomous weapon, no matter how much data it could process, to exercise this sort of judgment. As the Holy See observed in a statement at a CCW meeting on lethal autonomous weapon systems, “Prudential judgement cannot be put into algorithms.”^[6]

Humans also tend to feel the emotional weight and psychological burden of choosing to take away the life of other human beings.^[7] This is partly due to the fact that humans have the potential for empathy, which has been described as “the capacity to be profoundly touched by the misery of the other and to share in his burden.”^[8] Empathy can act as a check on killing, but only if humans have control over whom to target and when to fire. Christof Heyns, the UN special rapporteur on extrajudicial, summary or arbitrary executions, has written that: “Delegating this process [of deciding to use lethal force] dehumanizes armed conflict even further and precludes a moment of deliberation in those cases where it may be feasible. Machines lack morality and mortality, and should as a result not have life and death powers over humans.”^[9]

Ceding human control over decisions about who lives and who dies would deprive people of their inherent dignity.^[10] Inanimate machines, such as fully autonomous weapons, could truly comprehend neither the value of individual life nor the significance of its loss. Permitting them to make determinations to take life away would thus conflict with the principle of dignity and could “denigrate the value of life itself.”^[11] It would also endanger fundamental human rights. The concept of human dignity lies at the heart of the Universal Declaration of Human Rights and serves as a basis for all other rights.^[12] Because meaningful human control over weapons allows for ethical and unquantifiable factors to play a role in targeting decisions, it protects the dignity of civilians and soldiers alike.

Compliance with International Humanitarian and Human Rights Law

Meaningful human control over the use of weapons is consistent with and promotes compliance with the principles of international humanitarian law, notably distinction and proportionality. The ability to distinguish combatants from civilians or from wounded or surrendering soldiers as well as the ability to weigh civilian harm against military advantage require human qualities that would be difficult to replicate in machines, including fully autonomous weapons.^[13] Determining whether an individual is a legitimate target often depends on the capacity to detect and interpret subtle cues, such as tone of voice and body language. Humans usually understand such nuances because they can identify with other human beings and thus better gauge their intentions. Assessing proportionality entails a case-by-case analysis, traditionally based on a reasonable commander standard. Such an analysis requires “distinctively human judgement” and the application of reason, which takes into account both moral and legal considerations.^[14] Meaningful human control guarantees that human perception and judgment inform the decision about whether to use lethal force in a specific instance.

Because existing laws do not specifically address the prospect of weapons without meaningful human control, the Martens Clause is also relevant.^[15] The Martens Clause, a provision of international humanitarian law set out in Additional Protocol I to the Geneva Conventions, states:

In cases not covered by this Protocol or by other international agreements, civilians and combatants remain under the protection and authority of the principles of international law derived from established custom, from the principles of humanity and from the dictates of public conscience.^[16]

The International Committee of the Red Cross (ICRC) describes “humanity” as requiring compassion for others and an ability to protect.^[17] Machines cannot feel compassion and, as described above, fully autonomous weapons would have difficulty complying with international humanitarian law, an important tool for protecting civilians. Requiring meaningful human control over weapons would ensure that the principle of humanity can play a role in the selection and engagement of targets.

Meaningful human control is also crucial to compliance with human rights law. In addition to undermining human dignity, lack of control would threaten the right not to be arbitrarily deprived of life. Whether in a law enforcement or an armed conflict situation, upholding that right depends on human qualities of perception and judgment that are difficult to replicate in machines yet essential to assessing the necessity of force. In a 2015 general comment on the right to life under the African Charter of Human and Peoples’ Rights, the charter’s treaty body wrote, “Where advanced technology is employed, law enforcement officials must remain personally in control of the actual delivery or release of force.” With regard to armed conflict, it declared that “[a]ny machine autonomy in the selection of human targets or the use of force should be subject to meaningful human control.”^[18] A 2016 report from the UN special rapporteurs on extrajudicial, summary or arbitrary executions and on the rights to freedom of peaceful assembly and of association echoed the African Commission’s statement about meaningful human control in law enforcement, and recommended that “[a]utonomous weapons systems that require no meaningful human control should be prohibited.”^[19]

Promotion of Accountability

An obligation to have meaningful human control would allow for the imposition of legal liability and avoid the accountability gap associated with fully autonomous weapons.^[20] A weapon could not itself be punished because machines cannot experience suffering or be deterred. In addition, it could not possess the mental state or intentionality necessary for criminal responsibility. A human commander or operator that lacked meaningful human control over a weapon would also escape liability. He or she could not be held directly liable for a fully autonomous weapon’s unlawful actions because the robot would have operated independently. In most cases, the commander would also avoid indirect or command responsibility for harms caused by a robot because he or she could not prevent or punish its actions, a prerequisite for liability in international criminal law.^[21]

Mandating meaningful human control would close the accountability gap and ensure that someone could be punished for an unlawful act caused by the weapon used. A commander or operator could face direct liability because he or she would choose when weapons fire on targets. The commander would also have knowledge of the decision-making process and the ability to prevent unlawful attacks. Finally, if meaningful human control were a legal requirement, a commander could be held criminally liable for using any weapon without such control, regardless of whether it caused an unlawful act.

Views on the Concept of Human Control

Over the past two years many states parties to the Convention on Conventional Weapons have recognized and embraced the value of meaningful human control. The concept has been discussed since the first CCW meeting on lethal autonomous weapons systems was held in May 2014 and its currency has increased over the course of deliberations. Almost 30 countries have specifically addressed the concept of human control in statements at the CCW experts meetings, usually characterizing it as meaningful, appropriate, or effective. At least nine states referenced the concept during the 2014 Meeting of Experts, and this number jumped to at least 27 in 2015.^[22] Most of these states have expressed either explicit support for a requirement of meaningful human control or an interest in exploring the concept in greater depth. Such statements illustrate the growing belief that meaningful human control provides fruitful grounds for further discussion and common understandings. It could help direct state practice in the future, although states will have to agree on a clear and shared definition to make it a useful standard.

During the two experts meetings, at least a dozen states explicitly said they viewed human control over the use of weapons as necessary for various reasons.[23] Reflecting on this meeting and others, the ICRC concluded that “there appears to be broad agreement among States on the need to retain human control over the critical functions of weapon systems.”^[24] Colombia, for example, stated that “multilateral regulation is required” to ensure human control over deployed weapons.^[25] Croatia said, “[A]n international prohibition of weapons systems operating without meaningful human control should not be something unthinkable, particularly given the calls for a moratorium.”^[26] Denmark said that “[a]ll use of force must remain under meaningful human control.”^[27] Although not all states embraced the concept of meaningful human control, by November 2015 a total of nine states had called for a preemptive ban on fully autonomous weapons, which amounts to a requirement of meaningful human control over the use of weapons.^[28]

Several states have argued that there is already a moral duty to maintain human control. A 2015 paper from the Holy See, which has presented the most in-depth discussion of the ethical objections to autonomous weapons systems, explained, “It is fundamentally immoral to utilize a weapon the behavior of which we cannot completely control.”^[29] The previous year, Chile stated that significant human control over weapons is an “ethical imperative” rather than a technological problem.^[30] Germany argued that from a moral perspective, “it is indispensable to maintain human control over the decision to kill another human being.”^[31] Iraq noted that great ethical value, in addition to practical value, comes from human control over weapons systems.^[32] To states concerned about the moral problems with fully autonomous weapons, no technological improvements can solve the fundamental problem of delegating a human life-or-death decision to a machine.

In the CCW meetings in 2014 and 2015, states raised additional concerns about the lack of meaningful human control, notably the danger of an accountability gap. During the 2015 Meeting of Experts, for example, the Republic of Korea cited three problems of weapons without such control: “risk of malfunctioning, potential accountability gap, and ethical concerns.”^[33] Poland stated, “Human or institutional oversight upholds accountability, the rule of law and supports procedures through which our decisions may be verified.”^[34]

At least 18 CCW states have requested further discussions of the concept of human control.[35] Japan noted that “consensus is not easy” and called for “in-depth discussions” of meaningful human control in order to sharpen understanding of the concept and move toward a definition of lethal autonomous weapons systems.^[36] The Netherlands said, “We see the notion of meaningful human control as an important concept for the discussion on [lethal autonomous weapons systems].”^[37] While stating its belief that “the decision to end somebody’s life must remain under meaningful human control,” the Czech Republic noted that “[t]he challenging part is to establish what precisely ‘meaningful human control’ would entail.”^[38]

The United States and Israel have both advocated for using the term “appropriate human judgment” rather than meaningful human control in the discussion of lethal autonomous weapons systems.^[39] This phrase seems to aim at the same idea as meaningful human control, although it may differ in connotation. For example, control is more likely to ensure that humans have the power to reverse a machine’s decision on a particular attack. The United States has stated at CCW meetings that it is important to “ensure appropriate levels of human judgment over the use of force.”^[40] In addition, the US Department of Defense policy on autonomous weapons requires that they be designed to allow commanders and operators to exercise such judgment over the use of force.^[41] According to Israel, appropriate human judgment is already built into the development of weapons systems, including at the design, testing, and deployment phases, and thus requiring meaningful human control is unnecessary.^[42] Despite these references to human judgment, as of March 2016, CCW states parties who have spoken on the issue have used “control” more frequently than “judgment” in this context.     

Precedents in Disarmament Law

Although the specific term “meaningful human control” has not appeared in international arms treaties, the idea of human control is not new in disarmament law. Recognition of the need for human control is present in prohibitions of mines and chemical and biological weapons, which were motivated in part by concern about the inability to dictate whom they engage and when. After a victim-activated mine is deployed, a human operator cannot determine at what moment it will detonate or whom it will injure or kill. Although a human can choose the moment and initial target of a biological or chemical weapons attack, the weapons’ effects after release are uncontrollable and can extend across space and time causing unintended casualties. The bans on mines and chemical and biological weapons provide precedent for prohibiting weapons over which there is inadequate human control.

Mines

The idea of human control can be found in disarmament law as early as 1907, in Hague Convention No. VIII on automatic submarine contact mines. This Hague convention prohibited states parties from laying unanchored automatic contact mines “except when they are so constructed as to become harmless one hour at most after the person who laid them ceases to control them.”^[43] The text implies that these sea mines become unacceptably dangerous without human control.

Reflecting similar concerns, the 1997 Mine Ban Treaty banned the use, production, stockpiling, and transfer of antipersonnel landmines because they are also uncontrollable and thus indiscriminate.^[44] The treaty prohibits victim-activated antipersonnel mines, but not command-detonated mines, which a human operator detonates by remote control. Directional fragmentation “Claymore” mines, for example, are not covered by the treaty when they are used in a command-detonated mode, that is, without a tripwire.^[45]

The Mine Ban Treaty’s prohibition of victim-activated but not command-detonated landmines highlights that human control was a key factor in determining which weapons to ban. As one state party explained to the Landmine Monitor, command-detonated mines are “designed to be placed on the ground, aimed and controlled by a soldier who assesses the situation and makes a deliberate decision as to detonation.”^[46] It is the element of human control that distinguishes command-detonated mines from the antipersonnel mines covered by the Mine Ban Treaty. The treaty’s explicit prohibition of victim-activated mines reflects that they pose a greater threat to non-combatants than do command-detonated ones.^[47] It also demonstrates that states have objected to weapons that can operate and kill without human control. Similar objections have been raised about fully autonomous weapon, because they, and not a human operator or victim, would determine when to activate lethal force on their own.

Biological and Chemical Weapons

The international bans on biological and chemical weapons resulted in part from concern about the controllability of the weapons. After releasing such weapons, humans cannot control where they go or whom they kill. This lack of control can lead to unintended victims, which underlies many of the objections to biological and chemical weapons.

There have been long-standing moral and legal objections to these weapons, including from within the military. In 1964, a US Army Reserves officer wrote in his personal capacity that when biological warfare “escapes completely from human control, its use must be rejected as immoral.”^[48] Fifty years later, a US Air Force manual classified biological and chemical weapons as unlawful, grouping them with weapons that are “incapable of being controlled.”^[49]

On multiple occasions in the years leading up to the 1972 Biological Weapons Convention, UN officials and bodies expressed concern about the uncontrollability of biological and chemical weapons. A 1969 report from the UN secretary-general commented that “controllability … is a most important consideration in their [biological and chemical agents’] use as weapons.”^[50] Following this report, the UN General Assembly adopted, with only three dissenting votes, a resolution declaring the use of biological and chemical weapons to be counter to general principles of international law.^[51] The resolution explicitly gives as one reason for its position that biological and chemical weapons’ “effects are often uncontrollable and unpredictable and may be injurious without distinction to combatants and non-combatants.”^[52]

The effects of using fully autonomous weapons are also potentially uncontrolled because they, not their human operators, would make life-and-death determinations. While biological and chemical weapons are now banned, requiring meaningful human control over the use of weapons would address comparable concerns about fully autonomous weapons and other problematic means of war in the future.

The Concept of Control in Other Areas of International Law

Public international law has embraced the concept of control in several other areas. Under the rules of state responsibility and command responsibility, for example, “effective control” is a prerequisite for legal liability for unlawful conduct.^[53] In international environmental law, control becomes a positive obligation and must be exercised over substances that could harm unintended victims. These bodies of law show that control is a well-accepted legal concept that could be adapted to the arms context. In addition, an examination of how the frameworks use the term could inform discussions of the legal significance and elements of meaningful human control over weapons.

Responsibility for Internationally Wrongful Conduct

State Responsibility

International law holds a state responsible for conduct that can be attributed to it, including that of private actors and groups over which the state exercises a certain degree of control.^[54] The relevant standard is effective control as articulated by the International Court of Justice (ICJ) in the 1986 Military and Paramilitary Activities in and against Nicaragua case (“Nicaragua”).^[55] In that case, Nicaragua sued the United States for its involvement in acts committed by the contra rebels against the Nicaraguan government. Nicaragua alleged that the acts of the contras were attributable to the United States because the United States financed, organized, trained, supplied, and equipped the contras, as well as selected their targets and planned their operations.

While the ICJ ruled that this involvement violated the principle of non-interference with the affairs of another state, it found that the United States did not have enough control over the contras to be responsible for their internationally wrongful acts. The court argued that “[f]or this conduct to give rise to legal responsibility … it would … have to be proved that State had effective control of the military or paramilitary operations in the course of which the alleged violations were committed.”^[56] The court did not elaborate further on what would have amounted to effective control in this situation. In its 2007 judgment in the Case Concerning the Application of the Convention on the Prevention and Punishment of Genocide, however, the ICJ explained that a state must have effective control over the specific operation in which the alleged violations of international law occurred, not the “overall actions” of the private individuals or group.^[57]

The effective control standard under the rules of state responsibility establishes liability for breaches of the law. Without effective control, there is a danger of an accountability gap, especially if a machine, such as a fully autonomous weapon, was the actor committing the crime. The standard also recognizes that it is control over specific operations that puts a state in a position to ensure that its actions comply with international law. Similarly, requiring meaningful human control over the use of weapons in individual attacks would promote respect for international law and establish legal responsibility for breaches.

Command Responsibility

The doctrine of command responsibility, like that of state responsibility, considers control to be a prerequisite for assigning liability. Referred to as superior responsibility in non-military contexts, this mode of criminal liability places responsibility for the actions of a subordinate on an individual commander. It arises where a commander fails to prevent or punish the commission of a crime by one of his or her subordinates. The doctrine is laid out in the statutes of the International Criminal Tribunal for the former Yugoslavia (ICTY), the International Criminal Tribunal for Rwanda (ICTR), and the International Criminal Court (ICC).^[58] These statutes all identify three elements that must be satisfied to trigger command responsibility: a superior-subordinate relationship, knowledge or reason to know of the crime, and failure to prevent the crime or punish the perpetrator.

A commander must have “effective command and control” over subordinates to meet the first criteria and be held liable for subordinates’ actions.^[59] According to an ICTR judgment, the “material ability to control the actions of subordinates is the touchstone of individual [command] responsibility.”^[60] Under this doctrine, effective control entails the ability to prevent troops from committing unlawful acts or to punish them after the fact.^[61]

Meaningful human control similarly entails a capacity to prevent harm, in this case, the civilian casualties that weapons outside human control might cause. Such control gives humans the power to determine when to use force and as well as the potential to intervene if a weapon is hacked or malfunctions. In addition, like effective control, which makes criminal liability possible, meaningful human control would close the accountability gap created were weapons themselves to select and engage targets without a human in the loop.

Control as a Positive Obligation: International Environmental Law

In some areas of the law, control is a positive obligation imposed on states, rather than a threshold that triggers liability. International environmental law requires states to control pollution and other causes of environmental damage in order to prevent and minimize harm to the environment. For example, the 1982 United Nations Convention on the Law of the Sea devotes a section to states’ obligations to “prevent, reduce and control pollution of the marine environment.”^[62] State control advances environmental protection by decreasing pollution and helping to ensure that contamination does not spread across national borders.^[63] A number of other treaties exist specifically to control the transboundary movement of hazardous wastes and contaminants, and many of them use the word “control” in their title.^[64] In international environmental law, state control can be achieved through a range of means, including adoption of international and national laws, monitoring of the risks and effects of pollution, implementation of enforcement measures, and establishment of mechanisms for legal liability.^[65]

International environmental law shows that a state’s obligation to exercise control has found acceptance within the international community and offers a promising a model for a comparable duty in the context of regulating the use of weapons. Like state control over pollution, human control over weapons serves to prevent harm to unintended victims and across national borders.^[66] As a result, the requirement for meaningful human control over weapons could similarly be promulgated in international and national law, and would be enhanced by monitoring and enforcement mechanisms as well as through a means of accountability for violations.    

Conclusion

Mandating meaningful human control of weapons would help protect human dignity in war, ensure compliance with international humanitarian and human rights law, and avoid creating an accountability gap for the unlawful acts of a weapon. Discussions at CCW meetings suggest that such a requirement would be well received by many countries. Precedent shows that the concept of control is not new to disarmament or other areas of international law. As a result, it would likely be feasible and effective to apply it to the use of weapons. In light of these factors, Human Rights Watch and the International Human Rights Clinic call on all states to adopt a prohibition on the development, production, and use of fully autonomous weapons.