Killer Robots and the Concept of Meaningful Human Control

Introduction

Machines have long served as instruments of war, but historically humans have always dictated how they are used. The evolution of technology has the potential to change that reality, and the implications are profoundly disturbing. According to experts in artificial intelligence, fully autonomous weapons, which would select and engage targets without meaningful human control, could be developed for use within years, not decades.^{^[1]} Also known as “killer robots,” these weapons would have the power to make life-and-death determinations, a power previously reserved for humans. The prospect raises a host of moral, legal, and other concerns.^{^[2]}

States parties to the Convention on Conventional Weapons (CCW) have discussed these concerns at two meetings devoted to “lethal autonomous weapons systems,” the CCW term for fully autonomous weapons. A third informal meeting of experts is scheduled for April 2016. While states are still considering how to deal with the problems posed by these weapons, there is emerging agreement that the issue of meaningful human control should be a central point of discussion.

In the arms arena, the term “meaningful human control” signifies control over the selection and engagement of targets, that is, the “critical functions” of a weapon.^{^[3]} “This means when, where and how weapons are used; what or whom they are used against; and the effects of their use,” according to Article 36, a UK nongovernmental organization.^{^[4]} Humans should exercise control over individual attacks, not simply overall operations. Only by prohibiting the use of fully autonomous weapons can such control be guaranteed.

It is both important and feasible to require human control over the use of weapons. Mandating human control would resolve many of the moral and legal concerns that fully autonomous weapons raise. For this reason, the concept is gaining currency among CCW states parties, many of whom see it as central to avoiding the threats presented by this new class of weapons. International law offers models for making control a legal requirement. Disarmament law prohibits several weapons that lack human control. Other legal frameworks treat control as a threshold for liability or an obligation to prevent harm. Such precedents could inform application of the concept of control to law governing new weapons.

Recognizing meaningful human control as a viable means to address the problems posed by emerging weapons, Human Rights Watch and Harvard Law School’s International Human Rights Clinic call on states to ban weapons without such control.

In particular, states should:

Adopt an international, legally binding instrument that prohibits the development, production, and use of fully autonomous weapons; and
Adopt national laws or policies that establish similar prohibitions on fully autonomous weapons.

The Importance of Meaningful Human Control

A requirement to maintain human control over the use of weapons would eliminate many of the problems associated with fully autonomous weapons. Such a requirement would protect the dignity of human life, facilitate compliance with international humanitarian and human rights law, and promote accountability for unlawful acts.

A Moral Imperative

Mandating meaningful human control over weapons would help obviate threats to fundamental moral principles. Any decision to use force should be made with great care and respect for the value of human life. From a moral perspective, the power to come to such a decision should rest with humans because they are endowed with reason and possess “prudential judgment,” the ability to apply broad principles and past experience to particular situations.^{^[5]} Because the exercise of prudential judgment depends on more than numeric analysis of data on lawful and unlawful attacks, it would be very difficult for a fully autonomous weapon, no matter how much data it could process, to exercise this sort of judgment. As the Holy See observed in a statement at a CCW meeting on lethal autonomous weapon systems, “Prudential judgement cannot be put into algorithms.”^{^[6]}

Humans also tend to feel the emotional weight and psychological burden of choosing to take away the life of other human beings.^{^[7]} This is partly due to the fact that humans have the potential for empathy, which has been described as “the capacity to be profoundly touched by the misery of the other and to share in his burden.”^{^[8]} Empathy can act as a check on killing, but only if humans have control over whom to target and when to fire. Christof Heyns, the UN special rapporteur on extrajudicial, summary or arbitrary executions, has written that: “Delegating this process [of deciding to use lethal force] dehumanizes armed conflict even further and precludes a moment of deliberation in those cases where it may be feasible. Machines lack morality and mortality, and should as a result not have life and death powers over humans.”^{^[9]}

Ceding human control over decisions about who lives and who dies would deprive people of their inherent dignity.^{^[10]} Inanimate machines, such as fully autonomous weapons, could truly comprehend neither the value of individual life nor the significance of its loss. Permitting them to make determinations to take life away would thus conflict with the principle of dignity and could “denigrate the value of life itself.”^{^[11]} It would also endanger fundamental human rights. The concept of human dignity lies at the heart of the Universal Declaration of Human Rights and serves as a basis for all other rights.^{^[12]} Because meaningful human control over weapons allows for ethical and unquantifiable factors to play a role in targeting decisions, it protects the dignity of civilians and soldiers alike.

Compliance with International Humanitarian and Human Rights Law

Meaningful human control over the use of weapons is consistent with and promotes compliance with the principles of international humanitarian law, notably distinction and proportionality. The ability to distinguish combatants from civilians or from wounded or surrendering soldiers as well as the ability to weigh civilian harm against military advantage require human qualities that would be difficult to replicate in machines, including fully autonomous weapons.^{^[13]} Determining whether an individual is a legitimate target often depends on the capacity to detect and interpret subtle cues, such as tone of voice and body language. Humans usually understand such nuances because they can identify with other human beings and thus better gauge their intentions. Assessing proportionality entails a case-by-case analysis, traditionally based on a reasonable commander standard. Such an analysis requires “distinctively human judgement” and the application of reason, which takes into account both moral and legal considerations.^{^[14]} Meaningful human control guarantees that human perception and judgment inform the decision about whether to use lethal force in a specific instance.

Because existing laws do not specifically address the prospect of weapons without meaningful human control, the Martens Clause is also relevant.^{^[15]} The Martens Clause, a provision of international humanitarian law set out in Additional Protocol I to the Geneva Conventions, states:

In cases not covered by this Protocol or by other international agreements, civilians and combatants remain under the protection and authority of the principles of international law derived from established custom, from the principles of humanity and from the dictates of public conscience.^{^[16]}

The International Committee of the Red Cross (ICRC) describes “humanity” as requiring compassion for others and an ability to protect.^{^[17]} Machines cannot feel compassion and, as described above, fully autonomous weapons would have difficulty complying with international humanitarian law, an important tool for protecting civilians. Requiring meaningful human control over weapons would ensure that the principle of humanity can play a role in the selection and engagement of targets.

Meaningful human control is also crucial to compliance with human rights law. In addition to undermining human dignity, lack of control would threaten the right not to be arbitrarily deprived of life. Whether in a law enforcement or an armed conflict situation, upholding that right depends on human qualities of perception and judgment that are difficult to replicate in machines yet essential to assessing the necessity of force. In a 2015 general comment on the right to life under the African Charter of Human and Peoples’ Rights, the charter’s treaty body wrote, “Where advanced technology is employed, law enforcement officials must remain personally in control of the actual delivery or release of force.” With regard to armed conflict, it declared that “[a]ny machine autonomy in the selection of human targets or the use of force should be subject to meaningful human control.”^{^[18]} A 2016 report from the UN special rapporteurs on extrajudicial, summary or arbitrary executions and on the rights to freedom of peaceful assembly and of association echoed the African Commission’s statement about meaningful human control in law enforcement, and recommended that “[a]utonomous weapons systems that require no meaningful human control should be prohibited.”^{^[19]}

Promotion of Accountability

An obligation to have meaningful human control would allow for the imposition of legal liability and avoid the accountability gap associated with fully autonomous weapons.^{^[20]} A weapon could not itself be punished because machines cannot experience suffering or be deterred. In addition, it could not possess the mental state or intentionality necessary for criminal responsibility. A human commander or operator that lacked meaningful human control over a weapon would also escape liability. He or she could not be held directly liable for a fully autonomous weapon’s unlawful actions because the robot would have operated independently. In most cases, the commander would also avoid indirect or command responsibility for harms caused by a robot because he or she could not prevent or punish its actions, a prerequisite for liability in international criminal law.^{^[21]}

Mandating meaningful human control would close the accountability gap and ensure that someone could be punished for an unlawful act caused by the weapon used. A commander or operator could face direct liability because he or she would choose when weapons fire on targets. The commander would also have knowledge of the decision-making process and the ability to prevent unlawful attacks. Finally, if meaningful human control were a legal requirement, a commander could be held criminally liable for using any weapon without such control, regardless of whether it caused an unlawful act.

Views on the Concept of Human Control

Over the past two years many states parties to the Convention on Conventional Weapons have recognized and embraced the value of meaningful human control. The concept has been discussed since the first CCW meeting on lethal autonomous weapons systems was held in May 2014 and its currency has increased over the course of deliberations. Almost 30 countries have specifically addressed the concept of human control in statements at the CCW experts meetings, usually characterizing it as meaningful, appropriate, or effective. At least nine states referenced the concept during the 2014 Meeting of Experts, and this number jumped to at least 27 in 2015.^{^[22]} Most of these states have expressed either explicit support for a requirement of meaningful human control or an interest in exploring the concept in greater depth. Such statements illustrate the growing belief that meaningful human control provides fruitful grounds for further discussion and common understandings. It could help direct state practice in the future, although states will have to agree on a clear and shared definition to make it a useful standard.

During the two experts meetings, at least a dozen states explicitly said they viewed human control over the use of weapons as necessary for various reasons.[23] Reflecting on this meeting and others, the ICRC concluded that “there appears to be broad agreement among States on the need to retain human control over the critical functions of weapon systems.”^{^[24]} Colombia, for example, stated that “multilateral regulation is required” to ensure human control over deployed weapons.^{^[25]} Croatia said, “[A]n international prohibition of weapons systems operating without meaningful human control should not be something unthinkable, particularly given the calls for a moratorium.”^{^[26]} Denmark said that “[a]ll use of force must remain under meaningful human control.”^{^[27]} Although not all states embraced the concept of meaningful human control, by November 2015 a total of nine states had called for a preemptive ban on fully autonomous weapons, which amounts to a requirement of meaningful human control over the use of weapons.^{^[28]}

Several states have argued that there is already a moral duty to maintain human control. A 2015 paper from the Holy See, which has presented the most in-depth discussion of the ethical objections to autonomous weapons systems, explained, “It is fundamentally immoral to utilize a weapon the behavior of which we cannot completely control.”^{^[29]} The previous year, Chile stated that significant human control over weapons is an “ethical imperative” rather than a technological problem.^{^[30]} Germany argued that from a moral perspective, “it is indispensable to maintain human control over the decision to kill another human being.”^{^[31]} Iraq noted that great ethical value, in addition to practical value, comes from human control over weapons systems.^{^[32]} To states concerned about the moral problems with fully autonomous weapons, no technological improvements can solve the fundamental problem of delegating a human life-or-death decision to a machine.

In the CCW meetings in 2014 and 2015, states raised additional concerns about the lack of meaningful human control, notably the danger of an accountability gap. During the 2015 Meeting of Experts, for example, the Republic of Korea cited three problems of weapons without such control: “risk of malfunctioning, potential accountability gap, and ethical concerns.”^{^[33]} Poland stated, “Human or institutional oversight upholds accountability, the rule of law and supports procedures through which our decisions may be verified.”^{^[34]}

At least 18 CCW states have requested further discussions of the concept of human control.[35] Japan noted that “consensus is not easy” and called for “in-depth discussions” of meaningful human control in order to sharpen understanding of the concept and move toward a definition of lethal autonomous weapons systems.^{^[36]} The Netherlands said, “We see the notion of meaningful human control as an important concept for the discussion on [lethal autonomous weapons systems].”^{^[37]} While stating its belief that “the decision to end somebody’s life must remain under meaningful human control,” the Czech Republic noted that “[t]he challenging part is to establish what precisely ‘meaningful human control’ would entail.”^{^[38]}

The United States and Israel have both advocated for using the term “appropriate human judgment” rather than meaningful human control in the discussion of lethal autonomous weapons systems.^{^[39]} This phrase seems to aim at the same idea as meaningful human control, although it may differ in connotation. For example, control is more likely to ensure that humans have the power to reverse a machine’s decision on a particular attack. The United States has stated at CCW meetings that it is important to “ensure appropriate levels of human judgment over the use of force.”^{^[40]} In addition, the US Department of Defense policy on autonomous weapons requires that they be designed to allow commanders and operators to exercise such judgment over the use of force.^{^[41]} According to Israel, appropriate human judgment is already built into the development of weapons systems, including at the design, testing, and deployment phases, and thus requiring meaningful human control is unnecessary.^{^[42]} Despite these references to human judgment, as of March 2016, CCW states parties who have spoken on the issue have used “control” more frequently than “judgment” in this context.

Precedents in Disarmament Law

Although the specific term “meaningful human control” has not appeared in international arms treaties, the idea of human control is not new in disarmament law. Recognition of the need for human control is present in prohibitions of mines and chemical and biological weapons, which were motivated in part by concern about the inability to dictate whom they engage and when. After a victim-activated mine is deployed, a human operator cannot determine at what moment it will detonate or whom it will injure or kill. Although a human can choose the moment and initial target of a biological or chemical weapons attack, the weapons’ effects after release are uncontrollable and can extend across space and time causing unintended casualties. The bans on mines and chemical and biological weapons provide precedent for prohibiting weapons over which there is inadequate human control.

Mines

The idea of human control can be found in disarmament law as early as 1907, in Hague Convention No. VIII on automatic submarine contact mines. This Hague convention prohibited states parties from laying unanchored automatic contact mines “except when they are so constructed as to become harmless one hour at most after the person who laid them ceases to control them.”^{^[43]} The text implies that these sea mines become unacceptably dangerous without human control.

Reflecting similar concerns, the 1997 Mine Ban Treaty banned the use, production, stockpiling, and transfer of antipersonnel landmines because they are also uncontrollable and thus indiscriminate.^{^[44]} The treaty prohibits victim-activated antipersonnel mines, but not command-detonated mines, which a human operator detonates by remote control. Directional fragmentation “Claymore” mines, for example, are not covered by the treaty when they are used in a command-detonated mode, that is, without a tripwire.^{^[45]}

The Mine Ban Treaty’s prohibition of victim-activated but not command-detonated landmines highlights that human control was a key factor in determining which weapons to ban. As one state party explained to the Landmine Monitor, command-detonated mines are “designed to be placed on the ground, aimed and controlled by a soldier who assesses the situation and makes a deliberate decision as to detonation.”^{^[46]} It is the element of human control that distinguishes command-detonated mines from the antipersonnel mines covered by the Mine Ban Treaty. The treaty’s explicit prohibition of victim-activated mines reflects that they pose a greater threat to non-combatants than do command-detonated ones.^{^[47]} It also demonstrates that states have objected to weapons that can operate and kill without human control. Similar objections have been raised about fully autonomous weapon, because they, and not a human operator or victim, would determine when to activate lethal force on their own.

Biological and Chemical Weapons

The international bans on biological and chemical weapons resulted in part from concern about the controllability of the weapons. After releasing such weapons, humans cannot control where they go or whom they kill. This lack of control can lead to unintended victims, which underlies many of the objections to biological and chemical weapons.

There have been long-standing moral and legal objections to these weapons, including from within the military. In 1964, a US Army Reserves officer wrote in his personal capacity that when biological warfare “escapes completely from human control, its use must be rejected as immoral.”^{^[48]} Fifty years later, a US Air Force manual classified biological and chemical weapons as unlawful, grouping them with weapons that are “incapable of being controlled.”^{^[49]}

On multiple occasions in the years leading up to the 1972 Biological Weapons Convention, UN officials and bodies expressed concern about the uncontrollability of biological and chemical weapons. A 1969 report from the UN secretary-general commented that “controllability … is a most important consideration in their [biological and chemical agents’] use as weapons.”^{^[50]} Following this report, the UN General Assembly adopted, with only three dissenting votes, a resolution declaring the use of biological and chemical weapons to be counter to general principles of international law.^{^[51]} The resolution explicitly gives as one reason for its position that biological and chemical weapons’ “effects are often uncontrollable and unpredictable and may be injurious without distinction to combatants and non-combatants.”^{^[52]}

The effects of using fully autonomous weapons are also potentially uncontrolled because they, not their human operators, would make life-and-death determinations. While biological and chemical weapons are now banned, requiring meaningful human control over the use of weapons would address comparable concerns about fully autonomous weapons and other problematic means of war in the future.

The Concept of Control in Other Areas of International Law

Public international law has embraced the concept of control in several other areas. Under the rules of state responsibility and command responsibility, for example, “effective control” is a prerequisite for legal liability for unlawful conduct.^{^[53]} In international environmental law, control becomes a positive obligation and must be exercised over substances that could harm unintended victims. These bodies of law show that control is a well-accepted legal concept that could be adapted to the arms context. In addition, an examination of how the frameworks use the term could inform discussions of the legal significance and elements of meaningful human control over weapons.

Responsibility for Internationally Wrongful Conduct

State Responsibility

International law holds a state responsible for conduct that can be attributed to it, including that of private actors and groups over which the state exercises a certain degree of control.^{^[54]} The relevant standard is effective control as articulated by the International Court of Justice (ICJ) in the 1986 Military and Paramilitary Activities in and against Nicaragua case (“Nicaragua”).^{^[55]} In that case, Nicaragua sued the United States for its involvement in acts committed by the contra rebels against the Nicaraguan government. Nicaragua alleged that the acts of the contras were attributable to the United States because the United States financed, organized, trained, supplied, and equipped the contras, as well as selected their targets and planned their operations.

While the ICJ ruled that this involvement violated the principle of non-interference with the affairs of another state, it found that the United States did not have enough control over the contras to be responsible for their internationally wrongful acts. The court argued that “[f]or this conduct to give rise to legal responsibility … it would … have to be proved that State had effective control of the military or paramilitary operations in the course of which the alleged violations were committed.”^{^[56]} The court did not elaborate further on what would have amounted to effective control in this situation. In its 2007 judgment in the Case Concerning the Application of the Convention on the Prevention and Punishment of Genocide, however, the ICJ explained that a state must have effective control over the specific operation in which the alleged violations of international law occurred, not the “overall actions” of the private individuals or group.^{^[57]}

The effective control standard under the rules of state responsibility establishes liability for breaches of the law. Without effective control, there is a danger of an accountability gap, especially if a machine, such as a fully autonomous weapon, was the actor committing the crime. The standard also recognizes that it is control over specific operations that puts a state in a position to ensure that its actions comply with international law. Similarly, requiring meaningful human control over the use of weapons in individual attacks would promote respect for international law and establish legal responsibility for breaches.

Command Responsibility

The doctrine of command responsibility, like that of state responsibility, considers control to be a prerequisite for assigning liability. Referred to as superior responsibility in non-military contexts, this mode of criminal liability places responsibility for the actions of a subordinate on an individual commander. It arises where a commander fails to prevent or punish the commission of a crime by one of his or her subordinates. The doctrine is laid out in the statutes of the International Criminal Tribunal for the former Yugoslavia (ICTY), the International Criminal Tribunal for Rwanda (ICTR), and the International Criminal Court (ICC).^{^[58]} These statutes all identify three elements that must be satisfied to trigger command responsibility: a superior-subordinate relationship, knowledge or reason to know of the crime, and failure to prevent the crime or punish the perpetrator.

A commander must have “effective command and control” over subordinates to meet the first criteria and be held liable for subordinates’ actions.^{^[59]} According to an ICTR judgment, the “material ability to control the actions of subordinates is the touchstone of individual [command] responsibility.”^{^[60]} Under this doctrine, effective control entails the ability to prevent troops from committing unlawful acts or to punish them after the fact.^{^[61]}

Meaningful human control similarly entails a capacity to prevent harm, in this case, the civilian casualties that weapons outside human control might cause. Such control gives humans the power to determine when to use force and as well as the potential to intervene if a weapon is hacked or malfunctions. In addition, like effective control, which makes criminal liability possible, meaningful human control would close the accountability gap created were weapons themselves to select and engage targets without a human in the loop.

Control as a Positive Obligation: International Environmental Law

In some areas of the law, control is a positive obligation imposed on states, rather than a threshold that triggers liability. International environmental law requires states to control pollution and other causes of environmental damage in order to prevent and minimize harm to the environment. For example, the 1982 United Nations Convention on the Law of the Sea devotes a section to states’ obligations to “prevent, reduce and control pollution of the marine environment.”^{^[62]} State control advances environmental protection by decreasing pollution and helping to ensure that contamination does not spread across national borders.^{^[63]} A number of other treaties exist specifically to control the transboundary movement of hazardous wastes and contaminants, and many of them use the word “control” in their title.^{^[64]} In international environmental law, state control can be achieved through a range of means, including adoption of international and national laws, monitoring of the risks and effects of pollution, implementation of enforcement measures, and establishment of mechanisms for legal liability.^{^[65]}

International environmental law shows that a state’s obligation to exercise control has found acceptance within the international community and offers a promising a model for a comparable duty in the context of regulating the use of weapons. Like state control over pollution, human control over weapons serves to prevent harm to unintended victims and across national borders.^{^[66]} As a result, the requirement for meaningful human control over weapons could similarly be promulgated in international and national law, and would be enhanced by monitoring and enforcement mechanisms as well as through a means of accountability for violations.

Conclusion

Mandating meaningful human control of weapons would help protect human dignity in war, ensure compliance with international humanitarian and human rights law, and avoid creating an accountability gap for the unlawful acts of a weapon. Discussions at CCW meetings suggest that such a requirement would be well received by many countries. Precedent shows that the concept of control is not new to disarmament or other areas of international law. As a result, it would likely be feasible and effective to apply it to the use of weapons. In light of these factors, Human Rights Watch and the International Human Rights Clinic call on all states to adopt a prohibition on the development, production, and use of fully autonomous weapons.

[1] Future of Life Institute, “Autonomous Weapons: An Open Letter from AI & Robotics Researchers,” July 28, 2015, http://futureoflife.org/open-letter-autonomous-weapons/ (accessed March 18, 2016). As of March 18, 2016, the letter had been signed by more than 3,100 experts and 17,700 other endorsers.

[2] The concerns not discussed elsewhere in this paper include the likelihood of an arms race in the field of fully autonomous weapons and the threat of proliferation of such weapons, especially to dictatorships and non-state armed groups that have little regard for international law.

[3] Statement of the International Committee of the Red Cross (ICRC), CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/4CE346B40DDBF000C1257E2600616A59/$file/ICRC+general+statement+CCW+LAWS+expert+meeting+13+04+2015+FINAL.pdf (accessed April 3, 2016) (discussing critical functions).

[4] Article 36, Killing by Machine: Key Issues for Understanding Meaningful Human Control, April 2015, http://www.article36.org/autonomous-weapons/killing-by-machine-key-issues-for-understanding-meaningful-human-control/ (accessed March 18, 2016).

[5] Statement of the Holy See, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 16, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/4D28AF2B8BBBECEDC1257E290046B73F/$file/2015_LAWS_MX_Holy+See.pdf (accessed February 29, 2016), p. 5.

[6] Ibid.

[7] Report of the UN Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, to the Human Rights Council, A/HRC/23/47, April 9, 2013, http://www.ohchr.org/Documents/HRBodies/HRCouncil/RegularSession/Session23/A-HRC-23-47_en.pdf (accessed March 18, 2016), para. 94. See also Statement of the Holy See, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 16, 2015, p. 4.

[8] Statement of the Holy See, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 16, 2015, p. 4 (noting that the humans with empathy can reduce “the dehumanization and barbarity involved in all wars”).

[9] Report of the UN Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, to the Human Rights Council, A/HRC/23/47, April 9, 2013, para. 94.

[10] For further discussion of the threat to human dignity and international human rights, see Human Rights Watch and Harvard Law School’s International Human Rights Clinic (IHRC), Shaking the Foundations: The Human Rights Implications of Killer Robots, May 2014, https://www.hrw.org/report/2014/05/12/shaking-foundations/human-rights-implications-killer-robots.

[11] According to UN Special Rapporteur Christof Heyns, “[T]here is widespread concern that allowing [fully autonomous weapons] to kill people may denigrate the value of life itself.” Report of the UN Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, to the Human Rights Council, A/HRC/23/47, April 9, 2013, para. 109.

[12] Universal Declaration of Human Rights (UDHR), adopted December 10, 1948, G.A. Res. 217A(III), U.N. Doc. A/810 (1948), art. 1.

[13] For more information on the challenges fully autonomous weapons would have in complying with international humanitarian law, see Human Rights Watch and IHRC, “Advancing the Debate on Killer Robots: 12 Key Arguments for a Preemptive Ban on Fully Autonomous Weapons,” May 2014, https://www.hrw.org/news/2014/05/13/advancing-debate-killer-robots, pp. 4-8.

[14] Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, to the Human Rights Council, A/HRC/23/47, April 9, 2013, para. 72. See also Olivier Corten, “Reasonableness in International Law,” Max Planck Encyclopedia of Public International Law, updated March 2013, http://opil.ouplaw.com/view/10.1093/law:epil/9780199231690/law-9780199231690-

e1679?rskey=U1bcau&result=10&prd=EPIL (accessed March 18, 2016), para. 1.

[15] The Martens Clause first appeared in the preamble to the 1899 Hague Convention (II) on the laws and customs of war on land and reads:

Until a more complete code of the laws of war is issued, the High Contracting Parties think it right to declare that in cases not included in the Regulations adopted by them, populations and belligerents remain under the protection and empire of the principles of international law, as they result from the usages established between civilized nations, from the laws of humanity and the requirements of the public conscience .

Convention (II) with Respect to the Laws and Customs of War on Land and its Annex: Regulations concerning the Laws and Customs of War on Land (Hague II), adopted July 29, 1899, entered into force September 4, 1900, pmbl. (emphasis added). The clause is also found in a modified form in the 1977 Additional Protocol I to the Geneva Conventions of 1949. See also Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions, Christof Heyns, to the Human Rights Council, A/HRC/23/47, April 9, 2013, para. 89; Human Rights Watch and IHRC , “Advancing the Debate on Killer Robots,” pp.14-17.

[16] Protocol Additional to the Geneva Conventions of August 12, 1949, and Relating to the Protection of Victims of International Armed Conflicts (Protocol I), adopted June 8, 1977, 1125 U.N.T.S. 3, entered into force December 7, 1978.

[17] ICRC, “The Fundamental Principles of the Red Cross and Red Crescent,” 1996, http://www.icrc.org/eng/assets/files/other/icrc_002_0513.pdf (accessed March 9, 2016), p. 2.

[18] African Commission on Human and Peoples’ Rights, “General Comment No. 3 on African Charter of Human and Peoples’ Rights: The Right to Life (Article 4),” November 2015.

[19] Joint Report of the UN special rapporteur on the rights to freedom of peaceful assembly and of association and the UN special rapporteur on extrajudicial, summary or arbitrary executions on the proper management of assemblies, to the Human Rights Council, A/HRC/31/66, February 4, 2016, pp. 13, 15.

[20] For more information on the accountability gap created by fully autonomous weapons, see Human Rights Watch and IHRC, Mind the Gap: The Lack of Accountability for Killer Robots, April 2015, https://www.hrw.org/report/2015/04/09/mind-gap/lack-accountability-killer-robots. For a discussion of the challenges with imposing criminal responsibility in particular, see ibid, pp. 18-25.

[21] The doctrine of command responsibility and its elements are discussed in a separate section below.

[22] These figures were derived from statements that have been posted on the United Nation’s CCW webpage, and do not take into account statements that were not posted. 2014 states include: Argentina, Austria, Germany, Ireland, Japan, Mexico, Norway, Sweden, and Switzerland. 2015 states include: Argentina, Austria, Chile, Colombia, Croatia, the Czech Republic, Denmark, Ecuador, Finland, France, Germany, Greece, the Holy See, India, Iraq, Ireland, Israel, Japan, Mexico, the Netherlands, Poland, the Republic of Korea, South Africa, Spain, Sri Lanka, Sweden, and Switzerland. Across the two years, at least 28 states mentioned human control in a statement.

[23] 2014 states include: Austria, Germany, and Sweden. 2015 states include: Austria, Chile, Colombia, Croatia, the Czech Republic, Denmark, Germany, the Holy See, Ireland, the Republic of Korea, Spain, and South Africa.

[24] Statement of the ICRC, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13, 2015. The ICRC added that, “States should now turn their attention to agreeing [to] a framework for determining what makes human control of a weapon meaningful or adequate” (emphasis deleted).

[25] Statement by Colombia, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 17, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/036C8F5A520CB808C1257E2D002C09AA/$file/2015_LAWS_MX_Colombia_WA.pdf (accessed March 10, 2016).

[26] Statement of Croatia, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/4828C7F936CDAC9AC1257E26005D28BF/$file/2015_LAWS_MX_Croatia.pdf (accessed April 3, 2016).

[27] Statement of Denmark, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13-17, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/C5B8B0A4AD379822C1257E26005D7D20/$file/2015_LAWS_MX_Denmark.pdf (accessed March 18, 2016).

[28] Bolivia, Cuba, Ecuador, Egypt, Ghana, Holy See, Pakistan, State of Palestine, and Zimbabwe. See Campaign to Stop Killer Robots, “Report on Activities at the CCW Annual Meeting Held in Geneva,” November 2015, http://www.stopkillerrobots.org/wp-content/uploads/2013/03/KRC_ReportCCWannual16Dec2015_uploaded-1.pdf (accessed April 3, 2016).

[29] Statement of the Holy See, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 16, 2015, p. 8.

[30] Statement by Chile, CCW Meeting of States Parties, Geneva, November 13-14, 2014, http://www.unog.ch/80256EDD006B8954/(httpAssets)/928D4835FF2D3DF4C1257D970046D276/$file/Chile_LAWS_MSP.pdf (accessed February 28, 2016).

[31] Statement by Germany, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, May 13-16, 2014, http://www.unog.ch/80256EDD006B8954/(httpAssets)/9FB02F665072E11AC1257CD70066D830/$file/Germany+LAWS+2014.pdf (accessed March 8, 2016).

[32] Statement by Iraq, CCW Meeting of States Parties, Geneva, November 12-13, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/64597570B3352E0FC1257F0F004C35D4/$file/2015_CCWMSP_LAWS_Iraq.pdf (accessed March 8, 2016) (as translated by Human Rights Watch).

[33] Statement of the Republic of Korea, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/2A22908A9A03E949C1257E29005B90C1/$file/2015_LAWS_MX_RoK_GS+Corr.pdf (accessed February 28, 2016).

[34] Statement of Poland, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13-17, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/16BDFE48306133F6C1257E31002BA329/$file/2015_LAWS_MX_Poland_characteristics.pdf (accessed February 29, 2016). Note that although this quote uses the word “oversight,” Poland’s statement as a whole is comfortable with the language of “control.”

[35] 2014 states include: Ireland, Germany, Mexico, and Switzerland. 2015 states include: Argentina, Chile, the Czech Republic, Denmark, Ecuador, Finland, India, Ireland, Japan, Mexico, the Netherlands, Poland, South Africa, Sweden, and Switzerland.

[36] Statement by Japan, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/7C284472DDBBA998C1257E26005EA5DC/$file/2015_LAWS_MX_Japan.pdf (accessed February 29, 2016).

[37] Statement of the Netherlands, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13-17, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/D5A22AEF369C4ED4C1257E340053754B/$file/2015_LAWS_MX_Netherlands.pdf (accessed April 3, 2016).

[38] Statement of the Czech Republic, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, Geneva, April 13-17, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/2DD5110A33C9C2D2C1257E26005DD47B/$file/2015_LAWS_MX_Czech+Republic.pdf (accessed April 3, 2016).

[39] Statement of the United States, CCW Meeting of Experts on Lethal Autonomous Weapons Systems, April 13, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/8B33A1CDBE80EC60C1257E2800275E56/$file/2015_LAWS_MX_USA+bis.pdf (accessed February 28, 2016).

[40] Ibid.

[41] US Department of Defense, “Autonomy in Weapon Systems,” Directive 3000.09, November 21, 2012, p. 2, http://www.dtic.mil/whs/directives/corres/pdf/300009p.pdf (accessed April 3, 2016).

[42] Statement by Israel, CCW Meeting of States Parties, Geneva, November 13-14, 2014, http://www.unog.ch/80256EDD006B8954/(httpAssets)/A9D6A596BC5B169DC1257D9700471102/$file/Israe_LAWS_MSP.pdf (accessed March 8, 2016).

[43] Convention Relative to the Laying of Automatic Submarine Contact Mines (Hague VIII), adopted October 18, 1907, entered into force January 26, 1910, art. 1. See also Detlev F. Vagts, “The Hague Conventions and Arms Control,” American Journal of International Law, vol. 94, no. 31 (2000), p. 36.

[44] Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and Their Destruction (Mine Ban Treaty), adopted September 18, 1997, entered into force March 1, 1999.

[45] Human Rights Watch, “Claymore-Type and OZM-72 Command-Detonated Mines,” May 2006, http://the-monitor.org/media/1418623/Claymore_and_OZM_May_2006_photos.pdf (accessed March 21, 2016).

[46] International Campaign to Ban Landmines, “Country Profile: Canada,” in Landmine Monitor 2002, ed. Stephen Goose et. al (New York: Human Rights Watch, August 2002), http://archives.the-monitor.org/index.php/publications/display?url=lm/2002/canada.html#fnB858 (quoting “ILX0149: Response to Query,” email to MAC from Shannon Smith, DFAIT/ILX, May 2, 2002, and also citing “The Canadian Forces and Anti-Personnel Landmine,” DND document BG-02.007, February 13, 2002) (emphasis added).

[47] Stuart Maslen, Anti-Personnel Mines under Humanitarian Law: A View from the Vanishing Point, (Oxford: Intersentia, 2001), p. 57.

[48] Colonel Brungs differentiates between types of biological weapons, some of which escape from human control. Col. Bernard J. Brungs, “The Status of Biological Warfare in International Law,” Military Law Review, vol. 24, no. 47 (1964), p. 82.

[49] US Department of the Air Force, “The Military Commander and the Law,” 2015, http://www.afjag.af.mil/shared/media/document/AFD-151102-022.pdf (accessed March 9, 2016), p. 683.

[50] UN Secretary-General, “Chemical and Bacteriological (Biological) Weapons and the Effects of their Possible Use: Report of the Secretary-General” Doc.A/7575/Rev.1, (1969), http://repository.un.org/bitstream/handle/11176/75955/A_7575_Rev.1%3bS_9292_Rev.1-EN.pdf?sequence=10&isAllowed=y (accessed March 9, 2016), para. 28. The report observes that these weapons are deployed by being discharged into the atmosphere. Once in the atmosphere, the biological and chemical agents can be dispersed by elements of nature, such as the wind, and “control is thus possible only to the extent that the meteorological situation can be predicted.” In addition, biological agents can be carried by migratory animals, and chemical agents can spread through underground waters and soil.

[51] UN General Assembly, “Question of Chemical and Bacteriological (Biological) Weapons,” Resolution 2603 (XXIV) (1969) A/7890. The resolution was adopted by a vote of 80 in favor to 3 against; 36 states abstained from voting and 7 states were non-voting.

[52] Ibid.

[53] The law of belligerent occupation also uses the term “effective control,” but in that framework control triggers obligations rather than imputes liability. An occupation exists when the occupying power has effective control over a territory, which means that it can substitute its own authority for that of the occupied authorities. If an occupying power has effective control, it has duties to respect and protect the civilian population. Meaningful human control over weapons could lead to comparable duties to protect the civilian population under international law. Operators, for example, would have to ensure that attacks with weapons under their control were used discriminately and proportionately. See, for example, Convention (IV) respecting the Laws and Customs of War on Land and its Annex: Regulations concerning the Laws and Customs of War on Land (Hague IV), adopted October 18, 1907, entered into force January 26, 1910, art. 46; Geneva Convention relative to the Protection of Civilian Persons in Time of War, adopted August 12, 1949, 75 U.N.T.S. 287, entered into force October 21, 1950, arts. 4, 27, 47 – 62; Protocol I, art. 75. See also Yoram Dinstein, The International Law of Belligerent Occupation (Cambridge: Cambridge University Press, 2009), pp. 43-44.

[54] Draft Articles on Responsibility of States for Internationally Wrongful Acts (text adopted by the International Law Commission at its fifty-third session, in 2001, and submitted to the General Assembly as a part of the Commission’s report covering the work of that session (A/56/10)), pp. 47-49. States also have responsibility for the conduct of its organs of government or agents of the state. Ibid, p. 38.

[55] Military and Paramilitary Activities in and against Nicaragua (Nicaragua v. United States of America), International Court of Justice, Judgment, June 24, 1986.

[56] Ibid, para. 115 (emphasis added).

[57] Application of the Convention on the Prevention and Punishment of the Crime of Genocide (Bosnia and Herzegovina v. Serbia and Montenegro), International Court of Justice, Judgment, February 26, 2007, para. 400. The court stated: “It must however be shown that this ‘effective control’ was exercised, or that the State’s instructions were given, in respect of each operation in which the alleged violations occurred, not generally in respect of the overall actions taken by the persons or groups of persons having committed the violations.”

[58] Statute of the International Criminal Tribunal for the Former Yugoslavia (ICTY), S/RES/827, adopted May 25, 1993, art. 7(3); Statute of the International Criminal Tribunal for Rwanda (ICTR), S/RES/955, adopted November 8, 1994, art. 6(3); Rome Statute of the International Criminal Court (Rome Statute), A/CONF.183/9, July 17, 1998, entered into force July 1, 2002.

^{^[59]} For example, the Rome Statute reads:

A military commander or person effectively acting as a military commander shall be criminally responsible for crimes within the jurisdiction of the Court committed by forces under his or her effective command and control, or effective authority and control as the case may be, as a result of his or her failure to exercise control properly over such forces, where: (i) That military commander or person either knew or, owing to the circumstances at the time, should have known that the forces were committing or about to commit such crimes; and (ii) That military commander or person failed to take all necessary and reasonable measures within his or her power to prevent or repress their commission or to submit the matter to the competent authorities for investigation and prosecution.

Rome Statute, art. 28(a) (emphasis added).

^{^[60]} Prosecutor v. Kayishema, ICTR, Case No. ICTR-95-1-T, Judgment (Trial Chamber II), May 21, 1999, para. 229. See also Prosecutor v. Ignace Bagilishema, Judgment (Trial Chamber), June 7, 2001, para. 45 (“[T]he essential element is not whether a superior had authority over a certain geographical area, but whether he or she had effective control over the individuals who committed the crimes.”).

[61] Rome Statute, art. 28(a).

[62] United Nations Convention on the Law of the Sea (UNCLOS), adopted December 10, 1982, entered into force November 16, 1994, 1833 UNTS 3, part XII, section 5 (entitled “International Rules and National Legislation to Prevent, Reduce and Control Pollution of the Marine Environment”).

[63] Ibid, arts. 207-212, 194(2).

[64] Other treaties on pollution control include the Convention on the Control of Transboundary Movements of Hazardous Wastes and their Disposal, adopted March 22, 1989, entered into force May 15, 1992, 1673 UNTS 57; Convention to Ban the Importation into Forum Island Countries of Hazardous and Radioactive Wastes and to Control the Transboundary Movement and Management of Hazardous Wastes within the South Pacific Region, adopted 1995, entered into force 2001; Convention on the ban of the Import into Africa and the Control of Transboundary Movements and Management of Hazardous Wastes within Africa, adopted January 30, 1991, entered into force April 22, 1998, 30 ILM 773.

[65] See, for example, UNCLOS, part XII, section 5, arts. 207-212, 204-206, 213-222, and 235.

[66] Article 36, Killing by Machine.