November 19, 2012

I. Unmanned Robots and the Evolution toward Fully Autonomous Weapons

Robots are not new to the battlefield, but their expanding role encroaches upon traditional human responsibilities more than ever before. Most visibly, the use of US Predator, Reaper, and other drones in Afghanistan and elsewhere has provided an early sign of the distancing of human soldiers from their targets. Often piloted from halfway around the globe, these robotic aerial vehicles provide surveillance and identify targets before a human decides to pull the trigger, commanding the drone to deliver lethal force.

In keeping with the escalating use of aerial drones, government planning documents and spending figures indicate that the military of the future will be increasingly unmanned. In recent years, for example, the US Department of Defense has spent approximately $6 billion annually on the research and development, procurement, operations, and maintenance of unmanned systems for war, and that figure is likely to increase rapidly.[6] Drones are seen as just the beginning of a technological revolution. As robotic warfare expert Peter W. Singer suggests, “Predators are merely the first generation—the equivalent of the Model T Ford or the Wright Brothers’ Flyer.”[7]

Unmanned technology possesses at least some level of autonomy, which refers to the ability of a machine to operate without human supervision.[8] At lower levels, autonomy can consist simply of the ability to return to base in case of a malfunction. If a weapon were fully autonomous, it would “identify targets and … trigger itself.”[9] Today’s robotic weapons still have a human being in the decision-making loop, requiring human intervention before the weapons take any lethal action. The aerial drones currently in operation, for instance, depend on a person to make the final decision whether to fire on a target. As this chapter illustrates, however, the autonomy of weapons that have been deployed or are under development is growing quickly. If this trend continues, humans could start to fade out of the decision-making loop, retaining a limited oversight role—or perhaps no role at all.

Plans for Autonomy

Military policy documents, especially from the United States, reflect clear plans to increase the autonomy of weapons systems. In its Unmanned Systems Integrated Roadmap FY2011-2036, the US Department of Defense wrote that it “envisions unmanned systems seamlessly operating with manned systems while gradually reducing the degree of human control and decision making required for the unmanned portion of the force structure.”[10] The US plans cover developments in ground, air, and underwater systems. A US roadmap specifically for ground systems stated, “There is an ongoing push to increase UGV [unmanned ground vehicle] autonomy, with a current goal of ‘supervised autonomy,’ but with an ultimate goal of full autonomy.”[11] According to the US Air Force, “[i]ncreasingly humans will no longer be ‘in the loop’ but rather ‘on the loop’—monitoring the execution of certain decisions. Simultaneously, advances in AI will enable systems to make combat decisions and act within legal and policy constraints without necessarily requiring human input.”[12] A 2004 US Navy planning document on unmanned undersea vehicles (UUVs) stated, “While admittedly futuristic in vision, one can conceive of scenarios where UUVs sense, track, identify, target, and destroy an enemy—all autonomously.”[13] Other countries are also devoting attention and money to unmanned systems. [14]

While emphasizing the desirability of increased autonomy, many of these military documents also stress that human supervision over the use of deadly force will remain, at least in the immediate future. According to the US Department of Defense, “[f]or the foreseeable future, decisions over the use of force and the choice of which individual targets to engage with lethal force will be retained under human control in unmanned systems.”[15] The UK Ministry of Defence stated in 2011 that it “currently has no intention to develop systems that operate without human intervention in the weapon command and control chain.”[16] Such statements are laudable but do not preclude a change in that policy as the capacity for autonomy evolves.

Although the timeline for that evolution is debated, some military experts argue that the technology for fully autonomous weapons could be achieved within decades. The US Air Force predicted that “by 2030 machine capabilities will have increased to the point that humans will have become the weakest component in a wide array of systems and processes.”[17] The UK Ministry of Defence estimated in 2011 that artificial intelligence “as opposed to complex and clever automated systems” could be achieved in 5 to 15 years and that fully autonomous swarms could be available in 2025.[18] Other experts have quoted similar estimates while cautioning that intelligence for weapons that equals that of a human is much further off, and many experts believe it is impossible.[19]

The next two sections examine the development of increasingly autonomous weapons. They are not the focus of this report, which instead highlights the risks posed by fully autonomous weapons. They show, however, that autonomous technology already exists and is evolving rapidly. An analysis of these weapons also leads to the conclusion that development of greater autonomy should proceed cautiously, if at all.

Automatic Weapons Defense Systems

Automatic weapons defense systems represent one step on the road to autonomy. These systems are designed to sense an incoming munition, such as a missile or rocket, and to respond automatically to neutralize the threat. Human involvement, when it exists at all, is limited to accepting or overriding the computer’s plan of action in a matter of seconds.

The US Navy’s MK 15 Phalanx Close-In Weapons System is designed to identify and fire at incoming missiles or threatening aircraft. This automatic weapons defense system, shown during a live fire exercise, is one step on the road to full autonomy. Photograph by Chief Fire Controlman Brian Kirkwood, US Navy.

The United States has several such systems. The US Navy’s MK 15 Phalanx Close-In Weapons System, the earliest model, was first installed on a ship in 1980, and modified versions are still widely used by the United States and its allies.[20] It is designed to sense approaching anti-ship missiles or threatening aircraft and respond with fire from two 20mm guns with six rotating barrels.[21] The guns each fire 3,000 to 4,500 rounds per minute.[22] More recent models aim to defend against small gunboats, artillery, and helicopters.[23] The Navy describes the Phalanx as “the only deployed close-in weapon system capable of autonomously performing its own search, detect, evaluation, track, engage and kill assessment functions.”[24]

The Counter Rocket, Artillery, and Mortar System (C-RAM) is a US land-based version of the Phalanx. The United States first deployed it at forward operating bases in Iraq in 2005. Twenty-two systems reportedly had more than 100 successful intercepts of rockets, artillery, and mortars[25] and provided more than 2,000 warnings to troops.[26] Like the Phalanx, it can fire 20mm rounds from a six-barrel gun at an incoming munition.[27] According to one US Army publication, after the C-RAM detects a threat, “a human operator certif[ies] the target,”[28] but that would have to happen almost instantaneously in order for the C-RAM to destroy the incoming munition in time.

Other countries have developed comparable weapons defense systems. Israel has deployed its Iron Dome near the border with Gaza and in Eilat, near the Sinai Peninsula.[29] It uses radar to identify short-range rockets and 155mm artillery shells up to 70 kilometers away.[30] It is armed with 20 Tamir interceptor missiles to respond to such threats, and extended range versions of those missiles are reportedly scheduled to be available in early 2013.[31] Israel has received financial support for the Iron Dome from the United States, and the US Department of Defense stated that the system, which reportedly has more than an 80 percent success rate, “has shot down scores of missiles that would have killed Israeli civilians since it was fielded in April 2011.”[32] In a split second after detecting an incoming threat, the Iron Dome sends a recommended response to the threat to an operator. The operator must decide immediately whether or not to give the command to fire in order for the Iron Dome to be effective.[33]

An Iron Dome, an Israeli automatic weapons defense system, fires a missile from the city of Ashdod in response to a rocket launch from the nearby Gaza Strip on March 11, 2012.  The Iron Dome sends warnings of incoming threats to an operator who must decide almost instantly whether to give the command to fire. © 2012 Jack Guez, AFP/Getty Images

Another example of an automatic weapons defense system is the NBS Mantis, which Germany designed to protect its forward operating bases in Afghanistan. The “short-range force protection system will detect, track and shoot the projectiles within a close range of the target base.” Within 4.5 seconds after detecting targets about three kilometers away, it can fire six 35mm automatic guns at 1,000 rounds per minute.[34] The system has a “very high degree of automation, including automatic target detection and engagement processes which the operator only has to monitor.”[35] Sources were unclear whether “monitoring” also allowed the operator to override the process.

These weapon defense systems have a significant degree of autonomy because they can sense and attack targets with minimal human input. Technically, they fall short of being fully autonomous and can better be classified as automatic. Robotics professor Noel Sharkey defines an automatic robot as one that “carries out a pre-programmed sequence of operations or moves in a structured environment. A good example is a robot arm painting a car.” An autonomous robot, he continues, “is similar to an automatic machine except that it operates in open and unstructured environments. The robot is still controlled by a program but now receives information from its sensors that enable it to adjust the speed and direction of its motors (and actuators) as specified by the program.”[36] Nevertheless, while not the focus of this report, these automatic defense systems can be seen as a step toward greater autonomy in weapons.

As weapons that operate with limited intervention from humans, automatic weapons defense systems warrant further study. On the one hand, they seem to present less danger to civilians because they are stationary and defensive weapons that are designed to destroy munitions, not launch offensive attacks.[37] On the other hand, commentators have questioned the effectiveness of the human supervision in the C-RAM and other automatic weapons defense systems. Writing about the C-RAM, Singer notes, “The human is certainly part of the decision making but mainly in the initial programming of the robot. During the actual operation of the machine, the operator really only exercises veto power, and a decision to override a robot’s decision must be made in only half a second, with few willing to challenge what they view as the better judgment of the machine.”[38] When faced with such a situation, people often experience “automation bias,” which is “the tendency to trust an automated system, in spite of evidence that the system is unreliable, or wrong in a particular case.”[39] In addition, automatic weapons defense systems have the potential to endanger civilians when used in populated areas. For example, even the successful destruction of an incoming threat can produce shrapnel that causes civilian casualties.[40] Thus these systems raise concerns about the protection of civilians that full autonomy would only magnify.

Other Precursors to Fully Autonomous Weapons

Other unmanned weapons systems that currently retain humans in or on the loop are also potential precursors to fully autonomous weapons. Militaries have already deployed ground robots, and air models are under development.[41] If their potential for full autonomy in the use of lethal force were realized, these systems would pose a greater threat to civilians than automatic weapons defense systems. As currently designed, the systems discussed below would all have the capability to target humans. In addition, the increased mobility and offensive nature of the air systems in particular would give them more range and make them harder to control than weapons like the Phalanx.

South Korea and Israel have developed and started to use sentry robots that operate on the ground. South Korea installed SGR-1s, costing $200,000 each, along the Demilitarized Zone (DMZ) for testing in 2010.[42] These stationary robots can sense people in the DMZ with heat and motion sensors and send warnings back to a command center.

The South Korean SGR-1 sentry robot, a precursor to a fully autonomous weapon, can detect people in the Demilitarized Zone and, if a human grants the command, fire its weapons. The robot is shown here during a test with a surrendering enemy soldier. © 2007 Kim Dong-Joo/AFP/Getty Images

From there, human soldiers can communicate with the individual identified and decide whether to fire the robot sentry’s 5.5mm machine gun or 40mm automatic grenade launcher.[43] The SGR-1’s sensors can detect people two miles away during the day and one mile away at night. Its guns can hit targets two miles away.[44] At present, the sentry has autonomous surveillance capabilities, but it cannot fire without a human command.[45] The journal of the Institute of Electrical and Electronics Engineers reported, however, “[T]he robot does have an automatic mode, in which it can make the decision.”[46]

The Israel Defense Forces (IDF) has deployed Sentry Tech systems along Israel’s 60 kilometer border with Gaza. The sentry detects movement and sends signals to a facility “at a distant location.”[47] Soldiers then evaluate the data and decide whether to fire at the target. The Sentry Tech currently has a 12.7mm or a .50 caliber machine gun with a kill zone of about 1 to 1.5 kilometers.[48] To increase its range to several kilometers, the IDF is considering adding anti-armor missiles.[49] Sentry Tech is reportedly designed to defend against people trying to cross the border as well as sniper and rocket attacks.[50] In 2007, Jane’s Defence Weekly described Sentry Tech as “revolutionary” because it could not only detect threats but also engage them.[51] While the system is currently operated by remote control, an IDF division commander told Defense News: “[A]t least in the initial phases of deployment, we’re going to have to keep the man in the loop.”[52] The commander thus implied that human involvement may not always be the case.

Israel has also deployed the Guardium, “a semi-autonomous unmanned ground system,” which is reportedly used for patrolling Israel’s border with Gaza. It can carry lethal or non-lethal payloads. According to the manufacturer G-NIUS’s brochure, “[t]he Guardium UGV™ was designed to perform routine missions, such as programmed patrols along border routes, but also to autonomously react to unscheduled events, in line with a set of guidelines specifically programmed for the site characteristics and security doctrine.”[53] While the brochure implies there is some level of human oversight because it refers to stationary, mobile, and portable control terminals, it also notes that the Guardium can have “autonomous mission execution.”[54]

Unmanned aircraft are moving beyond existing drones to have greater autonomy. The US Navy has commissioned the X-47B, which will be able to take off from and land on an aircraft carrier and refuel on its own power.[55] It was tested multiple times in 2012 and is scheduled in 2013 to do a trial landing on a carrier, “one of aviation’s most difficult maneuvers.”[56] Although as a prototype it will not carry weapons, it has reportedly been designed for eventual “combat purposes,”[57] and it has two weapons bays with a total payload capacity of 4,500 pounds.[58] Humans remain on the loop for the time being, but their role in the flight of the X-47B is limited. Northrop Grumman described it as a system that “takes off, flies a preprogrammed mission, and then returns to base in response to mouse clicks from its mission operator. The mission operator monitors the X-47B air vehicle’s operation, but does not actively ‘fly’ it via remote control as is the case for other unmanned systems currently in operation.”[59] The Los Angeles Times called it “a paradigm shift in warfare, one that is likely to have far-reaching consequences. With the drone’s ability to be flown autonomously by onboard computers, it could usher in an era when death and destruction can be dealt by machines operating semi-independently.”[60]

The US Navy’s X-47B, currently undergoing testing, has been commissioned to fly with greater autonomy than existing drones. While the prototype will not carry weapons, it has two weapons bays that could make later models serve a combat function. Photograph by DARPA.

The United Kingdom unveiled a prototype of its Taranis combat aircraft in 2010. Designers described it as “an autonomous and stealthy unmanned aircraft” that aims to strike “targets with real precision at long range, even in another continent.”[61] Because Taranis is only a prototype, it is not armed, but it includes two weapons bays and could eventually carry bombs or missiles.[62] Similar to existing drones, Taranis would presumably be designed to launch attacks against persons as well as materiel. It would also be able to defend itself from enemy aircraft.[63] At this point, the Taranis is expected to retain a human in the loop. The UK Ministry of Defence stated, “Should such systems enter into service, they will at all times be under the control of highly trained military crews on the ground.”[64] Asked if the Taranis would one day choose its own targets, Royal Air Force Air Chief Marshal Simon Bryant responded, “This is a very sensitive area we are paying a lot of attention to.”[65] He thus left the door open to the possibility of greater autonomy in the future.[66]

The United Kingdom’s Taranis combat aircraft, whose prototype was unveiled in 2010, is designed to strike distant targets, “even in another continent.”  While the Ministry of Defence has stated that humans will remain in the loop, the Taranis exemplifies the move toward increased autonomy. © 2010 Associated Press

Other countries have also developed or procured unmanned aircraft that are precursors to fully autonomous weapons. The Israeli Harpy, for example, has been described as a combination of an unmanned aerial vehicle and a cruise missile. It is designed to fly “autonomously to the patrol area.” Once there, it seeks to detect hostile radar signals and then destroy a target with a high explosive warhead.[67]

The US military’s SWARMS technology would also involve autonomous aircraft, but in this case, many such aircraft would navigate in a synchronized way with a human controller directing them as a group “swarm” rather than individually.[68] While initially designed to gather intelligence,[69] SWARMS could one day undertake offensive operations. For example, their numbers, designed to be a force multiplier, could overwhelm an air defense system.[70] At least at this point, designers envision that SWARMS would have a human on the loop. Tests done in August 2012 showed that a single “operator on the ground, using only a laptop and a military radio, can command an unmanned aerial vehicle (UAV) ‘swarm.’”[71] The ability of a single operator to have effective oversight of dozens or even hundreds of aircraft seems implausible to many experts.[72] As a result, a swarm could operate as a de facto out-of-the-loop weapon.

Because humans still retain control over the decisions to use lethal force, the above weapons systems are not, at least yet, fully autonomous, but they are moving rapidly in that direction. Human oversight is minimal, especially in the case of SWARMS. At the same time, technology is developing that allows weapons to identify targets and travel to and around a battle zone on their own power. Proponents tout military advantages, such as a reduction in the number of human troops required for military operations, the availability of sentries not influenced by laziness or fear, and faster response time.[73] Technological developments combined with these advantages of autonomy create incentives for states to develop weapons with greater autonomy.

Critics have two major concerns, however. First, they question the effectiveness of the existing limited human oversight.[74] Second, they worry that the next step will be to grant these systems control over launching attacks. Speaking of Taranis, for example, Sharkey, a computer scientist and vocal critic of fully autonomous weapons, said, “But warning bells ring for me when they talk about Taranis being ‘a fully autonomous intelligent system’ together with applications in ‘deep missions’ and having a ‘deep target attack’ capability…. We need to know if this means the robot planes will choose their own targets and destroy them—because they certainly will not have the intelligence to discriminate between civilians and combatants.”[75] Control systems specialist Nick Jennings did not object to SWARMS technology as a surveillance tool, but he warned, “We don’t want UAVs selecting targets and working out how best to carry out an attack.”[76] Full autonomy would give weapons this power to decide when to fire.

Given that some believe that full autonomy could become a reality within 20 or 30 years, it is essential to consider the implications of the technology as soon as possible. Both supporters and skeptics have agreed on this point.[77] The UK Ministry of Defence wrote, “[I]f we wish to allow systems to make independent decisions without human intervention, some considerable work will be required to show how such systems will operate legally.”[78] Philip Alston, when serving as the UN special rapporteur on extrajudicial, summary or arbitrary executions, warned that “[u]rgent consideration needs to be given to the legal, ethical and moral implications of the development and use of robot technologies, especially but not limited to uses for warfare.”[79] The rest of this report will explore these implications, particularly as they relate to the protection of civilians during times of armed conflict.

[6] US Department of Defense, “Unmanned Systems Integrated Roadmap FY2011-2036,” Reference Number 11-S-3613, 2011, http://www.defenseinnovationmarketplace.mil/resources/UnmannedSystemsIntegratedRoadmapFY2011.pdf (accessed September 26, 2012), p. 13.

[7] Statement of Peter W. Singer, director, 21st Century Defense Initiative, Brookings Institution, UnManned Systems and Robotic Warfare: Hearing before the Subcommittee on National Security and Foreign Affairs of the House Committee on Oversight and Governmental Reform, March 23, 2010, 112th Congress (2010), http://www.brookings.edu/testimony/2010/0323_unmanned_systems_singer.aspx (accessed September 30, 2012).

[8] Krishnan, Killer Robots, p. 4.

[9] Ibid., p. 5.

[10] US Department of Defense, “Unmanned Systems Integrated Roadmap FY2011-2036,” p. 3.

[11] Robotic Systems Joint Project Office, “Unmanned Ground Systems Roadmap,” July 2011, http://contracting.tacom.army.mil/future_buys/FY11/UGS%20Roadmap_Jul11.pdf (accessed September 30, 2012), p. 39.

[12] US Department of the Air Force, “Unmanned Aircraft Systems Flight Plan 2009-2047,” May 18, 2009, http://www.govexec.com/pdfs/072309kp1.pdf (accessed September 30, 2012), p. 41.

[13] US Department of the Navy, “The Navy Unmanned Undersea Vehicle (UUV) Master Plan,” November 9, 2004, www.navy.mil/navydata/technology/uuvmp.pdf (accessed September 30, 2012), p. xvii.

[14] A Congressional Research Service report states, “Thus, some would argue that much new business is likely to be generated in the UAS [unmanned aerial systems] market, and if U.S. companies fail to capture this market share, European, Russian, Israeli, Chinese, or South African companies will.” See Jeremiah Gertler, “U.S. Unmanned Aerial Systems,” Congressional Research Service report, January 3, 2012, http://www.fas.org/sgp/crs/natsec/R42136.pdf (accessed September 30, 2012), p.28. A UK Ministry of Defence report noted, “For high end systems, it is likely that apart from the US, even major western countries will need to collaborate on UAS [unmanned aircraft systems] development.” See UK Ministry of Defence, The UK Approach to Unmanned Aircraft Systems (Shrivenham: UK Ministry of Defence, 2011), http://www.mod.uk/NR/rdonlyres/F9335CB2-73FC-4761-A428-DB7DF4BEC02C/0/20110505JDN_211_UAS_v2U.pdf (accessed September 30, 2012), p. 4-1. Two weapons reportedly in development are the Russian MiG Skat and the Chinese Invisible Sword. See Vladimir Karnozov, “MiG and Sukhoi to Join Forces on Russian UCAV,” Flight, August 11, 2011, http://www.flightglobal.com/news/articles/mig-and-sukhoi-to-join-forces-on-russian-ucav-360562/ (accessed September 30, 2012); email communication from Noel Sharkey, professor of artificial intelligence and robotics, University of Sheffield, to Human Rights Watch, September 4, 2012.

[15] US Department of Defense, “Unmanned Systems Integrated Roadmap FY2011-2036,” p. 17.

[16] UK Ministry of Defence, The UK Approach to Unmanned Aircraft Systems, p. 5-4.

[17] US Air Force Chief Scientist, “Report on Technology Horizons: A Vision for Air Force Science & Technology during 2010-2030,” May 15, 2010, http://www.af.mil/shared/media/document/AFD-100727-053.pdf (accessed September 30, 2012), p. 106. See also Robotic Systems Joint Project Office, “Unmanned Ground Systems Roadmap,” p.38. This roadmap predicts “full-autonomy packages” for payloads by 2020.

[18] UK Ministry of Defence, The UK Approach to Unmanned Aircraft Systems, pp. 5-4, 6-8. While the UK government seemed to be referencing robots with full autonomy, Sharkey noted that this estimate as expressed is misleading because the field of artificial intelligence developed decades ago. Human Rights Watch telephone interview with Noel Sharkey, September 6, 2012. See also Noel Sharkey, “Automating Warfare: Lessons Learned from the Drones,” Journal of Law, Information & Science (2011), http://www.jlisjournal.org/abstracts/sharkey.21.2.html, p. EAP 2.

[19] See, e.g., Krishnan, Killer Robots, p. 48. See also P.W. Singer, “Robots at War: The New Battlefield,” Wilson Quarterly, Winter 2009; Helen Briggs, “Machines ‘to Match Man by 2029,’” BBC News, February 16, 2008, http://news.bbc.co.uk/2/hi/americas/7248875.stm (accessed September 30, 2012); Human Rights Watch telephone interview with Noel Sharkey, September 6, 2012.

[20] Federation of American Scientists, “MK 15 Phalanx Close-In Weapons System (CIWS),” January 9, 2003, http://www.fas.org/man/dod-101/sys/ship/weaps/mk-15.htm (accessed September 30, 2012); “MK 15–Phalanx Close-In Weapons System (CIWS),” US Navy fact sheet, http://www.navy.mil/navydata/fact_print.asp?cid=2100&tid=487&ct=2&page=1 (accessed October 30, 2012). US allies using the Phalanx include Australia, Canada, Israel, Japan, and the United Kingdom. Raytheon reports that 890 Phalanxes have been built and are used in the navies of 25 countries. “Raytheon Awarded $57.8 Million Phalanx Contract,” Raytheon press release, May 18, 2012, http://raytheon.mediaroom.com/index.php?s=43&item=2101 (accessed September 30, 2012). There are also alternatives to the Phalanx platform, including the Thales Nederland Goalkeeper system. See “Phalanx CIWS: The Last Defense, On Ship and Ashore,” Defense Industry Daily press release, December 28, 2011,

http://www.defenseindustrydaily.com/phalanx-ciws-the-last-defense-on-ship-and-ashore-02620/ (accessed September 30, 2012). The Phalanx was involved in the shooting down of an Iranian airliner in the 1980s. For more information on that incident, see footnote 37 below.

[21] Federation of American Scientists, “MK 15 Phalanx Close-In Weapons System (CIWS).”

[22] Ibid.

[23] “Phalanx CIWS: The Last Defense, On Ship and Ashore,” Defense Industry Daily press release.

[24] “MK 15–Phalanx Close-In Weapons Systems (CIWS),” US Navy fact sheet.

[25] “Land-Based Phalanx Weapon System Completes Mission in Iraq,” Naval Sea Systems Command (NAVSEA) newswire, February 16, 2012, http://www.navsea.navy.mil/Lists/NewsWires/DispForm.aspx?ID=12 (accessed September 30, 2012).

[26] “Northrop Wins US Army’s C-RAM Contract,” January 31, 2012, http://www.army-technology.com/news/newsnorthrop-wins-us-armys-c-ram-contract (accessed September 30, 2012).

[27] “C-RAM Transforms Defense Tactics,” US Army news release, April 25, 2012, http://www.army.mil/article/78724/C_RAM_transforms_defense_tactics/ (accessed September 30, 2012).

[28] Ibid.

[29] Yaakov Katz, “Air Force to Get Two New Iron Dome Batteries,” Jerusalem Post, July 29, 2012, http://www.jpost.com/Defense/Article.aspx?id=279256 (accessed September 30, 2012).

[30] Rafael Advanced Defense Systems Ltd., “Iron Dome: Defense against Short Range Artillery Rockets,” 2010, http://www.rafael.co.il/Marketing/186-1530-en/Marketing.aspx (accessed September 30, 2012).

[31] Katz, “Air Force to Get Two New Iron Dome Batteries,” Jerusalem Post.

[32] Jim Garamone, “Iron Dome System Demonstrates U.S.-Israeli Partnership,” American Forces Press Service, August 1, 2012, http://www.defense.gov/news/newsarticle.aspx?id=117354 (accessed September 30, 2012).

[33] “Iron Dome Battle Management Demonstrated,” Defense Update, 2009, http://www.defense-update.com/photos/iron_dome_bms.html (accessed October 30, 2012).

[34] “NBS MANTIS Air Defence Protection System, Germany,” army-technology.com,

http://www.army-technology.com/projects/mantis/ (accessed September 30, 2012).

[35] “Germany Orders MANTIS C-RAM Base Defense Systems,” Defense Industry Daily,  January 17, 2011,

http://www.defenseindustrydaily.com/Germany-Orders-Skyshield-C-RAM-Base-Defense-Systems-05418/#deutschland-nbs (accessed September 30, 2012).

[36] Sharkey, “Automating Warfare,” p. EAP 2. Armin Krishnan, author of a comprehensive book entitled Killer Robots, uses a different breakdown but also determines this category of weapons falls short of being fully autonomous. He describes the Phalanx as an example of “pre-programmed autonomy,” which allows for limited independence in decision making. In other words, it “carries out a particular function by following instructions that have been inserted into the machine by a designer or user. Normally, a pre-programmed machine is computer-controlled and it does its work with very little variation. This means that such machines have no or little capacity to vary from the original instructions or from pre-programmed movements.” Krishnan, Killer Robots, pp. 43-44.

[37] The Phalanx, part of the Aegis System, was involved in a notable case of civilian casualties. On July 3, 1988, the USS Vincennes, which used the Aegis System, shot down an Iranian passenger airliner, killing 290 civilians. There was significant debate about the cause of the incident. A US Department of Defense report found that “combat induced stress on personnel may have played a significant role in this incident” and recommended study into “stress factors impacting on personnel in modern warships with highly sophisticated command, control, communications and intelligence systems, such as AEGIS.” It also, however, called for further investigation into certain design features of the Aegis system. Whatever the cause of the incident, it shows that great care should be employed in the use of automatic weapons defense systems. See US Department of Defense, “Formal Investigation into the Circumstances Surrounding the Downing of Iran Air Flight 655 on 3 July 1988,” July 28, 1988, http://homepage.ntlworld.com/jksonc/docs/ir655-dod-report.html (accessed September 30, 2012); see also Chris Hables Gray, “The Destruction of Iran Air Flight 655 by the USS Vincennes,” International Affairs, January 17, 2011, http://suite101.com/article/the-destruction-of-iran-air-flight-655-by-the-uss-vincennes-a333613 (accessed September 30, 2012).

[38] P.W. Singer, “War of the Machines: A Dramatic Growth in the Military Use of Robots Brings Evolution in Their Conception,”Scientific American, July 2010, p. 63.

[39] Peter M. Asaro, “Modeling the Moral User,” IEEE Technology and Society Magazine, Spring 2009, http://peterasaro.org/writing/Asaro%20Modeling%20Moral%20User.pdf (accessed September 30, 2012), p. 22.

[40] GlobalSecurity.org, “Counter Rocket, Artillery, and Mortar (C-RAM),”

http://www.globalsecurity.org/military/systems/ground/cram.htm (accessed September 30, 2012).

[41] The US Navy is also developing autonomous technology. The Proteus, for example, is a 25-foot-long underwater vehicle that can operate unmanned or manned. It is being designed to have capabilities for delivering weapons or laying sea mines. “Bluefin, Battelle, and the Columbia Group Investing to Propel Proteus Vehicle into the Seas,” Bluefin Robotics press release, February 21, 2012, http://www.bluefinrobotics.com/news-and-downloads/press/bluefin-battelle-and-the-columbia-group-investing-to-propel-proteus-vehicle-into-the-seas/ (accessed September 30, 2012); Jeff Smith (Bluefin Robotics) and Ross Lindman (The Columbia Group), “Proteus–Large Diameter Undersea Vehicle and its Applications for Undersea Warfare,” symposium presentation, May 9, 2012, http://www.10thsymposium.com/presentations/Wed%20pm%20B/1530-1600%20Smith%20Bluefin%20-%20Proteus%20-%20MINWARA%20-%20%20050912.ppt%20%5BRead-Only%5D.pdf (accessed September 30, 2012).

[42] Jon Rabiroff, “Machine Gun-Toting Robots Deployed on DMZ,” Stars and Stripes, July 12, 2010, http://www.stripes.com/news/pacific/korea/machine-gun-toting-robots-deployed-on-dmz-1.110809 (accessed September 30, 2012).

[43] “South Korea’s Military Technologies: Defensive Robots and Urine Powered Batteries,” Korea IT Times, July 14, 2010, http://www.koreaittimes.com/story/9750/south-koreas-military-technologies-defensive-robots-and-urine-powered-batteries (accessed September 30, 2012); Kim Deok-hyun, “Army Tests Machine-Gun Sentry Robots in DMZ,” Yonhap News Agency, July 13, 2010, http://english.yonhapnews.co.kr/national/2010/07/13/14/0301000000AEN20100713007800315F.HTML# (accessed September 30, 2012); Tim Hornyak, “Korean Machine-Gun Robots Start DMZ Duty,” CNET, July 14, 2010, http://news.cnet.com/8301-17938_105-20010533-1.html (accessed September 30, 2012).

[44] Rabiroff, “Machine Gun-Toting Robots Deployed on DMZ,” Stars and Stripes.

[45] Ibid.

[46] Jean Kumagai, “A Robotic Sentry for Korea’s Demilitarized Zone,” IEEE Spectrum, March 2007, http://spectrum.ieee.org/robotics/military-robots/a-robotic-sentry-for-koreas-demilitarized-zone (accessed September 30, 2012). According to Sharkey, the South Korean government originally made statements about the “automatic mode” consistent with this article, but it later changed its public position and removed the statements from its websites. Email communication from Noel Sharkey, September 4, 2012.

[47] Rafael Advanced Defense Systems Ltd., “Sentry Tech: Long Distance Stationary Remote Controlled Weapon Station,” http://www.rafael.co.il/marketing/SIP_STORAGE/FILES/4/1104.pdf (accessed September 30, 2012), p. 1.

[48] “Israel Deploying ‘See-Shoot’ RWS along Gaza,” Defense Industry Daily, June 7, 2007, http://www.defenseindustrydaily.com/israel-deploying-seeshoot-rws-along-gaza-03354/ (accessed September 30, 2012).

[49] Ibid.; Barbara Opall-Rome, “Israel Wants Robotic Guns, Missiles to Guard Gaza Border,” DefenseNews.com, June 1, 2007, http://www.kibush.co.il/show_file.asp?num=20516 (accessed October 30, 2012).

[50] “Israel Deploying ‘See-Shoot’ RWS along Gaza,” Defense Industry Daily.

[51] Robin Hughes, “IDF Deploys Sentry Tech on Gaza Border,” Jane’s Defence Weekly, June 6, 2007, p.1.

[52] Barbara Opall-Rome, “Israel Wants Robotic Guns, Missiles to Guard Gaza Border,” DefenseNews.com.

[53] G-NIUS Unmanned Ground Systems, “Guardium UGV,” http://g-nius.co.il/pdf/brochures/GuardiumUGV.pdf (accessed October 4, 2012), p.2; Yaakov Katz, “IDF Seeking Expanded Use of Robotic Systems,” Jerusalem Post, July 27, 2012, http://www.jpost.com/Defense/Article.aspx?id=279110 (accessed October 4, 2012). Katz also reported that the IDF has been seeking money for a new unmanned ground vehicle for “long-range reconnaissance missions” called the Advance Guard.

[54] G-NIUS Unmanned Ground Systems, “Guardium UGV,” p.2.

[55] Northrop Grumman, “Unmanned Combat Air Systems Carrier Demonstration (UCAS-D),” July 28, 2011, http://www.as.northropgrumman.com/products/nucasx47b/assets/X-47B_Navy_UCAS_FactSheet.pdf (accessed October 4, 2012), p. 1.

[56] W.J. Hennigan, “New Drone Has No Pilot Anywhere, so Who's Accountable?” Los Angeles Times, January 26, 2012, http://www.latimes.com/business/la-fi-auto-drone-20120126,0,740306.story (accessed October 4, 2012); “Northrop Grumman, U.S. Navy Conduct First East Coast Flight of X-47B Autonomous Unmanned Aircraft,” Northrop Grumman news release, July 30, 2012, http://www.irconnect.com/noc/press/pages/news_releases.html?d=10000044 (accessed October 4, 2012).

[57] “The X-47B Autonomous Drone,” BBC World Today (BBC World Service), January 31, 2012 (quoting US Representative Henry Cuellar, co-chair of US congressional Unmanned Systems Caucus) (transcript provided by Noel Sharkey).

[58] Northrop Grumman, “X-47B UCAS Unmanned Combat Air System Demonstrator,” 2012, http://www.as.northropgrumman.com/products/nucasx47b/assets/UCAS-D_Data_Sheet.pdf (accessed October 4, 2012), p. 1.

[59] Northrop Grumman, “Unmanned Combat Air Systems Carrier Demonstration (UCAS-D),” p. 2.

[60] Hennigan, “New Drone Has No Pilot Anywhere, so Who's Accountable?” Los Angeles Times.

[61] BAE Systems, “Taranis,” http://www.baesystems.com/product/BAES_020273/taranis (accessed October 4, 2012).

[62] “Taranis: The £143 Million Unmanned Stealth Jet that Will Hit Targets in Another Continent,” Daily Mail, July 13, 2010,

http://www.dailymail.co.uk/sciencetech/article-1294037/Taranis-The-143million-unmanned-stealth-jet-hit-targets-continent.html (accessed October 4, 2012); “Taranis UCAV Demonstrator Vehicle Rollout,” Defense Update, http://defense-update.com/features/2010/july/taranis_12072010.html (accessed October 4, 2012).

[63] “Taranis Unmanned Combat Air Vehicle (UCAV) Demonstrator, United Kingdom,”

http://www.airforce-technology.com/projects/tanaris/ (accessed October 4, 2012).

[64] “New Taranis Combat Aircraft Thunders into View,” UK Ministry of Defence news release, July 12, 2010, http://www.mod.uk/DefenceInternet/DefenceNews/EquipmentAndLogistics/NewTaranisCombatAircraftThundersIntoView.htm (accessed October 4, 2012).

[65] Paul Marks, “Warning Sounded over British Dogfighting Drone,” New Scientist, July 12, 2010, http://www.newscientist.com/article/dn19162-warning-sounded-over-british-dogfighting-drone.html (accessed October 4, 2012).

[66] Another autonomous UK aircraft developed by the same company is the Mantis. First tested in October 2009, it is designed to fly autonomously, even during bad weather, so that “ground station operators [can] focus on the overall task, rather than vehicle control.” While primarily intended for surveillance and military transport, it would reportedly be able to “identify and destroy targets using its missiles or bombs”; it can carry air-to-air missiles or precision guided bombs weighing 226 kg. The United Kingdom pulled funding before “armed variants” were tested, although the developer reports that it plans to continue work on the aircraft, which it describes as “intelligent … autonomous … unique.” See “Mantis MALE Unmanned Aerial Vehicle (UAV), United Kingdom,” Airforce-Technology.com, http://www.airforce-technology.com/projects/mantis-uav/ (accessed October 4, 2012); BAE Systems, “MANTIS,” http://www.baesystems.com/product/BAES_019760?_afrLoop=716320862883000&_afrWindowMode=0&_afrWindowId=null&baeSessionId=xK5TQJhZQFpJ2LY1p18g22z5YKRxNhJ1B9dzGSbnGjG8nTpjqpJn!92979655#%40%3F_afrWindowId%3Dnull%26baeSessionId%3DxK5TQJhZQFpJ2LY1p18g22z5YKRxNhJ1B9dzGSbnGjG8nTpjqpJn%252192979655%26_afrLoop%3D716320862883000%26_afrWindowMode%3D0%26_adf.ctrl-state%3D153ns2xf3k_4 (accessed October 4, 2012).

[67] See “Harpy Unmanned Aerial Vehicle,” SinoDefence.com, February 1, 2009, http://www.sinodefence.com/airforce/uav/harpy.asp (accessed October 4, 2012). Sharkey has critiqued the Harpy because it cannot distinguish between an anti-aircraft defense system and a radar placed on a school by an enemy force. Human Rights Watch telephone interview with Noel Sharkey, September 6, 2012. The Israeli Harop is similar to the Harpy but can be remotely controlled when in flight. India, Germany, and Turkey have reportedly purchased variants of this weapon. “Harop Loitering Munitions UCAV System, Israel,” Airforce-Technology.com, http://www.airforce-technology.com/projects/haroploiteringmuniti/(accessed October 4, 2012).

[68] “SWARMS Project: Swarming Drones to Sting the Enemy?” Defense Industry Daily, June 24, 2009, http://www.defenseindustrydaily.com/Swarming-Drones-Will-Be-Able-to-Sting-Enemy-05540/ (accessed October 4, 2012). SWARMS stands for Scalable sWarms of Autonomous Robots and Mobile Sensors. University of Pennsylvania, “SWARMS,” http://www.swarms.org/ (accessed October 4, 2012).

[69] “Boeing Team Demonstrates Expanded Control of Unmanned Aircraft Swarm,” Boeing news release, August 6, 2012, http://boeing.mediaroom.com/index.php?s=43&item=2379 (accessed October 4, 2012).

[70] Skyler Frink, “UAV Swarm Technology Emerges to Perform Varied Pperations,” The Aerospace & Defense Blog, August 13, 2010, http://www.militaryaerospace.com/blogs/aerospace-defense-blog/2012/08/uav-swarm-technology-emerges-to-perform-varied-applications.html (accessed October 4, 2012). See also Singer, Wired for War, pp. 232-34.

[71] “Boeing Team Demonstrates Expanded Control of Unmanned Aircraft Swarm,” Boeing news release.

[72] For more information on experts’ concerns with this technology, see “Robot Wars,” The Engineer, June 6, 2011.

[73] A Samsung Techwin spokesman said the South Korean robot sentry is designed to operate better than humans: “[T]hese robots have automatic surveillance, which doesn’t leave room for anything resembling human laziness. They also won’t have any fear [of] enemy attackers on the front lines.” Rabiroff, “Machine Gun-Toting Robots Deployed on DMZ,” Stars and Stripes. The IDF values its sentry robot because it reduces troops deployed, decreases response time, and increases intelligence gathering capabilities. Hughes, “IDF Deploys Sentry Tech on Gaza Border,” Jane’s Defence Weekly, p. 1.

[74] Human Rights Watch telephone interview with Noel Sharkey, September 6, 2012.

[75] Marks, “Warning Sounded over British Dogfighting Drone,” New Scientist. According to Sharkey, “‘deep mission’ is military speak for ‘beyond the reach of a remote pilot.’” Ibid.

[76] “Robot Wars,” The Engineer.

[77] There have been several books and articles starting to address this issue in recent years. See, e.g., Gary E. Marchant et al., “International Governance of Autonomous Military Robots,” The Columbia Science and Technology Law Review, vol. 12, 2011, http://www.stlr.org/cite.cgi?volume=12&article=7 (accessed October 30, 2012), pp. 272-315; Lin, Abney, and Bekey, eds., Robot Ethics;Krishnan, Killer Robots.

[78] UK Ministry of Defence, The UK Approach to Unmanned Aircraft Systems, p. 5-2.

[79] Interim report of the Special Rapporteur on extrajudicial, summary or arbitrary executions, Philip Alston, p. 21.