Collateral By Algorithm: When The Model Makes The Mistake

The US military’s first large programme to apply machine learning to targeting data was Project Maven, formally the Algorithmic Warfare Cross-Functional Team, created by a memorandum from Deputy Secretary of Defense Robert Work on 26 April 2017. Its stated job was to flag objects, classify threats and highlight anomalies in full-motion drone video so that human analysts could work faster.

The programme became public in March 2018 when reporting revealed Google engineers were helping build the computer-vision models. More than three thousand employees signed a petition against the work, and in June 2018 Google said it would not renew the contract. Palantir later developed the Maven Smart System, and responsibility for the programme was divided between the National Geospatial-Intelligence Agency and the Chief Digital and AI Office in 2022.

Department of Defense Directive 3000.09, reissued on 25 January 2023 and replacing the 2012 version, governs autonomy in weapon systems. It requires that such systems be designed so commanders and operators can exercise appropriate levels of human judgment over the use of force, that they pass senior review, and that they undergo verification, validation and realistic testing. It also calls for technologies and data that are transparent, auditable and explainable.

The directive does not prohibit autonomous weapons, and it does not require a human to approve every engagement. It sets a review and testing standard. The department’s August 2022 Civilian Harm Mitigation and Response Action Plan adds a separate set of commitments to assess and reduce civilian harm.

The most contested public allegations come from April 2024 reporting by +972 Magazine and Local Call. They described a system called Lavender that reportedly flagged about 37,000 Palestinians as suspected militants, and another called Gospel that recommended buildings as targets, with sources claiming human review was sometimes brief.

The Israel Defense Forces rejected that account. It said it does not use an artificial intelligence system that identifies terrorist operatives or predicts whether a person is a terrorist, and described the tool as a database for cross-referencing intelligence, with information systems acting as tools for analysts. United Nations human rights experts also raised concern in April 2024 about the reported use of AI in Gaza. The claims rest largely on anonymous sources and the IDF’s denial, and no independent audit of the systems or their error rates has been published.

On 22 December 2023 the UN General Assembly adopted resolution 78/241 on lethal autonomous weapons systems by 152 votes to 4, with 11 abstentions. It acknowledged serious challenges and concerns raised by new technologies in the military domain, including artificial intelligence and autonomy in weapons. The resolution is not binding and does not create a treaty, and the International Committee of the Red Cross has separately called for limits on autonomous weapon systems.

The record establishes that algorithmic tools support military targeting, that US policy sets testing and human-judgment requirements, and that serious allegations about their use exist. It does not establish how much any specific system influenced any specific strike, or that civilian harm followed from an algorithm instead of from human decisions, intelligence errors or the conduct of the conflict.