Surprise Machines for Harvard Art Museums


by Lins Derry, Douglas Duhaime, Jordan Kruguer, Christopher Pietsch, Dario Rodighiero, and Jeffrey Schnapp

Surprise Machines is a visual investigation that will take the form of a digital installation at the Harvard Art Museums (HAM) in Spring 2022. The project sets out to visualize and curate the entire universe of the museums’ collections, with the aim of opening up unexpected vistas on the more than 200,000 objects that make them up. To accomplish these surprise encounters, algorithms are curatorially employed to shape the visualizations, and a "choreographic interface" has been designed to connect the audience's movement with several unique views.


Can machines think? In 1950 Alan Turing famously answered this question in the affirmative by means of a so-called “imitation game” in the course of which an examiner is asked to distinguish between humans and machines while communicating via a typewriter. He argued that, once the responses appear indistinguishable, the machine can correctly be understood as engaged in thought. In an article entitled “Computing Machines and Intelligence,” Turing approached the subject of artificial intelligence from multiple perspectives, one of them inspired by the English mathematician Ada Lovelace. Here Lovelace figures as Turing’s foil, arguing that machines are incapable of thought because they are incapable of “tak[ing] us by surprise.” Turing counters by stating that machines are a frequent source of surprise, behaving unpredictably and, thereby, generating surprises.

Surprise Machines reprises Turing’s experiment some seven decades later by means of an AI-based curatorial experiment that relies upon “black box” algorithms whose behaviors, once set in motion, cannot be predicted by their programmers. It sets out to visualize and curate the entire universe of Harvard Art Museums, with the aim of opening up unexpected vistas on the more than 200,000 objects that make them up.

Surprise Machines is a visual investigation that takes the form of a digital installation for Curatorial A(i)gents, an exhibition that will take place at the Harvard Art Museums in Spring 2022. The goal of this exhibition is to present a series of machine-learning experiments curated by members of metaLAB (at) Harvard, a creative research group working in the networked arts and humanities.

From a technical point of view, Surprise Machines relies on the museums’ API, which provides access to images and metadata through the International Image Interoperability Framework (IIIF). Making use of Yale University DHLAB’s Pix Plot, the images of the digital collection are arranged in the Cartesian plane using UMAP, an algorithm that reduces the multi-dimensionality of pixel colors and positions on the flat surface of screens.

The data visualization is successively contextualized in the Lightbox Gallery, a high-tech exhibition area resulting from the recent Renzo Piano’s restoration. The area hosts a wall of conjoined high-resolution screens and, for the occasion, a camera pointed at the space in front of them. A machine-learning system connects the camera with the visitors, whose body becomes a sort of ‘choreographic interface’ for interacting with the collection. This sophisticated system designed by metaLAB allows the visitors to move in the visualization through precise, choreographed gestures: while raising and lowering an arm enables zooming in both directions, a hand-to-the-hip movement goes for selecting a specific image, and a circle above the head resets the images’ arrangement at the initial state.

Surprise Machines sets out to visualize the universe of the Harvard Art Museums' digital collection, opening up unexpected vistas on the objects that make them up.