Choreography, Design, and Technology: An Interview with Lins Derry from the metaLAB (at) Harvard

Dario Rodighiero: It is my pleasure to introduce Lins Derry, a colleague and friend since 2019. Lins brings an extraordinary blend of expertise to metaLAB (at) Harvard, merging the fields of human-computer interaction and choreography in innovative ways. Her unique trajectory from a professional dancer to a principal researcher at metaLAB showcases a fascinating journey of interdisciplinary exploration and achievement. At metaLAB, Lins is pioneering research on choreographic interfaces and the embodiment of data, seeking ways to make scientific information more accessible and engaging to the public. Her works, such as Data Sensorium, exemplify her commitment to translating complex datasets into movement scores, thereby opening new avenues for interaction and interpretation of data. This intersection of dance, technology, and design not only highlights the body as a site of investigation but also redefines the boundaries of how we engage with and understand information in the digital age. Lins, your diverse career path, encompassing professional dance and your innovative work in design and technology, is incredibly inspiring. Given our collaboration, I’ve had the privilege of witnessing first-hand the depth of your creativity and the impact of your research. Could you share more about your journey in dance, which seems to have influenced your career so far?

Lins Derry: Thank you, Dario! I began dancing at three and spent over a decade dancing professionally with companies in Montreal, New York, and San Francisco. I also ran my own platform, Linsdans, for around five years. Through Linsdans, I started to adopt a more design-oriented approach, taking on various roles including video, sound, and set design, and even graphic design for promotional materials. Transitioning to design toward the end of my dance career felt natural. Some highlights of my career include touring across North America and Europe, and occasionally in Asia. Performing and teaching in often non-tourist locations allowed for a deeper connection to the places and communities I visited. Being part of residencies and festivals also meant engaging with others at a more embodied level, which is a different experience from typical travel.

Dario: That’s a truly beautiful journey you’ve shared, Lins. Your transition from the world of professional dance to the realms of design and technology is fascinating and inspiring. As someone listening to your story, it strikes me that attending Harvard University, particularly the Graduate School of Design, represents a pivotal, perhaps game-changing moment in your narrative. Can you walk us through what inspired you to make such a significant transition? How did the idea to pursue studies at the Graduate School of Design at Harvard come about, and what were you hoping to achieve or explore through this academic endeavor?

Lins: It’s a bit of a funny story. My husband was at Google in Silicon Valley for an event where MIT Media Lab students were presenting work. He reflected, “I think you would thrive at the Media Lab. You seem to have a similar creative flair that intersects art and design. Maybe you should apply.” So, I prepared an application to a group at the MIT Media Lab, only to discover they weren’t accepting new applicants that year. Then, somewhat serendipitously, I looked “across the street” in Cambridge, metaphorically speaking through the internet, and found the Harvard Graduate School of Design (GSD). By then, I was already captivated by the concept of design. Upon entering the GSD, I was uncertain about the specific direction I wanted to take but found myself drawn to technology, both as a tool and subject for creation and critique. This interest led me to enroll in various art, design, and technical courses. Eventually, I met Sidney Skybetter, the founder of the Conference for Research on Choreographic Interfaces at Brown University. Discussing the idea of a choreographic interface with him was a pivotal moment. I realized that what I had been doing all along was integrating my background in choreography and performance into designing new spaces and media, and I wanted to delve deeper into that. Now, I identify as a choreographic designer, applying my lifelong choreographic sensibility to explore how we can design kinesthetic interactions with computer systems or engage with abstract content like data through the body. This has led to two areas that continue to excite me since graduating.

Dario: That’s great. I believe that represented a significant shift, but it was also a very stimulating change for you, especially moving from your background into a context of design and technology. You’ve started mentioning individuals who have guided you over these two years of study. And I understand that at a certain point, you encountered Jeffrey Schnapp, a leading figure in the fields of digital humanities and design. He was your teacher at the GSD and founder of the metaLAB (at) Harvard. How did you meet Jeffrey?

Lins: I met Jeffrey the first week I arrived at Harvard; he was kind enough to have coffee with me. I reached out to him after discovering that metaLAB is a GSD affiliate lab. I sensed I might find an exceptional community there and wanted to learn more about its initiatives. I like to say I began my membership as a fly on the wall. I simply attended the staff meetings and listened openly to what was happening. Later, after I graduated and gained more capacity to become involved, the opportunity was there to finally jump in, which was during the pandemic.

Dario: As we piece together your journey, we observe the blend of dance, design, and technology, all converging within the interdisciplinary community of the metaLAB. This unique milieu, which brings together scholars, artists, and more, seems to have served as a fertile ground for your master’s thesis. Did these elements mix up in your thesis work? Could you share more about the focus of your thesis at the university and how these diverse areas were integrated?

Lins: The requirements at the GSD are constantly evolving, but while I was there, I chose the option not to pursue a formal thesis, only coursework. However, I ended up conducting what amounted to an unofficial thesis, which began with a semester of independent study guided by Jeffrey Schnapp on the topic of choreographic interfaces. Then after graduating, an opportunity to experiment further with the idea came about. metaLAB was planning an exhibition at the Harvard Art Museums titled Curatorial A(i)gents. Slated to open in the wake of COVID-19, it was going to need a “touchless” solution for visitors to interact with the show’s screen-based projects. Jeffrey suggested it as an ideal chance to explore movement with computers, which is how the choreographic interface prototype for Curatorial A(i)gents came into being.

Dario: Yeah, I’m quite familiar with the context, as it coincided with our collaboration. However, I’d love to hear from you about the choreographic interface project created for the Curatorial A(i)gents exhibition. Can you highlight the concepts or aspects that most captivated you?

Lins: Curatorial A(i)gents was a group exhibition curated by metaLAB and held in the Harvard Art Museums’ experimental space called the Lightbox Gallery. The prompt for all contributing creatives was to employ AI, or machine learning, in connection to the digital image collection. This was to generate new perspectives on the collections and to creatively and critically question the role AI plays in curatorial practice. There were 11 works and about half of them would require desktop-style interactions like point and click. What captivated me about accomplishing those interactions using choreography was how the digital collections could be bridged to the physical museums through the body as medium.

Dario: And this leads us to how you integrated the choreographic interface with this image collection during the pandemic. Jeffrey offered you the opportunity to develop the interface for all the exhibition’s projects, collaborating also with me on Surprise Machines. I imagine that your dedication brought you into various roles. Would you mind sharing the range of roles you undertook during Curatorial A(i)gents?

Lins: Many hats, indeed. I actually project-managed the exhibition as well as led a team on the choreographic interface, all remotely from California. The interface team included Jeffrey Schnapp, Sidney Skybetter, and yourself as advisors; Jordan Kruguer as developer; Maximilian Mueller as composer (he sonified all the interactions); and myself as choreographic designer. I also contributed to the exhibition a data visualization project that analyzed the gender and sentiment of the Harvard Art Museum’s portrait collection (Second Look: Gender and Sentiment on Show). So, I took on three different roles during the show! There were significant challenges while working remotely during the pandemic. The choreographic interface team, for instance, couldn’t develop directly on the hardware in the Lightbox Gallery, leading to difficulties when trying to make the interface operate as smoothly and consistently as on our personal laptops. Nonetheless, the show came together in the spring of 2022.

Dario: And listen, I’m familiar with many details of this project, but what might be most intriguing, especially from your perspective, is the body vocabulary you developed within it (Fig. 1). I’m curious about how you developed and finalized the project’s postures and gestures, and how you began interacting with the machine. Could you describe the design process between your body and the machine?

Lins: That was truly the thrill of the research—developing this vocabulary. And that’s where the choreographic could really shine. To clarify, the project’s simple idea was to connect a movement vocabulary with conventional interactions like zooming in and out, scrolling up and down, and tracking and selecting. This typical control system was chosen to enable interaction across all the Curatorial A(i)gents projects that required it. Regarding the choreography of the vocabulary, we began with accessibility, conducting workshops with Judith Smith, a remarkable dancer who uses a wheelchair. We started by examining how pose estimation models respond to a dancer in a wheelchair. From there, we tested several vocabularies, creating classification systems for them using a couple of open-source pose estimation models. This helped us to understand which “poses” would have the least interference with each other and which models would work fastest. A particular hurdle was differentiating when the system should recognize gestures for commands like scroll versus when it should track the hand to move the cursor and select something. Though seemingly simple, this ended up being a significant obstacle we had to overcome.

Dario: And I wasn’t aware that you needed to clearly distinguish between poses to avoid overlapping positions. Moreover, knowing the project quite well, I was particularly fascinated by the concept of seeing oneself on the screen with a skeleton overlay. Could you describe your experience interacting with the interface in the video? It would be great to hear how you engaged with your own image on the monitor.

Lins: We opted to maintain, in some ways, what felt like a raw feedback system through OpenCV, then fine-tuned it for greater visual clarity. This approach was almost instructional, reinforcing the correct positions for users to trigger the desired interactions, such as zoom out. To further support a kind of dance literacy, we color-coded the skeletal overlay and added text labels that changed when the gestures were activated. So when the user spread their arms wide to zoom out, they’d see the color change and the text read zoom out, making the interaction exciting and reinforcing. This helped users connect with the computation behind the screens too; they got to see how AI was interpreting their movements and how those movements affected the projects being displayed on the huge monitor wall.

Dario: I have to say I’m a big fan of the refresh pose, which I find quite elegant—the one described by a circle formed above your head with your arms (Fig. 2). As you were describing the interaction, I couldn’t help but wonder about your personal reaction to interacting with the system versus observing others engage with it, especially considering that much of the development took place during the pandemic. Some metaLAB researchers, including Jeff Steward from the Harvard Art Museums, were working with the interface on-site. And then, at a certain point, you were at the museum testing the system. I’m curious about your feelings regarding the distance between experiencing the interface yourself and watching others interact with what you’ve created. Was it strange, or was it satisfying?

Lins: Absolutely satisfying! Coming from a dance background, I sometimes noticed that when people tried the circle for refresh, which was chosen because circling has this notion of washing the screen clean, their hands would be too low, forming a diamond instead. Though it looked imperfect to my trained eye, it was helpful to see because I discovered how others would interpret the vocabulary. Another example is when we were trying to decide on the selection process, people would do things like open and close their hands, leading us to revisit our decision to connect index finger to thumb for select. In these ways, seeing untrained bodies perform the movements in their own ways was wonderful, as it added a deeply human and intuitive interpretation to the gestures that often inspired iteration.

Dario: This reconnects with the idea that the most beautiful aspect of architecture is ultimately the people within it. Similarly, in design, you can create a very nice object, but it’s the people using it who truly bring out its interest—the usage and even the inconveniences, when objects are supposed to function one way end up being used differently. And, touching on another related point, especially considering our collaboration on the Surprise Machines project during the pandemic. Chris Pietsch was in Germany, Douglas Duhaime in Connecticut, Jeffrey in Vermont, you were in California, I was in Italy, and other metaLAB colleagues in Massachusetts—testing the projects entirely over Zoom must have been a bit strange. What were your impressions of that time?

Lins: It’s remarkable we actually managed to pull it off, considering we had such limited time to be physically together. This was especially true for Jordan, Max, and me, who handled the bulk of the choreographic interface development virtually (Fig. 3). Not being able to be in the actual space for which the system was being customized posed significant challenges. We simply didn’t have the bandwidth or time for the necessary debugging to overcome the hardware limitations of the space. I’d say the biggest lesson learned is that if you’re designing a new technology for a specific space, it’s crucial to be there often during the development phase. On the flip side, the pandemic collapsed geographic distances, leading us to form an international team for Surprise Machines, thanks to your leadership!

Dario: Moving on, I understand that beyond the choreographic interface, you’re also developing another project called Data Sensorium. This marks a bit of a shift, but I’m curious—are these two projects connected, and if so, how?

Lins: Yes. I began Data Sensorium with the desire to choreograph a dataset. The project uses a dataset from the Internal Displacement Monitoring Centre in Switzerland. It focuses on migratory bodies, which I felt would parallel nicely to dancing bodies, and examines how environmental disasters displace people within the borders of their countries/territories—a complex issue to track since these displacements don’t involve crossing borders where passport controls are in place. The bulk of the work involved developing a movement score that allowed dancer Isabelle Edgar to essentially perform the dataset row by row. Connecting back to the choreographic interface, Jordan Kruguer and I adapted the code to recognize the movement vocabulary associated with the disaster types like storm, volcano, and earthquake. The interface provides a similar visual feedback to the version discussed at the Harvard Art Museums (Fig. 4).

Dario: From a practical perspective, it’s fascinating that a gesture can trigger sounds, computer behaviors, and many other reactions. And going back for a moment to Curatorial A(i)gents, I think we forgot to mention that Maximilian Mueller did a fantastic job with the choreographic interface by adding a sonification layer on top of your movements. How was it working with sound?

Lins: I’m so happy you mentioned Max’s sonification. It was really cool: he designed multiple sound textures that could be selected then sonically manipulated by the gestures. The aural feedback really helped to open up the physical space. Without it, the experience would have felt flat, confined to feedback seen on the screen. It really brought a heightened awareness to the gallery as the sound filled the space. To me, that was one of its greatest contributions—a reinvigoration of the embodied experience.

Dario: What kind of feedback were you triggering with the movements in this project? Was it video, sound, or a mix of both?

Lins: At the moment, there’s a digital dashboard containing the movement score as text, two data visualizations, and the choreographic interface that all update as Isabelle moves through the dance. That’s the extent of the feedback system for now, but to your question, introducing some sound or video projection that’s less nerdy and more aesthetic would be an amazing next step. For instance, when the storm gesture is performed, there could be a thunderous sound in a sonification score.

Dario: If I understand correctly, the sensing aspect of these projects is confined to the camera’s presence. And we know we can create beautiful things with it. So, I’d like to ask if you could expand a bit on capturing through devices. What has the experience of using the camera been like? What are the limitations? And are there other methodologies or capture devices you’re interested in exploring?

Lins: We chose the webcam because it was simple to use and wouldn’t require any body contact, which was crucial for maintaining social distancing. This is what ultimately led us into the realm of gestural interfaces. I’m curious about experimenting with other motion capture devices, especially low-fidelity ones where their latencies could provide more space for creating interesting feedback loops. This would be less for a strict control system like we had for Curatorial A(i)gents and more for something artistic and expressive.

Dario: What strikes me about your work with technology is essentially a kind of gesture classification. You take a pose, and then that pose is recognized, classified, and essentially frozen by the computer. Could you discuss this aspect of working with static gestures versus the dynamism of dance? What do you feel is missing in terms of movement, and what possibilities do you see for integrating more dynamic movements and gestures with these systems?

Lins: I’m not sure if classification would be the proper approach for a system that truly involves motion because it would be so computationally heavy. Rather than assess one frame at a time for a gesture class, the system would have to compute a string of frames and their relationships to each other to make a similar assessment. Latency was already a major issue when running the choreographic interface and Surprise Machines together. Seems like using motion capture without any classification would work best for this. My main struggle with that option is that while there are many motion capture systems that do phenomenally well at tracking the body in realtime and producing immediate feedback, they rarely provide any direct perspective into the actual structure of the choreography. It’s this perspective that’s really important to me. Although limited, classification systems help connect choreographic thinking to computational systems by establishing a common ground or vocabulary.

Dario: Yeah, of course, it’s you speaking to technology, and your gesture changes based on what the machine gives back to you. It makes sense. Have you ever considered this connectivity and interactivity, with all these dependencies between humans and nonhumans? I spent some time working as a designer with Bruno Latour on AIME, his ERC Consolidator Grant (Latour 2013). His legacy often led me to view projects in terms of their complexity, considering all the contributing elements—from each individual involved to the smallest technologies used—and I was thinking about whether you’ve contemplated the complexity of all the project’s elements, from people to technology. Have you ever thought about the complexity of all the elements contributing to the project together?

Lins: Not entirely familiar with Bruno’s work, but I’ll respond by saying it’s been through the writing process that I’ve come to think more broadly about those connections. For instance, tracking a body with a webcam naturally connects to ideas of surveillance and being watched. Another topic or question that has come up while writing about Data Sensorium is who am I to represent all these people’s migratory journeys, especially in light of not knowing how their data was necessarily collected in the first place?

Dario: I think it’s beautiful to have this moment of writing in which, actually, you’re not moving, but you are expressing yourself through a different medium. I was just curious to know if you developed these arguments in the papers you publish about your projects.

Lins: I have! Perhaps a nice moment to highlight the following papers on the projects we discussed: “Data Embodiment: Approaching the Body as a Choreographic Medium for Performing Abstract Data” (Derry 2023), “Surprise Machines: Revealing Harvard Art Museums’ Image Collection” (Rodighiero et al. 2022), and “Designing a Choreographic Interface during COVID-19” (Derry et al. 2022).

Dario: Thank you for sharing your journey and insights with us, Lins. Your work is a vivid example of how interdisciplinary approaches can enrich our understanding and interaction with choreography, design, and technology.

Lins: Thank you, Dario, for guiding the conversation. Having the opportunity to reflect on the choreographic interface in relation to Surprise Machines, one of the Curatorial A(i)gents projects that set out to visualize the entire Harvard Art Museums’ image collection, and Data Sensorium made me realize a few things. That my work generally involves shaping movement as information for computer systems to understand or moving information, whether through interfacing with data visualizations or dancing data. Excited to see how these approaches might converge in future collaborations together, while continuing to locate new experiments in physical spaces like museums!

References

• Derry, Lins. 2023. “Data Embodiment: Approaching the Body as a Choreographic Medium for Performing Abstract Data.” International Journal of Performance Arts and Digital Media 19 (1): 60–82. doi:10.1080/14794713.2023.2175105.
• Derry, Lins, Jordan Kruguer, Maximilian C. Mueller, and Jeffrey Schnapp. 2022. “Designing a Choreographic Interface during COVID-19.” In Proceedings of the 8th International Conference on Movement and Computing (MOCO ’22). New York: Association for Computing Machinery. doi:10.1145/3537972.3538020.
• Latour, Bruno. 2013. An Inquiry into Modes of Existence: An Anthropology of the Moderns. Cambridge, MA: Harvard University Press.
• Rodighiero, Dario, Lins Derry, Douglas Duhaime, Jordan Kruguer, Maximilian C. Mueller, Christopher Pietsch, Jeffrey Schnapp, and Jeff Steward. 2022. “Surprise Machines: Revealing Harvard Art Museums’ Image Collection.” Information Design Journal 27 (1): 21–34. doi:10.1075/idj.22013.rod.