The Imagine team is a work in progress. Please visit us later.

Plan-view choreography notation from video recordings of theatre rehearsals.

Responsable : Remi Ronfard

Contact : ronfard at inria dot fr

Description : The goal of the thesis is to use video recordings as a means to automatically notate the choreography of a theatre performance or rehearsal in a "plan-view" representation with actor trajectories. Such representation is useful for comparing the evolution between successive versions during rehearsals, and as an archive of the final performance.

Using a top-view camera is not a practical option in this application because of the heavy lighting equipment used on a theatre stage. Instead, this thesis will investigate direct, offline methods for recovering a top view of all actors using high-resolution, wide angle video recordings of the stage from a few "central" positions in the audience. To do this, we will extend methods that have been proposed recently on the related problem of recovering "storyboards" of movie scenes (1,2). Such methods rely on manual initialization and labeling of actor's names. In this thesis, we expect to perform automatic initialization and naming of actors.

The thesis will use recently developed upper body detectors, which give precise but incomplete information about the actor's position on screen. The internship will be principally devoted to proposing original approaches for "online learning" of the actor's appearances, to make the detection faster and complete; and on "higher level" reasoning over long sequences for resolving ambiguous cases where actors approach or hide each other; enter or exit the stage; change appearances; or turn away from the audience. We will explore dynamic programming approaches for computing the "most likely" sequence of such events (5). Ideally, we would like to work from a single viewpoint. But for evaluation purposes, we will use a maximum of three viewpoints. Synchronization will be performed offline using audio channels from the three cameras. For evaluating methods based on binocular or trinocular stereo (3,4), the student will be given camera calibration matrices.

The internship is expected to lead to a PhD thesis on the topic of automatic annotation and online web publishing of theatre performances and rehearsals (conditional on funding). Based on the reconstructed choregraphy notation (plan view), we plan to investigate methods to automatically compare the recorded rehearsals and performances; to align them with the director's score; to reframe the recorded video with virtual "pan and scan" camera movements; and to dynamically generate summaries of the rehearsal process and the final performance.

Bibliography

  1. Dony, Mateer, Robinson, Day: Iconic versus naturalistic motion cues in automated reverse storyboarding. CVMP 2005.
  2. Goldman, Curless, Salesin and Seitz. "Schematic Storyboarding for Video Visualization and Editing." ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2006), Vol. 25, No. 3, pp. 862-871, July 2006.
  3. Darrell, Demirdjian, Checka, Felzenszwalb. Plan-view trajectory estimation with dense stereo background models. ICCV, 2001.
  4. Ivanov, Y., Bobick, A., and Liu, J. 1998. Fast Lighting Independent Background Subtraction. In Proceedings of the 1998 IEEE Workshop on Visual Surveillance.
  5. Nillius, P., Sullivan, J., and Carlsson, S. 2006. Multi-Target Tracking - Linking Identities using Bayesian Network Inference. CVPR, 2006.