Shadow Removal & Occlusion Detection in Front Projection Environments

This project involved prior contributions by M.Eng. students Daniel Sud and Nadia Hilario, and was brought to its present form by the M.Eng. research of Samuel Audet.

Overview

Multi-projector displays are often exploited for installations where large display surfaces are desired.
A seminal example is the CAVE (Cave Automatic Virtual Environment), which uses back projection with mirrors.
However, this approach requires dedicated space and limits adoption.

Front projection (e.g. in the Office of the Future) avoids that limitation but creates shadows when a person stands between projector and display.
Existing shadow removal methods either cannot handle dynamic video playback at 60 fps or require special hardware.

Our method uses cameras and projectors geometrically calibrated to the floor and display surfaces. Cameras operate in pairs with stereo correspondence algorithms, producing depth information that can be reprojected to detect occlusions. Alternatively, tracking data can be used. Knowing where people stand, our algorithm identifies occluded projector regions and directs another projector to fill them in with matching color and intensity.

We tested the method using a stereo camera from Tyzx and four projectors on one wall. Calibration was accurate and shadow removal worked correctly and efficiently.
Limitations: only one user can be handled, and thin objects (e.g. arms) cause flickering due to stereo algorithm sensitivity.

Figure 2. Rendered model of the room with cameras, projectors, and a person. Floor is dark gray, wall black, chosen display surface white. Colored lines denote camera/projector coverage.

• Audet, S. and Cooperstock, J.R. (2007).
  Shadow Removal in Front Projection Environments using Object Tracking.
  IEEE Projector-Camera Systems (ProCams), June 18, Minneapolis.

• Shadow removal with dynamic video playback
• Conference video for ProCams 2007
• Technical details

Overview

Front projection technology enables large immersive displays but suffers from occlusion shadows.
Our goal: develop a camera-projector system that detects occlusions dynamically using offline calibration and online view synthesis.

Occluded regions are detected by comparing predicted and captured camera images.
Applications include Active Virtual Rear Projection (AVRP) for shadow removal, projector light suppression, or augmenting the occluder itself.

Demonstration: dynamic shadow detection and removal using a dually overlapped projector display.

Occlusion detection has been studied for applications such as:

• Shadow removal
• Occluder light suppression
• Hand detection and tracking

Methods are divided into:

• Direct detection (locating the occluding object)
• Indirect detection (locating its shadow)

More details: related work.

Our algorithm involves two stages:

Step 1: Offline Camera-Projector Calibration

• Geometric Calibration: projector-to-camera warping homographies estimated for alignment and keystone correction.
• Color Calibration: projector-to-camera color transfer matrices estimated for more accurate synthesis.

Step 2: Online Occlusion Detection

• Camera view synthesis predicts unoccluded display appearance.
• Pixel-wise comparison with actual camera feed reveals occlusions.
• Shadows are compensated by overlapping projectors.

Improvements: variable thresholding for dark regions and morphological smoothing for noise reduction.

• Intensity blending at shadow edges using ramp functions to reduce visual noise.
• Improved inter-projector color calibration for seamlessness.

• Hilario, M.N. (2005). Occlusion Detection in Front Projection Environments Based on Camera-Projector Calibration.
  Master’s thesis, Electrical and Computer Engineering Department, McGill University.

• Hilario, M.N. and Cooperstock, J.R. (2004). Occlusion Detection for Front-Projected Interactive Displays.
  Pervasive Computing, Vienna, April 21–23. Appears in Advances in Pervasive Computing, Austrian Computer Society (OCG), ISBN 3-85403-176-9.

• Last update: 17 July 2008
• Last update: 22 June 2005