A 39,000 Subexposures/s CMOS Image Sensor with Dual-tap Coded-exposure Data-memory Pixel for Adaptive Single-shot Computational Imaging

Rahul Gulve*, Navid Sarhangnejad*, Gairik Dutta, Motasem Sakr, Don Nguyen, Roberto Rangel, Wenzheng Chen, Zhengfan Xia, Mian Wei, Nikita Gusev, Esther Y. H.Lin, Xiaonong Sun, Leo Hanxu, Nikola Katic, Ameer Abdelhadi, Andreas Moshovos, Kiriakos N. Kutulakos, Roman Genov

PRESENTATION(.pptx)       PAPER(.pdf)    VIDEO(.mp4)

Abstract

A dual-tap coded-exposure-pixel (CEP) image sensor is presented and demonstrated in several computational imaging applications. The data-memory pixel (DMP) yields 39,000 subexposures/s at 320×320 sensor resolution with 7 µm pitch.

The outputs of a frame-rate 12-bit ADC1 and a subexposure-rate 1-bit ADC2 are adaptively combined to boost the native dynamic range of coded-imaging modalities by over 57dB, demonstrating over 101dB dynamic range in intensity imaging. The CEP camera combined with machine learnt projection patterns enables single-shot structured-light 3D imaging at native resolution and video rate.

Fast Coded-Exposure

Application1: Scene-Adaptive HDR

This shows methodology and experimental results for subframe-rate scene-adaptive single-shot HDR imaging.

  • The two pictures on the left are captured by conventional camera with high and and low exposure settings.
  • The CEP image sensor captures the scene
  • and generates 1-bit output for all the pixels at the end of the first subframe
  • This output is fed back to the sensor as masks for the next subframe.
  • In the following subframe,
    some of the pixels have crossed the reference voltage and must stop integrating light any further to avoid saturation.
    So the corresponding masks are updated on the sensor.
  • We see more and more bright pixels start crossing the reference voltage during the frame.
  • At the end, ADC1 digitizes the raw output from the sensor
  • The effective per-pixel exposure time is calculated from ADC2 output.
  • The combination of ADC1 and ADC2 is then used to determine the intensity of each pixel.
  • The tone mapped HDR scene is shown in the bottom right corner

Adaptive masking and HDR image

Application 2: Single-Shot Structured Light Imaging

we demonstrate is single-shot structured light imaging with optimal projection patterns.

The goal is to acquire disparity map of the scene using only a single frame – in order to reduce motion artifacts We program our sensor to have a 4 Bayer-like mosaic pattern exposure codes in 4 subframes. The projector is synchronized with the camera and projects four optimal illumination patterns found using stochastic gradient discent. The resulting recorded two images, one for each tap … are used to reconstruct disparity.

To find the optimal projection patterns,

  • We start by projecting random patterns.
  • Then we introduce small variations these patterns and calculate mean error with respect to ground truth.
  • By the method of stochastic gradient descent, we minimize the error and obtain the optimal set of patterns
  • These patterns are then projected in synchronization with the image sensor subframes.
  • The bayer like exposure-codes encode the information from all different illuminations in single frame.
  • The fast sub-exposures and projections allow us to obtain disparity map of the scene with reduced motion blur and without extra power penalty.
More details can be found here: https://www.dgp.toronto.edu/optimalsl/