Assignment 4: HDR and Tone mapping from Multiple Images

Michael Tao


(a) Reinhard

(b) Bilateral

(C) My Tone Mapping

Figure: Tone mapping usually requires sacrifices on either realism or detail. In Reinhard's tone mapping algorithm, we can see that details are severely lost (a). In Bilateral tone mapping, edges are well preserves but the algorithm causes contrasty edges and unrealistic colors (b). I've derived a new tone mapping algorithm that preserves detail and maintains color (c).
(please click on images for full images)

DATA SET

note: All JPEG files have my camera's EXIF information.

Overview

I implemented a full HDR MATLAB pipeline that includes these elements:

Prerequisites:
- image downsampling
- image sampling
- bilateral filter (3D and 4D) in .c and converted to mex
- Fattal's HDR color compression
- response curve output for each camera

HDR: (HDR formation and implemented 3 tone mapping algorithms)
- complete and automatic EXIF file reading (any jpeg from cameras will do)
- response curve (output and input)
- radiance map recovery
- pixel sampling for radiance map (for efficiency)
- Tone mapping: Reinhard's tone-mapping
- Tone mapping: Bilateral filter
- Tone mapping: My own algorithm (full details below)

Relevant Equipment:
- Canon 5D Mark II

My tone-mapping algorithm provides the follow advantages:
1. reduces amount of parameter changes
2. produce very good results
3. fast and simple
4. maintains very good color

Also note all the photographs are taken by me except for the memorial sequence, which is generously provided by Paul Debevec ( http://ict.debevec.org/~debevec/Research/HDR/ ).

Radiance Map Recovery

figure: Over-exposed image



figure:
Average-exposed image

figure: Under-exposed image

 

Practical Implementations

In my system, the user can add jpeg images into the input folder and everything would work out of the box. The system automatically reads EXIF file information to extract aperture and exposure time. The information will later be used for exposure calculations. Moreover, I made an automatic downscaling (which can be turned off).

Sampling

Sampling is required before using the gsolve.m code that Paul provided. The reason for this is that with a fullsized image, the amount of RAM required is enormous for the matrix. Moreover, since the matrix is heavily overdetermined if all pixels are used, we can just simply sample. The amount of sampling is provided in the paper, which is number of pixels multiplied by exposures must be greater than the maximum difference from high to low (which in this case is 255). I uniformly sampled throughout the image.

Response Curve

With the sampled points, we can then use gsolve to get our response curve. Since each camera has one unique response curve, I computed the response function for one set of images and never computed it again. My system can save this information and display your camera response curve. Since I used my Canon 5D Mark II for all the images, here is the response curve:

figure: Canon 5D Mark II Response Curve

Radiance Map

By using Paul Debevec's formulation, I sum and normalize all of the pixel's mapped responses. I also subtracted the log(exposure time) to compensate for the exposure. Using the weight function derived for the response curve (which is simply equation (4) of Paul Debevec's paper- which you can see in acknowledgements below). The purpose of the weight function is to compensate for steep slopes near Z_min and Z_max. With the summation of all the images to map on the radiance map, I also used normalization and capping to make sure no values are 0 (which will be problematic in nomalization). I then exponent the result of the normalization. As an output, we have our radiance map shown here:

figure: Radiance Map
(note- might not be accurate colors because I used imshow- which scales output).

We then can feed the radiance map to tone mapping algorithms.


Tone Mapping : Reinhard's Algorithm

By using Reinhard's tone mapping algorithm, we first compute the luminance map. The luminance map is then taken into the log scale (taken account of the exposure scale we have used before). We use this luminance and adjust it with the log_summation of luminance. This is used to normalize the output of the luminance map from zero to one. I will refer this map as the scaled luminance map.

We then saturate the radiance map by simply dividing each of the radiance map by the luminance map. The author's intention is to normalize is color channel by its luminance. We then scale the saturated radiance by its scaled luminance map values. This is then clamped by 0 and 1- so no values go over or under the allowed range.

figure: Doe Library Result (please click for full image)

figure: Botanical Gardens Result (please click for full image)


Just by following the algorithm, we can see that the luminance map is distorted as part of the output. Unfortunately, in LUV space, L contains the most information about the high frequency data. As a result, the output becomes soft. But due to the normalization method of the radiance maps, we can also see that this algorithm preserves colors very well, which is favorable.


figure: Doe Library : details


figure: Soft Edges by Reinhard's Tone Map


figure: Botanical Gardens : details


figure: Soft Edges by Reinhard's Tone Map

On the result above, we can see that edge information is lost. We also see that regions of high brightness (such as the light), loses much of the high frequency data.

Summary:

Advantages:
+ colors look very natural - right out of the box!
+ fast

Disadvantages:
- lose details!
-- lose lots of high frequency data in bright areas


Tone Mapping : Bilateral Filter

In order to preserve details, Fredo and Dorsey introduced using bilateral filters for tone mapping. As a summary of what I have implemented, I first calculated the log intensity map (which is similar to have I have done for Reinhard's). I then use the result convolved it with a bilateral filter. I implemented the bilateral filter using the full 3-D space. Due to the speed issues with MATLAB, I programmed the bilateral filter using C and converted it into mex. To get the details map, we use the bilateral filted map subtracted by the log scaled intensity map. With this, I then use a gamma correction. The gamme correction includes scaling the intensity map by the power of the bilateral filtered map, multiplied by gamma, added with the detail map. To convert to RGB, I simply used the similar methods as Reinhard's, which is to multiply the input radiance map by the scaled intensity map with a normalization factor. The output is then scaled to [0 1], and clamping is used.


figure: Input image


figure: Bilateral filter output (respects edges)


figure: Detials Image

By analyzing the algorithm, with the detailed map, we can see that the output of the image files will preserve detail. This is because the algorithm has preserved the high frequency data before HDR color compression. Unfortunately, due to the color compression scheme, which also relies on the intensity map, the modified intensity map fused with the original color information is put through HDR compressed, which causes color shifts. Therefore, contrast is increased and hue and saturation are not from the original input. With parameters such as contrast, bilateral filter sigmas, and brightness, this algorithm requires tweaking of parameters to get what you want. Therefore, many times the images sometime look unnatural due to the boost of contrast and color shifts.

figure: Doe Library Result (please click for full image)

figure: Botanical Gardens Result (please click for full image)



figure: Doe Library : details


figure: Unnaturalness by Bilateral TM


figure: Botanical Gardens : details


figure: Unnaturalness by Bilateral TM

Summary:

Advantages:
+ greatly enhanced details!
+ fast

Disadvantages:
- parameters, parameters, parameters (hard to get it right)
- light blooms when a large area of bright areas
- halos! (adjusting parameters will prevent this from happening, but it is inconsistent- known problem of bilateral filters)
- unnatural images if parameters are off (color shifts)

Tone Mapping : My New Tone Mapping Algorithm

Analysis

Here we can see two fanstastic algorithms that are fast and practical. Unfortunately, one has great color information but poor high frequency information. Moreover, the high frequency data on the bilateral tone mapping seems amplified. Therefore, sacrificing some of the high frequency information to obtain naturalness is the key. If we extract too much information from the high frequency, we will start to see halos - which defeats the purpose of bilateral tone mapping.


figure: We want the colors of Reinhard's and high frequency data from bilateral's tone mapping. The broad story is to extract favorable low and high frequencies and composite them in a meaningful way to create a new image.

Because Cb and Cr are color information and usually contain low frequency data, we can see that Cb and Cr can be retained through low pass filters. Y contains most of the high frequency information due to its luminance measure. Therefore, we want to be able to composite them together. However, we cannot just take the result of Bilateral tone-mapping, extract Y, and place it on Reinhard's Cb and Cr because that would be a non-linear function and would cause colors to shift through the dominant bilateral tone-mapping's high frequency data.

So lets do something that immitates this process. We perform two operations with the luminance, one with the bilateral tone-mapping method and one with Reinhard's tone mapping method. We then use the same HDR color compression and use the output data from both. To extract the high frequency data from the bilateral filter, we use a small Gaussian kernel on the image. We then subtract the original output by the small Gaussian kernel. We add this information to the Reinhard's output convolved with the same Guassian kernel. Here, we effectively preserved colors, because since Cb and Cr are low frequency data, the low pass filter preserves this information. Since the subtraction of the Bilateral original output with its low frequency data gives us the high frequency information, we can then use this information the fuse the high frequency data and the low frequency data.

This simple technique gives us very impressive results, beating results from the bilateral tone-mapping and Reinhard's, across the board. This technique is simple and uses two fast luminance decomposition techniques. Moreover, because the Bilateral tone mapping's parameters mainly affect the output of the color, we can be more linient in the fine-tunning of the method's parameters, and still achieve great results.

Summary:

Advantages:
+ Fast
+ Simple
+ Good color - right out of the box!
+ high frequency data recovered
+ Doesn't need fine tuning (because process compensates for it)

Disadvantages:
- Some ringing is evident (but minimized due to small Gaussian Kernel)

Results

Here are all the results I ran. As stated before, my tone map algorithm produces natural colors and has high frequency data. To look at the results more carefully, please click on the images for full resolution. I also included Photoshop CS4's algorithm, which usually displays bad blooming effect in bright areas. Sharpness, correct contrast, and color should be noted for each.

Note: For bilateral tone mapping, I did not fine tune the parameters for each image for comparison sake. I used a single set of parameters to demonstrate the need for fine tuning the parameters.

Doe Library

figure: Photoshop CS4

figure: Reinhard Tone Mapping

figure: Bilateral Tone Mapping

figure: My Tone Mapping

Botanical Gardens : Desert

figure: Photoshop CS4

figure: Reinhard Tone Mapping

figure: Bilateral Tone Mapping

figure: My Tone Mapping

Botanical Gardens : Bench

figure: Photoshop CS4

figure: Reinhard Tone Mapping

figure: Bilateral Tone Mapping

figure: My Tone Mapping

Botanical Gardens : Desert

figure: Photoshop CS4

figure: Reinhard Tone Mapping

figure: Bilateral Tone Mapping

figure: My Tone Mapping

Doe Library : Athena

figure: Photoshop CS4

figure: Reinhard Tone Mapping

figure: Bilateral Tone Mapping

figure: My Tone Mapping

South Hall

figure: Photoshop CS4

figure: Reinhard Tone Mapping

figure: Bilateral Tone Mapping

figure: My Tone Mapping

Here, we do see a somewhat of a failure case. Our result has an unnatual blooming in the center. The Bilateral Tone Mapping seems to look better.

Doe Library : Athena

figure: Photoshop CS4

figure: Reinhard Tone Mapping

figure: Bilateral Tone Mapping

figure: My Tone Mapping

References

[1] Erik Reinhard , Michael Stark , Peter Shirley , James Ferwerda, Photographic tone reproduction for digital images, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002

[2] Frédo Durand , Julie Dorsey, Fast bilateral filtering for the display of high-dynamic-range images, ACM Transactions on Graphics (TOG), v.21 n.3, July 2002

© 2010 Michael Tao. All Rights Reserved.