Long exposure digital photo noise

July 4, 2016

For digital cameras, long exposures are affected by ‘hot pixels’ and ‘ noise’. This post investigates long exposure sensor noise using Raw Therapee for my new Canon EOS 100D  (Rebel SL1 / Kiss X7) digital SLR camera. All photos in this post were intentionally shot with the lens cap on!

Taking the images

Dark-frames (i.e. photos with the lens cap on) are long exposures that record long-exposure sensor noise.

Relevant camera settings were:

  • Daylight white balance (fixed white balance for comparisons between images).
  • Manual exposure with varying shutter speed. Lens aperture is not relevant to long exposure sensor noise.
  • Fixed ISO sensitivity (ISO 100 to start with).
  • Long exposure noise reduction OFF.
  • High ISO noise reduction OFF.

Processing the images

Processing parameters in Raw Therapee version 4.1.0 were:

  • Neutral profile (no adjustments).
  • Camera white balance (Daylight white balance, see above).
  • Input Profile = No profile
  • Output Profile = RT_sRGB (i.e. with gamma adjustment, like in real world photos.)

Each image was cropped to the same 400 × 400 pixel square that was selected to contain hot pixels. Results were exported as 8-bit TIFF files. The very small image size reduced further processing loads.

Images were converted into text files using ImageMagick. Text files were cleaned up and saved as .csv text files using LibreOffice Calc. Statistical analysis of the data used R. It was easier to evaluate the results using graphs rather than images (see example below).

Example crop of a dark-frame at ISO100 and 1/60 seconds (raw photo processed in Raw Therapee). A red hot pixel is 'just-noticeable' near the centre and with an amplitude of about 60 (having a display with low black level and turning off the lights helps). I used a 'just-noticeable' noise threshold of 50 (at 8-bits).

Example crop of a dark-frame at ISO100 and 1/60 seconds (raw photo processed in Raw Therapee). A red hot pixel is ‘just-noticeable’ up and left of centre, with an amplitude of about 60 (at 8-bits). Having a display with low black level and turning off the lights helps. I used a ‘just-noticeable’ noise level of 50.

Exposure time effect

I shot a series of dark-frames with increasing exposure. I let the sensor cool down for at least 12 minutes between shots.

Dark-frame results from 1/60 seconds to 16 minutes (2-stop increments) at ISO100. 160 000 individual Red, Green and Blue pixels are plotted in sequence as read by ImageMagick (horizontal axis). Pixel values can be read on the vertical axis.

Dark-frame results from 1/60 seconds to 16 minutes (2-stop increments) at ISO100. 160 000 individual Red, Green and Blue pixels are plotted in sequence as read by ImageMagick (horizontal axis). Pixel values can be read on the vertical axis. The dashed line is a ‘just noticeable’ level.

Hot pixels stand out above the background noise level –  they appear much brighter than they should. Hot pixels are a type of fixed-pattern noise – they always appear in the same location. Hot pixels result from leakage currents into sensor wells – they are manufacturing defects.  The (linear) level of hot pixels increases linearly with exposure time.

In the above series, red hot pixels can be seen increasing in amplitude until they saturate (255 at 8-bits). There were no blue hot pixels and no green hot pixels. Green hot pixels are rare for Bayer filter sensors, for which there are two green pixels for each red and blue pixel. I suggest that it is unlikely to have two adjacent green pixels defective.

I was disappointed to see hot pixels appearing from 1/60 seconds. Another Canon 100D of mine did not show hot pixels before 1 seconds exposure time.  My old Canon 400D (Digital Rebel XTi / Kiss Digital X) did not show hot pixels before 3 or 4 seconds exposure time. As pixel counts and pixel densities increase, I expect that sensor defects become more difficult to control.

The background noise level is dark current noise (also called ‘thermal noise’). Dark currents are small electric currents that occur even when no photons are arriving at the sensor. Dark currents are due to electrons dislodged by random thermal activity. Dark currents increase with temperature. Sensor temperature increases towards the end of long exposures.

In the above series, background noise increased strongly at four and 16 minutes. This noise includes dark current noise and additional hot pixels. Hot pixel noise also increases with temperature.

ISO effect

For each ISO, I shot a series of dark-frames with increasing exposure. I let the sensor cool down for at least 12 minutes between shots. Here, I present results only for one second and 4 minutes exposures.

Dark-frame results from ISO100 to ISO12800 at one second exposure.

Dark-frame results from ISO100 to ISO12800 at one second exposure.

Dark-frame results from ISO100 to ISO12800 at 4 minutes exposure.

Dark-frame results from ISO100 to ISO12800 at 4 minutes exposure.

Increasing ISO amplifies the signal before analog-to-digital conversion. Amplifying the signal will also amplify noise.

In the above series, hot pixel intensities and dark current noise increase at higher ISOs.  ISO greater than 800 should be avoided.

Interestingly, some strong hot pixels vanished above ISO 1600. I suspect there is some in-camera faking for ISO 3200 to ISO 12800 – these images appeared coarser-grained than lower ISO images. The high ISO noise reduction camera setting had no effect on images produced from the raw files.

Temperature effect

I shot a cold dark-frame in the morning, packed my camera in a black camera box and placed that in the sun for a few hours. The camera felt warm when I removed it from the camera box and shot a hot dark-frame. For both shots, I believe the camera sensor temperature was very close to the ambient temperature. Temperature was measured using a ‘fridge thermometer’ probe.

Dark-frame results at 12°C and 32°C, four minutes exposures at ISO100.

Dark-frame results at 12°C and 32°C, four minutes exposures at ISO100.

In the above comparison, background noise increased strongly from 12°C to 32°C. It’s much better to shoot long exposures in cool conditions.

Dark-frame subtraction

Hot pixel noise has a fixed-pattern and can be removed by subtracting a dark-frame. I prefer to do perform dark-frame subtraction in Raw Therapee. In-camera long exposure noise reduction can waste a lot of time shooting dark-frames with every photo.

From the preceding results and discussion, we need to match the level of the hot pixels in the photo and in the dark-frame:

  • Shoot at the same temperature.
  • Shoot at the same exposure time.
  • Shoot at the same ISO.

I evaluated dark-frame subtraction at four-minutes exposure time, for which there were moderate to large numbers of hot pixels at all ISO settings.

Dark-frame subtraction results from ISO100 to ISO800 at 4 minutes exposure.

Dark-frame subtraction results from ISO100 to ISO800 at 4 minutes exposure.

In the above series, dark-frame subtracted images had zero to very low noticeable noise up to ISO400. Dark-frame subtraction also was effective at ISO800, however there were many low level noisy pixels in the dark-frame subtracted ISO800 image.

Dark current noise results from random processes and cannot be removed by subtracting a dark-frame. In fact, for uncorrelated variables, the variance of the sum or difference is the sum of the variances (dark-current noise is random and uncorrelated). In the above series and at ISO100 in particular, observe that the background noise actually increased after dark-frame subtraction.

Shot-interval effect

When shooting multiple long exposures at very short intervals, the camera sensor could experience a cumulative heating effect similar to one very long exposure.

I shot sets of three four minute exposures at increasing shot intervals, using a plug-in intervalometer to control the camera shutter. I let the camera sensor cool down for about 16 minutes between each set. I compared noise levels for the third shot in each set.

Dark-frame results at shot intervals from zero to four minutes, three four minutes exposures at ISO100 in each set.

Dark-frame results at shot intervals from zero to four minutes, three four minutes exposures at ISO100 in each set.

In the above series, the zero second interval result is a single 12 minute exposure (4 + 0 + 4 + 0 + 4 minutes) and very noisy. Noise levels were much lower at shot intervals from one second to one minute and slightly lower again at a four minute shot interval. I was using a slow SD card and the real shot interval may have been longer than the intervalometer setting at one second. Nonetheless, these results clearly indicate that the sensor cooled rapidly. Shot intervals of just a few seconds are adequate.

Minimising long exposure noise

Following the above analysis, I classify long exposures into three groups:

  • Shorter long exposures from one second to about one minute. A dark-frame is required for removing hot-pixel noise. Dark current noise is not usually an issue.
  • Medium long exposures from about one minute to four minutes. With low ISO and avoidance of sensor heating, dark current noise can be minimised and dark-frame subtraction is effective.
  • Very long exposures greater than about four minutes. Consumer digital cameras are not designed for such long exposures and dark current noise becomes excessive. Image averaging may be helpful.

Also recognise that ‘live view’ and digital video are long exposures. Taking a long time to set-up a shot in live view will heat up the sensor and increase noise. Recording video with a digital SLR camera for several minutes will heat up the sensor and can increase noise. However, downscaling from sensor resolution to video resolution will average away much of the noise.


Zomei neutral density filters tested

March 19, 2015

I recently purchased a video-capable DSLR (Digital SLR) camera and wanted some ND filters (neutral density filters). I purchased some Zomei ND filters from ebay. Zomei is Chinese, but they make lots of filters and the products are nicely presented. There was very little feedback about Zomei on the web and so I wrote this review.

I found that the Zomei ND2 and ND4 filters were satisfactory. For the ND8 filter, exposure was inaccurate and colour errors were relatively large.

Filters tested

I tested Zomei ND2, ND4 and ND8 58 mm filters. The filters are well made and the threads are good. The filters are not multicoated.

Zomei ND2, ND4 and ND8 filters.

Zomei ND2, ND4 and ND8 filters.

I immediately noticed that the Zomei filters felt different, because they are made from high-density resin. Some Zomei filters use glass, but these ND filters were plastic! The ebay seller had incorrectly stated they were import optical glass (and promptly removed the listings after my negative feedback).

I also tested Kenko SMART ND8 52 mm and ND8 77 mm filters to compare with the Zomei results. The Kenko filters have a slim frame and are not multicoated. They were made in the Philippines, I think using glass from Japan.

Methods

I photographed an X-Rite ColorChecker Passport in sunlight on a clear afternoon (uniform and stable lighting is required) with a Canon EOS 350D DSLR and EF 85 mm f/1.8 lens. I set the base exposure (1/500 seconds shutter speed) with a 18% grey card and photographed the ColorChecker four times with fixed aperture f/8 and fixed ISO100:

  1. No filter and base exposure before.
  2. ND filter and base exposure.
  3. ND filter and exposure compensation (slower shutter speed: +1 stop for ND2, +2 stops for ND4 and +3 stops for ND8).
  4. No filter and base exposure after.

The remainder of this section is very technical and for specific software. You might prefer to jump over to the results.

Linear device RGB is required. I processed each image in Raw Therapee version 4.1 as follows:

  • Processing Profile = Neutral (disables most adjustments).
  • Input Profile = No profile (device RGB).
  • Output Gamma = linear_g1.0 (linear RGB).
  • Custom white balance on the third-lightest grey patch of the ColorChecker (any neutral light-grey patch should work).
  • For image 2, apply linear exposure White Point Correction (2× for ND2, 4× for ND4, 8× for ND8).
  • Crop to ColorChecker target.
  • Save as TIFF (16 bit).

Rather than applying exposure compensation to image 2, I could have simply used image 3. The shutter speeds should be accurate enough.

I used Argyll CMS to read the ColorChecker patches from each linear RGB TIFF image:

scanin -v -p -a -G1 -dipn input.tif ColorChecker.cht ColorChecker.cie
-v verbose.
-p compensate for perspective distortion.
-a chart orientation normal (not upside down).
-dipn generate diagnostic image.
-G1 Gamma encoding of image (linear).
input.tif is the input image and input.ti3 is the name of the output text file.
ColorChecker.cht is a chart recognition file.
ColorChecker.cie contains the the chart reference data (which is copied to the .ti3 output file).

I copied the scanin output to a spreadsheet. I compared white patch average RGB values with- and without filters (they should all be the same if lighting and exposure were the same).

Next, I used Argyll CMS to compute colour differences relative to the ColorChecker reference data. For this, I had to match the exposure of images 1 and 3 to minimise lightness differences increasing the colour differences. I used image 3 because it has a better signal to noise ratio than the underexposed image 2.

I had previously made my own camera profile for the Canon EOS 350D and matched white patch average RGB values to that profiling image. Alternatively, one could simply profile image 1 as the reference. After applying linear exposure White Point Corrections in Raw Therapee, I ran scanin again and then profcheck to evaluate colour differences:

profcheck -v2 -k image.ti3 profile.icc
-v verbosiy level 2.
-k report CIE Delta-E 2000 colour differences.
input.ti3 is the output from scanin.
profile.icc is the camera profile.

Argyll CMS profcheck takes the patch RGB values from the .ti3 file, applies the camera profile and computes colour differences with the chart reference data in the .ti3 file. I repeat: to minimise lightness errors, the exposure of the image must match the exposure of camera profiling image.

I copied the profcheck output to a spreadsheet. I compared average DE2000 differences with- and without- filters.

Exposure errors

After 3-stops exposure compensation, the Zomei ND8 filter was still underexposed −0.49 stop relative to the images without any filters. The ND4 filter was better (−0.17 stop) and the ND2 was good (+0.02 stop). The ND8 filter was effectively a ND11 filter (2^(3+0.49) = 2^3.49 = 11.2), i.e. the effect is stronger than 8×.

Zomei ND filters exposure errors. Exposure compensation was applied to the ND images (+1 stop for ND2, +2 stops for ND4, +3 stops for ND8) before making these comparisons. Numbers above the columns are exposure errors in stops = log(RGB/RGB_before) / log(2). The before and after measurements without any filters agree, which indicates that the lighting was constant during the tests.

Zomei ND filters exposure errors. Exposure compensation was applied to the ND images (+1 stop for ND2, +2 stops for ND4, +3 stops for ND8) before making these comparisons. Numbers above the columns are exposure errors in stops = log(RGB/RGB_before) / log(2). The before and after measurements without any filters agree, which indicates that the lighting was constant during the tests.

For comparison, exposure errors for two Kenko ND8 filters were much smaller at −0.14 and −0.12 stops.

Colour errors

The 3.3 average colour error for the Zomei ND8 filter exceeded a just noticeable difference (DE > 2) and the maximum was 6.4. Colour accuracy was satisfactory for the Zomei ND4 filter (average DE = 2.0) and good for the ND2 filter (average DE = 1.6).

Zomei ND filters colour errors. Numbers above the columns are average DE2000 colour errors (DE > 2 is a just noticeable difference). The before and after measurements agreed closely and average colour differences were small. This indicates: 1) the lighting (afternoon sun) was similar to the camera profile (based on morning sunlight) and, 2) the lighting was constant during the tests.

Zomei ND filters colour errors. Numbers above the columns are average DE2000 colour errors (DE = 2 is a just noticeable difference). The before and after measurements agreed closely and average colour differences were small (DE < 2). This indicates: 1) the lighting (afternoon sun) was similar to lighting for the camera profile (morning sunlight) and, 2) the lighting was constant during the tests.

For comparison, average colour errors for two Kenko ND8 filters were much smaller at 1.5 and 1.4 stops.

The following two charts highlight the colour differences when using the Zomei ND8 filter.

Colour differences for 24 ColorChecker patches. No filter. Most colour errors are small (DE < 2) and were similar to the camera profiling errors.

Colour differences for 24 ColorChecker patches. No filter. Most differences are small (DE < 2) and were similar to the camera profiling errors.

Zomei_colour2

Colour differences for 24 ColorChecker patches. Zomei ND8 filter. Many differences exceed a just noticeable difference (DE > 2).

Conclusion

We can use a ColorChecker chart, Raw Therapee and Argyll CMS to evalute ND filters.

The Zomei ND filters were increasingly inaccurate for higher densities. Results for the Zomei ND8 filter were unsatisfactory. This doesn’t mean that filter is useless, however it is not the 8× (3 stop) and neutral density (DE < 2) filter that we want. Results for Kenko filters showed that accurate ND8 filters are achievable.

Disregarding the ND8, I purchased two Zomei ND filters for AUD22 (AUD11 each). The Kenko filter was AUD19 (for 58 mm diameter). I think it is worth paying more for better quality glass ND filters.


Digital photo random noise and solutions

August 1, 2014

Noise in digital photos is very familiar to “pixel-peepers”. Noise obscures details, reduces dynamic range and looks bad when photos are viewed at large sizes. This post examines how ISO sensitivity, colour and exposure affect random noise levels. Random noise is usually dominant and more difficult to remove than fixed pattern noise (“dark noise”) and banding noise, which are not covered here. This post concludes with some tips for reducing random noise when taking digital photos.

Measuring noise with Raw Therapee and Argyll CMS

Camera profiling targets can be used for measuring digital photo noise. A fair evaluation requires “device RGB” images, without any colour processing, noise reduction and other enhancements. I used Raw Therapee to get raw photo linear RGB images:

  1. Processing Profile: Neutral.
  2. Input Profile: No profile.
  3. Output gamma: linear_g1.0 (this will disable the Output Profile).
  4. Spot White Balance on neutral grey patch.
  5. Crop to target and save as TIFF.

Next, I used Argyll CMS scanin to read the images. My chart was an X-Rite ColorChecker Digital SG:

scanin -v -p -a -dipn IMG_9999.tif ColorCheckerSG.cht ColorCheckerSGD50.cie

Argyll CMS generates a text file including mean patch RGB measurements and standard deviations (SD). I pasted these data into a spreadsheet and then computed coefficients of variation (CV = SD/mean). CVs measure relative noise levels. Engineers often prefer the signal to noise ratio (SNR ~ mean/SD = 1/CV), which is commonly reported in decibels: SNR = 10×log_10×((mean/SD)^2).

High ISO noise

Digital photographers know from experience that noise levels increase at higher ISO settings. I tested this by photographing my CCSG target at different ISO settings. The light source was direct afternoon sunlight, to minimise glare from the darker patches. I stopped the lens down to f/11, metered off a 18% grey card at ISO100 (shutter 1/250 s for this example) and took the first image. I then increased ISO in one stop increments (ISO200, 400 … 1600) and held the exposure value constant by halving the shutter speed for each successive image (1/500, 1/1000 … 1/4000). A constant exposure value is necessary for a fair comparison.

The graph below plots CVs versus reflectance. Dotted lines indicate middle-grey (18%) and 32 dB SNR. Middle-grey (L* = 50) is a good reference because most information in digital photos is usually present in the mid-tones. If middle-grey is noisy, people will notice. Middle-grey has a rather low 18% reflectance because human vision lightness response is non-linear. The SNR reference I have stolen from DxOMark: 24 dB (CV = 0.063 ) is bad SNR; 32 dB is good (CV = 0.025); and 38 dB (CV = 0.013) is excellent.

Coefficients of Variation (CV) for greyscale patches (reflectance = 0.7% to 91%) at different ISO settings. CVs for neutral grey patches computed as means of individual RGB channels. Camera: Canon EOS 400D Digital SLR. Target: X-Rite ColorChecker Digital SG.

Coefficients of Variation (CV) for greyscale patches (reflectance = 0.7% to 91%) at different ISO settings. CVs for neutral grey patches computed as means of individual RGB channels. Camera: Canon EOS 400D Digital SLR. Target: X-Rite ColorChecker Digital SG.

The ISO effect appears as a higher “noise floor” at higher ISO settings. This is random amplifier noise (“read noise”). Read noise has a fixed level at each ISO setting and is additive. ISO800 and ISO1600 for my Canon 400D are noisy, from the shadows to the highlights, even though this test was in good light (Exposure Value 14.9).

Noise also decreases very rapidly with increasing signal levels. This suggests “shot noise”, which results from variations in photon counts at each photosite and follows a Poisson distribution, where the CV is the reciprocal square-root of the mean.

DxOMark has measured raw photo noise data for most digital SLR cameras. They present SNR, like in the next graph. For the Canon EOS 400D, DxOMark suggested a maximum ISO664 for good middle-grey SNR (nearest setting ISO400) and my results agree.

Signal to Noise Ratios (SNR) for greyscale patches (reflectance = 0.7% to 91%) at different ISO settings. Presentation like Full SNR graph from DxOMark. Camera: Canon EOS 400D Digital SLR. Target: X-Rite ColorChecker Digital SG.

Signal to Noise Ratios (SNR) for greyscale patches (reflectance = 0.7% to 91%) at different ISO settings. Presentation like DxOMark’s Full SNR graph. Camera: Canon EOS 400D Digital SLR. Target: X-Rite ColorChecker Digital SG.

Colour noise

Colour noise (or “chroma” noise) should also be familiar to digital photographers, but I think less well understood. To investigate colour noise, I examined colour patches from the ISO100 photograph in afternoon sunlight used above.

CVs for individual RGB channels are compared in the graph below. For blue, the red signal is weak and just as noisy as for black. For red, the blue signal is weak and noisy. Green wavelengths are between blue (shorter wavelengths) and red (longer wavelength). The green channel has similar CVs for all three of these colour patches. Also observe that neutral patches reflect all wavelengths equally. A correct white balance is necessary.

Coefficients of Variation (CV) for red, green and blue patches plus white, middle-grey and black for reference. The dotted line equals 32 dB signal to noise. Camera: Canon EOS 400D Digital SLR. Target: X-Rite ColorChecker Digital SG.

Coefficients of Variation (CV) for red, green and blue patches plus white, middle-grey and black for reference. The dotted line equals 32 dB signal to noise. Camera: Canon EOS 400D Digital SLR. Target: X-Rite ColorChecker Digital SG.

Following the above results, in our photos we expect more red colour noise in blues and more blue colour noise in dark reds. The illuminant is also important. For example, tungsten lights have lots of red and very little blue wavelengths. Photographs in tungsten lighting will have higher blue colour noise compared to daylight, which has a more even spectrum.

Exposure effects

Increasing the signal (photons counted) reduces shot noise (“Poisson noise”). The following graph uses data from my linearity testing post. It demonstrates that over-exposure reduces noise. Shadows, represented by the black patch, are most sensitive to exposure.

Coefficients of Variation (CV) for white, middle-grey and black versus exposure. Camera: Canon EOS 400D Digital SLR at ISO100. Target: X-Rite ColorChecker Digital SG. Base exposure (delta EV = 0) metered using a 18% grey card.

Coefficients of Variation (CV) for white, middle-grey and black versus exposure. Camera: Canon EOS 400D Digital SLR at ISO100. Target: X-Rite ColorChecker Digital SG. Base exposure (delta EV = 0) metered using a 18% grey card.

Expose To The Right” (ETTR) has been promoted to increase resolution in shadows and critics have argued that ETTR is not necessary for newer, higher bit-depth cameras. However, ETTR also reduces noise, as demonstrated above. This is true for faster shutter speeds typical of hand-held photography and not for long exposures, where “dark noise” appears.

Solutions for random noise

There are several photographic techniques to reduce random noise:

  • Use a better camera, which usually means a larger image sensor (reduce shot noise).
  • Use a lower ISO setting (reduce read noise).
  • Expose to the right (reduce shot noise).
  • Supplement the natural light (e.g. flash, reflectors) if it is weak (reduce shot noise).
  • Prefer broad, even spectrum artificial lighting (reduce colour noise).
  • Use image averaging for static scenes and a tripod. This reduces noise levels everywhere.
  • Use High Dynamic Range software to merge an overexposed image (with reduced shot noise in shadows) with a correctly exposed image. Two images may suffice and hopefully HDR effects can be disabled when not wanted.

All of the above techniques reduce noise in the raw photo data. Further software noise removal is often required for the final image. However, software can’t restore details that are lost in noise. And image sharpening tends to amplify noise. Low noise images save processing hassles.

Finally, here is an example where ETTR has reduced digital photo noise at ISO400.

Cape York, Queensland. Photographed in the afternoon with a Canon EOS 400D at ISO400.

Cape York, Queensland. Photographed in the afternoon with a Canon EOS 400D at ISO400.

Shadow noise detail at 200% from the above photo at base exposure (shutter 1/100 s). Raw photo processed in Raw Therapee with no noise reduction and no sharpening. There is subtle chroma noise in the shadows. There was more noise in deeper shadows towards the bottom of the photo, but this would be difficult to see on lower contrast ratio displays and for everyday viewing conditions.

Shadow noise detail at 200% from the above photo at base exposure (shutter 1/100 s). Raw photo processed in Raw Therapee with no noise reduction and no sharpening. There is subtle chroma noise in the shadows.

Shadow noise detail at 200% with ETTR (shutter 1/60 s). Exposure corrected by -0.74 stops (linear ×0.60) in Raw Therapee.

Shadow noise detail at 200% with ETTR (shutter 1/60 s). Exposure corrected by -0.74 stops (linear ×0.60) in Raw Therapee. If you have a decent display, you should be able to see an improvement over the base exposure.

 Raw photo histogram for ETTR image above. ETTR can be recommended for this example, where the raw photo RGB channels are not clipping.

Raw photo histogram in Raw Therapee for ETTR image above. ETTR can be recommended for this example, where the raw photo RGB channels are not clipped.


Checking digital camera system linearity

July 9, 2012

When making custom digital camera profiles, it’s helpful to know if the system response is linear. For example, a two-fold increase in exposure time should result in a two-fold increase in linear device RGB. This can easily be tested by taking multiple different exposures of a test chart, a grey card, or even a piece of white paper.

The following procedure tests the complete system linearity, i.e. camera + lens + raw photo software. I have tested two Canon Digital SLR (DSLR) cameras using an X-Rite ColorChecker Digital SG (CCSG) target and Raw Therapee version 4.0.8 (RT). Other charts and raw photo software (e.g. dcraw, Darktable) can be used. The raw photo software must be able to output linear device RGB (UFraw does some weird scaling and is not suitable).

Photograph the target

I photographed the target as follows:

  1. Setup the test chart in a dimly lit environment, with a stable light source. I did this indoors during the day, with natural window light.
  2. Meter the correct exposure. I metered off an 18% grey card. Exposure is not critical, only the differences are used for the results.
  3. Shoot the grey card and set custom white balance (WB) in the camera. This step is optional: WB can also be set in raw processing.
  4. Shoot the target from about 5 stops under the metered exposure to about  5 stops over the metered exposure. Watch out for glare.

It’s best to shoot with low ISO, for minimal noise. You will need to use a tripod, mirror lock-up and a remote shutter release or a timed shutter release for long exposures. It is impossible to vary shutter speed by 10 stops and shoot everything hand-held: camera shutters are not fast enough in bright light and camera shake is a problem in low light.

Process the raw photos

Open each image in RT and load the neutral profile, which should switch off all adjustments. Then adjust the Color settings as follows:

  • Input Profile: No profile.
  • Working Profile: Irrelevant (no adjustments will be made in the working space and there is no Output Profile).
  • Output gamma: linear_g1.0 (this will disable the Output Profile).

If required, custom WB on a neutral patch and set the same custom WB for all photos. I did an Auto WB on my grey card photo and saved a temporary RT processing profile to make setting up the remaining images easier.

Some underexposed shots will be almost black and impossible to read. These can be brightened by increasing the white point linear correction factor (Raw tab): 2x = +1 stop, 4x = +2 stops, 8x = +3 stops, 16x = +4 stops. When the white point linear correction factor is at maximum, you might further increase brightness with exposure compensation (Exposure tab): 32x = 16 + 1 EV.

Almost done, crop each image to the target edges and export 16-bit uncompressed TIFF files.

8 stops of dynamic range with my Canon 400D. Each photo displays linear device RGB (gamma = 1).

Measure the camera response

I used Argyll CMS version 1.4.0 to read the patches of each TIFF file:
scanin -v -p -a -dipn filename.tif ColorCheckerSG.cht ColorCheckerSG.cie
-v Verbose output.
-p Perspective correction.
-a Recognize chart in normal orientation (not upside-down). This speeds up chart recognition.
-d ipn Generate diagnostic output. For checking the chart has been read correctly.
ColorCheckerSG.cht is the recognition template file from the Argyll CMS /ref folder.
ColorCheckerSG.cie was generated with spec2cie.

Always check the diag.tif diagnostic output to make sure patches have been identified correctly – this will save confusion later on. Argyll CMS will fail to read noisy and very dark images. For my cameras, photos more than 4 or 5 stops under-exposed had insufficient signal-to-noise.

Finally, I copied the scanin .ti3 text file outputs into a spreadsheet and plotted the RGB measurements for patches E5 (white), H5 (middle-grey) and E6 (black) versus exposure time in seconds. Use the exact exposure times reported by the camera. Some image viewers might report inaccurate exposure times because of rounding. And don’t forget to reverse any RGB scaling of under-exposed images that was applied in raw processing.

Behold, my Canon 350D and 400D DSLRs are linear over at least 9 stops!

Raw photo linear RED response of my Canon 400D for CCSG patches E5 (white, diamonds), H5 (middle grey, squares) and E6 (black, triangles). White-balance was made with a grey card (middle-grey). Grey diamonds, squares and triangles are clipped. Some minor variation is surely due to inaccuracies in camera shutter speed and exposure increments.

Raw photo linear GREEN response of my Canon 400D for CCSG patches E5 (white, diamonds), H5 (middle grey, squares) and E6 (black, triangles).

Raw photo linear BLUE response of my Canon 400D for CCSG patches E5 (white, diamonds), H5 (middle grey, squares) and E6 (black, triangles).

Raw photo linear RED response of my Canon 350D for CCSG patches E5 (white, diamonds), H5 (middle grey, squares) and E6 (black, triangles). These data were obtained at a different time of day, with more ambient light than the 400D data.

Raw photo linear BLUE response of my Canon 350D for CCSG patches E5 (white, diamonds), H5 (middle grey, squares) and E6 (black, triangles).

Raw photo linear GREEN response of my Canon 350D for CCSG patches E5 (white, diamonds), H5 (middle grey, squares) and E6 (black, triangles).


A guide to printing digital photos via the internet

June 1, 2012

Common 6×4 inch prints are for wimps. If you want to impress people with your photographs then make big prints. This article can help you get excellent prints and good service at a reasonable price.

Avoid kiosks and printing in-store

We had one local store that was good at printing digital photos, but, like most small camera stores, they’ve shut down. I’ve tried printing in-store at BIG W once and the prints came back with an ugly greenish cast. Sure, you might demand reprints but why waste your time in the first place? Discount department stores rarely have experienced photo professionals running the machines.

What to look for when printing via the internet

There are many online photo printing services in Australia and pricing is highly variable. The following table compares only a few, just to illustrate some points for the discussion that follows. This post is not a complete review.

BIG W Photos, Harvey Norman Photo Centre and photoenlargements.com.au are consumer labs. They print on digital minilabs (Fuji Frontier) and maximum print size is 12 x 18 inches. Professional labs like Digilab Professional, Pixel Perfect and RGB DIGITAL PRO PHOTO LAB feature large-format printers, e.g. Durst  or Chromira, which can expose 50 or 30 inch roll paper.

Inches BIG W
Harvey Norman
photoenlarge (<20) Digilab
(1-5)
Pixel Perfect RGB DIGITAL
4×6 0.15 0.12 2.00 0.80 0.52 0.25
5×7 0.35 1.60 0.75 0.70
5×7.5 1.60 0.99 0.80
6×8 0.75 2.00 1.95 1.27 0.95
8×10 2.95 2.00 3.10 4.29 1.50
8×12 3.75 2.00 3.60 5.15 2.00
10×15 9.84 8.95 7.00 3.75
11×14 13.84 12.95 8.20 8.27 6.00
12×16 15.84 14.95 9.90 10.30
12×18 17.95 5.00 10.50 11.59 9.00
16×20 15.90 26.42 12.00
20×24 17.90 39.63 17.00
20×30 18.85 49.54 17.50
Shipping 0.00 0.00 10.00 6.50 25.00 12.00
Upload Flash Flash Web FTP Web FTP/App.
Comparison of photo printing costs at three consumer labs and three professional labs. Lowest prices in bold, valid May 2012. Minimum shipping and handling shown. Table ends at 20×30 inches although most pro labs can print much larger. Unit costs at photoenlargments.com.au and digilab will decrease with larger orders. This table provides examples for the discussion. It is not comprehensive and not updated. You must shop around for the current best prices.

Reliable quality

Poor colour-balance is common from consumer labs. I’ve experienced better results from BIG W’s internet photo service, where the products are made in a print centre somewhere and then delivered to my local store. However, my recent online prints from Harvey Norman were unsatisfactory. For small to medium prints, most pro labs run digital minilabs similar to BIG W and Harvey Norman, but the quality is consistently good.

A good selection of print sizes

I thought BIG W Photos was satisfactory but they no longer offer my favourite print sizes, like 5 x 7, 8 x 10 and 8 x 12 inches.

Low cost prints

If you’re printing lots of photos, the cost accumulates quickly. For small prints, consumer labs can’t be beaten for price. For medium and large prints, pro labs are usually cheaper. I don’t waste my time with labs whose price lists aren’t published openly on their website.

Low cost shipping

Total order cost includes shipping. Again, consumer labs win because they can deliver prints to your local store for free. Some pro labs are rather expensive for shipping and handling and some don’t offer regular post.

Easy upload

Web-based uploaders can be slow and tedious. The flash-based Fujifilm application (BIG W, Harvey Norman) is probably the worst. I prefer FTP which allows me to upload a whole order in one click. Some labs provide upload and ordering software but I prefer not to install more applications on my personal computer.

How to evaluate different photo labs

Evaluating digital photo labs is easy, just send them a test image. Check that colours are natural, with no colour casts in the neutral tones. Pay attention to skin-tones, which are especially sensitive to poor colour balance. Note that most pro labs will use a minilab machine for smaller prints and not the large-format printers you might be more interested in evaluating.

Personally

There are no good consumer labs in my area, which is why I print via the internet. I’ve printed at Digilab a couple of times now and have been happy with the service and results. In preparing this article, I noticed that photoenlargements.com.au can be good value for mini-lab prints and RGB DIGITAL also looks worth trying. Feel free to comment about your favourite Australian photo lab.


An introduction to digital camera profiling

January 7, 2012

This article introduces camera profiling and potential sources of error. Successful profiling requires good photographic technique, a reliable target and a model that fits the device behaviour. I use Argyll CMS for camera profiling.

Overview

Digital camera sensors do not “see” colour in the same way as human vision. Raw photo processing applications need input device profiles to interpret camera RGB.

There are five or six elements in profiling a digital camera:

  1. Camera and lens.
  2. Camera profiling target.
  3. Illuminant.
  4. Photographic technique.
  5. Profile model.
  6. Spectrophotometer.

Each of these elements has some impact on the accuracy of the resulting profile. Camera profiling targets were evaluated in another post. I don’t have a spectrophotometer. I use manufacturer’s reference data and assume that manufacturing is precise.

Photographic technique

Some people are good with computers but only mediocre photographers. Getting a good photo of the target is the most difficult step in camera profiling. Garbage in = garbage out.

Reflected light (e.g. trees, buildings, vehicles and, indoors, furniture) will contaminate the colour of the light source. Set-up the target away from any light coloured objects and don’t wear brightly coloured clothing.

I set the target on a large plywood sheet (roughly 1 m x 1 m) which is spray-painted matte black. I also use “shutters” to block light from the sides and reflections from the ground or floor.

A setup for shooting the camera profiling target in sunlight.

A setup for shooting the camera profiling target in sunlight. Just cheap plywood, two lift-off hinges and some screws.

Glare is reflection of the light source and imparts a lightness to the image. Viewed at at 100%, glare can be seen as light-speckling in darker parts of the image. Glare should not be mistaken for dust or noise.

Strong glare example. 100% crop of X-Rite ColorChecker Digital SG target photographed in direct sunlight with Canon EOS 400D.

Weak glare example. 100% crop of a photograph taken from a different angle and 10 minutes later than the previous example.

A 0/45 degree geometry is used for spectrophotometer measurements to exclude glare. Likewise, the camera profiling target should be photographed from a different angle to the direction of the light source. In direct sunlight, I face the target towards the sun and shoot from below. I sometimes lay on the ground to get a satisfactory angle.

0/45 degree spectrophotometer geometry. Source: X-Rite.

0/45 degree spectrophotometer geometry. Source: X-Rite.

Camera geometry in direct sunlight. Glare is reflected away from the camera. Shooting from below and in front, left-right perspective distortion is prevented.

Camera geometry in direct sunlight. Shooting from in front of and below the target, left-right perspective distortion is prevented.

With artificial lighting, it’s usually not possible to place the light source high above the target and photograph from a low angle. I place the light directly in front of the target and then photograph from the side.

A setup for shooting the camera profiling target with flash. I have used an umbrella to give an even light.

A setup for shooting the camera profiling target with flash. The umbrella gives an even light.

My home-made light panel, here with nine tungsten bulbs (made neuatral white with a custom white balance). The circular patches are a second layer of diffuser material to attenuate direct light. I made a light panel because I could not get very uniform light in a light tent.

My home-made light panel. The circular patches are a second layer of diffuser material to attenuate direct light. I made a light panel because I could not uniform light inside a light tent.

Use a lens hood to avoid flare. Stand back to avoid casting a shadow and lens distortion at wide focal lengths. Fill only the centre 1/2 to 2/3 of the frame to avoid any vignetting and resolution loss towards the edges of the frame. Shoot with low ISO to minimise noise.

Illuminant

The reference illuminant for the ICC profile connection space is D50. Most camera profiling targets are supplied with D50 reference data only.

Direct sunlight is a convenient approximation to illuminant D50 except that sunlight has more ultraviolet, especially around noon and in summer. I’ve found that applying manual white balance makes a very good D50 approximation from direct sunlight at any time between about mid-morning and mid-afternoon.

I prefer to shoot with direct sunlight at about one or two hours after sunrise:

  • When the sky is clear and sunlight more directional, with less diffuse daylight. It’s easier to control glare and maximise contrast in the photo.
  • When sunlight is nearest to D50 and UV is lower than at noon.

Colour temperature of direct sunlight in the morning. I used my Canon 400D to photograph an X-Rite ColorChecker Digital SG target every 15 minutes between 30 minutes and two hours after sunrise. Then I used Canon Digital Photo Professional to estimate colour temperatures. I adjusted the colour temperature slider until RGB were approximately equal on a neutral grey patch of the target. Beware that third-party raw photo software may not estimate colour temperature accurately.

Strictly speaking, colour changes with illuminant and a camera profile is accurate only for a specific illuminant. In practise, camera profiles created for sunlight can deliver pleasing results for most other daylight situations and some other continuous spectrum light sources.

Exposure

The sunny 16 exposure rule predicts a shutter speed of 1/(4 × ISO) = 1/400 at f/8 and ISO 100. A faster shutter speed suggests glare or very bright sunlight.

I actually set my base exposure by metering off a 18% reflectance grey card. I then shoot a series of photos over the base exposure (e.g. 1/400s, 1/320s, 1/250 s, 1/200s). The raw photo RGB channels should be bright, but not clipped. The response is very linear and final exposure adjustments will be made during profiling.

Profile models

Choosing a profile model depends on the behaviour of the camera and how much data is available for characterisation. Argyll CMS v 1.6.0 has seven profiling models to choose from (in order of complexity):

  1. Matrix.
  2. Single gamma + matrix.
  3. RGB gamma + matrix.
  4. Single shaper + matrix.
  5. RGB shaper + matrix.
  6. XYZ colour look-up table.
  7. L*a*b* colour look-up tabl.

A matrix profile is simply a 3×3 matrix that transforms white-balanced linear RGB to D50 XYZ. I recommend simple matrix profiles:

 


Targets for camera profiling

September 30, 2011

Third-party raw photo processing applications need custom input profiles to make nice colour from camera RGB. For a long time now, I have making my own camera profiles with Argyll CMS. For this we need a profiling target (a test chart). All targets are not equal, as you will find in this review.

Profiling targets examined

I have evaluated the following reflective targets which are supported by Argyll CMS. I found reference data on the manufacturers websites for most charts. I don’t quote batch numbers for reference data because I believe the manufacturing is precise.

Chart Number of patches Cost
X-Rite ColorChecker Classic 24 USD 69
X-Rite ColorChecker Digital SG 140 USD 259
LaserSoft DCPro Studio Target 140 EUR 79
Wolf Faust IT8.7/2 C1 288 USD 30
HutchColor HCT (Fuji/Kodak) 528 USD 255
CMP Digital Target 4 570 EUR 92
Summary of camera profiling targets examined

Colour gamut

Modern digital SLR (DSLR) cameras have very wide colour gamuts. A wide gamut test chart is preferred, to exercise the camera’s response.

Below are two-dimensional gamut plots for seven targets (including two from HutchColor). The calculation of xyY is very simple from XYZ reference data. For the X-Rite charts I could only get Lab data and conversion to XYZ and then xyY is more involved (equations on Bruce Lindbloom’s site).

All targets are weak towards green. The HutchColor targets have the widest gamut. The ColorChecker Digital SG and LaserSoft DCPro targets have similar gamuts. The IT8.7/2 target provides satisfactory coverage with 288 patches. Surprisingly, the CMP Digital Target 4, with 570 patches, has a small gamut and is weak in blues and reds. Is the CMP Digital Target 4 simply a home-printed target?

Gamut plot for X-Rite ColorChecker Classic. This chart has rather weak RGB patches.

Gamut plot for X-Rite ColorChecker Digital SG. A decent gamut but this chart is expensive and has just 140 patches.

Gamut plot for LaserSoft DCPro Studio Target. A similar gamut to the X-Rite ColorChecker Digital SG.

Gamut plot for Wolf Faust IT8.7/2 C1. A good result from a simple and inexpensive chart.

Gamut plot for HutchColor HCT (Fuji). The gamut of the chart touches WideGamutRGB in cyans and yellows.

Gamut plot for HutchColor HCT (Kodak). Green is a little less intense than the HCT Fuji chart.

Gamut plot for CMP Digital Target 4. This chart is weak in blues and reds.

Dynamic range and white point

Modern DSLR cameras have very high dynamic range and a high contrast test chart is preferred. The white patch should be bright and neutral. The dark patch is less important because it is difficult to avoid glare.

The graphs below summarise brightness, contrast and white point for the seven targets. The ColorChecker Digital SG seems best overall, with a bright (L* = 96.5), neutral (Delta-ab = 1.1) white patch and not the worst contrast ratio (White/Black = 15).

White patch lightness for the seven targets considered in this review.

Black patch lightness for the seven targets considered in this review.

Also Observe that the medium makes a difference: the HutchColor HCT target on Kodak paper (Delta-ab = 0.7) is more neutral than Fuji paper (Delta-ab = 2.3). The Wolf Faust IT8.7/2 C1 is also printed on Kodak paper (perhaps different to HutchColor) and the white patch is slightly bluish (Delta-ab = 2.2).

White patch deviation from perfect neutral (Delta-ab=0) for the seven targets considered in this review.

Spectral problems

Camera profiling targets are commonly photographed in sunlight, which includes ultraviolet wavelengths (UV). Photographic papers can contain “fluorescent whitener additives” (or “optical brightening agents”) which makes these papers appear more blue. These papers and printed camera profiling targets can show colour shifts towards blue when there is ultraviolet in the light source.

Metamerism is another problem for photographic and printed targets. The processes are optimised to produce colour, usually from just three of four colourants, that look natural to the human eye but actually might be composed of quite different spectra. This is a problem if the camera spectral sensitivities are different to the human eye. ColorChecker targets are made using multiple different pigments, giving reflective spectra that are more representative of the real world.

Conclusions

Modern DSLR cameras surpass the gamut and dynamic range of any camera profiling target. For real world photography, forget about buying the “biggest and best” camera profiling target and consider more general matrix profile models.

I have found the simple and low-cost ColorChecker Classic 24 to be satisfactory for matrix profiles. The bigger ColorChecker Digital SG is expensive, but the increased number of patches may be useful for more detailed modelling (which I am yet to attempt).

ColorChecker charts are made using multiple different pigments, giving reflective spectra that are more representative of the real world. They are not printed and I don’t expect there are any fluorescent whitener additives in the pigments. I sold my Coloraid IT8.7/2 target because the Kodak photographic paper has fluorescent brightening additives.