Robust automatic jitter and smoothed bootstrap in R

July 3, 2017

The jitter {base} R function is useful for smoothed bootstrapping, but manual tuning of the jitter amount is often necessary. Should we change the jitter factor, select the alternative jitter amount estimator or directly specify the jitter amount? I thought of using order statistics for robust estimation of the jitter amount using the range-based estimator. This robust automatic jitter is demonstrated for two smoothed bootstrap examples.

Robust automatic jitter

From the R documentation, if jitter amount = NULL (which is the default) then amount = factor * d/5 where d is the smallest difference between adjacent unique x values. This jitter amount is typically smaller than the alternative estimator examined in this post.

If jitter amount = 0 then amount = factor * (max(x) – min(x)) / 50. When the range (max(x) – min(x)) is large, the jitter amount is large. Knowing that the sample range is strongly affected by sample size, outliers and not a robust estimator of scale, I thought to use the interquartile range IQR(x) instead.

To investigate the relationship between the range and IQR(x), I assumed a normal distribution for x and simulated some data below. The sample range increases with the sample size and for smaller sample sizes the range is about three times IQR(x). Accordingly, the proposed robust jitter amount = 1*3*IQR(x)/50 ≈ IQR(x)/17 (with factor = 1).

n IQR Range Range/IQR
10 1.2 2.9 2.8
20 1.2 3.7 3.1
30 1.2 4.1 3.4
40 1.3 4.3 3.4
50 1.3 4.6 3.5
100 1.3 4.9 3.8
Interquartile range (IQR) and range for the normal distribution. For smaller sample sizes, the range is about three times the IQR.

Here’s the code for the above table:

res=matrix(nrow=99, ncol=4) # results matrix
print(paste(“n”,”IQR”, “Range”, “Range/IQR”)) # header
for (i in c(10,20,30,40,50,100)){ # sample sizes
   for (j in 1:99){ # simulated data sample sizes
     x=rnorm(i)
     res[j,1]=I # sample size
     res[j,2]=IQR(x) # inter-quartile range
     res[j,3]=max(x)-min(x) # range
     res[j,4]=(max(x)-min(x))/IQR(x) # ratio of above
   }
   # print results for sample size i
   print(paste(min(res[,1]), round(mean(res[,2]),1), round(mean(res[,3]),1), round(mean(res[,4]),1)))
}

Two examples

In the two examples below, the objective was to estimate the shift in location for some paired data that were strongly skewed and included outliers. I used the median paired difference as a robust estimator for the shift in location and bootstrapping to estimate the confidence interval for the median paired difference.

For small samples, the bootstrap distribution for the median often appears with “holes” in it because the median of a bootstrap resample is usually one of the few observations in the middle of the original sample. Adding a small amount of jitter to each resample helps to fill the “holes”. Adding too much jitter results in wide bootstrap distributions. The examples below show that the proposed automatic jitter is robust against outliers and offers a reasonable compromise between smoothing and detail.

The bootstrap distribution is fairly smooth (bias = +0.04, Percentile 95% CI = −0.8450, −0.1000), jitter amount = NULL had little effect, jitter amount = 0 is perhaps too smooth (bias = +0.03, 95% CI = −0.8750, −0.0816), robust jitter is intermediate (bias = +0.05, 95% CI = −0.8141, −0.0956).

The bootstrap distribution is “holy” and percentile confidence intervals are confined to the spikes (bias = −0.08, Percentile 95% CI = −0.79, −0.13), jitter amount = NULL had little effect, jitter amount = 0 is too smooth because the range (max(x) − min (x)) was inflated by outliers in the data (bias = −0.12, 95% CI = −0.7708, −0.0805), robust jitter has the narrowest confidence interval (bias = −0.09, 95% CI = −0.7597, −0.0776).

Here’s some of the R code for the above plots:

## packages
library(MASS) # for truehist
library(boot) # for bootstrapping

## median functions for boot
# median difference function
dif.median1=function(data, i) {median(data$x1[i]-data$x2[i])}
# median amount=NULL jittered difference function for boot
# this is the R default
dif.median2=function(data,i) {median(jitter(data$x1[i]-data$x2[i]))}
# median amount=0 jittered difference function for boot
# this is the S default
dif.median3=function(data,i) {median(jitter(data$x1[i]-data$x2[i], amount=0))}
# median robust jittered difference function for boot
dif.median4=function(data,i) {median(jitter(data$x1[i]-data$x2[i], amount=myamount))}

## bootstrap median difference
# replace dif.median1 with any of the above median functions
dif.boot=boot(mydata,dif.median1,R=999)
mean(dif.boot$t) # bootstrap estimate, should be close to the median difference
boot.ci(dif.boot) # bootstrap confidence intervals
truehist(dif.boot$t, nbins=”FD”)
abline(v=median(dif.boot$t), lty=”dashed”)


Long exposure digital photo noise

July 4, 2016

For digital cameras, long exposures are affected by ‘hot pixels’ and ‘ noise’. This post investigates long exposure sensor noise using Raw Therapee for my new Canon EOS 100D  (Rebel SL1 / Kiss X7) digital SLR camera. All photos in this post were intentionally shot with the lens cap on!

Taking the images

Dark-frames (i.e. photos with the lens cap on) are long exposures that record long-exposure sensor noise.

Relevant camera settings were:

  • Daylight white balance (fixed white balance for comparisons between images).
  • Manual exposure with varying shutter speed. Lens aperture is not relevant to long exposure sensor noise.
  • Fixed ISO sensitivity (ISO 100 to start with).
  • Long exposure noise reduction OFF.
  • High ISO noise reduction OFF.

Processing the images

Processing parameters in Raw Therapee version 4.1.0 were:

  • Neutral profile (no adjustments).
  • Camera white balance (Daylight white balance, see above).
  • Input Profile = No profile
  • Output Profile = RT_sRGB (i.e. with gamma adjustment, like in real world photos.)

Each image was cropped to the same 400 × 400 pixel square that was selected to contain hot pixels. Results were exported as 8-bit TIFF files. The very small image size reduced further processing loads.

Images were converted into text files using ImageMagick. Text files were cleaned up and saved as .csv text files using LibreOffice Calc. Statistical analysis of the data used R. It was easier to evaluate the results using graphs rather than images (see example below).

Example crop of a dark-frame at ISO100 and 1/60 seconds (raw photo processed in Raw Therapee). A red hot pixel is 'just-noticeable' near the centre and with an amplitude of about 60 (having a display with low black level and turning off the lights helps). I used a 'just-noticeable' noise threshold of 50 (at 8-bits).

Example crop of a dark-frame at ISO100 and 1/60 seconds (raw photo processed in Raw Therapee). A red hot pixel is ‘just-noticeable’ up and left of centre, with an amplitude of about 60 (at 8-bits). Having a display with low black level and turning off the lights helps. I used a ‘just-noticeable’ noise level of 50.

Exposure time effect

I shot a series of dark-frames with increasing exposure. I let the sensor cool down for at least 12 minutes between shots.

Dark-frame results from 1/60 seconds to 16 minutes (2-stop increments) at ISO100. 160 000 individual Red, Green and Blue pixels are plotted in sequence as read by ImageMagick (horizontal axis). Pixel values can be read on the vertical axis.

Dark-frame results from 1/60 seconds to 16 minutes (2-stop increments) at ISO100. 160 000 individual Red, Green and Blue pixels are plotted in sequence as read by ImageMagick (horizontal axis). Pixel values can be read on the vertical axis. The dashed line is a ‘just noticeable’ level.

Hot pixels stand out above the background noise level –  they appear much brighter than they should. Hot pixels are a type of fixed-pattern noise – they always appear in the same location. Hot pixels result from leakage currents into sensor wells – they are manufacturing defects.  The (linear) level of hot pixels increases linearly with exposure time.

In the above series, red hot pixels can be seen increasing in amplitude until they saturate (255 at 8-bits). There were no blue hot pixels and no green hot pixels. Green hot pixels are rare for Bayer filter sensors, for which there are two green pixels for each red and blue pixel. I suggest that it is unlikely to have two adjacent green pixels defective.

I was disappointed to see hot pixels appearing from 1/60 seconds. Another Canon 100D of mine did not show hot pixels before 1 seconds exposure time.  My old Canon 400D (Digital Rebel XTi / Kiss Digital X) did not show hot pixels before 3 or 4 seconds exposure time. As pixel counts and pixel densities increase, I expect that sensor defects become more difficult to control.

The background noise level is dark current noise (also called ‘thermal noise’). Dark currents are small electric currents that occur even when no photons are arriving at the sensor. Dark currents are due to electrons dislodged by random thermal activity. Dark currents increase with temperature. Sensor temperature increases towards the end of long exposures.

In the above series, background noise increased strongly at four and 16 minutes. This noise includes dark current noise and additional hot pixels. Hot pixel noise also increases with temperature.

ISO effect

For each ISO, I shot a series of dark-frames with increasing exposure. I let the sensor cool down for at least 12 minutes between shots. Here, I present results only for one second and 4 minutes exposures.

Dark-frame results from ISO100 to ISO12800 at one second exposure.

Dark-frame results from ISO100 to ISO12800 at one second exposure.

Dark-frame results from ISO100 to ISO12800 at 4 minutes exposure.

Dark-frame results from ISO100 to ISO12800 at 4 minutes exposure.

Increasing ISO amplifies the signal before analog-to-digital conversion. Amplifying the signal will also amplify noise.

In the above series, hot pixel intensities and dark current noise increase at higher ISOs.  ISO greater than 800 should be avoided.

Interestingly, some strong hot pixels vanished above ISO 1600. I suspect there is some in-camera faking for ISO 3200 to ISO 12800 – these images appeared coarser-grained than lower ISO images. The high ISO noise reduction camera setting had no effect on images produced from the raw files.

Temperature effect

I shot a cold dark-frame in the morning, packed my camera in a black camera box and placed that in the sun for a few hours. The camera felt warm when I removed it from the camera box and shot a hot dark-frame. For both shots, I believe the camera sensor temperature was very close to the ambient temperature. Temperature was measured using a ‘fridge thermometer’ probe.

Dark-frame results at 12°C and 32°C, four minutes exposures at ISO100.

Dark-frame results at 12°C and 32°C, four minutes exposures at ISO100.

In the above comparison, background noise increased strongly from 12°C to 32°C. It’s much better to shoot long exposures in cool conditions.

Dark-frame subtraction

Hot pixel noise has a fixed-pattern and can be removed by subtracting a dark-frame. I prefer to do perform dark-frame subtraction in Raw Therapee. In-camera long exposure noise reduction can waste a lot of time shooting dark-frames with every photo.

From the preceding results and discussion, we need to match the level of the hot pixels in the photo and in the dark-frame:

  • Shoot at the same temperature.
  • Shoot at the same exposure time.
  • Shoot at the same ISO.

I evaluated dark-frame subtraction at four-minutes exposure time, for which there were moderate to large numbers of hot pixels at all ISO settings.

Dark-frame subtraction results from ISO100 to ISO800 at 4 minutes exposure.

Dark-frame subtraction results from ISO100 to ISO800 at 4 minutes exposure.

In the above series, dark-frame subtracted images had zero to very low noticeable noise up to ISO400. Dark-frame subtraction also was effective at ISO800, however there were many low level noisy pixels in the dark-frame subtracted ISO800 image.

Dark current noise results from random processes and cannot be removed by subtracting a dark-frame. In fact, for uncorrelated variables, the variance of the sum or difference is the sum of the variances (dark-current noise is random and uncorrelated). In the above series and at ISO100 in particular, observe that the background noise actually increased after dark-frame subtraction.

Shot-interval effect

When shooting multiple long exposures at very short intervals, the camera sensor could experience a cumulative heating effect similar to one very long exposure.

I shot sets of three four minute exposures at increasing shot intervals, using a plug-in intervalometer to control the camera shutter. I let the camera sensor cool down for about 16 minutes between each set. I compared noise levels for the third shot in each set.

Dark-frame results at shot intervals from zero to four minutes, three four minutes exposures at ISO100 in each set.

Dark-frame results at shot intervals from zero to four minutes, three four minutes exposures at ISO100 in each set.

In the above series, the zero second interval result is a single 12 minute exposure (4 + 0 + 4 + 0 + 4 minutes) and very noisy. Noise levels were much lower at shot intervals from one second to one minute and slightly lower again at a four minute shot interval. I was using a slow SD card and the real shot interval may have been longer than the intervalometer setting at one second. Nonetheless, these results clearly indicate that the sensor cooled rapidly. Shot intervals of just a few seconds are adequate.

Minimising long exposure noise

Following the above analysis, I classify long exposures into three groups:

  • Shorter long exposures from one second to about one minute. A dark-frame is required for removing hot-pixel noise. Dark current noise is not usually an issue.
  • Medium long exposures from about one minute to four minutes. With low ISO and avoidance of sensor heating, dark current noise can be minimised and dark-frame subtraction is effective.
  • Very long exposures greater than about four minutes. Consumer digital cameras are not designed for such long exposures and dark current noise becomes excessive. Image averaging may be helpful.

Also recognise that ‘live view’ and digital video are long exposures. Taking a long time to set-up a shot in live view will heat up the sensor and increase noise. Recording video with a digital SLR camera for several minutes will heat up the sensor and can increase noise. However, downscaling from sensor resolution to video resolution will average away much of the noise.


DC cable selection guide

June 11, 2016

I have recently installed solar and dual bettery systems on two buses, refreshed my own installation and am starting to think like an electrical engineer. After some extensive research, I have produced some cable selection tables for my toolbox. These tables can be used for auto electrics, marine electrics and small solar power systems. Before presenting the tables, I discuss the data sources.

Ampacity

Ampacity (amps capacity) is the maximum continuous current an electrical cable can carry without the insulation melting. Some major factors that determine ampacity are:

  • Cable size (small conductors have high resistance and become hot at low currents).
  • Insulation temperature rating.
  • Ambient temperature (heat dissipation from hot cables is slow at high ambient temperatures).
  • AC or DC current.

Also beware that old cables with corroded conductors and weathered or perished insulation will not perform like new.

There can be large differences between ampacity data from different sources. The following plot compares five standards. These curves show that AC ratings tend to be lower than DC ratings (e.g. NEC versus ABYC). For DC applications, the JASO and ABYC ratings are similar and the ISO ratings are about 15% lower. I selected the ABYC data, which seems reliable and covers a broad range of cable sizes and conditions.

Ampacity versus conductor area for copper power cables with 90°C insulation, at 30°C ambient temperaute. The American Boat and Yacht Council E-11 data I found at Blue Sea (marine small craft, DC). The JASO D609 data are from the manufacturer Tycab (automotive, DC). The ISO 10133 data I found at Energy Solutions LINK (marine small craft, < 50 VDC). The NFPA 70 NEC 2014 data are from Wikipedia (AC, ≤ 3 conductors). The AS/NZS 3008 data are from the manufacturer Olex (single-phase AC, single conductor).

Ampacity versus conductor area for copper cables with 90°C insulation, at 30°C ambient temperature. The American Boat and Yacht Council E-11 data I found at Blue Sea Systems (marine small craft, DC). The JASO D609 data are from the manufacturer Tycab (automotive, DC). The ISO 10133 data I found at Energy Solutions (marine small craft, < 50 VDC). The NFPA 70 NEC 2014 data are from Wikipedia (AC, ≤ 3 conductors). The AS/NZS 3008 data are from the manufacturer Olex (single-phase AC, single conductor).

Resistance

Voltage losses usually decide power cable size rather than ampacity. Resistance is largely determined by copper cross-sectional area. The following plot compares DC resistance data from five sources and the differences are not substantial. I selected the ABYC data, which covers a broad range of cable sizes.

Resistance versus conductor area for copper power cables, for 0 to 35 mm2. The American Boat and Yacht Council E-11 data (30°C) data I found at Blue Sea. The Tycab (20°C) data are from the manufacturer. The IEC 60827 data (20°C) I found at myElectrical Engineering LINK. The NEC data (20°C, solid core) are from Wikipedia. The Olex data are from the manufacturer.

Resistance versus conductor area for copper cables. The ABYC E-11 (30°C) data I found at Blue Sea Systems. The Tycab (20°C) data are from the manufacturer. The IEC 60827 data (20°C) I found at myElectrical Engineering. The NEC data (20°C, solid core) are from Wikipedia. The Olex data are from the manufacturer.

Cable ratings table

I merged all data for common power cable sizes into one reference table below. For cables not in the ABYC E-11 table (e.g. ‘auto cables’), I estimated ratings from quadratic curves fitted to the ABYC data.

Area DC Ampacity (A) DC Resist.
Cable (mm2) 30°C 60°C (ohm/km) Terminal
0.5 mm2 0.5 8 6 36.04 RED
20 AWG 0.52 8 6 34.65 RED
2 mm auto 0.56 9 7 32.18 RED
2.5 mm auto 0.64 9 7 28.16 RED
0.75 mm2 0.75 10 7 23.92 RED
18 AWG 0.82 10 8 21.88 RED
1 mm2 1 13 10 17.94 RED
3 mm auto 1.13 14 10 15.95 RED
16 AWG 1.32 15 11 13.70 RED
1.5 mm2 1.5 16 12 11.96 RED / BLUE
4 mm auto 1.84 18 14 9.79 BLUE
14 AWG 2.1 20 15 8.63 BLUE
2.5 mm2 2.5 21 16 7.18 BLUE
5 mm auto 2.9 24 18 6.21 YELLOW
12 AWG 3.3 25 19 5.42 YELLOW
4 mm2 4 34 25 4.49 YELLOW
6 mm auto 4.59 38 29 3.93 YELLOW
10 AWG 5.32 40 30 3.41 YELLOW
6 mm2 6 53 40 2.99 YELLOW
8 AWG 8.5 65 49 2.14
10 mm2 10 79 60 1.79
6 AWG 13.5 95 71 1.35
16 mm2 16 105 79 1.12
4 AWG 21.3 125 94 0.85
25 mm2 25 141 106 0.72
2 AWG 33.7 170 128 0.51
35 mm2 35 173 130 0.51
Cable ratings chart sorted by wire size (white = IEC/ISO cables, yellow = AWG cables, grey = ‘Auto cables’). Ampacity for 30°C and 60°C (‘engine room) ambient temperatures. Most data are from ABYC E-11 found at Blue Sea Systems. Auto cable cross-sectional areas are from the manufacturer Tycab. Blue values are from quadratic interpolation of the ABYC data. Red values are extrapolated. I have also added a column for insulated crimp terminal selection.

The most common insulation for copper power cables seems to be PVC, which is rated for 75°C conductor temperature (V-75). Usage up to 90°C (V-90) is limited. I selected ABYC data for 75°C insulation.

Ampacity should be derated for ‘engine room’ conditions (e.g. inside the engine bay of a vehicle). The ABYC data decrease ampacity by 25% at 60°C ambient temperature. Tycab recommend a larger derating, 40% at 60°C, perhaps because they provide ampacities for V-90 insulation.

When routing cable inside an engine room or engine bay, be careful not to run PVC insulated cables near exhausts, cyclinder heads, radiators and other hot parts > 75°C. PVC insulation will melt at high temperatures!

To use the above chart, one needs to identify the cable size and insulation:

  • Sometimes, the insulation of a power cable is marked with the conductor area (mm2 or AWG) and temperature rating of the insulation (°C or maybe °F).
  • When buying cable off the reel, a label on the reel should identify the conductor area (mm2 or AWG) and insulation type (e.g. V-75).
  • For scrap pieces of cable, I have to judge whether the insulation is PVC or not and then estimate the copper area using a wire stripper. PVC insulation is ‘opaque’ (never clear), ‘firm’ (not soft), fairly tough (when cutting and stripping) and softens when heated lightly (but doesn’t burn, melt away or shrink by a large amount).

Beware that the size of ‘auto cables‘ refers to the total cable diameter, including the insulation. Plastic is cheaper than copper and the cross-sectional conductor area of auto cables may not be well standardised. For Australia, Electra auto cables and Tycab auto cables have the same cross-sectional area as in the above table. Avoid buying cable from auto parts stores. Better cable and better prices are found at electrical trade stores and sometimes on Ebay.

Also note that ‘gas cable’ is double-insulated cable for hazardous applications. It’s better than single insulated cable, but is slightly more expensive.

Voltage losses and cable selection charts

Resistance in a power circuit results in voltage losses and less power reaching the load (DC power = voltage × current). For example, the compressor in my Evakool fridge may have starting problems and runs slower when voltage is low.

Voltage loss is a function of current and resistance (Ohm’s law: volts = amps × ohms). Cable resistance is a function of cable length and cross-sectional area (ohms = ohms/m × m; see the cable ratings table above).

Below are some cable selection tables for both 12 V and 24 V systems. I calculated these tables in five steps:

  1. Calculate maximum cable resistance (ohms/km) from voltage drop, cable length and current.
  2. Calculate minimum copper conductor cross-sectional from cable resistance (using power function fitted to ABYC data, see above).
  3. Calculate ampacity for 75°C insulation and 60°C ambient temerature (using quadratic function fitted to ABYC data, see above).
  4. Upgrade results smaller than 0.5 mm2 to 0.5 mm2.
  5. Upgrade results with ampacity less than the current in Step 1 above.

These cable selection tables specify the minimum conductor area. An appropriate cable can then be selected by reference to the cable ratings table above.

mm2 1 2 5 10 15 20 30 40 50 75 100 A
1 0.5 0.5 0.5 0.5 1.0 2.1 4.0 5.3 6.0 10.0 16.0 25.0
2 1.0 0.5 0.5 0.8 1.5 2.3 4.0 5.3 6.0 10.0 16.0 25.0
5 2.5 0.5 0.8 1.9 3.8 5.6 7.5 11.3 15.0 18.8 28.2
10 5.0 0.8 1.5 3.8 7.5 11.3 15.0 22.5 30.0
15 7.5 1.1 2.3 5.6 11.3 16.9 22.5 33.8
20 10.0 1.5 3.0 7.5 15.0 22.5 30.0
25 12.5 1.9 3.8 9.4 18.8 28.2
m Single m Twin
Cable selection chart for 12 V systems and 2% voltage loss (sensitive loads). Read minimum cross-sectional copper area (mm2) at the intersection of current (top row) and cable length (one of the two left-rows). For single core cables, it is assumed the chassis return has zero resistance.
mm2 1 2 5 10 15 20 30 40 50 75 100 A
1 0.5 0.5 0.5 0.5 1.0 2.1 4.0 5.3 6.0 10.0 16.0 25.0
2 1.0 0.5 0.5 0.5 1.0 2.1 4.0 5.3 6.0 10.0 16.0 25.0
5 2.5 0.5 0.5 0.9 1.9 2.8 4.0 5.6 7.5 9.4 16.0 25.0
10 5.0 0.5 0.8 1.9 3.8 5.6 7.5 11.3 15.0 18.8 28.2
15 7.5 0.6 1.1 2.8 5.6 8.4 11.3 16.9 22.5 28.2
20 10.0 0.8 1.5 3.8 7.5 11.3 15.0 22.5 30.0
25 12.5 0.9 1.9 4.7 9.4 14.1 18.8 28.2
m Single m Twin
Cable selection chart for 12 V systems and 4% voltage loss (normal loads). Equivalent to 24 V and 2% voltage loss (sensitive loads). Read minimum cross-sectional copper area (mm2) at the intersection of current (top row) and cable length (one of the two left-rows). For single core cables, it is assumed the chassis return has zero resistance.
mm2 1 2 5 10 15 20 30 40 50 75 100 A
1 0.5 0.5 0.5 0.5 1.0 2.1 4.0 5.3 6.0 10.0 16.0 25.0
2 1.0 0.5 0.5 0.5 1.0 2.1 4.0 5.3 6.0 10.0 16.0 25.0
5 2.5 0.5 0.5 0.5 1.0 2.1 4.0 5.3 6.0 10.0 16.0 25.0
10 5.0 0.5 0.5 0.9 1.9 2.8 4.0 5.6 7.5 9.4 16.0 25.0
15 7.5 0.5 0.6 1.4 2.8 4.2 5.6 8.4 11.3 14.1 21.1 28.2
20 10.0 0.5 0.8 1.9 3.8 5.6 7.5 11.3 15.0 18.8 28.2
25 12.5 0.5 0.9 2.3 4.7 7.0 9.4 14.1 18.8 23.5
m Single m Twin
Cable selection chart for 24 V systems and 4% voltage loss (normal loads). Read minimum cross-sectional copper area (mm2) at the intersection of current (top row) and cable length (one of the two left-rows). For single core cables, it is assumed the chassis return has zero resistance.

Cable selection tips

When selecting electrical cables, biggest is not best. Bigger cables are more difficult to work with (e.g. routing, making connections), heavier and more costly. Use electrical engineering know-how and always select the thinnest cable that is fit for purpose. For example, you will not find many thick cables in OEM installations because they are not necessary for most applications.

When routing cables, shortest is not best. Leave a generous amount of extra length to allow for future work. Investing in a little extra cable is cheaper than having to replace a whole section of cable that is found to be too short or better than having to join two pieces of cable.

Exceed the ampacity and risk an electrical fire! Use appropriate fuses, especially for smaller cables with low current ratings.


Testing USB power banks

March 20, 2016

USB charging is very common in portable electronics today and USB ‘power bank’ batteries are popular for topping up devices on the go. Beware that some USB power banks have low capacity and charge slowly. This post explains how USB power banks work, how to test them using simple equipment and power bank sizing.

How USB power banks work

A common USB power bank contains Li-ion cells, a boost converter, a charge controller and, hopefully, a protection circuit (Li-ion cells usually do not have built-in protection).

Nominal cell voltage for Lithium nickel manganese cobalt cells is 3.7 V and maximum charging voltage is 4.2 V. Nominal USB voltage is 5 V. The boost converter steps up voltage from 3.7 V to 5 V output. The charge controller stops USB charging at 4.2 V.

The capacity rating for USB power banks usually refers to the Li-ion cells and is substantially greater than the output capacity at USB 5 V.

Block diagram of USB Power Bank. Measure points indicated with *

Block diagram of USB Power Bank. Measure points indicated with *

The output USB voltage is higher than the Li-ion cell voltage. The cell current is greater than the output USB current:

(Power out) = (Power in) × (Boost converter efficiency)
(Volts out) × (Current out) =  (Volts in) × (Current in) × (Efficiency)

Substituting nominal voltages:

5 × (Current out) =  3.7 × (Current in) × (Efficiency)
(Current out) =  3.7 × (Current in) × (Efficiency) ÷ 5
(Capacity out) = 3.7 × (Capacity in) × (Efficiency) ÷ 5

where capacity is measured in mAh. Substituting the nominal 3.7 V voltage is slightly inaccurate. The Li-ion voltage actually declines during discharge to about 3 V when empty (see example below).

Substituting DC-to-DC switching converter efficiency of between 75% and 98%:

(Capacity out) = 3.7 × (Capacity in) × (0.75 to 0.98) ÷ 5
(Capacity out) = (0.56 to 0.73) × (Capacity in)

This equation is important because the mAh at USB 5 V (see block diagram above for measuring points) is substantially smaller than the mAh drawn from the Li-ion cell.

Known-capacity device testing method

Charging a known-capacity device is the cheapest method to test discharge capacity of a USB power bank. My smart phone is new, the battery is new and the battery meter seems to be reasonably accurate:

Discharge capacity = (End % − Start %) × (Device capacity) ÷ 100
Start % = Device battery meter when a full-charged power bank is plugged in to the device.
End % = Device battery meter when the power bank is empty.
Device capacity is for 3.7 V nominal Li-ion cell voltage, the same voltage as the power bank cell.
The factor 100 is used to convert percentages to fractions.

Average discharge rate = (Discharge capacity) ÷ (Discharge duration)

To discharge in one step, the device capacity should be greater than the USB power bank capacity at 5 V (see previous section). The device should be switched off, so that it drawing current for charging and little else. This method will be unreliable for devices with old batteries having reduced and unknown capacity. This method does not measure voltage.

USB meter testing methods

USB meters can be used to measure both discharging and charging. I purchased a simple USB voltage and current meter on ebay for AUD 2.

Simple USB voltage and current meter.

Simple USB voltage and current meter.

To measure capacity, I record time, voltage and current at regular intervals, enter the data in a spreadsheet and integrate current over time. Longer sampling intervals can be used for high capacity batteries. Accuracy is limited by the meter accuracy (±2 % current accuracy for my meter) and sampling frequency.

I later purchased a USB voltage, current and capacity meter for 5 AUD, which integrates current over time and reports capacity.

Integrating USB voltage, current and capacity meter.

Integrating USB voltage, current and capacity meter.

Example: Laser 2200 mAh Powerbank

The Laser 2200 mAh USB power bank is popular in Australia, where it retails for between AUD 8 and AUD 15.

The Laser 2200 mAh USB power bank contains one 18650 Li-ion cell. Maximum current ratings are 1 A output and 0.5 A input at 5 V. I removed the lid to measure cell voltage with a multimeter.

The Laser 2200 mAh USB power bank contains one 18650 Li-ion cell. Maximum current ratings are 1 A output and 0.5 A input at 5 V. I removed the lid to measure cell voltage with a multimeter.

I discharged the Laser power bank into my phone. Average output voltage of 4.96 V was very close to 5 V nominal USB voltage. Average current of 684 mA was typical for my phone. Maximum current of 870 mA was close to rated 1 A output for this power bank. Integrated discharge capacity was 1254 mAh. My phone’s battery meter indicated 1383 mAh.  Another discharge test with an integrating meter gave 1230 mAh. Converting to 3.7 V nominal Li-ion cell voltage and assuming 75% to 98% boost converter efficiency (see introduction above), the estimated cell capacity was 1718 to 2240 mAh. This range includes the 2200 mAh rated capacity.

Discharge curve for Laser 2200 mAh USB power bank. Integrated discharge capacity was 1254 mAh at an average output voltage of 4.96 and current 684 mA.

Discharge curve for Laser 2200 mAh USB power bank. Integrated discharge capacity was 1254 mAh. USB voltage should be between 4.75 and 5.25 V.

I charged the Laser power bank with a mains charger (maximum rated output current 1 A at 5 V). The average charging current of 503 mA agreed with 500 mA rated charging current. Integrated charging capacity was 2305 mAh (no voltage conversion is applied to charge capacity because current out = current in). Assuming 80% to 90% Li-ion charge/discharge efficiency, the estimated cell capacity was 1844 to 2075 mAh, which agrees with the discharge capacity.

This Laser power bank performed as rated but the low capacity charges my phone only 50%. Nearly identical USB power banks can be purchase on ebay for AUD 1, excluding the battery. High capacity 2900 to 3600 mAh Panasonic 18650 batteries cost about AUD 10. We can buy these componenets and assemble a higher capacity power bank for about the same price as the 2200 mAh Laser power bank.

Sizing USB power banks

From the introduction:

(Capacity out) = (0.56 to 0.73) × (Capacity in)

This capacity equation can be used for sizing USB power banks (i.e. capacity of the Li-ion cells). Li-ion charge/discharge efficiency is between 80 and 90%:

(Device capacity) = (0.80 to 0.90) × (Capacity out)
(Device capacity) = (0.80 to 0.90) × (0.56 to 0.73) × (Rated capacity)
(Device capacity) = (0.80 × 0.56  to 0.73 × 0.90) × (Rated capacity)
(Rated capacity) = (1.53 to 2.25) × (Device capacity)

An easy to remember rule of thumb is (USB power bank rated capacity)= 2 × (Device capacity). Now you understand why USB power banks often seem to be inadequate!


Could Activ Energy rechargeable batteries be Eneloop killers?

February 6, 2016

All of my rechargeable nickel-metal hydride AA cells are Eneloops. I was surprised to find that Activ Energy cells from ALDI give similar performance at half the price.

Active Energy NiMH rechargeable AA cell.

Active Energy NiMH rechargeable AA cell.

Product description

The Activ Energy AA cells tested were 2400 mAh capacity, low self-discharge and “extra long life”. Here is a comparison with Eneloop specifications:

Cell Origin Date Min. cap. (mAh) Max. cycles Diam. (mm) Length (mm) Mass (g)
Activ Energy MSN 5004 9143 China Jan. 2015 2400 14.4 50.4 27
Sanyo Eneloop HR-3UTGA (2nd Gen.) Japan 1900 1500 14.3 50.4 27
Panasonic Eneloop BK-3MCCE (4th Gen.) China 1900 2100 14.3 50.2 26
Panasonic Eneloop Pro BK-3HCC Japan Feb. 2014 2450 500 14.4 50.3 30
Activ Energy AA cells compared to Eneloops. Minimum rated capacity reported. The standard eneloops I own and measured. The Panasonic Eneloop Pro dimensions and mass are from lygte-info.dk

The mass of the Activ Energy AA cell is about the same as for 1900 mAh Eneloops and lower than 2450 mAh Eneloop Pros. The capacity of the Activ Energy cells appears over rated in this comparison.

Maximum cycles was not specified for the Activ Energy AA cells. I assume maximum cycles is about 500, like Chinese Eneloops

Self-discharge performance was not specified for the Activ Energy AA cells. I assume 85% capacity retention in one year, like the Eneloop Pros and 2nd generation Eneloops.

The Activ Energy packaging included an LGA tested quality mark. It was not explained what product features were tested.

Capacity

I don’t have a battery tester. I discharged the cells using a 2.4 ohm resistor and recorded voltage and current at regular intervals. Discharge current was about 400 mA and discharge duration was about 5 hours. I stopped the discharge when voltage decreased below 0.9 V under load. Estimates of capacity are imprecise because of low sampling frequency and variable end-points.

I charged the cells with an Olympus Ni-MH Battery Charger BU-100 (around 2002 or 2003 vintage) at 490 mA per cell.

The Activ Energy cells were tested in the following sequence:

  1. Initial charge (top-up).
  2. Discharge 1.
  3. Recharge.
  4. Discharge 2.
  5. Recharge and one week rest.
  6. Discharge 3.

For comparison, I also tested one new Panasonic Eneloop and one well-used Sanyo Eneloop:

Capacity (mAh)
Cell Test Mean Low High
Activ Energy 1 1830 1743 1899
Activ Energ 2 1950 1880 2120
Activ Energ 3 1897 1849 1944
Panasonic Eneloop (new) 1 1755
Sanyo Eneloop (used) 1 1708
Discharge capacities. Four Active Energy cells were discharged three times. One Sanyo Eneloop and one Panasonic Eneloop were tested one time.

Average discharge capacity for Activ Energy cells was similar to Eneloops. Other tests of Activ Energy cells have reported mean capacities of 2045 mAh after a few break-in cycles (previous generation, 2300 mAh cells) and 2220 mA. Altogether, these tests show 200 to 500 mA less than rated capacity.

Voltage

The Activ Energy cells had flat discharge curves and mid-discharge voltage was close to nominal 1.20 V for NiMH cells.

Discharge curves for Active Energy Cell 1. Discharge test 3 had the highest voltage.

Discharge curves for Active Energy Cell 1. Discharge test 3 had the highest voltage.

Discharge curves for Active Energy Cell 1. Discharge test 3 had the highest voltage.

Discharge curves for Active Energy Cell 2. Discharge test 3 had the highest voltage.

Discharge curves for Active Energy Cell 3. Discharge test 3 had the highest voltage.

Discharge curves for Active Energy Cell 3. Discharge test 3 had the highest voltage.

Discharge curves for Active Energy Cell 4. Discharge test 3 had the highest voltage.

Discharge curves for Active Energy Cell 4. Discharge test 3 had the highest voltage.

Self discharge

Activ Energy cell voltages initially were 1.29 V at 12 months after manufacture. This was not far below freshly charged 1.42 V and well above nominal 1.20 V. Although I did not measure initial capacities, these voltage measurements strongly indicate that the Activ Energy cells are low self-discharge cells.

Referring to the previous section, discharge capacity after one week of rest was 3% lower than the preceding test. Another test of Activ Energy cells reported 3% capacity loss in 5 days and 8% capacity loss in one month. The self-discharge rate declines over time and maybe these Activ Energy cells can retain 80% of their capacity after one year.

Cost comparison

In four-cell packs, Activ Energy AA cells (AUD 1.75 per cell) cost less than the Australian retail price of Chinese Eneloop AA cells (about AUD 5.00 per cell). However, I am not confident about cycle life for either of these products. I would choose Activ Energy cells if I urgently needed to buy NiMH cells. I would choose Activ Energy cells over Chinese Eneloops if cycle life was similar. The best choice remains quality Japanese Eneloops.


Simple nickel-metal hydride solar charger

February 6, 2016

Nickel-metal hydride rechargeable batteries can be recharged with a simple trickle charger. I have made a one AA solar charger for my headlamp and a two AA solar charger for other devices.

solar_circuit

Solar trickle charger wiring diagram. The blocking diode may not be required if reverse current is very small.

Caution: The simple charger circuit described here is not for charging cells in parallel. When charging cells in parallel, I think there should be some technology to distribute currents and balance charge among the cells.

Charging current

Self-catalysis can recombine gas formed when NiMH cells are overcharged if the charging current is less than 0.1 × capacity.

The cells I want to charge are 1900 mAh AA Eneloops:

Safe trickle charging current = 0.1 × 1900 = 190 mA.

The same charging rate would apply for these cells in series (where each cell experiences the same charging current).

Caution: The simple charger circuit described here is not for charging mismatched cells in series (e.g. cells with different capacities, different ages or different states of discharge). Matched cells should be used and assigned to specific devices. This will maximise cycle life (i.e. avoids one bad cell affecting the good cells) and allow safe series charging.

Charging voltage

Cell voltage increases towards full-charge and maximum charging voltage for a NiMH cell is 1.4 to 1.6 V.

Solar panels are over-rated at standard testing conditions. Voltage decreases by about 25% at typical operating temperatures and solar irradiance:

Maximum charging voltage = 1.5 V
Solar panel (maximum power) voltage = 1.5 ÷ (1 − 0.25) = 2 V (at standard testing conditions).

The minimum solar panel is about 2 V and 200 mA = 2 V × 0.2 A = 0.4 W. I have some 2 V solar cells and they are capable of charging NiMH cells.

Voltage upgrade

Low voltage circuits are sensitive to voltage losses. A higher solar panel voltage is recommended to overcome contact resistances at the battery terminals and to allow a blocking diode in case of reverse-current leakage.

Schottky diodes have low forward voltage drop of 0.15 to 0.45 V:

Schottky diode forward voltage = 0.3 V.
Solar panel voltage = 2 + 0.3 = 2.3 V.

From the solar panels available to me, I selected a 3 V 260 mA (= 0.8 W panel). These ratings are greater than the 1.4 to 1.6 V maximum charging voltage and 190 mA safe trickle charging current for AA Eneloops. However, self-catalysis handles gases produced by over-charging and I have not observed over-voltage or high cell temperatures using these panels.

My one AA solar charger

1AAcharger

My solar trickle charger for one AA NiMH battery

My one AA solar charger is assembled from the following components:

  • 3 V 260 mA solar panel
  • 1N5819 Schottky diode (rated for 40 V, 1 A)
  • Twin-core cable
  • 12 mm, 2-pin “aviation plug”
  • Small project box
  • AA battery holder

The diode forward current and reverse voltage ratings should be greater than the maximum charging current and battery voltage respectively. I soldered the positive leg of the diode to the panel. I covered the solar panel contacts with acrylic sealant. Acrylic sealant is opaque, dries harder than RTV silicon and is easier to peel and scrape off if necessary.

The solar panel has a two metre lead, so that the battery can be kept out of the sun and rain. The detachable lead is for convenience in packing away the charger.

The project box provides a mount for the aviation plug and protects the cheap and flimsy AA battery holder. I inserted rubber spacers to help support the ends of the battery holder. A rigid battery holder with stronger spring contacts would be an improvement.

There is no fuse because voltage is low and contact resistances limit the the short-circuit current. I measured about 2.5 A when shorting the battery holder leads. Heating of the battery holder contacts and leads was insufficient to result in a fire.

A battery tester is helpful to check the state of charge before and after charging. An ammeter would be useful to check charging rates, but is relatively expensive.

AAtester

Battery tester bought on ebay.

In strong sunlight and with the panel facing the sun, my charger delivers about 260 mA. In weak, afternoon sunlight the current is about 65 mA. Charging current does not vary with cell voltage because the solar cell is operating in its plateau region.

solar_parameters

Solar panel operating parameters. Current changes very little for voltages less than maximum power voltage.

A one-day charge resulted in 1460 mAh of capacity (77% of 1900 mAh rated capacity). A full-charge could be achieved with 340 mA panels or by moving the panel to track the sun or with an extra 2 hours of charging.

My two AA solar charger

2AAcharger

My solar trickle charger for two AA NiMH batteries.

My two AA solar charger uses two 3 V 260 mA panels in series (6V 260 mA) and a two AA battery holder (charging voltage 2.8 to 3.2 V).

Compared to spare batteries

Weight and cost are relevant criteria for choosing between a solar charger and carrying spare batteries.

The cost of my one AA solar charger was AUD 4.59 without the battery tester or AUD 6.57 with the tester; approximately equivalent to the cost of two Eneloops. The solar charger is cheaper than four Eneloops.

The mass of my one AA solar charger was 121 g with the lead; approximately equivalent to the mass of four Eneloops. For short trips, it is more convenient to bring spare batteries and no charger.


Zomei neutral density filters tested

March 19, 2015

I recently purchased a video-capable DSLR (Digital SLR) camera and wanted some ND filters (neutral density filters). I purchased some Zomei ND filters from ebay. Zomei is Chinese, but they make lots of filters and the products are nicely presented. There was very little feedback about Zomei on the web and so I wrote this review.

I found that the Zomei ND2 and ND4 filters were satisfactory. For the ND8 filter, exposure was inaccurate and colour errors were relatively large.

Filters tested

I tested Zomei ND2, ND4 and ND8 58 mm filters. The filters are well made and the threads are good. The filters are not multicoated.

Zomei ND2, ND4 and ND8 filters.

Zomei ND2, ND4 and ND8 filters.

I immediately noticed that the Zomei filters felt different, because they are made from high-density resin. Some Zomei filters use glass, but these ND filters were plastic! The ebay seller had incorrectly stated they were import optical glass (and promptly removed the listings after my negative feedback).

I also tested Kenko SMART ND8 52 mm and ND8 77 mm filters to compare with the Zomei results. The Kenko filters have a slim frame and are not multicoated. They were made in the Philippines, I think using glass from Japan.

Methods

I photographed an X-Rite ColorChecker Passport in sunlight on a clear afternoon (uniform and stable lighting is required) with a Canon EOS 350D DSLR and EF 85 mm f/1.8 lens. I set the base exposure (1/500 seconds shutter speed) with a 18% grey card and photographed the ColorChecker four times with fixed aperture f/8 and fixed ISO100:

  1. No filter and base exposure before.
  2. ND filter and base exposure.
  3. ND filter and exposure compensation (slower shutter speed: +1 stop for ND2, +2 stops for ND4 and +3 stops for ND8).
  4. No filter and base exposure after.

The remainder of this section is very technical and for specific software. You might prefer to jump over to the results.

Linear device RGB is required. I processed each image in Raw Therapee version 4.1 as follows:

  • Processing Profile = Neutral (disables most adjustments).
  • Input Profile = No profile (device RGB).
  • Output Gamma = linear_g1.0 (linear RGB).
  • Custom white balance on the third-lightest grey patch of the ColorChecker (any neutral light-grey patch should work).
  • For image 2, apply linear exposure White Point Correction (2× for ND2, 4× for ND4, 8× for ND8).
  • Crop to ColorChecker target.
  • Save as TIFF (16 bit).

Rather than applying exposure compensation to image 2, I could have simply used image 3. The shutter speeds should be accurate enough.

I used Argyll CMS to read the ColorChecker patches from each linear RGB TIFF image:

scanin -v -p -a -G1 -dipn input.tif ColorChecker.cht ColorChecker.cie
-v verbose.
-p compensate for perspective distortion.
-a chart orientation normal (not upside down).
-dipn generate diagnostic image.
-G1 Gamma encoding of image (linear).
input.tif is the input image and input.ti3 is the name of the output text file.
ColorChecker.cht is a chart recognition file.
ColorChecker.cie contains the the chart reference data (which is copied to the .ti3 output file).

I copied the scanin output to a spreadsheet. I compared white patch average RGB values with- and without filters (they should all be the same if lighting and exposure were the same).

Next, I used Argyll CMS to compute colour differences relative to the ColorChecker reference data. For this, I had to match the exposure of images 1 and 3 to minimise lightness differences increasing the colour differences. I used image 3 because it has a better signal to noise ratio than the underexposed image 2.

I had previously made my own camera profile for the Canon EOS 350D and matched white patch average RGB values to that profiling image. Alternatively, one could simply profile image 1 as the reference. After applying linear exposure White Point Corrections in Raw Therapee, I ran scanin again and then profcheck to evaluate colour differences:

profcheck -v2 -k image.ti3 profile.icc
-v verbosiy level 2.
-k report CIE Delta-E 2000 colour differences.
input.ti3 is the output from scanin.
profile.icc is the camera profile.

Argyll CMS profcheck takes the patch RGB values from the .ti3 file, applies the camera profile and computes colour differences with the chart reference data in the .ti3 file. I repeat: to minimise lightness errors, the exposure of the image must match the exposure of camera profiling image.

I copied the profcheck output to a spreadsheet. I compared average DE2000 differences with- and without- filters.

Exposure errors

After 3-stops exposure compensation, the Zomei ND8 filter was still underexposed −0.49 stop relative to the images without any filters. The ND4 filter was better (−0.17 stop) and the ND2 was good (+0.02 stop). The ND8 filter was effectively a ND11 filter (2^(3+0.49) = 2^3.49 = 11.2), i.e. the effect is stronger than 8×.

Zomei ND filters exposure errors. Exposure compensation was applied to the ND images (+1 stop for ND2, +2 stops for ND4, +3 stops for ND8) before making these comparisons. Numbers above the columns are exposure errors in stops = log(RGB/RGB_before) / log(2). The before and after measurements without any filters agree, which indicates that the lighting was constant during the tests.

Zomei ND filters exposure errors. Exposure compensation was applied to the ND images (+1 stop for ND2, +2 stops for ND4, +3 stops for ND8) before making these comparisons. Numbers above the columns are exposure errors in stops = log(RGB/RGB_before) / log(2). The before and after measurements without any filters agree, which indicates that the lighting was constant during the tests.

For comparison, exposure errors for two Kenko ND8 filters were much smaller at −0.14 and −0.12 stops.

Colour errors

The 3.3 average colour error for the Zomei ND8 filter exceeded a just noticeable difference (DE > 2) and the maximum was 6.4. Colour accuracy was satisfactory for the Zomei ND4 filter (average DE = 2.0) and good for the ND2 filter (average DE = 1.6).

Zomei ND filters colour errors. Numbers above the columns are average DE2000 colour errors (DE > 2 is a just noticeable difference). The before and after measurements agreed closely and average colour differences were small. This indicates: 1) the lighting (afternoon sun) was similar to the camera profile (based on morning sunlight) and, 2) the lighting was constant during the tests.

Zomei ND filters colour errors. Numbers above the columns are average DE2000 colour errors (DE = 2 is a just noticeable difference). The before and after measurements agreed closely and average colour differences were small (DE < 2). This indicates: 1) the lighting (afternoon sun) was similar to lighting for the camera profile (morning sunlight) and, 2) the lighting was constant during the tests.

For comparison, average colour errors for two Kenko ND8 filters were much smaller at 1.5 and 1.4 stops.

The following two charts highlight the colour differences when using the Zomei ND8 filter.

Colour differences for 24 ColorChecker patches. No filter. Most colour errors are small (DE < 2) and were similar to the camera profiling errors.

Colour differences for 24 ColorChecker patches. No filter. Most differences are small (DE < 2) and were similar to the camera profiling errors.

Zomei_colour2

Colour differences for 24 ColorChecker patches. Zomei ND8 filter. Many differences exceed a just noticeable difference (DE > 2).

Conclusion

We can use a ColorChecker chart, Raw Therapee and Argyll CMS to evalute ND filters.

The Zomei ND filters were increasingly inaccurate for higher densities. Results for the Zomei ND8 filter were unsatisfactory. This doesn’t mean that filter is useless, however it is not the 8× (3 stop) and neutral density (DE < 2) filter that we want. Results for Kenko filters showed that accurate ND8 filters are achievable.

Disregarding the ND8, I purchased two Zomei ND filters for AUD22 (AUD11 each). The Kenko filter was AUD19 (for 58 mm diameter). I think it is worth paying more for better quality glass ND filters.