Looks like GHCN v3 is no longer beta

I was just reading the GHCN v3’s changelog and the data set is no longer beta.


GHCNM v3.0.0 is officially released.  The “beta” status is removed.  Users may notice for the next week or so a few dangling references within dataset files that still contain the word “beta” (including documentation and web site).  We are working to remove those references.

How did I miss this? I’ll have to update my Fortran gridding program to accommodate the new official GHCN.

A Quick Preview of GISTEMP with GHCN v3-beta

Last week I checked the GISTEMP directory on the NASA-GISS FTP server to verify that I had the latest gridded binary files. I saw that I did, but I found files whose names implied they contained the GISTEMP analysis using the new beta version of GHCN. I downloaded the files, inspected their meta data and found that the data appeared to corresponded to GHCN v2 data. I sent an inquiry and was told that the GISTEMP file headers had not been updated and that the data did correspond to GHCN v3-beta. I was told that GISTEMP’s analysis should begin updating with GHCN v3-beta in a month or two. Until then, we can get a preview of what kind of changes to expect from GISTEMP relative to the current analysis. I’m not going to look into this in great detail because the data I’ve obtained may change substantially upon the official release.

To analyze the differences between the two GISTEMP analyses, I differenced both gridded data sets. I calculated global averages for the two analyses plus their difference in a way similar to GISTEMP’s zonal averaging, expect I didn’t implement the reference station method. All three global averages were calculated as a weighted average of four latitude zones: -90:-23.6, -23.6:0, 0:23.6, 23.6:90 with weights of 0.3, 0.2, 0.2 and 0.3, respectively. The figure below shows these three global averages as well as the linear trend in the differenced data.

I calculated linear trends over the entire length of the time series as well as three distinct warming/cooling periods:

  1. 1880-1939 – late 19th/early 20th century warming
  2. 1940-1974 – mid-20th century cooling
  3. 1975-2011 – modern warming period

The table below summarizes the trends (in °C/decade) for the three global averages for all four periods.

Data/Period 1880-2011 1880-1939 1940-1974 1975-2011
v2 0.0582 0.0562 -0.0402 0.2047
v3 0.0721 0.0804 -0.0206 0.2182
v3 – v2 0.0142 0.0271 0.0166 0.0109

I have previously looked at the two GHCN data sets and have found some increased warming in v3-beta relative the v2, so these results aren’t surprising. I also calculated grid cell linear trends for all three periods for the differenced data. First, we’ll look at the whole period 1880-2011.

There are many large patches of strong differential warming/cooling. Let’s remember that the GISTEMP analysis assimilates station data within 1,200 km around a given grid cell center. So all other things being equal, differences in the trend of one station will cause differential warming/cooling to spread out over 1,200 km with a linearly decreasing impact. Now look at the first period, 1880-1939.

These circular patches of differential warming are approximately (I eyeballed it using multiple maps) 1,200 km in radius. The second period, 1940-1974 gives us more sharply-defined patches.

There are four distinct alternating warming/cooling patches over the northern half of South America. Looking elsewhere, these 1,200 km sized patches a pretty numerous and well-defined. Now here’s the modern warming period, 1975-2011.

This image reminds me of Jupiter with its big red spot. Earth’s big red spot over Australia, while comparable to other hot spots elsewhere, is very coherent over the entire continent and suggests that a significant proportion of the v3-beta Australian data exhibits significantly more warming than its soon-to-be obsolete counterpart.

The next step would be to pick a few large areas of differential warming/cooling, extract the station data for both GHCN data sets and investigate further. I’ll leave that up to the few of you who are properly motivated as I have a couple of other projects competing for my time.

Base Period vs. Trend vs. Spatial Coverage

This post will address some issues I’ve known about for a while, but never got around to discussing formally or systematically. The issues are

  1. how the chosen base period of a spatially-averaged temperature time series affects the trend
  2. how the spatial coverage, which is a function of data coverage and base period choice, affects the trend.
  3. how adjusting the base period of one series of spatially-averaged anomalies to another base period affects its trend

This issue is very relevant because many blogs compare multiple global temperature time series that use different base periods so all but one time series has to be adjusted. Adjusting a time series of anomalies causes all of the anomalies for a specific month to move up or down by a constant amount. While the shifting is month-specific, the difference between the two series is an annual cycle which has no effect on the resulting trend.

For a single time series representing a single station or point on the Earth, adjusting the series incurs no error. However, when dealing with spatially-averaged anomalies, the logic behind adjusting the base period is erroneous. Consider a time series of spatially-averaged temperature anomalies relative to 1961-1990. We want to compare it to another time series of spatially-averaged anomalies that are relative to 1981-2010. To adjust the first series to the base period of the second series, we calculate the first series’ mean January anomaly from 1981-2010, the mean February anomaly from 1981-2010, etc. Then this long-term monthly mean series is subtracted from the first series. Now both series average to zero over 1981-2010. The problem is that there is an implicit assumption that the January anomalies from 1981-2010 all represent the same spatial extent. Furthermore, when subtracting this mean January anomaly from all the January anomalies, that same assumption is being made for each January anomaly. This means that the January 1880 anomaly represents the same spatial extent as the January 1956 or January 1987 anomaly. This is obviously wrong. But what difference does it make?

To investigate, I’ll use GHCN v2, CRUTEM 3 and GISTEMP’s (1,200 km) gridded data. I interpolated the GISTEMP data into a fixed-offset 5° x 5° grid to match GHCN and CRU’s spatial resolution. I calculated anomalies at the grid cell level relative to 1901-1930 up to 1981-2010 in steps of ten years and required that at least 20 years of data to be present to calculate a valid long-term mean. To properly calculate the spatial averages of these land-only data, I used a land mask to adjust the grid cell weights to avoid overweighting grid cells with both land and ocean present. However, for the GISTEMP data, I also calculated an additional average without adjusting the grid cell land fraction. This is done because GISTEMP’s land-only data extends far beyond land because of the 1,200 km smoothing algorithm and I don’t want that data masked out. The figure below shows the GISTEMP temperature anomaly for December 2010 to illustrate how far beyond land the data extends.

The presence of a few islands within 1,200 km of a grid cell center can fill in a lot of ocean area. The global averages of the three data sets I’m using are calculated differently by their respective creators. From IPCC’s AR4 WGI report,

The global average for CRUTEM3 is a land-area weighted sum (0.68 × NH + 0.32 × SH). For NCDC it is an area-weighted average of the grid-box anomalies where available worldwide. For GISS it is the average of the anomalies for the zones 90°N to 23.6°N, 23.6°N to 23.6°S and 23.6°S to 90°S with weightings 0.3, 0.4 and 0.3, respectively, proportional to their total areas.

As an aside, the GISTEMP method isn’t as simple as it sounds. They apply the reference station method at all 100 equal area sub-grid cells inside the primary 80 equal-area grid cells. From these data, they calculate zonal means and then the global mean. Also, as per Dr. Ruedy, the formulation above is no longer used. They compute the global mean as 0.3*T(23.6:90) + 0.2*T(0:23.6) + 0.2*T(-23.6:0) + 0.3*T(-90:-23.6). This change was made to make the global mean consistent with the mean of the hemispheric means since the GISTEMP group had receive lots of inquires about apparent inconsistencies with these means. For this analysis, I will calculate the global means using the old formulation and I won’t attempt to implement any reference station method.

I ran my calculations using all three methods to see how sensitive the results are to the spatial averaging method. I calculated trends for two periods: 1880-2010 and 1981-2010. The figure below shows the trends calculated for these two periods as a function of the chosen base period for all three averaging methods, starting with simple area-weighting.

Recall that adjusting the base period of spatially-averaged anomalies doesn’t change the trend. Here we see that the trend should change to reflect the differing resultant spatial coverage. Comparing trends among adjusted anomalies might lead to false conclusions about their relative sizes. The most proper way is to adjust anomalies at the grid cell level.

GHCN and CRUTEM show very similar variability in their trends for the 1880-2010 period though CRUTEM is consistently larger. These two data sets generally show increasing warming given more recent base periods for the 1981-2010 period. Masking out the “excess” data in GISTEMP consistently increases its trend. For the four recent base periods, starting with 1951-1980, the three data sets (including GISTEMP masked) show convergence for the 1981-2010 period.

Calculating the global mean from hemispheric means, GHCN and CRUTEM still show a strong correlation over longest period as well as increasing trends in the recent period. GISTEMP’s base period sensitivity diminishes significantly relative to the simple area-weighting case. The above noted convergence remains for GHCN and CRUTEM, but GISTEMP pulls a bit higher than before.

The zonal mean-derived global averages show the trends for the recent 30-year period in the strictly land-only data are noticeably lower relative to the other two averaging methods. To try to understand what explains some of the differences between the trends among the four data sets, we can look at the fraction of land area accounted for in each series’ spatial average. First, let’s look at GHCN’s spatial coverage for each base period.

This figure should be familiar. It is very similar to the well-known GHCN station count. Note that the earlier the base period, the more stable the spatial coverage is from about 1920 to 1990. The two base periods that offer similar and the best overall coverage are 1951-1980 and 1961-1990. The presence of a strong annual cycle in the 1971-2000 series suggests that for many grid cells, calculating the climatology failed for at least one month and those grid cells go missing for that same month or months each year. All the series show drops in coverage at 1990 ranging from small to large. Now let’s look at CRUTEM’s spatial coverage.

CRU and GHCN show strong similarities but what stands out is that the 1990s drop is more gentle in CRUTEM than GHCN. One could use the word “decline” instead of “drop”. A prominent annual cycle also shows up in the 1971-2000 base period. Now we’ll look at GISTEMP’s coverage.

Yes, you are reading the graph correctly. GISTEMP’s land surface area is greater than the total land surface! I won’t dwell on this one because of the obvious difficulties in interpreting such numbers. To be able to make more realistic interpretations, we can look at the spatial coverage of the GISTEMP data when it’s restricted to land-only.

GISTEMP’s 1,200 km smoothing method gives it a head start of almost 0.5 over CRUTEM and GHCN. At about 1920, the spatial coverage stabilizes for the first five base periods. The remainders jump at about 1955 to full coverage. The masked GISTEMP trends for 1981-2010 corresponding to the four most recent base periods are very stable because the spatial coverage isn’t changing nearly as much as the other two data sets.

The figure below shows the calculated linear trends (°C/decade) vs. the maximum spatial coverage (fraction of total land area) for each data set for each trend for each spatial averaging method. Also given are the correlation coefficient and a p-value.

Over the full 1880-2010 period using data calculated with simple area-weighted averages, CRUTEM and GISTEMP (masked) show numerically and statistically insignificant correlation between their respective data sets’ trends and maximum spatial coverage. GHCN and GISTEMP however do show numerically and somewhat statistically significant correlations. Interestingly, the more “spatial” coverage GISTEMP has, the lower the trend.

The hemispheric-mean derived data show a similar correlation for GHCN while GISTEMP’s correlation goes up a bit. GISTEMP continues to show lower trends with greater spatial coverage.

With the zonal-mean derived data, GISTEMP masked how behaves more like its unmasked counterpart. GHCN and CRUTEM’s figures remain very constant across the different spatial averaging techniques. Now we’ll look at the figures for the recent 30-year period.

With simple averaging, the linear trends are all showing numerically and statistically significant correlation with the spatial coverage. GISTEMP masked and unmasked show an almost perfect negative correlation!

The hemispheric-mean derived data show that relationship between GHCN, CRU and GISTEMP’s trends and spatial coverage don’t noticeably change with this averaging method. The stand-out is GISTEMP masked whose correlation has flipped sign!

The zonal-mean derived data show similar correlations in magnitude and sign as the other two methods. I’ll conclude that the trend vs. spatial coverage relationship is not particularly sensitive to the spatial-averaging method. This came as a surprise to me because I thought that calculating global means from at least hemispheric means if not zonal means would greatly dampen the variability in the temperature anomaly due to changing spatial coverage. I guess I was wrong.

GHCN v3-beta Looks to the Future

I was doing some routine calculations with the gridded GHCN v3-beta anomalies when I noticed something bizarre. The first three months of 2011 had non-missing spatial averages, but December had a non-missing value! I immediately thought “*#$)%!, another bug to squash!” But the error wasn’t on my part. I opened up the grid-mntp-1880-current-v3.dat file and sure enough, on line 58,582 there was one single non-missing grid point corresponding to December 2011 with a value of 302 or 3.02 °C. I sent an email to NOAA. They don’t call it v3-beta for nothing. Back to work.

Posted in GHCN | Comments Off on GHCN v3-beta Looks to the Future

GHCN v2’s lead over v3-beta – Mystery solved!

In my previous post, I found a strange anomaly (pun). GHCN’s v3-beta accounted for less of the Earth’s land area than the current operational version, v2, from 1880-1990. After which, v3-beta over takes v2. v3-beta had more data than v2, not in the form of new stations, but more data for v2 stations. I figured out a simple way to sort out the differences in v2 and v3-beta’s coverage. First, I found which grid cells in both data sets both have non-missing values. Then I looked for for grid cells that were present in v2, but not v3-beta and grid cells that were present in v3-beta, but not v2. The figure below (from my previous post) shows the land fraction accounted for by both data sets individually and in common.

Now here’s the land fraction accounted for by both data sets. “v2 and v3-beta” in red below is the same as “v3-beta minus v2” in green above.

The “v2 only” and “v3-beta only” are very small in comparison to their common coverage (“v2 and v3-beta”). If you add the common coverage to the “v2 only”, you get the v2 red line in the first graph. Add the common coverage to the “v3-beta only” data and you get the blue line in the first graph. Now that the contributions have been separated out, we can say the following:

  1. Relative to the common coverage, v2 only adds coverage from 1880 to about 1920, after which is declines and stagnates after 1980.
  2. Relative to the common coverage, v3-beta adds little coverage over 1880-1930, declines a bit till about 1955, then gradually grows till about 1980. Thereafter, v3-beta’s additional coverage skyrockets.
  3. Both data sets show a decline in coverage circa 1990, but the spike in “v3-beta only” coverage decreases this decline as is apparent in the first graph.

V3-beta’s additional data is mostly after the 1990s decline. The v3-beta’s pre-1990s lose of coverage relative to v2 is probably related to new homogeneity adjustments removing enough bad data that some grid cells will have missing values, whereas previously, the corresponding data in v2 was used.

An Intercomparison of GHCN v2 and v3-beta

GHCN v3-beta has been out for some time now. I recently noticed that they put up an official gridded product. I know that others have already gridded the station data and studied the spatial averages. I’m primarily interested in how the two gridded products differ at the grid cell level. To start, I downloaded the gridded data files. These files contain temperature anomalies gridded at 5°x5° resolution, relative to the 1961-1990 mean and cover 1880-2011. The analysis period will be 1880-2010, since I’m computing annual averages (2011 isn’t over yet).

In calculating global averages, I’ll need to use a land mask to adjust the grid cells’ weights to account for the fact that some grid cells contain both land and ocean. Multiplying the temperature anomaly of such a grid cell by its area will overweight its contribution to the global average, possibly biasing the regional and/or global trend. To construct a land mask, I downloaded the TerrainBase high-resolution elevation data. Where the elevation is greater than zero, the grid cell land fraction is equal to one, elsewhere, it’s equal to zero. I used first-order conservative interpolation to get this 5′ x 5′ grid into 5° x 5° resolution, as shown below.

After adjusting the area weights by the corresponding land fraction, I calculated global averages for the v2, v3-beta data and their difference. The differenced data is calculated from the v2 and v3-beta global averages, not their grid cell differences. Before we consider the global averages, let’s look at the spatial coverage of both data sets.

v2 has greater spatial coverage from the beginning of the series up to the station drop of the early 1990s. After that point, v3-beta overtakes v2. GHCN v3-beta doesn’t contain any new stations. But it contains more data, or less missing values than v2. So why does it account for less land for most of the time series? To get an idea of what I might be missing, I looked at a post by Zeke from late last year. The station count and land fraction correlate pretty decently, but Zeke’s graph shows that v3-beta is consistently above v2 after 1895.

How do we reconcile the apparent mismatch between the two graphs? Consider a grid cell that contains multiple stations, each with missing data, but enough observations to calculate anomalies relative to 1961-1990 and enough such that the multi-station average always has at least one observation, then a full time series can be calculated. If some missing values can be filled in with new records, that doesn’t change the fact that this grid cell’s average always has non-missing values and that it is always contributing to the global average. The station count may go up, but that doesn’t necessarily mean more of the Earth is being sampled. So at least, all other things being equal, v3-beta and v2 should be at least neck-and-neck in land area even if their station counts differ significantly. But that doesn’t explain why v3-beta is below v2. This will take me a while to figure out, but for now, I’ll have to just speculate. Perhaps homogeneity adjustments wiped out enough data to result in a number of grid cells to go from non-missing in v2 to missing in v3-beta. Some stations that disappeared in the 1990s probably now have additional data in this period to the present and that may explain why v3-beta takes the lead over v2 after the station drop off. Now take a look at the annual global averages for the for period 1880-2010 and 1975-2010.

Comparing to Zeke’s calculations based on the adjusted data, I find noticeable differences in the trend. I re-ran my numbers to match Zeke’s time period (1880-2009, 1975-2009). Here’s how our numbers compare (units are °C/decade).

Data/Period Chad Zeke
v2 (1880-2009) 0.077 0.079
v3 (1880-2009) 0.086 0.074
v2 (1975-2009) 0.300 0.322
v3 (1975-2009) 0.301 0.310

For both periods, Zeke finds v3 to be smaller than v2, while I’m finding the opposite. Although, the difference between the 1975-2009 period for my data is a mere 0.001 °C/decade, nothing to write home about. For all intents and purposes, they’re the same. I wonder if v3-beta has changed enough since Zeke ran his numbers to account for this difference. Another mystery to solve.

Now we’ll look at grid cell trends for both periods. To calculate a valid linear trend, I require that at least half of the data in a given grid cell must be present. If not, then the calculated trend is masked out.

For the 1880-2010 period, the U.S. data shows some increased warming in v3-beta relative to v2. The sparsity of data in Latin America and Africa make it hard to discern any systematic continental or regional differences. My less-than-objective eye doesn’t see any widespread systematic differences in the Eurasian continent. For the recent 1975-2010 period, more data is available in Latin America and Africa so we can get a better idea of any spatially-consistent differences. Africa significantly appears to show less warming in v3-beta than v2. Latin America’s southern half also gives a similar impression. The differences in the U.S. are less consistent and more variable. Chinese data appears to show very consistent weakened warming in v3-beta. Neglecting three grid cells, Western Australian data also shows less warming. I should note that if the regression were done without checking the number of observations, the trends would be all over the place! There are grid cells with extremely large (relatively speaking) warming and cooling. I don’t think that most of these large trends are realistic. When I first did this, I had trends ranging from -13 °C/decade to 15 °C/decade. These results are undoubtedly caused by the fact that if a time series has a lot of missing data, it is possible that a few highs and lows end up dominating the trend. If I tightened my restriction on the minimum number of observations, it might reduce the magnitude of the largest trends, but I might loose a lot of grid cells making it harder to judge widespread changes.

I’ll continue looking at GHCN coverage issues over the next few days. Ideas are welcome.