NCEP/NCAR Reanalysis Problems List

(updated: Sunday, 17-Dec-2017 18:01:19 MST)

The NCEP/NCAR Reanalysis Problem page at NCEP.
  • Corrupted cloud layer pressures for 1958-1959.
    The 1958 and 1959 cloud layer pressures in the other_gauss/"other flux" category had zeros where they should have missing_value. The use of missing_value in these variables is to indicate the absence of that cloud layer at that place and time. The zeros, otherwise a nonsensical pressure value, were replaced by missing_value and the Daily Averages and Monthly Means recalculated. Names: pres.?c?.gauss.195[89].nc
    Files were updated 2017/12/15.
  • Bad relative humidity values for three time steps in 2004.
    The last three time steps on 9 March in 2004 had the constant nonsense value of -25.01 for relative humidity at the eigth (AKA last, AKA 300 mb) level. GRIB1 data for these times and level was found at NCAR and used to replace these three grids. The Daily Averages and Monthly Means were recalculated. Name:
    Files were updated 2017/12/15.
  • NCEP reran the model for certain times because the snow boundary was input incorrectly. We obtained the corrected data and have replaced all the impacted files. Most of the effect was near the surface, particuraly for temperature. The fix was done 10/12/2012. The days impacted were
    1. 06Z May 17 2012
    2. 12Z May 17 2012
    3. 18Z May 17 2012
    4. 00Z May 18 2012
    5. 06Z May 18 2012
    6. 12Z May 18 2012
    7. 18Z May 18 2012
    8. 00Z May 19 2012
    9. 06Z May 19 2012
    10. 12Z May 19 2012
    11. 18Z May 19 2012
    12. 18Z Jul 18 2012
  • A computer glitch caused corruption of the 4X daily 2012 pressure level air temperature file for dates after April. The corrupted file was public on Oct 2nd2012. It was fixed the evning of Oct 2nd.
  • Bad VAIR values for spectral coefficients for Jan 21 12z for levels 17-21.
  • We uncovered bad values in the 2007 4 times daily NetCDF file for Jan 21st 12z, levels 17-21 for the vair variable. No cause was determined. The file was was replaced on 6/19/2012 using GRIB values from NCEP. No other variables or level/date seemed to be affected (in 2007).

  • Bad pressure-level omega values for parts of the 200x years.
  • We uncovered bad values in our NetCDF files from mid-2003 to mid-2004, mid-2005 through the end of 2006, and at the tail end of 2007. All 8 files (daily and six-hourly) for 2000-2007 have been replaced. The monthly means were also updated. Unfortunately, no cause has been determined. Regenerating the data from GRIB files provided by NCAR cured whatever the problem was. The diagnostic a user suggested was global means at various levels. Sudden discontinuities were seen in this diagnostic and have been eliminated.

  • Change in snow cover analyses lowers some temperatures.
  • NCEP originally used NESDIS snow cover analyses for the northern hemisphere and climatological snow cover for the southern hemisphere. The NESDIS analyses were discontinued. In anticipation of that, NCEP began using a global US Air Force set of snow cover analyses instead starting in 1998. This had a variety of effects, no doubt, but the one that was noticed was lower pressure level air temperatures (at lower levels of the atmosphere at least), for certain southern hemisphere land points especially. It's not believed to affect oceanic points. NCEP plans no data replacement.

  • Polar temperatures shift 1998-2004.
  • Different sea ice analyses were used in that period, and it appears in some cases those analyses disagreed with the R1 model as to whether a given point was land or ocean, especially in the Arctic. New analyses used starting in 2004 corrected the problem. So, skin temperatures and 2 meter air temperatures (at least) were significantly higher at some polar region points for the period. NCEP plans no data replacement.

  • Seven months of "skt" (skin temperature - SST over oceans) were wrong (Aug. 2004 through Feb. 2005).
  • A user noticed the problem, and the file was repaired 2007/09/06. Only the monthly means were affected. The daily and six-hourly values were not affected.

  • A few months of monthly means for a few variables had zeros instead of missing_value flag values.
  • For July, 2002, through October, 2002, the variables gflux, pevpr, soilw, and soilw200 in the surface_gauss/"surface flux" category, had zero values instead of missing_value flag values in the monthly means. The 4x daily and daily mean files were unaffected. The monthly means were recomputed and replaced 2005/09/07.

  • A day or two was missing at the end of Aug., 2005, and is now in place.
  • In the other_gauss/"Other flux" category, data for 30 Aug. 2005 were discovered to be missing, due to a server glitch here, and has been replaced. This was also true of the spectral coefficient category. In the sigma level derived category (psi & chi), this same day and 29 Aug. 2005 were missing and have been replaced. The means that were based on this data were re-done with the replacement data. All replacement files were installed 2005/09/07.

  • NCEP re-does 2004/08 - 2005/02 over sea-ice/SST data ingest problem
  • Data replaced at CDC on 2005/04/26. NCEP has details on their web site on this page.

  • Bad specific humidity values for Feb. 1972
  • In the pressure level category, a user noted bad values in parts of Feb., 1972. The month was reingested from GRIB originals the file updated 2004/10/27. The daily mean and monthly mean files were regenerated the same day.

  • Total Clouds (TCDC) missing 2003/08/01-03
  • In the "other flux"/other_gauss category, a user noted that the data for the time steps covering 01 Aug. to 03 Aug. were missing in the 2003 tcdc file. They were reingested from the GRIB originals and the file updated 2004/06/07. The daily mean file for tcdc in 2003 was regenerated as well the same day.

  • CDAS rerun 2003/05/09 0Z to 2003/06/26 18Z because of missing ATOVS input data.
  • Details are under the heading "7-7-2003" on NCEP's Reanalysis web page. The data were replaced here in August, 2003, if not earlier (probably in July).

  • Missing grids and time values in surface_gauss/
  • The file surface_gauss/ (surface flux category; variable is downward solar radiation flux) has zeros for time values and data values for the time steps 1972-06-01 00:00:00 and 1972-06-11 12:00:00. They will be repaired eventually, if possible.

  • Problem with ingest turning MISSINGs into zeroes resurfaces.
  • A previous problem, wherein the ingest process failed to detect a missing value situation in the input GRIB file for some variables and placed a zero value in the output NetCDF file, was rediscovered after reloading the Reanalysis data from old tape backups. The variables are the six cloud level boundary pressure variables in the other_gauss category, and {gflux, pevpr, runof, sfcr, soilw.0-10cm, soilw.10-200cm} in the surface_gauss category. The years affected in this situation were 1948-1957, and 1960-1987. The data have been corrected as of 2003/09/07. If you downloaded the four times daily data for these variables since early August, 2003, you may wish to reacquire them.

  • Restoration after disk array failure.
  • The entire disk array holding CDC's Reanalysis archive (and all of our other data sets) was lost in early August, 2003. Some data were restored from old backups, some were regenerated from original inputs.

  • 4 bad timesteps in pressure/
  • In pressure/, the values for 18Z 16 Feb through 12Z 17 Feb were wrong, and have been corrected. The corresponding daily average file was recalculated.

  • Problem with ingest turning MISSINGs into zeros 1948-1957.
  • The ingest process, which turns GRIBs into NetCDFs, had an error which changed what the GRIB source considered to be a MISSING value, into a zero value. This affected the pres.?c? cloud layer top and bottom pressure variables in the other_gauss/"other flux" category, and the following variables in the surface_gauss/"surface flux" category:
            gflux, pevpr, runof, sfcr, soilw.0-10cm, soilw.10-200cm
    This data, including daily averages and monthly means, was replaced 2003/05/28.

  • Slightly mispacked daily average prate data replaced.
  • As of 2003/05/09 prate data in the surface_gauss directory has been replaced for the years 1997-2001. These data files were originally created with software that erroneously caused zeros to mapped to 1.0e-7 instead. This error has been corrected and updates to prate from 2002 forward did not and will not have this problem. The monthly mean was also re-done, as it is based on the daily means.

  • Missing Total Cloud Cover Data replaced.
  • As of 2003/05/02 data for the dates 2002/01/14, 2002/04/21, 2002/05/08, 2002/06/24, 2002/10/25, 2002/11/26, 2003/02/23 and 2003/03/25 for the variable tcdc were replaced. These data files were originally released with missing data for these dates due to a processing error. This error has been corrected and future updates to tcdc will not have missing data.

  • Update on TOVS problem:
  • A problem was discovered regarding TOVS satellite retrievals with impact on high-level (physically above 150mb) temperatures and apparently geopotential heights. This resulted in a temporary suspension of CDAS processing, which has long since been restored. A number of variables may have some impact from this problem (period affected is March, 1997 through September, 2001), reportedly including surface winds. The good news is, CDC has the replacement data online as of 2002/04/23. Here is a web page at NCEP with details of this issue.

  • Omega values for levels above 300mb (250mb through 100mb) were missing in 2002.
  • The values of omega for 2002 at all three temporal resolutions (4x daily, daily mean, and monthly mean) were missing in 2002, because new software used in 2002 for daily updates was not configued to extract them. This has been corrected, and the data has been re-extracted and replaced at all three temporal resolutions. The new files have values for the last four vertical levels (250mb through 100mb). 2002/03/21

  • Some monthly means had slightly off time values.
  • Perhaps derived from the "2001/01/01 00:00:00" bug in the earlier version of the UDUNITS library we were using at the time, some monthly means files had times starting with that month and thereafter that were off by one minute. These files are the ones that have been repaired: pressure/, tropopause/, other_gauss/, other_gauss/, surface_gauss/, surface_gauss/, surface/, and surface/

  • Problem with wspd monthly means.
  • It was learned that some of the values (not the most recent ones) were miscalculated by an older version of the processing software. Both versions of this file, in the pressure and in the surface categories, were recomputed from the beginning using the newer software, and they have been verified correct.

  • Problem with ingest turning MISSINGs into zeros 1999-2001/06.
  • The ingest process, which turns GRIBs into NetCDFs, was erroneously changed, beginning with 1999 data, and for 30 months thereafter, to change what the GRIB source considered to be a MISSING value, into a zero value. This affected the pres.?c? cloud layer top and bottom pressure variables in the other_gauss/ "other flux" category, and the following variables in the surface_gauss/ "surface flux" category:
            gflux, pevpr, runof, sfcr, soilw.0-10cm, soilw.10-200cm
    This data, including daily averages, was replaced 2001/06/24, and the monthly means the following day (truncated monthly means ending in 1998 were installed 2001/06/24).

  • Problem with 1987 omega data conversion.
  • There was a problem with the pressure level omega data for 1987. The original GRIB data from NCEP was used to recreate this file (and the associated daily and monthly averages). This has eliminated the problem (the symptom was overlarge global averages). The corrected data was installed on 13 September 2000.

  • (Not a problem per se) Some 1948-1957(?) data is 8x daily.
  • Read about it here.

  • Problems with the data conversion from GRIB to netCDF.

  • In early May 2000 CDC became aware of a problem with some of the 4-times daily reanalysis files for 2000.  The problems and their resolution are described in the tables below.
    Problem Description: Files affected: Date Problem Occurred: Date Problem Resolved:
    Duplicate data for 2000/1/1/2000 0Z and 1/1/2000 4-times Daily
    1. surface/
    2. surface/
    3. surface/
    4. surface/
    Early February 2000 when the data was released at CDC. May 4, 2000
    Duplicate data for 2000/1/1 0Z and 2000/1/1 6Z and missing data for 2000/02/01 0Z through 2000/02/19 18Z  4-time Daily
    1. surface/
    Early February 2000 and early March 2000 when the data was released at CDC. May 4, 2000
    Missing data for 2000/02/01 0Z through 2000/02/19 18Z 4-times Daily
    1. pressure/ 
    2. pressure/
    3. surface/
    4. surface/
    5. surface/
    6. surface/ 
    7. surface/
    8. tropopause/ 
    9. tropopause/
    Early March 2000 when the data was released at CDC. May 4, 2000

    The following files are derived from the 4-times daily files above and were thus affected by the data conversion problems.  They have all been repaired as indicated in the table below.

    Problem Description: Files affected: Date Problem Occurred: Date Problem was Resolved:
    Files derived from and affected by problems with files listed in the table above. Daily Averages in /Datasets/ncep.reanalysis.dailyavgs/
    1. pressure/
    2. pressure/
    3. surface/
    4. surface/
    5. surface/
    6. surface/
    7. surface/
    8. surface/
    9. surface/
    10. surface/
    11. surface/
    12. surface/
    13. tropopause/
    14. tropopause/
    Beginning of February 2000 and beginning of March 2000. May 4, 2000
    Files derived from and affected by problems with files listed in the table above. Monthly Averages in:
    1. pressure/
    2. pressure/
    3. surface/
    4. surface/
    5. surface/
    6. surface/
    7. surface/
    8. surface/
    9. tropopause/
    10. tropopause/
    Beginning of February 2000 and beginning of March 2000. May 4, 2000

  • Some observational data missing from 12/31/1999.
  • Due to a Y2K bug a few observational data were left out of the analysis for 12/31/1999. The problem has been corrected and all subsequent data incorporates all available observational data. (Resolved on 2000/1/8.)

  • Problem ingesting the surface and mean sea level pressure data affecting the period 1948-1967.
  • A problem has been uncovered with the encoding of surface and mean sea level pressure that affect the period 1948-1967. The problem originated in the erroneous conversion of a portion of the TD13 dataset into the bufr ADPSFC type messages. Details about the effects of the problem are available here.

  • February 1997 CDAS was run without some input data.
  • Bob Kistler (EMC/NCEP) found that the Feb 1997 CDAS analyses was missing 3 days of observed SST and 13 days of snow cover analyses. CDAS used persistence for the last 3 days of SST analyses and used the forecast snow field for the last 13 days of February. Using the persisted SST is presumed to have almost no effect on the analyses. Using forecasted snow cover will have minimal effect in the boundary layer in regions where the analyzed and forecasts snow cover differ.

    The February CDAS analyses will be not be rerun. This is different than reported earlier.

  • Problems with large 1997 Indian monsoon rainfall.
  • New data to correct this problem has been installed and is available from CDC as of 2 April 1998.

    In response to a question by Dr. Rupa Kumar (visiting scientist at ENM/URF/UMT Meteo France) concerning the large 1997 Indian monsoon rainfall, Bob Kistler found a problem with our handling of the Indian station data since March 1997. These analyses are not "wrong" but use more Indian station data than intended.

    CDAS/Reanalysis did not use all the Indian station data (i.e., stations were "blacklisted") which was NCEP practice when Reanalysis was developed. Since then, operational practice has changed so that Indian stations are not blacklisted and PAOBS are not used. In March 1997, CDAS's GTS decoder had to be changed to the new operational decoder because NCEP's mainframes were removed from service. Through an oversight, the Indian data were not blacklisted for processing by CDAS. This oversight will be corrected. At present, there are no plans to add PAOBS decoding to the new decoder.

    The effect of using the "blacklisted" Indian stations was to drastically increase the Indian monsoon precipitation. According to Dr. Kumar, the assimilation's too low precipitation became too large. Removing the "blacklisted" stations is necessary for the sake of temporal continuity over the Indian subcontinent. On the bright side, Dr. Kumar found the interannual variability was "reasonably well captured" (about 0.8 correlation between the total Indian JJAS precipitation anomalies and Reanalysis for the years 1974-1996.)

  • Problems with aircraft data January 1976 to December 1978.
  • The corrected data have been on-line at CDC since 14 March 1998.

    Prior to March 14, 1998 the data for January 1976 to December 1978 were corrupted due to a miscommunication among the US Air Force, NCAR and NCEP regarding a data set of aircraft reports for this period only. Along the way the sense of E-W longitude was flipped, causing a sizable portion of these aircraft data for these three years to be mispositioned.

  • Problems with the June 1997 (CDAS) data.
  • The Climate Data Assimilation System (CDAS) data for the NCEP/NCAR Reanalysis project for June 1997 was recalculated after it was originally released at the beginning of July. This recalculation was necessitated by three factors:

    1. The strong El Niño.
    2. Problems with the radiances coming form NOAA-12 (started ~ 1 June).
    3. The grib identification for the ingest ice anl's were changed during the month - the original run ceased ingesting these data as of the date of the change.
    The corrected version of the June 1997 CDAS data has been on-line at CDC since 20 Oct 1997.

  • Problems with packing of the /Datasets/ncep.reanalysis/
  • A problem was recently discovered with the packing of the 1986 pressure level omega data ( A new file has been created and is available as of 1 October 1997 both on tape and by FTP. You can FTP the file from our anonymous FTP server, The directory and file name are /Datasets/ncep.reanalysis/pressure/

  • Large differences between the Reanalysis data and the climatology for some variables during the TOGA COARE period.
  • NCEP has found an error in the "Complex Quality Control" (CQC) module written for the TOGA-COARE period. Apparently the temperature to virtual temperature conversion was never done. The extent of the error is being investigated and a revised TOGA-COARE period reanalysis is anticipated. NCEP has created a page with further information about this.

  • Missing specific humidity for March 1995 at 500, 400, & 300 mb levels replaced
  • We at CDC were informed that this data was missing (specifically, set to the _FillValue for netCDF) by a user of our data. This data has been replaced in /Datasets/ncep.reanalysis/pressure/ Also, this month alone is available in /Datasets/ncep.reanalysis/shum_mar95/ for anonymous FTP. No explanation for the occurrence has been uncovered. (Resolved 1996/11/15.)

  • Tape problems with October, November and December 1985
  • NCEP discovered that the tapes that were written for October and November 1985 were corrupted. In order to correct the problem, the assimilation was re-done for the last three months of 1985. The problem associated with the corrupted tapes manifests itself as "drop outs" in the data. That is, occasional data points which are drastically different from their neighbors.

    This latest correction for the last three months of 1985 was run on new computing hardware with new operating system software. As a result, there are differences in all fields of the data, though the differences are largely limited to the least significant digit for that variable.

    As of Monday, September 16, 1996 CDC has placed on-line the most recent (and final) version of the October, November and December 1985 data.

    NCEP's explanation of the problem can be found here.

  • Southern Hemisphere Surface Pressure Input Data
  • A problem has been discovered with data used in the Reanalysis affecting the southern hemisphere for the years 1979-1992. The error involves the use of the PAOBS data (SH surface pressure bogus data) produced by Australia. A detailed description of the problem can be found here.. The model was not rerun with corrected values. NCEP/DOE Reanalysis II does use the corrected values.

  • Polar winds.
  • As of 17 July 1996 all of the winds on-line at CDC have been corrected. In the original distribution NCEP incorrectly smoothed the wind components rather than smoothing div and vort and computing u and v from the smoothed fields. This has resulted in problems in the winds near the poles. This problem only affected data from the years 1982-1983, 1986-1993. These data years have been replaced at CDC with the corrected data.

  • 1985 spin-up.
  • The new 1985 is on-line. The original 1985 began from a dead-start which created a longer than expected period of spinup that is in evidence in the soil moisture fields in particular, and may have been present in other fields.

  • High, Middle and Low Clouds.
  • Summary: Do not use the high/middle/low cloud fractions except for calculating the total cloud fraction. (Since the total cloud fraction is a standard output, it is best not to use the high/middle/low cloud fractions.) The clouds as seen by the radiation codes are not available which is an oversight on our part.

    Background: Historically the MRF model saved cloud data for high, middle and low clouds. One could then assume the clouds had random overlap to get the total cloud fraction (CF).

       total CF = 1 - (1 - high CF) * (1 - middle CF) * (1 - low CF)
                "random cloud overlap"

    The Reanalysis model recognizes two basic types of clouds, convective and RH. The convective clouds are based on the convective parameterizations and the RH clouds occur when the RH exceeds a critical value. While the cloud cover from RH clouds can be modeled as "random overlap", the convective clouds are better modeled as "maximum overlap". (At least that is how the radiation code works.) Unfortunately this change occurred before NCEP moved to grib output. At that time, changing the binary output files was a hanging offense. Imagine changing the binary files and suddenly all the code that read those files suddenly failed. In addition, any new code would be incompatible with the old binary files. Since the total cloud cover is the most commonly used diagnostic, the high/middle/low cloud fractions were redefined so that a random overlap would produce the approximate total cloud cover as seen by the radiation code. The redefinition is

    low_CF =  low_CF_RH            			if no convective clouds
           =  0.80 * (low_CF_RH + low_CF_cn - low_CF_RH*low_CF_cn)
    mid_CF =  mid_CF_RH            			if no convective clouds
           =  0.35 * (mid_CF_RH + mid_CF_cn - mid_CF_RH*mid_CF_cn)
    high_CF =  high_CF_RH            		if no convective clouds
           =  0.35 * (high_CF_RH + high_CF_cn - high_CF_RH*high_CF_cn)
    CF_RH = cloud fraction from large-scale clouds
    CF_cn = cloud from from convective clouds

    As you can see from the redefinition, the CF shows a large discontinuity depending on whether convective clouds are present. In addition, a deep convective cloud would show up as a combination of deep and shallow clouds. This makes the the high/middle/low CF useless EXCEPT for calculating the total cloud fraction. The total cloud fraction computed as above is the only cloud data that CDC is distributing.

  • Albedo (ALBDO) - Surface Flux data on gaussian grid
  • The albedo (ALBDO) variable that was distributed with 1985 to 1989 data is not quite correct, in that it doesn't take zenith angles into account. NCEP suggests calculating albedo from short wave up/downward fluxes. NCEP is no longer distributing the albedo data.

  • Convection Cloud Concentration (CDCON) - Surface Flux data on gaussian grid
  • We plotted CDCON and noticed very strange looking distributions of values. NCEP told us that the convective cloud concentration (CDCON) variable wasn't supposed to be written to tape, and is "meaningless". Therefore, this variable won't be converted or put on-line.

  • Pressure Tendency (PTEND) - Surface data on 2.5 degree grid
  • We have detected abnormally large PTEND values and reported this NCEP. They have agreed that the values are too large, but are not sure what the explanation for this is. Because PTEND is not reliable, we will not put this variable on-line and previously converted PTEND files will not be accessible.

  • Surface temperature (air.sfc), 2 meter temperature (air.2m), and TMAX temperature (tmax.2m).
  • The skin temperature is determined diagnostically. It is the temperature required to balance the fluxes at the surface.

    In air, sensible heat flux, long wave radiation (up and down) depend on the skin temperature. In the ground, the heat flux into the soil depends on the skin temperature.

    The problem occurs when the winds are weak (less than 0.75 m/s), and the stability conditions are just right. Under these conditions, the parameterization for the thermal exchange coefficient can break down. This caused the model to blow-up (divide by zero). (After running this code for years, the bug surfaced.) A bug fix was introduced (instead of dividing by zero, dividing by a small number). However, the fundamental problem remained. Under light winds, the thermal exchange coefficient can be close to zero. For argument's sake, let's assume that the exchange coefficient is zero. In this case, the sensible heating is zero, and the solar heating has to be balanced by latent heat flux, the net longwave flux and a ground heat flux. Since the latent heat flux is not dependent on the skin temperature, the reduction of the sensible heat flux must be balanced by an increased upward long wave flux and a downward ground heat flux. In order to make this balance work, the skin temperature will become unreasonably hot. On the next time step, the winds may pickup and skin temperature will return to normal (skin temperature has no inertia). The affect on the lowest sigma level is minor as the net fluxes remain reasonable. This phenomenon will not happen over water as the skin temperature (sst) is fixed.

    TMAX and possibly TMIN are affected by this problem. TMAX is the worst possible case - it will capture spikes that only last 1 time step.

    The surface temperature will be affected.

    The 2m temperature is interpolated from the skin temperature and the bottom sigma-layer and thus will be affected.

    Sensible heat flux and the upward long wave radiation flux are also affected.

    A suggested work around is to reject instantaneous data if the following two conditions are true:

    • The data is over land.
    • The 10 meter wind is less than .75 m/s.

    This test rejects ~ 2% of the total number of grid points. It will not work for averaged quantities and Tmax, Tmin. In addition, this test rejects an awful amount of good data. In the monthly means produced at NCEP this screening test will not be used because ejecting so much good data would be worse than including a few bad points (which get smoothed out in the monthly averages).

    It may also be possible to reject data greater than N standard deviations from the mean. No tests have been conducted with this screening method.