Wind Profiler Database: Data Description
General Information about Wind Profilers:
- General Description of Profilers
- Profiler Parameters Pertinent to the Recorded Doppler Velocity Spectra
- Determination of Moments from Doppler Spectra
Wind Profiler Data Description:
Data Processing Techniques
To analyze wind profiler data collected during the TOGA-COARE Intensive Operating Period (IOP) (November, 1992, through February, 1993), Dr. Anthony Riddle of NOAA's Aeronomy Lab developed a series of comprehensive data analysis and quality control algorithms. Since then, these same processing techniques have been applied to all Aeronomy Lab wind profiler data, with the resulting post-processed, research-quality wind profile data sets being called "TOGA-COARE-processed" data. This section describes the procedures involved in TOGA-COARE processing. More details can be found in the ISS Data Reports, Volume IA.
At each wind-profiler site, Doppler spectra of the sampled radar returns are recorded on removable storage media. Also recorded are real-time computations of the average wind vector profile. These data constitute a "quick-look" data set. When the original Doppler spectra is received at the Aeronomy Lab the TOGA-COARE processing is applied. The post-processing proceeds in three stages, which differ from the quick-look processing in several ways.
In the first stage the velocity values are recomputed from the recorded spectra. During this stage, algorithms are used which attempted to recognize (and avoid) spectral peaks due to various extraneous sources such as radio frequency interference, birds and ground clutter. Consequently, the resultant velocity values contain fewer erroneous measurements than the values computed at the site for the "quick-look" data set.
In the second stage further selection takes place based on a battery of "reasonableness" tests. These tests include constraints based on height and time continuity of the velocity values, as well as changes in signal-to-noise value and widths of the echoes. The constraining parameters are individually adjusted according to previous values in the data set and changes in the profiler operating parameters. Only values which pass the test criteria are passed on to the half-hour averaging process.
The second stage processing also attempts to identify the presence of hydrometeors in the profiler data. This detection is accomplished by a combination of tests based mainly on velocity signatures. For the vertical beam downward velocities of greater than 0.5 m/s above the melting layer, or 1.5 m/s below the melting layer, are taken to be indicative of hydrometeors, if observed in several height gates. On the oblique beams corresponding changes in velocity plus an enhancement in signal-to-noise ratio are required to indicate hydrometeors.
In the final stage, the velocity values which have passed the previous tests are processed. For horizontal winds the measured vertical velocities are used to correct the oblique velocities only when hydrometeors appeared to be present. In RASS mode the vertical velocity correction is made in clear air situations (which improves accuracy) but not when hydrometeors were present (which degrades accuracy). In quick-look processing those corrections are made in all cases. A further difference in the RASS post-processing is the use of more accurate values for the "constants" used to convert sound velocity to virtual temperature. The new values result in an overall reduction in the estimated virtual temperature by about 0.3 degrees.
Also in the final stage, comparisons are made with velocities (temperatures) measured at adjacent times and heights. By grouping together like measurements, possible outliers are identified and assigned quality flags according to their degree of dissimilarity. For the profilers aboard the Research Vessels, additional quality flags indicate when rapid changes in the ship speed or heading (and other problems) make the corrections for those quantities suspect.
Data File Formats
This profiler database contains half-hourly or hourly post-processed (i.e. research quality) data from 915 and 50 MHz wind profilers, together with some pertinent documentation and ancillary data.
The data appears in two different file formats, the so-called TOGA-COARE file format and a gridded file format. In the latter format, the original data has been interpolated onto a uniform vertical 100-m grid. The format of the interpolated files is given in this gridded file format document. The format of the TOGA-COARE files is described below.
TOGA-COARE-format Half Hour Wind Profiler Data
Each file is in ASCII (UNIX) format. The first ten lines describe pertinent experimental details. Then follow two lines describing the columns in the data table to follow. Finally comes the data table, one line for each height sampled.
The descriptive text starts off with the file name, site name and data mode. Then the starting time for data collection is given as calendar date followed by Universal time. The day of year count starts at 1 on Jan 1. The ending time for data used in the average is given. There is no indication of how uniformly the time interval is filled.
The site latitude, longitude and altitude may not agree exactly with those given in the quick look data files. The ones used here are considered more accurate (even though the shipboard altitudes are just informed guesses). It should be noted here that the latitude and longitude given in the shipboard site quick look data was entirely fictitious.
The quality number Q1 serves two different purposes. Normally, like all other quality flags, it is zero. However, if all the data in this record came from quick look data files, then it is incremented by 1. On shipboard systems it is incremented by 2 if the latitude and longitude were interpolated over an interval such that there might be an error in the least significant digit or it is incremented by 8 in circumstances where a larger error is possible.
The quality number Q2 takes on non-zero values only for shipboard data. It has 1 added whenever either of the ship's velocity components changed by more than 1m/s, or the heading by more than 30 degrees, between two consecutive profiles, 2 added whenever the ship's velocity changed by more than 2 m/s during the data collection period and 4 added whenever the ship's direction of motion changed by more than 90 degrees during the data collection period.
The number of profile sets (a triad of three measurments) is given along with the number of those characterised as being clear air sets, and the number characterised as having hydrometeors present in the measurements, which are designated as rain sets. When quick look data has been used exclusively for this record then there is no information available about the number of profile sets. The total number is then set, arbitrarily, to 1 and the number of clear air and rain sets are both put to 0.
The signal to noise threshold numbers are provided to give a reference for use with the signal to noise numbers in the table. The threshold is determined by several instrumental settings and it is unlikely that any valid measuments could be made at s/n ratios below the threshold.
The data table description lines indicate the column content and measurement units. Users should note that heights are above sea level, whereas the quick look reports had height above the radar. The horizontal velocities, u and v, have been corrected for vertical motions in the rain triads, but not in the clear air triads. For shipboard systems they are also corrected for ship heading and, when the velocity is above 1 m/s, for ship velocity. The columns labeled 'dev' contain the root mean square difference from the mean for the data sets used and hence are deviations of the data and not deviations of the mean. The columns labeled 'widn' contain the average of the width parameter for the that beam. When n is 1 it is for the oblique beam that points most nearly E-W and when 2 it is for the other oblique beam, which points most nearly N-S. For the vertical beam n is set to w. The signal to noise ratios give the arithmetic average of the signal to noise powers, expressed in decibels. The column labeled 'n' gives the total number of triads used at that height while nr shows how many of those were rain triads. The quality parameter Q ranges from 0 to 9, where 0 is most likely to be reliable and 9 is least likely. The odd quality numbers indicate data that came from the quick look processing, but are otherwise equivalent to the next lower even quality number. Users may wish to use this number as an aid in selecting data for use; for many purposes data with quality numbers of 3 or below will be suitable. Additionally they may consider using the number of triads used in the average as an additional quality control; an average of only one datum may often be suspect.
Users should note that for the vertical data there is a separate height column. This is required because the data samples are taken uniformly in range. Hence the inclined oblique beams have a different range to height conversion than does the vertical beam. As only clear air (non-rain) data is used to compute vertical velocity there is only one count of sets used.
The convention used for labelling heights is that the height given is the height of the midpoint of the transmitted pulse. In a homogeneous medium the reflected energy would fall off inversely with range squared and hence the weighted average response would correspond to a height lower than the labeled height. The effect would be greater at the shorter ranges. As the medium is unlikely to be homogeneous the situation is even more complicated than described above, and thus even more uncertain. Hence the convention has been not to apply any inverse square weighting. In the high height coverage modes the first sample is often taken before the pulse has completely left the antenna. In this case the height label will be an underestimate.
The user should be aware that the profiler radars require a non-zero recovery time after the pulse is transmitted before reliable wind measurements can commence. In order to make measurements at the lowest possible heights the instrument settings are often such that the lowest height or two produce unreliable results. Special attention has been paid to the data in the lowest heights and the bulk of the obviously erroneous measurements have been removed. However it is certain that some erroneous results have been included in the data sets and the user should exercise greater care when analysing data at the lowest heights.
Throughout the table 9999 is used to flag missing data. When the data comes from post processing, missing data will be associated with a count of zero items in the average. However, when quick look data are used, some columns can be missing data while others contain valid data on the same line.
Half-Hour RASS Data
The organization of the RASS half hour average data files is essentially the same as for the wind. The column headings differ to reflect the fact that virtual temperature and vertical wind velocity are being measured. As both measurements are made with the same beam there is only one height column. The suffixes 1 and 2 on the width and s/n columns refer to the spectral regions looked at, 1 for the vertical wind echo and 2 for the sound echo. For temperature there are two data count columns, one labeled 'n' for the total number of measurements and a second 'nr' to indicate those effected by rain. Note that for temperature estimation the vertical velocity is subtracted from the sound velocity in clear air cases, but not in rain cases. The clear air vertical velocity estimate is made only from the clear air cases.
Users should note that refinements to the constants used in calculating the virtual temperature result in numbers that are about 0.3 degrees lower than those calculated for the quick look data set using the same sound and air velocities!
Daily Wind Vector Plots
For each day a summary wind vector plot was produced separately for the high height mode and the low height mode. The vectors show the speed and direction of the wind at each height and time. The quality number for each vector is indicated by a filled circle at the base of the vector, increasing in size corresponding to the quality numbers 2, 4, 6 and 8, or a filled square, increasing in size for the quality numbers 1, 3, 5, 7 and 9.
In a panel below the vector display is a simple indicator of the clear air vertical wind. A set of horizontal bars are drawn, one for each height, with length proportional to the vertical velocity w (positive to the right). In order to avoid overlap all velocities above 0.5 m/s are represented as though they were 0.5 m/s.
In a panel below the vertical velocity display is an indicator of the occurrence of rain (or snow) as determined by the profiler. The length of the vertical bar is proportional to the fraction of data sets that are thought to indicate rain compared to the total number of data sets in the half hour. When the data has been taken from the quick look data set there is no rain information available; this plot always shows a full length bar under those circumstances.
Each file is a self contained Postscript text named ssmyyddd.00p with ss, m, yy and ddd as described above. They too were placed in tar files, which were named ssmyyddp.tar where the dd corresponded to the first two digits of the day number of the first plot. Again compression added a '.Z' suffix.
Daily RASS Temperature Anomaly Plots
For the RASS data the main panel shows not the virtual temperature but the difference (anomaly) between the measurement and a straight line standard temperature. This straight line standard starts at 30 degrees and has a slope of -0.4 degrees for each range gate. Hence the anomaly is given by A = T - 30 - 0.4*i where A is the anomaly , T the virtual temperature and i the height number (i=1 being the lowest height). The standard is not meant to represent any particular model atmosphere but was chosen to keep the plots looking reasonable.
The temperature anomaly is plotted as a horizontal line (positive to the right). A circle or square at the base of the line indicates the quality indicator for that measurement as described under the daily wind vector plot description. When measurements were obtained at adjacent heights then the tips of the horizontal lines were joined.
The panels below the temperature anomaly display indicate clear air vertical velocity and rain as in the daily wind vector plots.
Data Availability
Files have been generated which enable the user to determine quickly whether data in a particular mode is available at a particular site for any given time. These ASCII files have names of the form sssyyddd.dat where sss is a site designator and yyddd gives the first day of the period covered by the file. After the first two descriptive lines the files contain one line for each day of data. Each line commences with yyddd which gives the year and day being reported. Then follow 48 columns, one for each half hour of the day. Within each column is a number ranging from 0 (no data) to 7 (all modes). The number is composed by adding 1 if there is low height coverage wind data, 2 if there is high height coverage wind data and 4 if there is RASS data.
Data Quality Count
Files have been generated which enable the user to assess quickly the quality of data in a particular mode at a particular site. These ASCII files have names of the form ssmyyddd.tot where ss is a site designator, m indicates the mode and yyddd gives the first day of the period covered by the file. After the first three descriptive lines the files contain one line for each height in profiler record. Heights are indicated just by a number, 1 being the lowest. Then follow 20 columns, 10 for the horizontal wind or virtual temperature quality number and ten for the clear air vertical wind. Within each column is the count of number of occurrences of that quality number. Each file contains totals for the complete data set unless there are mutiple files, in which case the first file contains counts for data prior to the second file and so on.
Ship Position, Speed and Heading Data
For the three profilers on board the ships Kexue #1, Shiyan #3 and Moana Wave corrections had to be made to the data for the ship motion and heading. In order to illustrate the corrections made, files have been prepared for each ship giving half hour average values of the ship's position, velocity and heading. The time given in each row is the start time of the half hour average.
As corrections were made on a profile by profile basis, and because the profiles were not always made uniformly throughout the half hour (RASS for instance usually occurred only within the first 6 minutes), these files only contain illustrative numbers, not the actual corrections applied to to individual records.
The files are named sss.pos where sss gives the site.
The files were prepared using the raw heading information, so the corrections subsequently applied to Kexue #1 and Shiyan #3 heading data have not been applied in these files!
Caveats
We would suggest that only data with quality flags 0-3 be used for research. We would also suggest that for the low-mode wind data at Kapingamarangi, RV Kexue #1, and RV Shiyan #3, only data at and above the 4th, 7th, and 7th range gates, respectively, be used (see Hartten, 1998, for details). Note also that pulse lengths, sample spacings, and heights of the initial range gate are sometimes changed at sites, so these parameters need to be checked by the user.
Our experience using the data from several sites and time periods, together with recent research, leads us to believe that the interaction between long pulse lengths and atmospheric reflectivity gradients may be causing the "high-mode" 915-MHz winds and the 50-MHz winds to be assigned 200-400m too high in altitude; in other words, winds reported as being valid at 2500m may actually be representative of conditions at about 2100-2300m. The exact shift is a function of the reflectivity gradient, which may vary with season and altitude. We continue to use the 915-MHz high mode data, as presented, in our own work. We are currently shifting the 50-MHz winds at Christmas Island down 200m for our own dynamical research; to a first approximation, this brings them into reasonable agreement with the 915-MHz winds. This is an area of very active study, and we will update known users of the data on any future developments. (10/1999)
Recent comparisons and site visits have revealed a minor problem with the 915-MHz data at Christmas Island. More precise measurements of the beam positions for the original 915-MHz profiler (in place through February 1998) were made in September 1998. Until such time as we reprocess the 915-MHz data (an activity not currently planned), all 915-MHz winds collected before March 1998 must be rotated 2 degrees clockwise, e.g. a wind currently listed as coming from due east, 90 degrees, is really coming from 92 degrees. Since the data files give wind components rather than speed and direction, users will have to convert the format to make this change. We continue to use the data, with this correction applied, in our own work, and will update known users of the data on any future developments. (10/1999, 01/2000) Note: This correction has been made in the new gridded data files, but not in the original TOGA-COARE format files. (04/2003)
More precise measurements of the beam positions for the 915-MHz profiler at Tarawa reveal that winds collected before 0502:53 on 9 November 1999 (day 313) must be rotated by 1 degree counterclockwise, e.g. a wind currently listed as coming from 265 degrees was really coming from 264 degrees. Since the data files give wind components rather than speed and direction, users will have to convert the format to make this change. We continue to use the data, with this correction applied, in our own work, and will update known users of the data on any future developments. (01/2000)
* * * * * SUMMARY -- SPECIFIC RECOMMENDATIONS * * * * *
updatedApril 2003
- Only use data with quality flags 0-3.
- Use low-mode data at Kapingamarangi, RV Kexue #1, and RV Shiyan #3 at and above the 4th, 7th, and 7th range gates, respectively.
Increase Christmas Island 50-MHz winds before December, 1993,by 5%. (This correction was made to our data files in September, 2001.)- Move Christmas Island 50-MHz winds down in altitude by 200m.
- Rotate Christmas Island 915-MHz winds before March, 1998,2 degrees clockwise (TOGA-COARE files only).
- Use 915-MHz high-mode winds from any site as presented, but keep in mind that the heights assigned to the winds may be 200-400m too high.