7.6.0 |
S3 Bucket Entry
|
7.6.1 |
More Configuration
|
7.6.2 |
S3 Harvesting
|
7.6.3 |
Specifying Entry Types
|
7.6.4 |
S3 Tools
|
7.6.0 S3 Bucket Entry
To create a new S3 entry go to the File->Pick a Type menu and select "AWS S3". In the create form entry the S3 URL, a name and an optional description:
After adding the entry you can edit it to configure how the S3 buckets are displayed.- Enable caching: Turn this off if you are working on the settings as the entries are normally cached and any changes you make might not take effect. Note: turn this back on once you are done.
- Exclude patterns: Regular expression patterns, one per line, that are used to exclude certain S3 objects.
- Max per folder: Max number of folders or files listed per folder
- Percentage filter: Used to sub-sample the number of folders/files shown per folder. Value is a probablility that a folder/file will be included with range from 0-1
- Max file size: Max size of a file to include
- Convert dates: If yes then RAMADDA will see if the name of a folder and file matches certain basic patterns for dates (e.g. yyyy for year, yyyy/mm for year then month, etc)
- Date patterns- a ";" seperated 3-tuple, one per line. Each 3-tuple has a:
pattern;template;date format
and is used to match against a folder or file name. The pattern needs to contain "(...)" groups. If it matches then the matching groups are extracted and the template (the 2nd part of the 3-tuple) is used to create a date string. This date string is then parsed by the date format.
7.6.1 More Configuration
key1=value1 key2=value2 ...If it is a .txt file then the first line is the key that is matched against the S3 folder names and the rest of the file is a wiki display template:
key_or_pattern wiki text ...
For the NEXRAD example we will upload a nexrad.csv file. This has the following structure:
id,fullname,lat,lon KAPD,KAPD - Fairbanks/Pedro Dome,65.035,-147.50167 KAEC,KAEC - Fairbanks/Nome,64.51139,-165.295 KACG,KACG - Juneau/Sitka,56.85278,-135.52917 KAIH,KAIH - Anchorage/Middleton Island,59.46139,-146.30306 ...The first column is the station ID and this is used to match up (either exact or pattern) with the S3 folders. Upload the nexrad.csv file and select Alias and Exact: Now, the station folders have the full name. Next, upload the same file and select "Location". This adds the spatial metadata to the station folders.
Note: we have combined the aliases with the locations. The aliases file could have been a properties file: nexrad_aliases.properties file.
Next, we want to specify the wiki template to be used for the folder that contains the stations, i.e., the year/month/day folder. Here we will upload this nexrad_templates.txt file. It has the format:
^\d\d/\d\d$The first line is a pattern that matches NN/NN, i.e., the month/day folder names. The remainder of the file is the wiki text to be used for the folders that match the pattern. Upload this file, select "Template" and for match type select "Pattern". Now, for the day folder we should have: And finally, we want to show the files as "Level 3 Radar" files, a file format that RAMADDA provides. To do this go to the Add Properties menu and under Miscellaneous select "Entry Type Patterns". This property allows you to specify an entry type (e.g., Level 2 Radar File) and one or more patterns to match. For the NEXRAD data collection we know that all of the files are of this type so we specify as a pattern "file:" Now, with that the files should be shown as this entry type:+section title="NEXRAD Stations - { {name entry=parent} } { {name} } { {name entry=grandparent} }" +inset left=20px right=20px { {map listentries=true hideIfNoLocations=true} } ...
7.6.2 S3 Harvesting
7.6.3 Specifying Entry Types
7.6.4 S3 Tools
sh s3.sh -help Usage: S3File <-key KEY_SPEC (Key spec is either a accesskey:secretkey or an env variable set to accesskey:secretkey)> <-download download the files> <-nomakedirs don't make a tree when downloading files> <-overwrite overwrite the files when downloading> <-sizelimit size mb (don't download files larger than limit (mb)> <-percent 0-1 (for buckets with many (>100) siblings apply this as percent probablity that the bucket will be downloaded)> <-recursive recurse down the tree when listing> <-search search_term> <-self print out the details about the bucket> ... one or more bucketsAnother utility is provided in the RAMADDA repository runtime at the /aws/s3/list entry point. This provides an interface to the above tool to do a recursive listing of a bucket store. Check it out on https://ramadda.org/repository/aws/s3/list. Try this out with an example:
s3://first-street-climate-risk-statistics-for-noncommercial-use/