访问美国国家气象局(NWS) Forecasting API

sgenzer

So many of you know how much I love APIs, and as much as I love the Web Mining extension with Get Page and Enrich Data via Webservice, they have limitations. I have often dreamt really robust JSON parser in RapidMiner...well my friend@landat Old World Computing has created a beauty in his newWeb Automation Extension- currently in beta as of this writing.

Fortunately for me, Sebastian gave me a copy of this beta to run through its paces and I thought, "Hey! I've been meaning to tackle that huge, nasty National Weather Service Forecast API for ages. This will be PERFECT!"

And it was. It's not pretty but it gets the job done. So here it goes...

If you're not familiar with the NWS API, it's fairly simple on the surface (pardon the pun). Just send latitude and longitude and you get a weather station. Then send a second query to that weather station and poof - you get forecast information in nice JSON. Like this:

Get the URL for the Weather Station covering Norwich, Vermont USA = 43.7022 latitude, 72.2978 longitude

URL:https://api.weather.gov/points/43.7022,72.2978

Response:

{ "@context": [ "https://raw.githubusercontent.com/geojson/geojson-ld/master/contexts/geojson-base.jsonld", { "wx": "https://api.weather.gov/ontology#", "s": "https://schema.org/", "geo": "http://www.opengis.net/ont/geosparql#", "unit": "http://codes.wmo.int/common/unit/", "@vocab": "https://api.weather.gov/ontology#", "geometry": { "@id": "s:GeoCoordinates", "@type": "geo:wktLiteral" }, "city": "s:addressLocality", "state": "s:addressRegion", "distance": { "@id": "s:Distance", "@type": "s:QuantitativeValue" }, "bearing": { "@type": "s:QuantitativeValue" }, "value": { "@id": "s:value" }, "unitCode": { "@id": "s:unitCode", "@type": "@id" }, "forecastOffice": { "@type": "@id" }, "forecastGridData": { "@type": "@id" }, "publicZone": { "@type": "@id" }, "county": { "@type": "@id" } } ], "id": "https://api.weather.gov/points/43.7022,-72.2978", "type": "Feature", "geometry": { "type": "Point", "coordinates": [ -72.297799999999995, 43.702199999999998 ] }, "properties": { "@id": "https://api.weather.gov/points/43.7022,-72.2978", "@type": "wx:Point", "cwa": "GYX", "forecastOffice": "https://api.weather.gov/offices/GYX", "gridX": 8, "gridY": 48, "forecast": "https://api.weather.gov/gridpoints/GYX/8,48/forecast", "forecastHourly": "https://api.weather.gov/gridpoints/GYX/8,48/forecast/hourly", "forecastGridData": "https://api.weather.gov/gridpoints/GYX/8,48", "observationStations": "https://api.weather.gov/gridpoints/GYX/8,48/stations", "relativeLocation": { "type": "Feature", "geometry": { "type": "Point", "coordinates": [ -72.305660000000003, 43.718015999999999 ] }, "properties": { "city": "Norwich", "state": "VT", "distance": { "value": 1868.6929186969621, "unitCode": "unit:m" }, "bearing": { "value": 160, "unitCode": "unit:degrees_true" } } }, "forecastZone": "https://api.weather.gov/zones/forecast/NHZ005", "county": "https://api.weather.gov/zones/county/NHC009", "fireWeatherZone": "https://api.weather.gov/zones/fire/NHZ005", "timeZone": "America/New_York", "radarStation": "KCXX" } }

There's a lot of information there but all I really care about is the URL:

forecastGridData": "https://api.weather.gov/gridpoints/GYX/8,48

So now I put that into OWC's fancy JSON parser like this:


The OWC JSON parser uses a subprocessbefore the actual Parse JSON operatorto specify how and where to parse. I chose "Process Object" because the root of this JSON was in this form:

{ "foo" : "bar", "foo2" : "bar2", ... }

So this is a list of对象sinside one main对象(objects are shown as {} in JSON). So if I only wanted to extract simple items likeidandtype, I would just put this inside of Process Object:





Which I do. But I also want that forecaseGridData URL which is inside the 'properties' object, so I do this instead:



And finally inside of Process Object (3) just another Extract Properties and Commit Row with forecastGridData. Easy!

Get the Weather Forecast from this NWS URL

So once you get this URL, you just send another Get Page and voilà - a MASSIVE JSON with all sorts of goodies. It's too long to post here so I'll just abbreviate it...

{ "@context": [ "https://raw.githubusercontent.com/geojson/geojson-ld/master/contexts/geojson-base.jsonld", { "wx": "https://api.weather.gov/ontology#", "s": "https://schema.org/", "geo": "http://www.opengis.net/ont/geosparql#", "unit": "http://codes.wmo.int/common/unit/", "@vocab": "https://api.weather.gov/ontology#", "geometry": { "@id": "s:GeoCoordinates", "@type": "geo:wktLiteral" }, etc... "county": { "@type": "@id" } } ], "id": "https://api.weather.gov/gridpoints/GYX/8,48", "type": "Feature", "geometry": { "type": "Polygon", "coordinates": [ [ [ 120.5517605, 90 ], [ 120.5517605, 90 ], [ 120.5517605, 90 ], [ 120.5517605, 90 ] ] ] }, "properties": { "@id": "https://api.weather.gov/gridpoints/GYX/8,48", "@type": "wx:Gridpoint", "updateTime": "2018-12-18T19:44:04+00:00", "validTimes": "2018-12-18T13:00:00+00:00/P8DT6H", "elevation": { "value": 160.93440000000001, "unitCode": "unit:m" }, "forecastOffice": "https://api.weather.gov/offices/GYX", "gridId": "GYX", "gridX": "8", "gridY": "48", "temperature": { "sourceUnit": "F", "uom": "unit:degC", "values": [ { "validTime": "2018-12-18T13:00:00+00:00/PT2H", "value": -6.6666666666666288 } etc... { "validTime": "2018-12-26T17:00:00+00:00/PT1H", "value": -3.8888888888888573 }, { "validTime": "2018-12-26T18:00:00+00:00/PT1H", "value": -2.7777777777777715 } ] }, "dewpoint": { "sourceUnit": "F", "uom": "unit:degC", "values": [ { "validTime": "2018-12-18T13:00:00+00:00/PT2H", "value": -12.222222222222229 }, etc... { "validTime": "2018-12-26T18:00:00+00:00/PT1H", "value": -9.4444444444444571 } ] } A LOT GOES HERE.... }, "stability": { "values": [ ] }, "redFlagThreatIndex": { "values": [ ] } } }

Then you use the same JSON parsing tools (I also did some post-processing to convert units - yes I still think in inches and feet), and you get a five-day forecast in a nice RapidMiner ExampleSet:



Easy!

I will post the process on the Community Repository soon. Stay dry!

Scott

Telcontar120

Excellent use case! I would love to see that process to play around with it as well.
I have also been testing the OWC JSON extension and it is great.

sgenzer

I'm going to attach the process here...not sure when I can get it on the repo.

Scott