Halogen Labs Weblog

I’ve been uncovering more information. The NWS does have plans for Web services. Check out this PDF I found, page 4:

• Machine to machine queries to retrieve a “slice” of the database – specific locations and elements
• XML based queries, results in XML
• Uses W3C standard protocols and formats: UDDI, WSDL, SOAP, XML
• Currently in planning
• Experimental services available spring/summer

I wonder where these bad boys are hiding… check out all the other PDF files hiding in the same directory. Also check out National Digital Forecast Database.

LaughingMeme: Weather RSS and the Dangers of Screen Scraping

Kellan over at LM describes why screenscraping sucks: Obscure URLS, inconsistent HTML, poor data, etc etc.

I describe what needs to be done:

The real way to do this is to get hired by the NOAA and be given a license to do whatever you want with their data. Because this is highly unlikely (ATTN NOAA: Hire me plz), the best we can do for now is to manually decode metar data that we directly download from the NOAA.

RSS could bring NOAA into the 21st century, they just need to realize it.

(via MoreLikeThis)

Over at MoreLikeThis, there’s some discussion about Screen-scraping using XSL and Tidy. I’ve done stuff like this already, but entirely within Cocoon using the HTMLGenerator, which happens to JTidy the input, and I specify an XPath statement to “dig” into the page a bit to get to the goodies I want. After that, It’s all a matter of using XSL to transform to an intermediate format, and then another XSL to output to the format I want.

I’m using this not only in openWeather, but also in a personal little project of mine which aggregates job listings from sites like Techies.com, Dice.com, Monster, etc, and displays everything in a nice quick to read format. Now, instead of checking 4 sites, all I do is hit my private Cocoon page and I can scan through all the listings in about 10 seconds. Very efficient.

Edit: Oh yeah, I should turn those job listings into RSS feeds. I could be lightyears ahead of any of the job sites! :)

Just been playing around with the new scheduler component which was added recently to the Cocoon CVS scratchpad. It’s everything I wanted! It’s an Avalon component so it fits in perfectly with everything already!

Here’s a sample of the log output:

Target aggregatorTrigger triggered at Mon Aug 25 01:14:16 CDT 2003

Target aggregatorTrigger triggered at Mon Aug 25 01:14:26 CDT 2003

Target aggregatorTrigger triggered at Mon Aug 25 01:14:36 CDT 2003

Target aggregatorTrigger triggered at Mon Aug 25 01:14:46 CDT 2003

I have a trigger setup to dump output to STDOUT every 10000ms (10sec). A big part of openWeather was going to be the service which ran periodically to download weather reports, now I can have this all within Cocoon! Sweet!

Scheduler docs coming soon to the Wiki :)