APIMASH Quakes & Astronomy

A bit overdue, but I just realized I never really announced the APIMASH project we’ve been working on.  It’s up on github.   The purpose of the project is to illustrate how to go about constructing a mashup application from a variety of data sources.  We’re adding to the project all the time, and have both Windows 8 and Windows Phone app starter kits to get you going.  What makes the APIMASH concept so fun, to me, is that we’re consuming real-world data.   Whether you use one of our APIs we consume in the kits, or find one through a site like Mashery, Programmable Web, the Azure Datamarket, or others, mashup applications are a LOT of fun to create.  Because many mashup applications use similar patterns, consume similar JSON data, etc., you’ll see it’s easy to adapt the code to just about any scenario. I’ve got a couple of examples in the APIMASH project, and more to come soon.  The first one is an earthquake sample that illustrates how to consume a data feed from the USGS that contains earthquake information for a given time frame.  That data is then plotted on a Bing map.   This starter kit is available as both a Windows 8 app and Windows Phone app.   The starter kits are very bare-bones (intentionally) – a more complete version based off these templates is available in the Windows Store as an app called Earthquake Explorer.   Conceptually, the apps are the same, but the finished app in the store illustrates what you can do by adding a bit of polish to the app.  You can read more about creating a great mashup using this app as an example here in these blog posts. Starter Kit: Completed app: The other project I created in the APIMASH project is called Messier Sky Objects, which is available in the Windows Store as Messier Object Explorer.   This is a mashup (of sorts) that combines Worldwide Telescope with the data of all Messier objects, a collection of objects like nebulae, galaxies, clusters, et. al.  By using the Worldwide Telescope JavaScript API, it’s simple to create great-looking astronomy apps (in fact, many of my apps, listed in my side-bar on my home page, use this as a template). So what’s next?  I just released Brew Finder for Windows 8.   This app uses the brewerydb.com API to show local breweries and is available in the Windows Store here.   I hope to have a nice sample in the APIMASH project soon based on this project. Have another sample you’d like to see?  Leave a comment and we’ll review it on the next team meeting!

Creating a Great Mashup Part 3

This post details features used by Earthquake Explorer, a Windows 8 app that displays earthquake information on Bing maps.   Earthquake Explorer was based off of the Earthquakes mashup starter kit on github.  Read part 1 here.  Read part 2 here.  We left off Part 2 talking about creating a compelling live tile in Windows 8.   This really is a must-do feature in Windows 8 – primarily because a live tile creates a more compelling app experience.  It encourages the user to keep the live tile ‘pinned-left’ (that is, more likely on their primary screen), and encourages the user to open the app more frequently.  That equals higher app usage, higher ad impressions, etc.  Because the earthquake app is consuming earthquake data on our backend using Windows Azure Mobile Services, it’s trivial to send notifications when new earthquakes are detected.  We can send a nice live tile that looks like so: Clearly, this is better than displaying _nothing_, and the map adds a professional look to it.   We can also combine this with badges – for example, in Dark Skies, I do something similar when users add new locations to the map, adding a badge notification that displays how many new locations are nearby: The first step in doing this is to determine which tile template to use.  Because the map is the main image, this is a pretty simple as we’d want the map to take up the main tile.  The place to look is the tile template catalog on MSDN.  In this case, TileWideImageAndText02 seems to fit the bill for wide tiles, and TileSquareImage is the bet bet for smaller, square-sized tiles.  From the template page: The key here is the tile XML – it’s the same if we’re doing a push notification, if the app is updating its own tile, or if Windows is updating the tile using the TileUpdater using periodic updates.  To get the image, we’ll use the Bing Maps REST API, which requires a valid Bing Maps key from the portal site.  For a wide tile, we can get a static map using a URL like so, specifying the zoom level (2), centered location (36 latitude, –79 longitude), pushpin location (also at 36 latitude, –79 longitude), and a size of 310x100: http://dev.virtualearth.net/REST/v1/Imagery/Map/Road/36,-79/2?mapSize=310,100&pushpin=36,-79;48&key={your_api_key} Which returns an image like so: From the Windows Azure Mobile Service, it’s fairly simple to send a notification with some code like the below.  I’m removing some of the dynamic text to shorten it up a little, but realistically we’d be setting the location of the pin(s) and text of the tile itself dynamically: 1: function sendNotification(channelUri) { 2:   3: var baseImageUrl = "http://dev.virtualearth.net/" + 4: "REST/v1/Imagery/Map/Road/36,-79/2?[mapsize]" + 5: "&pushpin=36,-79;48&key={your_api_key}"; 6: 7: var smallImage = baseImageUrl.replace("[mapsize]", "mapSize=150,150"); 8: var bigImage = baseImageUrl.replace("[mapsize]", "mapSize=310,100"); 9:   10: var payload = "<tile><visual><binding template='TileWideImageAndText02'>" + 11: "<image id='1' src='" + xmlEscape(bigImage) + "' alt='map'/>" + 12: "<text id='1'>Some top line text here</text>" + 13: "<text id='2'>Some bottom line text here</text>" + 14: "</binding>" + 15: "<binding template='TileSquareImage' branding='none'>" + 16: "<image id='1' src='" + xmlEscape(smallImage) + 17: "' alt='map'/></binding></visual></tile>"; 18:   19: var theTag = 'some_identifier'; 20: push.wns.send(channelUri, payload, 'wns/tile', 21: { 22: client_id: 'your ms-app:// id', 23: client_secret: 'your client secret', 24: headers: { 'X-WNS-Tag': theTag } 25: }, 26: function (error, result) { 27: if (error) { 28: console.log("(" + error.statusCode + ") Error sending notification"); 29: } 30: } 31: ); 32: }  .csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; } (Note: the xmlEscape method is omitted, but is a simple method that escapes special characters for the XML document.)  What we’re doing here is using the push.wns (Windows Notification Service) object to send the tile XML directly.   If you review the script reference, it is certainly simpler to use the push.wns.sendTileWideImageAndText02 method, but I generally prefer to build the XML directly as done above.  The reason is that it allows you to bundle multiple notifications together – such as including both the wide and narrow tile versions.  (The other side benefit is that this allows you to enter the application package ID and secret, which should allow a single WAMS to send notifications to multiple applications.) We’ll look at other mashup examples in future posts!

Creating a Great Mashup Part 2

This post details features used by Earthquake Explorer, a Windows 8 app that displays earthquake information on Bing maps.   Earthquake Explorer was based off of the Earthquakes mashup starter kit on github.  Read part 1 here.  In the first part, we created a Windows Azure Mobile Service to store data as a back end for our mashup.  A big benefit of this approach is that it lets us filter/sort/query the data any way we’d like, and our application is now resilient against outages and changes in the schema.   In this part, we are going to modify our existing scheduled task to consume the actual USGS data feed and store it in a table. Getting the data is simple, thanks to the nice GeoJSON feed provided by the USGS.  Read more about that here: http://earthquake.usgs.gov/earthquakes/feed/v1.0/ From within our scheduled job, we can query it like so: function fetchUSGS() { console.log('Starting job'); var httpRequest = require('request'); var uri = 'http://earthquake.usgs.gov/earthquakes/feed/v0.1/summary/all_hour.geojson'; httpRequest(uri, function (err, response, body) { if (err) { console.warn('Error connecting to service.'); } else if (response.statusCode !== 200) { console.warn('Unexpected reply.'); } else { //got reply, process the results var theData = JSON.parse(response.body); processData(theData); } }); } .csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; } We’re making a request to the feed (uri) and specifying the callback as the inline function.   There really isn’t any robust error handling, but then, there really isn’t much you can do in the event of an error except just try again the next time the task runs.  Calling JSON.parse() will, naturally, parse the response text into a nice object collection we can parse.  If we look at the JSON in fiddler, you can see the structure of the document: The JSON contains a collection of features, where each feature contains a geometry object, an id string, and a properties object.  Either through the documentation on the USGS site, or through examining the results, we’ll need to know the structure of the data. To make working with the data a bit easier we can create a simple javascript class to hold the data: function earthquake(item) { this.latitude = item.geometry.coordinates[1]; this.longitude = item.geometry.coordinates[0]; this.depth = item.geometry.coordinates[2]; this.usgsId = item.id; this.usgsCode = item.properties.code; this.tz = item.properties.tz; this.mag = item.properties.mag; this.time = item.properties.time; this.updated = item.properties.updated; this.place = item.properties.place; this.timestamp = new Date(); } The one tricky part here is that the identifier of the earthquake is the USGS ID, but all data in Azure Mobile Services uses a bigint as a primary key.  When we’re updating a row in Mobile Services, we’d pass the object into the update method with the appropriate ID (bigint).  (Deletes require the bigint ID only.)  Yes, this is a slight inefficiency because WAMS uses the surrogate bigint, and the USGS prefers the natural key of the USGS ID, and we will, for simplicity sake, use both depending on the operation.  Because we’re requesting the data frequently, each item in the feed will have one of a few states:  it may be new (not in the database), it may be in the database already, or it may be in the database but the feed contains updated/revised values. function processData(json) { var quakesTable = tables.getTable('earthquakes'); var featureList = json.features; featureList.forEach( function (item) { var eq = new earthquake(item); quakesTable.where({ usgsId: eq.usgsId }).read({ success: function (results) { if (results.length > 0) { //record exists eq.id = results[0].id; eq.timestamp = new Date(); quakesTable.update(eq); } else { //record doesn't exist quakesTable.insert(eq); //send notification? //see next blog post } } }); } ); } .csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; } As we iterate through the features, we create an earthquake object from each item.   We then check the table for a record with the same USGS ID property.  If it already exists, we’ll update the row with the information because details may have changed.  Otherwise, we’ll insert it.  There’s a general inefficiency here that we can deal with if we’re so motivated.  Each item results in a query to see if the item exists, and then a query to either update or insert.  We could simply build a stored procedure that handles either scenario so it’s one call, but because each feed contains about 10 items and this scheduled task is a server-side job, it’s not high on my priority list.  With higher volume apps and/or with client apps, this is a modification you should make. The other nice thing we can do is send notifications when a new earthquake is found.  For example, if a new earthquake is found in the feed, we can send a live tile notification which would update the tile to look like so:   For Windows 8 apps, this is really a must-do feature.  Do _something_ (anything!) with the live tile – this will differentiate your app.  Mobile Services can send notifications to Windows Phone, Windows 8, iOS, and Android, so this is a great way to take the app cross platform.  In the next post, we’ll look at how to do a live tile update like the one above.

Bing Maps Mashup in Dark Skies

I’ve gotten a few requests about how to do a mashup in Dark Skies … so, thought I’d do a blog post on how it was put together.  First, what’s a mashup?   A mashup is an application that combines data from multiple sources into what is, hopefully, a more useful or interesting way to view the data.  They are often done as web applications because the data is often pulled from online sources.  But, mashups are ideal for Windows 8 and Windows Phone apps, too.  Dark Skies is simply a mashup that combines 3 main sources of data.  maps (from Bing), light pollution data (from sources like these: Save The Night, NOAA, and P. Cinzano), and favorite astronomy viewing locations and looks like so: What makes this interesting is the level of zooming, so you get really specific: Specific is good, but add in data sharing and you, hopefully, have an ideal mashup experience. Overlaying pins is easy as the map.  In fact, there’s a small sample on putting pins on a map here.   The harder part is storing the data, and for this, Windows Azure Mobile Services works really well.  I’ve talked about that in previous posts. The light pollution data is available from a variety of sources but in this case, a high res version with a color that can be made transparent (black) is ideal.  Credit: P. Cinzano, F. Falchi (University of Padova), C. D. Elvidge (NOAA National Geophysical Data Center, Boulder). Copyright Royal Astronomical Society. Reproduced from the Monthly Notices of the RAS by permission of Blackwell Science.   There are two fundamental problems in overlaying the two: first, the image should be sliced into small tiles to make it bandwidth sensitive (the uncompressed TIF file is 200MB).   The second is that the image doesn’t exactly line up.  It’s close, but there are subtle errors that get introduced despite both maps appearing to be Mercator projections.  There are a number of tools that can help you solve this problem, but in this case, I used MapCruncher.  It’s from Microsoft, and it hasn’t been updated since 2007, but … it works.  The idea is that MapCruncher can both transform/skew an image to fit the projection of the map, and carve it up into nice, little tiles.  Let’s take the following image: If I want to overlay this image at a specific point on the map. Let’s say that the two eyes should be where Lake Superior and Lake Huron are.  In MapCruncher, we’d bring in the image as an asset, and start defining matching points in the image, like so:     It’s not until we hit the lock button near the bottom left that the image is scaled/skewed/transformed to fit.  The closer an image is to being the same scale, the better/few points are required to get it look right.  Once you hit the Render button in the bottom left, it will go to town and slice and dice the image up, based on the zoom depth requested.  More on that in a second. In this case, it generated some 92 tiles.  When we preview the results, we can see the data is overlaid nicely: The tiles produced are laid out and named format known as quadkeys.   You can read a lot more about it here, but in short, each tile is scaled to fit the appropriate dimensions based on the current zoom level:   The pattern to this approach makes it very easy to know exactly what tile you need, and at what zoom level (detail).  Additionally, all of these files can be stored in a flat data structure (like a folder) which makes storage quite simple.  The best part is, Bing maps already knows how to build a quad key, so all you have to do is create a tile layer, and specify the quadkey as a parameter by putting it in curly braces, like so: MapTileLayer pollutionTileLayer = new MapTileLayer(); pollutionTileLayer.TileSource = "myurl.com/{quadley}.png";   But, this isn’t quite ideal.  One problem that I ran into is that if there are a large number of quad keys you don’t have (which is the case even in my app), there are a HUGE number of 404’s because the app has no way of knowing there isn’t a tile available for a specific quad key.  While the user doesn’t directly see this, it’s a lot of wasted traffic and just not clean.  The way to solve that is to roll your own GetTileUri handler. In my next post, I’ll detail the steps involved in setting that up!

My Apps

Dark Skies Astrophotography Journal Vol 1 Explore The Moon
Mars Explorer Moons of Jupiter Messier Object Explorer
Brew Finder Earthquake Explorer Venus Explorer  

My Worldmap

Month List