Technical details of the Android app Weather Alert Canada

EDIT: Google has since demised GCM and now Environment Canada has its own app to push notification to mobile devices. This app has been unpublished from the Play store.

In late August 2015, there was a massive wind storm in Metro Vancouver that knocked out power to many people.  I relied on radio and Twitter to get updates but I thought it would be better if I could get notifications directly from Environment Canada.  A quick search on Google Play store turned up void.  And hence I spent some time to implement an Android app.

These are the technical details behind the app.

Getting the data

Meteorological data can be retrieved easily from Environment Canada.  The question is, what data to use?  My initial thought was to parse the forecast files and extract alert messages from there.  However, the forecast data is broken down into many files, each for specific area.  And these files are generated frequently.  As I planned to host the logic on Google App Engine (more on that later), I would like to minimize the network traffic.

Luckily, Environment Canada also provides public weather alerts in CAP format.  The CAP files are categorized by issuing offices.  Inside the file, the affected area is described by polygons and geocodes that are intended to overlap directly on a map.  However, what I needed was to somehow link each affected area to provinces / territories. Good thing is Environment Canada has geo data that can map the area description in CAP to specific provinces.

Push mechanism

A few years ago when I first played with push notification, I tried Amazon SNS.  The advantage is that it supports Google Cloud Messaging (GCM), Apple Push Notification (APN) and many more.  But it turns out that since Oct 2013, GCM also supports iOS devices.  So, to keep it simple this time, the backend utilized GCM to push out notifications.

To be specific, GCM Topic is used as the downstream mechanism to broadcast messages.  It is much simpler than keeping registration id of each device and sending out targeted GCM messages.  The downside is, users cannot request to receive notification on certain areas only (well, technically it can be done with topic subscription too, but that means I need to create lots of GCM topics for different areas...)

Backend logic

Before implementing the server side with App Engine, a quick prototype  was created using Python.  urllib2 is used to fetch CAP files from Environment Canada site. After processing the xml file, GCM messages are pushed out with urllib2 in JSON format.

A script was also created to load mapping between areas and provinces into a database.  DBM was used as the API is simple and made it easy to port to Google Datastore.

Another DBM file was also used to keep track of processed CAP files.

Once the prototype is working, porting it to App Engine was easy.  Cron jobs were defined to pull CAP data periodically.  There are also cron jobs to clean datastore of processed files.

Datastore and memcached

When moving the Python scripts to Google App Engine, the DBM databases were ported to  Datastore. As I intended to use the App Engine free tier, there was a need to keep the I/O low.  Luckily App Engine provides memcache as a free service.  With proper caching (over 98% hit rate), I could keep the datastore utilization as low as 2%  of the daily quota.

Android app

The android app was a shameless clone of the GCM sample application.  On the navigator bar, users can choose to show notifications for specific provinces only.  The main window is a WebView.  A HTML file and some JavaScript are included in the app to fetch Environment Canada WeatherLink.