Fastest way to warm an AWS Redis Elasticache with JSON data

After creating an a Elasticache instance for Redis and loading it with data dynamically, it became apparent that it would be useful to warm the cache with a large dataset to further improve performance.  Redis recommends using their Luke protocol to accomplish this by creating a text file with the data and then use the redis-cli to import that into Redis directly. This approach is significantly faster than doing it programmatically. Here I’ll elaborate on the steps I took to do that.

The first step is to create a text file with the data you want to import.  The basic structure is:

set [key] "[JSON string]"

set [key] "[JSON string]"

set [key] "[JSON string]"
...

However, the devil is in the details. First, you must escape all the quotes in your JSON string. Also, you must be sure to have a carriage-return/line-feed after each line or your data won’t import. This is required even after the last line in your file.

Once you have your file, you then need to create (or use an existing) EC2 instance to call your redis instance and import the data.  On your EC2 instance you first need to install gcc to build the redis-cli and then make the redis-cli code.  Here are the steps:

sudo yum install gcc

wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz
cd redis-stable
make

Once you do this, you can then pipe your text file into your Redis instance using the following command:

redis-stable $ cat import.txt | ./src/redis-cli \
-h [elasticache hostname from replication group] -p 6379 --pipe

That’s it! If you want to schedule this on interval, you can simply create a cron job to refresh your cache as needed or use an event driven architecture to update the cache based on changes from the source system.

Advertisements

Protecting Node.js services written with Restify using Passport

I’ve been using Restify to create all my REST endpoints for the various services that I write. Invariably, I need to protect these services end-to-end. For client-facing UI’s I do that with OAuth, but my OAuth proxy needs to access the actual underlying service. To do that safely, I do this over HTTPS and I created a service account and used the passport-http module to do BASIC authentication with the backend service.

Here’s a smallish node.js module that I wrote to accomplish this:

    var restify = require('restify'),
        userId = process.env.ID,
        pwd = process.env.PWD,
        passport = require('passport'),
        BasicStrategy = require('passport-http').BasicStrategy;

    passport.use(new BasicStrategy(
        function (username, password, done) {
            findByUsername(username, function (err, user) {
                if (err) {
                    return done(err);
                }
                if (!user) {
                    return done(null, false, { message: 'Incorrect username.' });
                }
                if (user.password !== password) {
                    return done(null, false, { message: 'Incorrect password.' });
                }
                return done(null, user);
            });
        }
    ));

    // Just use a single user that can access this service set from deployment var
    var users = [
        { id: 1, username: userId, password: pwd}
    ];

    function findByUsername(username, fn) {
        for (var i = 0, len = users.length; i < len; i++) {
            var user = users[i];
            if (user.username === username) {
                return fn(null, user);
            }
        }
        return fn(null, null);
    }

    exports.authenticate = function (req, res, next, callback) {
        passport.authenticate('basic', function (err, user) {
            if (err) {
                return next(err);
            }
            if (!user) {
                var error = new restify.InvalidCredentialsError("Failed to authenticate.");
                res.send(error);
                return next();
            }

            callback(req, res, next);
        })(req, res, next);
    };

To use the module, all you have to do is “require” it and make a call to it passing in a callback function to the REST service that you’ll end up calling once authenticated. This is a simple example of that:

    var basicAuth = require(__dirname + '/../utils/authn.js');

    exports.viewPostsByUid = function (req, res, next) {
        basicAuth.authenticate(req, res, next, function() {
            validateParam(req.params.uid, next);
            var url = '/posts/' + encodeURIComponent(req.params.uid);

            processRequest(req, res, next, url);
        });
    };

As you can see, it’s fairly trivial to protect your services with BASIC authentication and is a good line of defense for any attacks directly to the service itself.

Deploying Meteor 1.0 apps with a Heroku Buildpack that does NOT use Meteorite

I’ve been using Meteor since 0.6.x and not too long ago Meteor was using a package system called meteorite. Now with Meteor 1.0, meteorite has been deprecated so this ultimately changes the way you can deploy your application should you want to do that with a Heroku buildpack.

I was getting exceptions with fibers and a recommendation to add ROOT_URL to my config which I didn’t have to do previously on version 0.9.3.  But once I updated to this buildpack: https://github.com/AdmitHub/meteor-buildpack-horse, life was good.  I also added the ROOT_URL: [http://myapp.com] to my config and I was back to successfully deploying my Meteor app using this new buildpack which has no dependency on meteorite.

Updating an existing Cordova App to iOS 8

Here are a few tips when you are ready to upgrade your existing Cordova App to iOS 8.  I migrated my Cordova 3.3.0 app this morning successfully following these instructions.

Step 1

Run npm install to upgrade Cordova to the latest with the following sequence of commands:

npm install -g cordova
cd my_project
cordova platform update android

Step 2

Update your project’s “Build Settings” for the target as follows:

    Build Active Architecture Only: “Yes”
    Valid Architectures: “arm64 armv7”

Step 3

If you’re updating an older version of Cordova, you will need to comment out or remove these lines from the “MainViewController.m” class:

- (BOOL)execute:(CDVInvokedUrlCommand*)command
{

    return [super execute:command];
}

Step 4 (optional)
If you need to have your WebView adjust the top 20px to account for the iOS 7/8 status bar, install the following plugin and update your config.xml accordingly. I have mine set as follows:

    <feature name="StatusBar">
        <param name="ios-package" onload="true" value="CDVStatusBar" />
    </feature>
    <preference name="StatusBarOverlaysWebView" value="false" />
    <preference name="StatusBarStyle" value="lightcontent" />
    <preference name="StatusBarBackgroundColor" value="#000000" />

Shazron’s blog goes into more details on a few other essential plugins here.

By Loutilities Posted in General

2013 in review

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 35,000 times in 2013. If it were a concert at Sydney Opera House, it would take about 13 sold-out performances for that many people to see it.

Click here to see the complete report.

By Loutilities Posted in General

QConSF 2012: MongoDB to Scale

Kenny Gorman presents on the gotchas with MongoDB and how to scale it appropriately. 

Scale = very big, busy, tricky system

Denormalize till it works to get to true scale; split vertically and then horizontally

Scaling today is focused on rapid development

MongoDB is an OPS nightmare

built-in horiz. scaling and replication 65% deployments are in the cloud

MongoDB challenges:

  • Course lock scope; 2.2 has DB lock scope; 2.0 was global
  • Visibility
  • Schema – still need to business stuff
  • When bad things happen…it goes down fast

Design for scale:

  • Denormalize into diff. DB’s
  • Tune your workloads
  • NoORM; dump it
  • Shard early (Think about your shard key in advance)
  • Replicate (How to get availability)
  • Load test pre-prod

Other thoughts:

  • Embed vs. not
  • Indexing the right amount (btree index 10% drop in performance for every index added; covered indexes are cool)
  • Atomic ops; use them! (set, pop, push)
  • Use profiler and explain(); you can mine the profiler collection for optimizing

Shard Keys:

  • Tradeoffs – uses range-based shard key
  • Local vs. Scattered
  • Figure this out at design time

Architecture:

  • Engage all processors
  • Replication is election based and has fault zones so you can plan; “shell game” – rebuild slaves due to fragmentation
  • Client connections, getLastError

I/O

  • Need fast response time from disk when passivating for large DBs

Perform work on slave and rotate that back in for the primary to do the “shell game” 

Add arbitar to replicaSets so it prevents replica set down from voting for itself error

Random partitioning tips:

  • Monitor elections and who is the primary
  • Write scripts to kill sessions
  • automate or die
  • mongostat (like iostat)
  • historical performance and serverstatus to get histogram of perf. (or just use MMS)

Gotchas:

  • Logical schema corruption – use docs to manage it across teams especially types
  • Locks are DB so split DB’s if required; 50% lock status is considered slow
  • Not enough I/O perf
  • Engage all processors (use test harness and optimize for scaling)
  • Visibility (save data over time; monitoring; MMS, etc.)
  • Not understanding how MongoDB works; use it when it makes sense
  • Don’t believe the FUD

@kennygorman; kgorman@objectrocket.com 

rocketstat a good replacement for mongostat

Mobile GeoLocation App in 30 minutes – Part 3: Deploying to OpenShift

Overview

In the previous two parts, you saw how to create a Node.js application and store JSON data in MongoDB (Part 1). You also saw how to present this data in a mobile Web application using Sencha Touch complete with Google Maps and geospatial searching (Part 2). Now, we are going to take both of these two pieces and deploy them on a Platform as a Service (PaaS). I looked at three different provides in this space: OpenShift, Heroku and Nodejitsu. The latter is Node.js specific while the other two you can deploy other types of applications (Java EE, Ruby, PHP, Python, etc.). I really liked RedHat’s OpenShift model. It has several nice dev features that I thought worked well which I will go over in this post. Nodejitsu worked well too but didn’t use Git (a distributed SCM tool) to manage changes. Heroku required a credit card at the point I wanted to introduce MongoDB, so I opted not to continue with Heroku. It doesn’t hurt to experiment with multiple vendors since you can sign-up for free accounts on the respective sites.

One of the main motivations to use a PaaS is that it wouldn’t be very maintainable or production worthy SSH’ing to a server and running your node process every time you wanted to launch your app. Moreover, even if you run your process, it eventually will shutdown when you close your SSH session. There is an npm package forever that can keep this from happening, but I think running Node on a PaaS is ideal.

Using OpenShift to create a Node App

The first thing you’ll want to do is create a free OpenShift account here. Once you have an account created, you can create the application directly from the website (very convenient) or you can install Ruby and the RHC gem (sudo gem install rhc) that allows you to do the same thing from command line. The website is pretty intuitive so I won’t cover this and should you want to use the CLI, the set-up is sufficiently covered here. Note that there are a few prerequisites to use the CLI, namely installed Ruby and Msysgit. Be sure to have those installed as per the provided link before continuing.

For the purposes of deploying the app we created in Part 1, you would do the following after you’ve established your OpenShift domain and added your SSH public keys to commit code from Git:

First, we create “dogtag” as a nodejs app:

$ rhc app create -a dogtag -t nodejs-0.6

Second, we add the MongoDB cartridge:

$ rhc app cartridge add -a myMongo -c mongodb-2.0

Last, we add the RockMongo cartridge to administer MongoDB from the browser (context URI: /rockmongo from your domain)

$ rhc app cartridge add -a myMongo -c rockmongo-1.1

Working with your application in OpenShift

After you have created the app, the next part is copy the DogTag Node.js code to the “dogtag” directory that was created by OpenShift. Since Git is used to manage changes, you can run the following commands to push the code to the server:

$ git add .

$ git commit -am ‘Initial check-in’

$ git push

What will happen then is that the code changes will push to the server, stop the node.js instance, deploy your code and restart the server…very cool! It will pick-up any NPM packages you need as long as you define them in the package.json file (in our case, we need express and mongoose). After that you can go to the provided URL from OpenShift and test the various URL routes we defined in Part 1 (e.g., http://nodetest-loutilities.rhcloud.com/dogtag). You’ll notice that there is no data, so I created a JSON file that you can import into MongoDB to have some initial data. To do the import, you need to run the following command when you SSH to your server’s node:

$ mongoimport –host $OPENSHIFT_NOSQL_DB_HOST –port $OPENSHIFT_NOSQL_DB_PORT –username $OPENSHIFT_NOSQL_DB_USERNAME –password $OPENSHIFT_NOSQL_DB_PASSWORD –db dogtag –collection dogtags –file dogs.json –jsonArray

Note the $OPENSHIFT* provided constants so that you don’t have to figure out the actual host, port, username and password you are on which is especially helpful in a cloud environment running on multiple nodes. Once you import, you can check the files and add a geospatial index, which is required for the geospatial queries as follows:

$ mongo
> use dogtag
> db.dogtags.find() //will return results
> db.dogtags.find().forEach( function (e) { e.coords = [ e.longitude, e.latitude ]; db.dogtags.save(e); }); //will add the required format for a coordinates field to index
> db.dogtags.ensureIndex( { coords : “2d” } ) //indexes the coords field for geospatial queries
> exit

Now, if you go to the URL again you will have JSON data returned in a JSONP callback. You can change the callback simply by adding &callback=[your callback name]. This is necessary to prevent cross-site scripting issues and required when using the JSONP proxy in Sencha Touch.

Deploying the Mobile App

Since the Mobile App was created with Sencha Touch, I just used a “DIY” (Do it yourself) app from OpenShift to deploy my app there. The process is very similar:

$ rhc app create -a dogtags -t diy-0.1

This creates a very basic Web app. You can cd to the “diy” directory that is created and copy the code from the Mobile app there. It will prompt you to replace index.html, which is fine. There is one thing you need to do in the existing code; you will need to update the store’s proxy to point to your “dogtag” node.js app you created in the previous section as well as the Pettracker controller which also has these references. To do that, simply update the URL with the one OpenShift provides and append “dogtag” on the end to get the list as follows:

 proxy: {  
       type: 'jsonp',  
       url: 'http://nodetest-loutilities.rhcloud.com/dogtag',  
       reader: {  
           type: 'json',  
           idProperty: '_id',  
           rootProperty: 'records',  
           useSimpleAccessors: true  
       }  
}  

Lastly, use git to push the updates to the server. It will deploy the new code and restart the server. Launch your app from a Webkit browser or from your mobile device and you should be in business similar to what you see here.

Once the app is deployed to OpenShift, you can tail the logs by running the following:

$ rhc app tail -a <app-name>

You can also start and stop the server as follows:

$ rhc app [start | stop ] -a <app-name>

And to snapshot a particular revision and go back to it you run the following:

$ rhc app snapshot save -a <app-name>

$ rhc app restore -a <app-name> -f <path to snapshot file>

Conclusion

I hope you found this three part series useful. In addition to creating a Node.js app and front-ending it with a Mobile application, you saw how easy it is to deploy such an application to the cloud.

Mobile GeoLocation App in 30 minutes – Part 2: Sencha Touch

Overview

This is the second part of a three part series to show how we can use Sencha Touch (a mobile web JavaScript framework) to create the GUI to integrate with our Node.js REST API that we built in Part 1. As you may recall from the previous post, I’m tracking dogs and storing their coordinates in a MongoDB. Now we will create a mobile app that will list the dogs and when selected will show them on a Google Map relative to your current location. Let’s get started.

Connecting to Node.js using MVC

Connecting to Node or really any REST/JSONP service is fairly trivial. Since Sencha Touch supports MVC on the client side, I will illustrate how to create the application using that approach. This is very useful because it keeps with one of the main programming tenets: Separation of Concerns. In this way we can create concise JavaScript applications that break-up the application along the lines of model, view and controller components. First step is to create the model as follows:

Ext.define('MyApp.model.Pet', {
    extend: 'Ext.data.Model',
    config: {
        fields: [
            {
                name: 'name'
            },
            {
                name: 'description'
            },
            {
                name: 'longitude',
                type: 'float'
            },
            {
                name: 'latitude',
                type: 'float'
            },
            {
                name: 'coords'
            },
            {
                name: 'date',
                type: 'date'
            },
            {
                name: '_id'
            }
        ]
    }
});

Notice above that I’m defining a “Pet” model by extending the Ext.data.Model class and assigning the fields to the model. Additionally, I set types as needed so they aren’t considered strings. Next, we create the store that will populate the model:

Ext.define('MyApp.store.PetTracker', {
    extend: 'Ext.data.Store',
    requires: [
        'MyApp.model.Pet'
    ],
    config: {
        autoLoad: true,
        model: 'MyApp.model.Pet',
        storeId: 'PetTracker',
        proxy: {
            type: 'jsonp',
            url: 'http://localhost:8080/dogtag',
            reader: {
                type: 'json',
                idProperty: '_id',
                rootProperty: 'records',
                useSimpleAccessors: true
            }
        }
    }
});

The PetTracker Store uses a JSONP proxy and binds the output of the service to the model we previously created. Notice, that I simply put the URL to the Node.js URL binding that will list out all dogtags and use a JSON reader to put them in the model.

Binding the Model with the View and Controller

Now that I have the data, where do I put it in the GUI? Sencha Touch provides several mobile GUI widgets that look very much like native mobile apps. In my case, I decided to use the panel with a docked toolbar at the top. Let’s take a look:

Ext.define('MyApp.view.PetPanel', {
    extend:'Ext.Panel',
    alias: 'widget.petListPanel',
    config:{
        layout:{
            type:'fit'
        },
        items:[
            {
                xtype:'toolbar',
                docked:'top',
                title:'Pettracker'
            },
            {
                xtype:'list',
                store:'PetTracker',
                id:'PetList',
                itemId:'petList',
                emptyText: "<div>No Dogs Found</div>",
                loadingText: "Loading Pets",
                itemTpl:[
                    '<div>{name} is a {description} and is located at {latitude} (latitude) and {longitude} (longitude)</div>'
                ]
            }
        ],
        listeners:[
            {
                fn:'onPetsListItemTap',
                event:'itemtap',
                delegate:'#PetList'
            }
        ]
    },
    onPetsListItemTap:function (dataview, index, target, record, e, options) {
        this.fireEvent('petSelectCommand', this, record);
    }
});

In the example above, I create a panel with fit layout (aka, responsive web design) and put a toolbar and list component in it. To get the data in the list, all I have to do is set the store to the store name I created earlier (line 16). On line 22, you can see that I’m using expression language to templatize the output of the store data in each list row (look Ma, no FOR loops!). Finally, we need to create the Controller to respond to events in the code. In the View code, you see on line 36 that I fire the ‘petSelectCommand’ when a list item is tapped. Let’s look at the controller code for this.

Ext.define('MyApp.controller.PetTracker', {
    extend: 'Ext.app.Controller',
    markers: [],
    directionsDisplay: null,
    directionsService: null,
    config: {
        stores: ['PetTracker'],
        refs: {
            petListPanel: 'petListPanel',
            petList: '#PetList',
            petMap: 'petMap',
            radiusPicker: 'radiusPicker'
        },
        control: {
            petListPanel: {
                petSelectCommand: "onPetSelected"
            },
            petMap: {
                backButton: "onBackButton",
                mapRender: "onMapRender",
                nearButton: "onNear"
            },
            radiusPicker: {
                pickerChanged: "onPickerRadiusChange"
            }
        }
    },

    launch: function () {
        // Initialize Google Map Services
        this.directionsDisplay = new google.maps.DirectionsRenderer();
        this.directionsService = new google.maps.DirectionsService();

        var mapRendererOptions = {
            //draggable: true,  //Allows to drag route
            //hideRouteList: true,
            suppressMarkers: true
        };

        this.directionsDisplay.setOptions(mapRendererOptions);
    },

    // Transitions
    slideLeftTransition: { type: 'slide', direction: 'left' },
    slideRightTransition: { type: 'slide', direction: 'right' },

    onPetSelected: function (list, record) {
        var mapView = this.getPetMap();
        mapView.setRecord(record);
        Ext.Viewport.animateActiveItem(mapView, this.slideLeftTransition);


        this.renderMap(mapView, mapView.down("#petMap").getMap(), record.data);
    },

    onBackButton: function () {
        var store = Ext.getStore('PetTracker');
        store.getProxy().setUrl('http://nodetest-loutilities.rhcloud.com/dogtag/');
        store.load();
        Ext.Viewport.animateActiveItem(this.getPetListPanel(), this.slideRightTransition);
    },

    renderMap: function (extmap, map, record) {
        // erase old markers
        if (this.markers.length > 0) {
            Ext.each(this.markers, function (marker) {
                marker.setMap(null);
            });
        } 
        var position = new google.maps.LatLng(record.latitude, record.longitude);

        var dynaMarker = new google.maps.Marker({
            position: position,
            title: record.name + "'s Location",
            map: map,
            icon: 'resources/img/yellow_MarkerB.png'
        });

        this.markers.push(dynaMarker);
    }
});

Covering all that’s inside this controller is beyond the scope of this article and besides there’s already really good articles on this such as the one here. If we pick-up from the event we fired in the view, you can see on line 16 where I’ve bound the onPetSelected() method to the event. On line 47, you can see the implementation of that method where I do the slide transition to the map view panel and then render the map in that panel. Finally, there’s this small piece of code to bootstrap the application and launch it.

Ext.Loader.setConfig({
    enabled: true
});


Ext.application({
    models: [
        'Pet'
    ],
    stores: [
        'PetTracker'
    ],
    views: [
        'PetPanel',
        'MapPanel',
        'RadiusPicker'
    ],
    name: 'MyApp',
    controllers: [
        'PetTracker'
    ],
    launch: function() {
        var petList = {
            xtype: 'petListPanel'
        };
        var petMap = {
            xtype: 'petMap'
        };
        var radiusPicker = {
            xtype: 'radiusPicker'
        };
        Ext.Viewport.add([petList, petMap, radiusPicker]);
    }
});

Conclusion

Well there you have it: how to create a mobile web application integrated with Node.js and Google Maps. You can see this application in action by pointing your mobile device to this site, http://dogtags-loutilities.rhcloud.com/, or internally here. I hope you see the value in implementing client side MVC so that your code is concise and maintainable, especially if multiple developers were to work on it. You can view the complete code base here. Look for the final part of this series where I will show you how easy it is to deploy this application to the cloud using a PaaS like OpenShift, Heroku or Nodejitsu.

Mobile GeoLocation App in 30 minutes – Part 1: Node.js and MongoDB

Overview

Are you interested in learning how to use Node.js with a NoSQL database like MongoDB to create REST services? Would you like to know how to create a mobile app with Google Maps that can perform geospatial queries with ease? Do you want to see how you can deploy all this to a PaaS with the click of a few buttons and a rudimentary understanding of Git? The answers to this and more are part of my three part series where I will cover the details on how you can create a “Pettracker” like application using Node.js, MongoDB, and Sencha Touch hosted on OpenShift. In this first part I will discuss how to build the middle tier and data store with Node.js and MongoDB.

What is Node.js and MongoDB?

Node.js is a JavaScript framework (API to a C library) for creating highly scaling, non-blocking IO applications that does not create multiple process threads for each invocation incurring a lot less O/S overhead under load. This is accomplished using an event-driven programming model that executes an IO bound call but does not block; it continues to the next line of code. It uses a callback mechanism to conclude the operation with the response once the event is complete. JavaScript is perfect to accommplish this because its native support for closures. Node by itself isn’t that interesting, but akin to Maven in the Java world, Node has npm so that you can add modules to it and expand its functionality further to all sorts of different use cases. In this article you’ll see how I leverage npm for working with MongoDB and creating a Web server router. Finally, Node together with Socket.IO can be used to create real-time Web applications (more on this in the future).

MongoDB is a highly scalable, high performance, open source NoSQL database that stores documents (rows in SQL parlance) into collections (tables in SQL) and is maximized for performance using in-memory cache, replica sets and sharding. It has a very nice query API and supports map/reduce functionality found in Hadoop/HBase. One of the pecular things with NoSQL DB’s is that there is no schema defined ahead of time. So other than installing MongoDB, there isn’t much else that you need to do. Given the fact that there’s no schema it doesn’t mean that we don’t have structure to our data. NoSQL allows for a very flexible schemas that can change easily as your application needs changing. Finally, MongoDB has built-in geospatial querying to support “near” and polygon searches. We’ll explore this in detail.

The DogTag Application

As you may be aware, Pettracker (aka Tagg) is a Qualcomm device/application that can be used to track lost Pets. I work for Qualcomm but I didn’t work on this project;however, I thought it would be interesting to see how I could replicate this application using Node.js and MongoDB, since it would be the perfect use case for something that would require high throughput and geospatial searching. Node.js will provide the middleware both to consume the JSON data from the emitters (i.e., the dog tag transponder) and it will provide the REST interface so that the mobile device can locate the dogs or perform geospatial queries of what’s near the user.

The first step to do all this is to create a Node.js application. I used the Express and Mongoose Node modules to allow me to create a website and to interface with MongoDB. You can add these modules using npm to your node install or add them to your package.json. I’ll talk more about this later in Part 3. Also, you can download the code from my GitHub account here.

Let’s take a look at the model.js code:

var mongoose = require('mongoose')
  , Schema = mongoose.Schema;

var dtSchema = new Schema({
    name:String,
    description:String,
    date: {type: Date, default: Date.now},
    longitude: Number,
    latitude: Number,
    coords: [Number, Number]
});

module.exports = mongoose.model('DogTag', dtSchema);

Mongoose is a nice Node module that allows me to work with MongoDB data in JavaScript, so in the code above you can see that I am defining “DogTag” schema (a document in Mongo) and setting fields for what should be stored. Mongoose will take care of the unmarshalling/marshalling the JSON data to MongoDB. Next, I need to define the api for the operations I wish to perform shown as follows:

var DogTag = require('../model/dogtag.js');

exports.save = function(req, res) {
    var dogtag = new DogTag({name: req.params.name, description: req.params.descr,
        longitude: req.params.longitude, latitude: req.params.latitude});
    dogtag.save(function (err) {
        if (err) throw err;
        console.log('Dogtag saved.');

        res.send('Dogtag saved.');
    });
}

exports.list = function(req, res) {
    DogTag.find(function(err, dogtag) {
          res.setHeader('Content-Type', 'text/javascript;charset=UTF-8');
        res.send(req.query["callback"] + '({"records":' +  JSON.stringify(dogtag) + '});');
    });
}

exports.show = (function(req, res) {
    DogTag.findOne({name: req.params.name}, function(error, dogtag) {
        res.send([{Dog: dogtag}]);
    })
});

// first locates a dog by its name
exports.show = (function(req, res) {
    DogTag.findOne({name: req.params.name}, function(error, dogtag) {
        res.send([{Dog: dogtag}]);
    })
});

exports.near = function(req, res) {
    DogTag.find({coords : { $near : [req.params.lon, req.params.lat], $maxDistance : req.params.dist/68.91}}, 
      function (error, dogtag) {    
        res.setHeader('Content-Type', 'text/javascript;charset=UTF-8');
        res.send(req.query["callback"] +'({"records":' + JSON.stringify(dogtag) + '});');
    })
}

This code provides the CRUD operations to work with MongoDB. I used the schema I created previously to create dogtag objects and I use their save method to persist them into MongoDB. It’s that simple! If I need to find all the dogs, I use find(), or to find one dog by name, I use show(petname). Now, the last API, “near”, is doing a geospatial query using $near and $maxDistance against my “coords” field in the database. One thing to note is that you have to tell MongoDB to index fields that contain lat/lon. For details on this, see here, but all you really do is run this command from a mongo prompt: db.[collection_name].ensureIndex({ [field_name]: "2d" }).

One more thing to note is that I need to return JSONP back to the client by providing a callback name. To do that, I need the client to pass my a queryParameter (called “callback”) that will be the name of the callback method in the JSONP that is returned. I also need to call the built-in JavaScript function JSON.stringify to convert the schema object to JSON. You can see this put to use on lines 17 and 38.

The last piece is do build the node server. Yes, that’s right…you’re actually going to create the HTTP server that will respond to the request. Let’s take a look to see how:

var express = require('express');
var mongoose = require('mongoose');
var app = express();

var ipaddr  = '127.0.0.1';
var port    =  8081;

mongoose.connect('mongodb://localhost/dogtag');

app.configure(function () {
    app.use(express.bodyParser());
    app.use(express.methodOverride());
    app.use(app.router);
});

// set up the REST API handler methods are defined in api.js
var api = require('./controller/api.js');
app.post('/dogtag', api.post);
app.get('/dogtag/near/:lon/:lat/:dist?', api.near);
app.get('/dogtag/:name/:descr/:latitude/:longitude?', api.save);
app.get('/dogtag/:name.:format?', api.show);
app.get('/dogtag', api.list);

//  And start the app on that interface (and port).
app.listen(port, ipaddr, function() {
   console.log('%s: Dogtag server started on %s:%d ...', Date(Date.now() ),
               ipaddr, port);
});

As described earlier, I need to tell node that I’m going to use express and mongoose in my application. One line 3, I simply create the HTTP server. On lines 6-9, I use mongoose to connect to the database. On lines 18-23, I configure URL routing to the controller api’s from the previous example. Note how I use colons to express parameters on the URL and then bind them to a given api. Finally, I tell the server to listen on a certain port and host spit out a start message if everything went successful.

So that’s really all there is to create the necessary middleware to support our DogTag application. It takes in JSON data from the emitters to store in MongoDB and then uses the Express framework to create a router to respond to requests for finding dogs whether by name or through geospatial searching. In the next part I’ll explore how we can use Sencha Touch to create a mobile application that will help a user locate their dog. For a little preview of that application, you can go here to experiment with it here.

JackBe Presto Webinar – Enterprise App Store

The presentation includes Qualcomm’s vision behind the adoption of a dynamic Enterprise Information Delivery Architecture. Qualcomm discusses the internal and external forces driving demand for ‘live, fast’ information delivery through an Enterprise App Store. The demonstration includes how Qualcomm is creating a personalized portal experience, aptly named “My Places”, for users to have a self-service means of aggregrating information into a personalized dashboard.

Presentation Slides

Webinar Video Download