Building an Amateur Radio Packet to Twitter bridge: Part 3 – preventing duplicate tweets

In my last post (part 2, also see part 1 here), I talked about Twitter’s API rate limits. Since many Packet Radio transmissions are duplicates by their nature, for example, beacon packets and ID packets, it’s important to have some kind of mechanism to prevent sending these through to Twitter.

The approach I used was to insert each received packet into a MongoDB database, storing the received packet data, who the packet was from and who to, and additional metadata about the packet, for example, when last sent, and when it was last received.

Here’s an example of what each document stored looks like:

{
 "_id" : ObjectId("5909828e5a2f130fc8039882"),
 "firstHeard" : ISODate("2017-05-03T07:11:10.051Z"),
 "from" : "AE6OR ",
 "heardTimes" : 1,
 "infoString" : "Š¤¤@à–¤ˆŽ@@à–„Š¤¤@ஞžˆ²@`–„Š¨@`¨‚žŠ@a\u0003ðHello from 5W Garage packet node AUBURN 73's Steli !\r",
 "lastHeard" : ISODate("2017-05-03T07:11:10.051Z"),
 "lastTweet" : ISODate("2017-05-03T07:11:10.051Z"),
 "to" : "BEACON",
 "tweet" : "Received station: AE6OR  sending to: BEACON : Š¤¤@à–¤ˆŽ@@à–„Š¤¤@ஞžˆ²@`–„Š¨@`¨‚žŠ@a\u0003ðHello from 5W Garage packet node AUBURN 73's Steli !\r"
}

My current logic to check for duplicates and record when a tweet is last sent is:

    1. search for a matching document (tweet details) with a $lt condition that lastTweet is before ‘now – 12 hours’:
      document.lastTweet = {'$lt' : moment().utc().subtract(12, 'hours').toDate() };
    2. This is executed as a findOne() query:
      db.collection('tweets').findOne(document).then(function (olderResult) {
         ...
      }
    3. If an existing document older than 12 hours is found, then update properties to record that this same packet was seen now, and the time is was resent was also now (we’ll resend it to the Twitter api after the db updates):
      if (olderResult != null) {
          sendTweet = true;
          updateProperties = {
              lastHeard: now,
              lastTweet: now
          };
      }
      else {
          updateProperties = {
              lastHeard: now
          };
      }

      If not older than 12 hours, set properties to be updated to indicate just the lastHeard property

    4. To either update the existing document or insert a new document if this was the first time this packet was heard, do an ‘upsert’:
      db.collection('tweets').update(
          document,
          {
              $set: updateProperties,
              $inc: {heardTimes: 1},
              $setOnInsert: {
                  firstHeard: now,
                  lastTweet: now
              }
          },
          {upsert: true}
      ).then(function (response) {
      ...
      }
    5. Depending on the result indicating if we inserted or updated, set a property so on return we know whether to send a new Tweet or not:
      if(response.upserted != null || sendTweet) {
          response.sendTweet = true;
      }
      else{
          response.sendTweet = false;
      }

The approach is not yet completely foolproof, but it is stopping the majority of duplicate Tweets sent to Twitter so far.

For the full source check the project on github here: https://github.com/kevinhooke/PacketRadioToTwitter .

angular-seed and node.js http-server for local development

Angular-seed is a skeleton AngularJS project with a recommended folder structure and configuration including all the recommended development tools, jasmine and karma for unit tests, Protractor for end-to-end tests, and node.js for your dev server. Check out the project here.

If you’ve already installed node.js then you may notice in the readme for angular-seed it mentions that they have configured a test webserver using node, but you can also install this as a global module and use it elsewhere. This is a useful and quick way to spin up a test server in a given directory to support development testing. Install the server globally with:

sudo npm install -g http-server

and then start it up in a directory that you want to be the root, with:

http-server