mongodb – Page 2 – Kev's Development Toolbox

May 15, 2017

Building an Amateur Radio Packet to Twitter bridge: Part 3 – preventing duplicate tweets

In my last post (part 2, also see part 1 here), I talked about Twitter’s API rate limits. Since many Packet Radio transmissions are duplicates by their nature, for example, beacon packets and ID packets, it’s important to have some kind of mechanism to prevent sending these through to Twitter.

The approach I used was to insert each received packet into a MongoDB database, storing the received packet data, who the packet was from and who to, and additional metadata about the packet, for example, when last sent, and when it was last received.

Here’s an example of what each document stored looks like:

{
 "_id" : ObjectId("5909828e5a2f130fc8039882"),
 "firstHeard" : ISODate("2017-05-03T07:11:10.051Z"),
 "from" : "AE6OR ",
 "heardTimes" : 1,
 "infoString" : "¤¤@à¤@@à¤¤@à®²@`¨@`¨@a\u0003ðHello from 5W Garage packet node AUBURN 73's Steli !\r",
 "lastHeard" : ISODate("2017-05-03T07:11:10.051Z"),
 "lastTweet" : ISODate("2017-05-03T07:11:10.051Z"),
 "to" : "BEACON",
 "tweet" : "Received station: AE6OR  sending to: BEACON : ¤¤@à¤@@à¤¤@à®²@`¨@`¨@a\u0003ðHello from 5W Garage packet node AUBURN 73's Steli !\r"
}

My current logic to check for duplicates and record when a tweet is last sent is:

1. search for a matching document (tweet details) with a $lt condition that lastTweet is before ‘now – 12 hours’:
```
document.lastTweet = {'$lt' : moment().utc().subtract(12, 'hours').toDate() };
```
2. This is executed as a findOne() query:
```
db.collection('tweets').findOne(document).then(function (olderResult) {
   ...
}
```
3. If an existing document older than 12 hours is found, then update properties to record that this same packet was seen now, and the time is was resent was also now (we’ll resend it to the Twitter api after the db updates):
```
if (olderResult != null) {
    sendTweet = true;
    updateProperties = {
        lastHeard: now,
        lastTweet: now
    };
}
else {
    updateProperties = {
        lastHeard: now
    };
}
```
  If not older than 12 hours, set properties to be updated to indicate just the lastHeard property
4. To either update the existing document or insert a new document if this was the first time this packet was heard, do an ‘upsert’:
```
db.collection('tweets').update(
    document,
    {
        $set: updateProperties,
        $inc: {heardTimes: 1},
        $setOnInsert: {
            firstHeard: now,
            lastTweet: now
        }
    },
    {upsert: true}
).then(function (response) {
...
}
```
5. Depending on the result indicating if we inserted or updated, set a property so on return we know whether to send a new Tweet or not:
```
if(response.upserted != null || sendTweet) {
    response.sendTweet = true;
}
else{
    response.sendTweet = false;
}
```

The approach is not yet completely foolproof, but it is stopping the majority of duplicate Tweets sent to Twitter so far.

For the full source check the project on github here: https://github.com/kevinhooke/PacketRadioToTwitter .

April 29, 2017

MongoDB on the Raspberry Pi

I looked at running MongoDB on the Pi several months back, and while you can download the latest source and compile it yourself which takes a while, there didn’t seem to be any prebuilt binaries (at least not when I last looked).

It seems the 32bit version for ARM is now in the Rasbian repos, so you can just do (from here):

sudo apt-get install mongodb-server

and get it installed and setup ready to go.

If you’re looking for a later 64bit version then some people have rebuilt these for you. Take a look at here, or for more info, also try Andy Felong’s posts, e.g. here and here.

July 3, 2016

MongoDB Java api: using a sequence collection with findAndModify()

MongoDB doesn’t have an equivalent of sequences typically used in relational databases, since documents are automatically given a unique ObjectId value for the “_id” property when inserted.

To return data from documents via a REST api, it may be more useful to use a sequential unique key. The MongoDB docs have an example of how you could use a Collection to hold a document per sequence that you need, and increment a value property each time you retrieve it with findAndModify().

There’s a number of questions on StackOverflow related to this approach, most seem related to the approach doc linked above (e.g. here, here, and articles elsewhere, like here).

To implement this approach using the Java api, using findAndModify() seems to be key, as you need to ensure you are querying and incrementing the sequence value in a document in a single, atomic step. After that point once you have the updated/next value from your document holding your sequence, the fact that you use that value in a subsequent atomic insert to another document seems to be safe (please leave me a comment if this assumption is not correct!), as every call to findAndModify() to increment the sequence value is atomic (making an assumption here based on my limited MongoDB knowledge, but I think this is correct!).

Here’s how I implemented the approach using the Java api:

[code]
public int getNextSequence() throws Exception {
DB db = MongoConnection.getMongoDB();

DBCollection sequences = db.getCollection("sequences");

// fields to return
DBObject fields = BasicDBObjectBuilder.start()
.append("_id", 1)
.append("value", 1).get();

DBObject result = sequences.findAndModify(
new BasicDBObject("_id", "addressId"), //query
fields, // what fields to return
null, // no sorting
false, //we don’t remove selected document
new BasicDBObject("$inc", new BasicDBObject("value", 1)), //increment value
true, //true = return modified document
true); //true = upsert, insert if no matching document

return (int)result.get("value");
}
[/code]

June 19, 2016

docker-compose.yml for 1x MongoDB container plus 1x data volume container

I’m starting to dabble with docker-compose to link up some container goodness. If you’ve followed my other Docker related posts the past couple of days (here), then you might have noticed that I have something Raspberry Pi related cooking 🙂

Although I’ll have to redo these with a Pi compatible base images, to get MongoDB running in one container with a separate data volume container is actually pretty simple. Here’s what I ended up with. To start it up, ‘docker-compose up -d’ and you’re up and running (copy this to docker-compose.yml):

mongodata: 
 image: mongo:3.2
 volumes:
 - /data/db
 entrypoint: /bin/bash
mongo: 
 image: mongo:3.2
 volumes_from:
 - mongodata
 ports:
 - "27017:27017"