Kev’s Blog – Page 87 – Kev's Development Toolbox

October 12, 2019January 21, 2020

MongoDB on MacOS Catalina 10.15: “exception in initAndListen: 29 Data directory /data/db not found”

By default, MongoDB usually stores it’s data files when running on MacOS under /data/db. After upgrading to Catalina 10.15, when starting MongoDB I get this error:

STORAGE  [initandlisten] exception in initAndListen: 29 Data directory /data/db not found., terminating

According to this question and answer here, Catalina no longer allows apps to read or write to non-standard folders beneath / so you need to move the data files elsewhere. After my upgrade, the files were moved to a ‘Relocated Items/Security’ folder. Moving them into my user dir and then starting up with the suggested:

mongod --dbpath ~/data/db

fixes the issue.

October 11, 2019December 17, 2019

Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 6: Redesigning to take advantage of the Cloud

Following on from Part 1 and subsequent posts, I now have the app deployed locally on WildFly 17, up and running, and also redeployed to a small 1 cpu 1 GB VPS: http://www.spotviz.info . At this point I’m starting to think about how I’m going to redesign the system to take advantage of the cloud.

Here are my re-design and deployment goals:

monthly runtime costs since this is a hobby project should be low. Less that $5 a month is my goal
take advantage of AWS services as much as possible, but only where use of those services still meet my monthly cost goal
if there are AWS free tier options that make sense to take advantage of, favor these services if they help keep costs down

Here’s a refresher on my diagram showing how the project was previously structured and deployed:

As of September 2019, the original app is now redeployed as a monolithic single .war again to WildFly 17, running on a single VPS. MongoDB is also running on the same VPS. The web app is up at: http://www.spotviz.info

There’s many options for how I could redesign and rebuild parts of this to take advantage of the cloud. Here’s the various parts that could either be redesigned, and/or split into separate deployments:

WSJT-X log file parser and uploader client app (the only part that probably won’t change, other than being updated to support the latest WSJT-X log file format)
Front end webapp: AngularJS static website assets
JAX-WS endpoint for uploading spots for processing
MDB for processing the upload queue
HamQTH api webservice client for looking up callsign info
MongoDB for storing parsed spots, callsigns, locations
Rest API used by AngularJS frontend app for querying spot data

Here’s a number of options that I’m going to investigate:

Option 1: redeploy the whole .war unchanged as previously deployed to OpenShift back in 2015, to a VM somewhere in the cloud. Cheapest options would be to a VPS. AWS LightSail VPS options are still not as a cheap as VPS deals you can get elsewhere (check LowEndBox for deals), and AWS EC2 instances running 24×7 are more expensive (still cheap, but not as cheap as VPS deals)

Update September 2019: COMPLETE: original app is now deployed and up and running

Option 2: Using AWS services: If I split the app into individual parts I can incrementally take on one or more of these options:

Route 53 for DNS (September 2019: COMPLETE!)
Serve AngularJS static content from AWS S3 (next easiest change) (December 2019: COMPLETE!)
AWS API Gateway for log file upload endpoint and RestAPIs for data lookups
AWS Lambdas for handling uploads and RestAPIs
Rely on scaling on demand of Lambdas for handling upload and parsing requests, removing need for the queue
Refactor data store from MongoDB to DynamoDB

Option 3: Other variations:

Replace use of WildFly queue with AWS SQS
Replace queue approach with a streams processing approach, either AWS Kinesis or AWS MSK

More updates coming later.

October 8, 2019October 8, 2019

Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 5: MongoDB Aggregate Queries

Now I have my http://www.spotviz.info app up and running with some collected spot data for testing, one of the first things I want to enhance is the date range selection for visualization, and specifically, adding more data to the heatmap to help you pick a good date range with available data.

The heatmap display I originally added gives you a visual representation of which days have data to view, but it doesn’t help you pick a time range within that day where there’s data. Here’s what the heatmap looks like right now:

The darker the color the more data there is for that day. The problem as it is right now though is that say for one day you ran WSJT-X for 1 hour between 1pm and 2pm and received spots from 2000 stations. Without knowing that there’s only data between 1pm and 2pm, you could select a range of 9am to 6pm for playback, and you’d get nothing displayed on the map until it animated through the 1pm to 2pm block and then a gap of nothing again.

My first enhancement here is to show the earliest and latest signal received times for a given day, to help you pick a good range.

The original MongoDB query (which I discussed here) retrieves a count of spots per hour for a given day:

db.Spot.aggregate( [   
   {$match: {spotter: "kk6dct"}},
   {"$group": {
     "_id": {
       "$subtract": [
         { "$subtract": [ "$spotReceivedTimestamp", new Date("1970-01-01") ] },
         { "$mod": [ { "$subtract": [ "$spotReceivedTimestamp", new Date("1970-01-01") ] }, 1000 * 60 * 60 * 24 ] }
       ]
     },
     count:{$sum: 1 }
   }
 }
])

There’s only one calculated value returned in the results in this query, count, but in order get the earliest and latest spots per day, this is a simple as adding two more calculated values following the return of

count : { $sum : 1 }

so the last part of the query is now:

    count : { $sum : 1 },
    "firstSpot" : { "$min" : "$spotReceivedTimestamp" },
    "lastSpot" : { "$max" : "$spotReceivedTimestamp" },

Done!

September 25, 2019October 3, 2019

systemd quick reference

sudo systemctl start|stop|restart|status servicename

sudo systemctl enable servicename : configures service to start at boot

If changing service config files in /etc/systemd/system/servicename.conf, rerun systemctl daemon-reload after changes and restart changed services