Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 6: Redesigning to take advantage of the Cloud

Following on from Part 1 and subsequent posts, I now have the app deployed locally on WildFly 17, up and running, and also redeployed to a small 1 cpu 1 GB VPS: http://www.spotviz.info . At this point I’m starting to think about how I’m going to redesign the system to take advantage of the cloud.

Here are my re-design and deployment goals:

  • monthly runtime costs since this is a hobby project should be low. Less that $5 a month is my goal
  • take advantage of AWS services as much as possible, but only where use of those services still meet my monthly cost goal
  • if there are AWS free tier options that make sense to take advantage of, favor these services if they help keep costs down

Here’s a refresher on my diagram showing how the project was previously structured and deployed:

As of September 2019, the original app is now redeployed as a monolithic single .war again to WildFly 17, running on a single VPS. MongoDB is also running on the same VPS. The web app is up at: http://www.spotviz.info

There’s many options for how I could redesign and rebuild parts of this to take advantage of the cloud. Here’s the various parts that could either be redesigned, and/or split into separate deployments:

  • WSJT-X log file parser and uploader client app (the only part that probably won’t change, other than being updated to support the latest WSJT-X log file format)
  • Front end webapp: AngularJS static website assets
  • JAX-WS endpoint for uploading spots for processing
  • MDB for processing the upload queue
  • HamQTH api webservice client for looking up callsign info
  • MongoDB for storing parsed spots, callsigns, locations
  • Rest API used by AngularJS frontend app for querying spot data

Here’s a number of options that I’m going to investigate:

Option 1: redeploy the whole .war unchanged as previously deployed to OpenShift back in 2015, to a VM somewhere in the cloud. Cheapest options would be to a VPS. AWS LightSail VPS options are still not as a cheap as VPS deals you can get elsewhere (check LowEndBox for deals), and AWS EC2 instances running 24×7 are more expensive (still cheap, but not as cheap as VPS deals)

Update September 2019: COMPLETE: original app is now deployed and up and running

Option 2: Using AWS services: If I split the app into individual parts I can incrementally take on one or more of these options:

  • Route 53 for DNS (September 2019: COMPLETE!)
  • Serve AngularJS static content from AWS S3 (next easiest change) (December 2019: COMPLETE!)
  • AWS API Gateway for log file upload endpoint and RestAPIs for data lookups
  • AWS Lambdas for handling uploads and RestAPIs
  • Rely on scaling on demand of Lambdas for handling upload and parsing requests, removing need for the queue
  • Refactor data store from MongoDB to DynamoDB

Option 3: Other variations:

  • Replace use of WildFly queue with AWS SQS
  • Replace queue approach with a streams processing approach, either AWS Kinesis or AWS MSK

More updates coming later.

WSJT-X reports time difference > 2 seconds on Windows 10

All of the JT modes (JT65, JT9), and now the new mode FT8 are sensitive to correct system time within a couple of seconds. This is because the transmit/receive cycle starts and stops at a predetermined time, and the software of each transmitting station and each remote receiving station need to be in sync in order for each transmission to get successfully decoded.

FT8 seems to be forgiving within about 1.5 seconds of difference, but above this it flags the delta between stations in the DT column:

On a new laptop with Windows 10, even though the date and time is set to set automatically using a remote time server on the internet, almost every FT8 transmission received was consistenly > 1.5 seconds in difference.

To fix this I downloaded and installed Dimension 4 and set it to start automatically. Sync’d the time and now we’re all set – you can see the received calls below in the DT column are all now within a couple of 0.1s after the sync:

Configuring WSJT-X to log to N3FPJ for ARRL Field Day (part 1)

My amateur radio club (RCARCS) uses N3FPJ logger for Field Day (we run N3FPJ on one laptop and the log from each station over WiFi). This year I want to have a go at setting up a digital mode station to log automatically to N3FPJ from WSJT-X to run some FT8, and also from fldigi to work PSK and RTTY. First up, let’s look at getting WSJT-X working.

WSJT-X doesn’t log directly to N3FPJ (it does to N1NM though), but it does indirectly through the intermediate helper app JT-Alert. The setup we’re going to do then is:

WSJT-X -> JT-Alert -> N3FPJ Field Day logger.

Starting with N3FPJ, the first step is to enable the Server API from Settings / Application Programming Interface:

The next step after installing JT-Alert is to select ACLog (N3FPJ’s main logging app) in the settings. The options give you a setting to connect to a locally running ACLog, and while this works for a local N3FPJ’s ACLog app, it doesn’t for N3FJP’s Field Day logger. This part is not obvious, but to configure JT-Alert to log to the Field Day logger even if on the same PC you configure the remote connection settings and then also point it to the .mdb log file of the locally running N3FPJ:

Here you also select the log type is ‘Field Day’. This approach for the config is in the Help docs for JT-Alert here:

Next, configuring network options in WSJT to allow JT-Alert to connect. Without making any other changes, if you start JT-Alert and then start WSJT you’ll see these two dialogs which tell you what settings you need to change in WSJT:

In WSJT settings on the reporting tab, select all 3 checkboxes in the UDP section, and replace the default 127.0.0.1 with your current IP address, e.g.

To use FT8 for Field Day, WSJT has a setting to customize the exchange to include the required class and ARRL section:

To test out the config so far, start the apps in this order: N3FPJ, JT-Alert, WSJT. To test logging a QSO, enter a test call in the DX Call field and press the Log QSO button:

… note here when running Field Day mode you should enter the correct exchange sent and exchange received so the QSO is logged correctly in N3FPJ.

The log is automatically sent across to N3FPJ via JT-Alert:

I noticed if you don’t manually enter the exchange sent and received then it will populate just the callsign field with the sent and received still blank. If you complete both as you log from WSJT then the log entry is saved into N3FPJ automatically.

Note: in JTAlert v2.13.6 I noticed the above steps work fine on Windows 10 if you are logged in as an Admin user. If you log in as a regular user, JTAlert will not copy logs across to N3FPJ, unless you start JTAlert with ‘Run as Administrator’. Also, it will pop up the warning message saying that it failed to check if the log was successfully written, even though it is written to N3FPJ successfully. There is a timing setting in the JTAlert Performance settings to increase the length of time before the N3FPJ is checked for a successful log, but it doesn’t appear to make any difference for this issue.

Next up, I’ll look at getting fldigi setup to also log direct to N3FPJ.