I completed some of my planned updates recently, in particular moving the AngularJS static content to AWS S3 to serve as a static website, and then also updated AWS Route 53 to point www.spotviz.info to the S3 bucket for this front end content, and then api.spotviz.info is pointing to a VPS running the REST backend. At the same time moving the frontend to S3, I also spent a crazy amount of time migrating to use Webpack to build the frontend, which I covered here.
I’ve now started to pick up some enhancements to the app. The first enhancement is to add a new heatmap to show spots per hour, as a drilldown feature from the heatmap per day. What surprised me at this point is how crazy the original MongoDB query looks now it’s been a couple of years since I was last playing with this. Luckily the update to do counts per hour is only a minor change from the counts per day, so should have that complete soon.
Following on from Part 1 and subsequent posts, I now have the app deployed locally on WildFly 17, up and running, and also redeployed to a small 1 cpu 1 GB VPS: http://www.spotviz.info . At this point I’m starting to think about how I’m going to redesign the system to take advantage of the cloud.
Here are my re-design and deployment goals:
monthly runtime costs since this is a hobby project should be low. Less that $5 a month is my goal
take advantage of AWS services as much as possible, but only where use of those services still meet my monthly cost goal
if there are AWS free tier options that make sense to take advantage of, favor these services if they help keep costs down
Here’s a refresher on my diagram showing how the project was previously structured and deployed:
As of September 2019, the original app is now redeployed as a monolithic single .war again to WildFly 17, running on a single VPS. MongoDB is also running on the same VPS. The web app is up at: http://www.spotviz.info
There’s many options for how I could redesign and rebuild parts of this to take advantage of the cloud. Here’s the various parts that could either be redesigned, and/or split into separate deployments:
WSJT-X log file parser and uploader client app (the only part that probably won’t change, other than being updated to support the latest WSJT-X log file format)
Front end webapp: AngularJS static website assets
JAX-WS endpoint for uploading spots for processing
MDB for processing the upload queue
HamQTH api webservice client for looking up callsign info
MongoDB for storing parsed spots, callsigns, locations
Rest API used by AngularJS frontend app for querying spot data
Here’s a number of options that I’m going to investigate:
Option 1: redeploy the whole .war unchanged as previously deployed to OpenShift back in 2015, to a VM somewhere in the cloud. Cheapest options would be to a VPS. AWS LightSail VPS options are still not as a cheap as VPS deals you can get elsewhere (check LowEndBox for deals), and AWS EC2 instances running 24×7 are more expensive (still cheap, but not as cheap as VPS deals)
Update September 2019: COMPLETE: original app is now deployed and up and running
Option 2: Using AWS services: If I split the app into individual parts I can incrementally take on one or more of these options:
Now I have my http://www.spotviz.info app up and running with some collected spot data for testing, one of the first things I want to enhance is the date range selection for visualization, and specifically, adding more data to the heatmap to help you pick a good date range with available data.
The heatmap display I originally added gives you a visual representation of which days have data to view, but it doesn’t help you pick a time range within that day where there’s data. Here’s what the heatmap looks like right now:
The darker the color the more data there is for that day. The problem as it is right now though is that say for one day you ran WSJT-X for 1 hour between 1pm and 2pm and received spots from 2000 stations. Without knowing that there’s only data between 1pm and 2pm, you could select a range of 9am to 6pm for playback, and you’d get nothing displayed on the map until it animated through the 1pm to 2pm block and then a gap of nothing again.
My first enhancement here is to show the earliest and latest signal received times for a given day, to help you pick a good range.
The original MongoDB query (which I discussed here) retrieves a count of spots per hour for a given day:
There’s only one calculated value returned in the results in this query, count, but in order get the earliest and latest spots per day, this is a simple as adding two more calculated values following the return of
I still have some refactoring to do as I move the app to the cloud, but as the first starting point I have the original app up and running on WildFly on a VPS. It’s back up and live on the original domain, http://www.spotviz.info
The uploader has been updated to read current 2.x WSJT-X log files, but I haven’t uploaded the .jar to GitLab yet, but I’ll do that soon. I’ve some test log data uploaded under my callsign, KK6DCT.
More changes to come soon.
If you’re catching up, here’s my previous posts on getting the project up and running again: