Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 6: Redesigning to take advantage of the Cloud

Following on from Part 1 and subsequent posts, I now have the app deployed locally on WildFly 17, up and running, and also redeployed to a small 1 cpu 1 GB VPS: http://www.spotviz.info . At this point I’m starting to think about how I’m going to redesign the system to take advantage of the cloud.

Here are my re-design and deployment goals:

  • monthly runtime costs since this is a hobby project should be low. Less that $5 a month is my goal
  • take advantage of AWS services as much as possible, but only where use of those services still meet my monthly cost goal
  • if there are AWS free tier options that make sense to take advantage of, favor these services if they help keep costs down

Here’s a refresher on my diagram showing how the project was previously structured and deployed:

As of September 2019, the original app is now redeployed as a monolithic single .war again to WildFly 17, running on a single VPS. MongoDB is also running on the same VPS. The web app is up at: http://www.spotviz.info

There’s many options for how I could redesign and rebuild parts of this to take advantage of the cloud. Here’s the various parts that could either be redesigned, and/or split into separate deployments:

  • WSJT-X log file parser and uploader client app (the only part that probably won’t change, other than being updated to support the latest WSJT-X log file format)
  • Front end webapp: AngularJS static website assets
  • JAX-WS endpoint for uploading spots for processing
  • MDB for processing the upload queue
  • HamQTH api webservice client for looking up callsign info
  • MongoDB for storing parsed spots, callsigns, locations
  • Rest API used by AngularJS frontend app for querying spot data

Here’s a number of options that I’m going to investigate:

Option 1: redeploy the whole .war unchanged as previously deployed to OpenShift back in 2015, to a VM somewhere in the cloud. Cheapest options would be to a VPS. AWS LightSail VPS options are still not as a cheap as VPS deals you can get elsewhere (check LowEndBox for deals), and AWS EC2 instances running 24×7 are more expensive (still cheap, but not as cheap as VPS deals)

Update September 2019: COMPLETE: original app is now deployed and up and running

Option 2: Using AWS services: If I split the app into individual parts I can incrementally take on one or more of these options:

  • Route 53 for DNS (September 2019: COMPLETE!)
  • Serve AngularJS static content from AWS S3 (next easiest change) (December 2019: COMPLETE!)
  • AWS API Gateway for log file upload endpoint and RestAPIs for data lookups
  • AWS Lambdas for handling uploads and RestAPIs
  • Rely on scaling on demand of Lambdas for handling upload and parsing requests, removing need for the queue
  • Refactor data store from MongoDB to DynamoDB

Option 3: Other variations:

  • Replace use of WildFly queue with AWS SQS
  • Replace queue approach with a streams processing approach, either AWS Kinesis or AWS MSK

More updates coming later.

Moving GoDaddy DNS to AWS Route 53

Before changing the DNS settings on GoDaddy, set up Route 53 to manage DNS for your domain first, because you’ll need the AWS DNS server names when updating the GoDaddy DNS config.

In the AWS Console, go to Route 53, then Hosted Zones, and press the ‘Created Hosted Zone’ button:

Enter your domain name, select ‘Public Hosted Zone’ and press ‘Create’:

At this point Route 53 will have assigned a list of DNS nameservers for your domain – remember this list for when you update the GoDaddy config

Next, press ‘Create Record Set’, select Type A record and enter the name of your subdomain, e.g. www, and enter the IP address of your server that this name should resolve to:

Now let’s update the GoDaddy config. By default, here’s the DNS entries managed by GoDaddy when you register a domain with them:

First step, cancel the GoDaddy nameservers. Do this from the ‘Change’ button:

Change the type to Custom:

Then enter the dns server names from Route 53 and press ‘Save’. You should now be done. Assuming the setup on Route 53 has propagated, hit your domain name in a browser and hopefully you’re up and running.

Creating a new AWS Lambda using the AWS CLI

It’s pretty easy to set up and configure a new AWS Lambda with the AWS Console, but if you’re iterating on some changes and need to redeploy a few times, the AWS CLI makes it pretty easy.

To create a new Lambda, assuming you have a .zip packaged up and ready to go:

aws lambda create-function --function-name example-lambda --zip-file fileb://example-lambda.zip --handler index.handler --runtime nodejs8.10 --role arn:aws:iam::role-id-here:role/lambda-role

Building a React frontend for my AWS Lambda Sudoku solver

Over the past few months I built an implementation of Donald Knuth’s Algorithm X using Dancing Links in Java to solve Sudoku puzzles.

This was a fascinating exercise in itself (you can read more my experience here), but the next logical step would be to package it up in a way to share it online.

Since I’m pursuing my AWS certifications right now, one interesting and low cost approach to host the the Solver implementation is to package it as an AWS Lambda. Sudoku Solver as a Service? Done. I exposed it through AWS API Gateway. It accepts an request payload that looks like this:

{"rows":["...81.67.","..749.2.8",".6..5.1.4","1....39..","4...8...7","..69....3","9.2.3..6.","6.1.743..",".34.69..."]}

and returns a response with a solution to the submitted puzzle request like this:

{"rows":["349812675","517496238","268357194","185723946","493681527","726945813","972538461","651274389","834169752"]}

The request and response payloads are an array of Strings, where each item represents a String of values concatenated together for one row in the grid, with ‘.’s for unknowns.

I’m still learning React as I go, and while building this front end for my Lambda Sudoku Solver I learnt some interesting things about React and Javascript. The source for the app is shared here.

I used Flux to structure the app, so there main parts of the app are:

  • a main, highlevel Container component,
  • a CellComponent that renders each cell in the Sudoku grid,
  • an Action that handles the interaction with the AWS Lambda
  • a Store that holds the results from calling the Lambda

I don’t want to focus on the pros and cons of using React or Flux (and this is not intended to be a how-to on building an app using React) as there were some other specific issues I ran into that were interesting learning opportunities. A couple of these I already captured in separate posts, so I’ll include these links below.

Iteration 1: onChange handler per row

My first approach to maintaining the state for the display of the grid and the handler for changes to each cell was to keep it simple and have a seperate array of values per row, and a separate onChange handler for each row. This is not a particularly effective way to structure this as there’s duplication in each of the 9 handlers.

The State looked like this:

this.state =
{
row1 : [],
row2 : [],
row3 : [],
row4 : [],
row5 : [],
row6 : [],
row7 : [],
row8 : [],
row9 : []
};

And each of the handlers looked like this, one handler per row, so handleChangeRow1() through handleChangeRow9():

handleChangeRow1(index, event){
console.log("row 1 update: " + event.target.value);
var updatedRow = [...this.state.row1];
updatedRow[index] = event.target.value
this.setState( { row1 : updatedRow } );
}

This approach needed 9 versions of the function above, each one specifically handling updates to the state for a single row. We’ll come back to improving this later.

The interesting thing to notice at this point that to update an array in React state, you need to clone a copy of the array, and then update the copy. I used the spread operator ‘…’ to clone the array.

Each row in the grid I rendered separately like this (so this approach needed 9 of these blocks):

<div>
{
this.state.row1.map( (cell, index) => (
<CellComponent key={index} value={this.state.row1[index]}
onChange={this.handleChangeRow1.bind(this, index)}/>
)
)}
</div>

This was my first working version of the app, at least at the point where I could track the State of the grid as a user entered or changed values in the 9×9 grid. Next steps was to improve the approach.

Iteration 2: Using an array of arrays for the State

The first improvement was to improve the State arrays, moving to an array of arrays. This is easily setup like this:

this.state =
{
grid: []
};

for (var row = 0; row < 9; row++) {
this.state.grid[row] = [];
}

Iteration 3: One onChange handler for all rows

Instead of a handler per row, I parameterized the onChange handler to reused for all rows. This is what I ended up with:

handleGridChange(row, colIndex, event) {
console.log("row [" + row + "] col [" + colIndex + "] : " + event.target.value);
var updatedGrid = [...this.state.grid];
updatedGrid[row][colIndex] = event.target.value;

//call Action to send updated data to Store
SudokuSolverAction.updatePuzzleData(updatedGrid);
}

Using .map() on each of the rows in State, I then rendered each row of the grid like this, passing the current row index and column index as params into handleGridChange():

<tr>
{
this.state.grid[0].map((cell, colIndex) => (
<td key={"row0" + colIndex}>
<CellComponent value={this.state.grid[0][colIndex]}
onChange={this.handleGridChange.bind(this, 0, colIndex)}/>
</td>
)
)}
</tr>

I’m sure there’s a way to use a nested .map() of the results of a .map() or some other clever approach to render the whole grid in a single go, but rendering each of rows individual is an ok approach with me since there’s only 9 rows. If the number of rows were much more than 9 then I’d spend some time working on a better approach, but I’m ok with this for now.

Flux Action and Store

The Action to call the Lambda, and maintaining the state of the responses in the Store was pretty simple. You can check out the source here if you’re interested.

CSS styling for the grid

One last thing to do was to style the grid so it looks like a usual Sudoku grid, with vertical and horizontal lines at 3 and 6, to divide the grid in 3×3 of the 3×3 squares. This took some reading to find out how to easily do this, but turns out CSS nth-child() psuedoclass handles this perfectly. I covered this in this post here.

Take a look at the app

I might move this to a more permanent home later, but if you want to check out the app, you can take a look here.