Category: Kev’s Blog

Posted on

Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 6: Redesigning to take advantage of the Cloud

Following on from Part 1 and subsequent posts, I now have the app deployed locally on WildFly 17, up and running, and also redeployed to a small 1 cpu 1 GB VPS: http://www.spotviz.info . At this point I’m starting to think about how I’m going to redesign the system to take advantage of the cloud.

Here are my re-design and deployment goals:

  • monthly runtime costs since this is a hobby project should be low. Less that $5 a month is my goal
  • take advantage of AWS services as much as possible, but only where use of those services still meet my monthly cost goal
  • if there are AWS free tier options that make sense to take advantage of, favor these services if they help keep costs down

Here’s a refresher on my diagram showing how the project was previously structured and deployed:

As of September 2019, the original app is now redeployed as a monolithic single .war again to WildFly 17, running on a single VPS. MongoDB is also running on the same VPS. The web app is up at: http://www.spotviz.info

There’s many options for how I could redesign and rebuild parts of this to take advantage of the cloud. Here’s the various parts that could either be redesigned, and/or split into separate deployments:

  • WSJT-X log file parser and uploader client app (the only part that probably won’t change, other than being updated to support the latest WSJT-X log file format)
  • Front end webapp: AngularJS static website assets
  • JAX-WS endpoint for uploading spots for processing
  • MDB for processing the upload queue
  • HamQTH api webservice client for looking up callsign info
  • MongoDB for storing parsed spots, callsigns, locations
  • Rest API used by AngularJS frontend app for querying spot data

Here’s a number of options that I’m going to investigate:

Option 1: redeploy the whole .war unchanged as previously deployed to OpenShift back in 2015, to a VM somewhere in the cloud. Cheapest options would be to a VPS. AWS LightSail VPS options are still not as a cheap as VPS deals you can get elsewhere (check LowEndBox for deals), and AWS EC2 instances running 24×7 are more expensive (still cheap, but not as cheap as VPS deals)

Update September 2019: COMPLETE: original app is now deployed and up and running

Option 2: Using AWS services: If I split the app into individual parts I can incrementally take on one or more of these options:

  • Route 53 for DNS (September 2019: COMPLETE!)
  • Serve AngularJS static content from AWS S3 (next easiest change) (December 2019: COMPLETE!)
  • AWS API Gateway for log file upload endpoint and RestAPIs for data lookups
  • AWS Lambdas for handling uploads and RestAPIs
  • Rely on scaling on demand of Lambdas for handling upload and parsing requests, removing need for the queue
  • Refactor data store from MongoDB to DynamoDB

Option 3: Other variations:

  • Replace use of WildFly queue with AWS SQS
  • Replace queue approach with a streams processing approach, either AWS Kinesis or AWS MSK

More updates coming later.

Posted on

Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 5: MongoDB Aggregate Queries

Now I have my http://www.spotviz.info app up and running with some collected spot data for testing, one of the first things I want to enhance is the date range selection for visualization, and specifically, adding more data to the heatmap to help you pick a good date range with available data.

The heatmap display I originally added gives you a visual representation of which days have data to view, but it doesn’t help you pick a time range within that day where there’s data. Here’s what the heatmap looks like right now:

The darker the color the more data there is for that day. The problem as it is right now though is that say for one day you ran WSJT-X for 1 hour between 1pm and 2pm and received spots from 2000 stations. Without knowing that there’s only data between 1pm and 2pm, you could select a range of 9am to 6pm for playback, and you’d get nothing displayed on the map until it animated through the 1pm to 2pm block and then a gap of nothing again.

My first enhancement here is to show the earliest and latest signal received times for a given day, to help you pick a good range.

The original MongoDB query (which I discussed here) retrieves a count of spots per hour for a given day:

db.Spot.aggregate( [   
   {$match: {spotter: "kk6dct"}},
   {"$group": {
     "_id": {
       "$subtract": [
         { "$subtract": [ "$spotReceivedTimestamp", new Date("1970-01-01") ] },
         { "$mod": [ { "$subtract": [ "$spotReceivedTimestamp", new Date("1970-01-01") ] }, 1000 * 60 * 60 * 24 ] }
       ]
     },
     count:{$sum: 1 }
   }
 }
])

There’s only one calculated value returned in the results in this query, count, but in order get the earliest and latest spots per day, this is a simple as adding two more calculated values following the return of 

count : { $sum : 1 }

so the last part of the query is now:

    count : { $sum : 1 },
    "firstSpot" : { "$min" : "$spotReceivedTimestamp" },
    "lastSpot" : { "$max" : "$spotReceivedTimestamp" },

Done!

Posted on

systemd quick reference

sudo systemctl start|stop|restart|status servicename

sudo systemctl enable servicename : configures service to start at boot

If changing service config files in /etc/systemd/system/servicename.conf, rerun systemctl daemon-reload after changes and restart changed services

Posted on

Revisiting my spotviz.info webapp: visualizing WSJT-X FT8 spots over time – part 2: jaxb with Java 9 and beyond

Continuing from my efforts to get my SpotViz.info site up and running again on a current version of WildFly before refreshing the tech used to run the app, I ran into the first major issue: jaxb and support on Java 9 and beyond.

This is the first time I’ve run into this in practice, but was aware of the move to remove jaxb from Java SE in releases after Java 9. Deploying and running the app locally on WildFly17 and Java 8 everything deploys and works unchanged, but now I’m deploying to a test Ubuntu18.04 server in a VM on my local rack server before moving to the cloud, I’m running into all sorts of exceptions at runtime with part of the app that uses jax-ws and jaxb to call a remote xml based webservice.

At runtime as the MDB is picking up messages to process from the queue, I’m getting this exception:

Caused by: java.lang.NullPointerException
     at org.jboss.resteasy.resteasy-cdi@3.7.0.Final//org.jboss.resteasy.cdi.CdiInjectorFactory.createConstructor(CdiInjectorFactory.java:91)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.spi.ResteasyProviderFactory.injectedInstance(ResteasyProviderFactory.java:2809)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$AbstractInterceptorFactory.createInterceptor(JaxrsInterceptorRegistry.java:170)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.initialize(JaxrsInterceptorRegistry.java:188)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.checkInitialize(JaxrsInterceptorRegistry.java:203)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$OnDemandInterceptorFactory.getInterceptor(JaxrsInterceptorRegistry.java:214)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry$AbstractInterceptorFactory.postMatch(JaxrsInterceptorRegistry.java:158)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.core.interception.JaxrsInterceptorRegistry.postMatch(JaxrsInterceptorRegistry.java:421)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientConfiguration.getRequestFilters(ClientConfiguration.java:112)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.getRequestFilters(ClientInvocation.java:408)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:582)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:440)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:464)
     at org.jboss.resteasy.resteasy-jaxrs@3.7.0.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:189)
     at deployment.callsignviz-1.0.war//kh.hamqthclient.HamQTHClient.logon(HamQTHClient.java:70)
     at deployment.callsignviz-1.0.war//kh.callsign.spotcollector.service.CallsignProcessorService.(CallsignProcessorService.java:35)

The clue here is the NullPointerException is coming from the HamQTHClient.logon() call, which is my library using jax-ws to call the remote HamQTH service.

Running my JUnits locally, this is definitely a JDK version issue. On Java 8 this code runs fine, but on Java 10 and 11 I get this:

Caused by: java.lang.ClassNotFoundException: javax.xml.bind.PropertyException
     at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)

… and now I’m remembering the jax-ws removal timeline from JEP-320:

  • Java 9 : deprecated for future removal
  • Java 11: complete removal of javax.xml.bind (jaxb) and javax.xml.ws (jax-ws) (also see here)

I don’t currently have Java 9 installed locally, but I suspect the code would work, but my test local Ubuntu 18.04 has OpenJDK 11, so this is currently my target JRE.

Ok, so what are the migration steps for using jax-ws after Java 8? A quick search found this useful article on migration, and adding explicit dependencies on the jaxb-api and the Glassfish reference implementation was exactly what I needed to get the code to run unmodified on Java 11:

<dependency>
    <groupId>javax.xml.bind</groupId>
    <artifactId>jaxb-api</artifactId>
    <version>2.3.1</version>
</dependency>
		
<dependency>
    <groupId>org.glassfish.jaxb</groupId>
    <artifactId>jaxb-runtime</artifactId>
    <version>2.3.1</version>
</dependency>

 

Proudly powered by WordPress