Kev’s Blog – Page 119 – Kev's Development Toolbox

July 19, 2018July 22, 2018

Using AWS Sagemaker to train a model to generate text (part 1)

If you’ve followed any of my recent posts, you’ll know I have been using RNN models to generate text from a model trained with my previous tweets, and the text from all of my previous blog posts, and feeding this into a Twitter bot: @kevinhookebot

The trouble I have right now is the scripts and generate models are running using Lua, and although I could install this to an EC2 instance, I don’t want to pay for an EC2 instance being up 100% of the time. Currently when I generate a new batch of text for my Twitter bot, I startup a local server running the scripts and the model, generate new text, and then stage it to DynamoDB to get picked up by the bot when it’s scheduled to next run. With the AWS provided Machine Learning services, there has to be something out of the box I can use on AWS that would automate these steps.

Let’s take a look at using AWS SageMaker.

First I created a SageMaker notebook with a new role, to access S3 buckets with ‘sagemaker’ in the name.

Then I created an S3 bucket – sagemaker-kevinhooke-ml – and uploaded a copy of my data file (all my previous posts from this blog, concatenated into a single file).

Next I created a new Training Job.

You need to pick an algorithm for the training and there’s a selection of provided algorithms for different purposes. To generate new text ‘in the style of’ the text that I’m going to training the model with, the ‘Sequence2Sequence’ looks like it does what I need.

On completing the Training Job, I got this error:

Ok, so let’s change the instance type. I picked the smallest of the instances before:

And it looks like you can’t change the type on the Notebook. So let’s create a new Notebook. Looking at the instance types, the ones with GPU support are on the large side, so let’s pick the smallest of the options and try again.

At this point I realized the instance type it’s talking about is for the training job not the notebook, and it’s specified here:

So let’s pick one of the GPU types and try again.

First training job is running:

Next error:

Hmm, off to do some reading in the docs to see what’s needed to run this training job. The docs here describe what’s needed for the sequence2sequence algorithm and I’m clearly missing some steps, so taking a pause here and will come back with an update later.

June 29, 2018June 29, 2018

Piping audio between applications: Configuring ham radio apps on Mac OS using SoundFlower (virtual audio cables)

You’re running some digital mode software like WSJT-X on your Mac. Normally you would use a physical audio cable between your radio to your Mac, either via a soundcard interface like a Rigblaster, or even a direct USB connection to your Mac and your radio. What happens though if you want to route your audio from one application to another? For example, can you pipe the audio from a Web SDR running in your browser straight into WSJT-X (or any other digital mode software)? What you need are ‘virtual audio cables’.

On Windows you have a product called VB-Cable (the approach for Windows is similar to what’s described here). On MacOS you have a couple of options. There’s a commercial product called Loopback from Rogue Amoeba, or an open source alternative called Soundflower.

Follow the instructions to download and install. Once installed, you’ll find a couple of extra sound devices in your System Preferences:

Think of the Soundflower device as your cable. Instead of configuring Speakers for output and Mic for input, if you configure the input for one app as Soundflower (one end of the virtual cable) and the output for another app also as Soundflower (the other end of the cable), and sound output from one app is now directed into input of the other.

Let’s give this a go to connect the output from a WebSDR with the input to WSJT-X.

First, from System Preferences, select the Output to be Soundflower (shown above).

Start up a browser and pick a Web SDR station from http://websdr.org/

Here’s KFS and we’re tuned in to 7.074Mhz USB to receive some FT8:

Next, start up WSJT-X and go to Preferences, Audio:

Note that with Input = Soundflower we’re routing the Output audio from the WebSDR running in the browser into the Soundflower virtual cable. From WSJT-X we’re then taking the audio from this virtual cable as the input into WSJT-X, effectively routing the audio from the web browser into WSJT-X.

Also note that with Output = Soundflower in WSJT-X, if we transmit on WSJT-X the audio will also go out on the virtual cable. With WebSDR we can’t obviously transmit, but if you have access to a remote rig like remotehamradio.com, you can route the audio from WSJT-X into the remote rig app. More on that coming next.

You might note that with this current configuration there’s no actual audio coming out of your speakers. With some virtual cables you have the option to monitor the audio passing over the virtual cable. On MacOS you also have the ability to create composite audio devices using the Audio MIDI Setup app:

This shows a ‘Multi-Output Device’ comprising both the regular built-in audio (your speakers) and Soundflower. Now you’ve got the best of both worlds. More on this next step, and also configuring to use remotehamradio.com with WSJT-X coming up next.

June 19, 2018June 19, 2018

Eclipse Oxygen with Atlassian Connector plugin for accessing Jira issues

I’ve been kicking the tires in my local dev setup running my own Jira and GitLab installations. I’ve been meaning to take a look at how to access Jira tickets from within Eclipse, and then the next logical step is to look at the Jira to GitLab integration.

First up, let’s look at accessing Jira tickets in Eclipse. Docs on the Atlassian Connector are here: https://confluence.atlassian.com/ideplugin/atlassian-connector-for-eclipse. The installation guide gives an Eclipse Update Sites for Eclipse version up to Luna but not more recent versions (Mars, Neon, Oxygen), but questions online (e.g. here) suggest the Luna version still installs and works with Oxygen (using update site: http://update.atlassian.com/atlassian-eclipse-plugin/rest/e3.7)

Integration within Eclipse is using Mylyn. After installing the plugin by adding the update site above, open the Mylyn/Task Repository view:

and select the Jira option:

Press Next and enter the URL for your Jira server:

When prompted to create a new Query, select Yes – this is what retrieves your assigned tasks from Jira:

Press Next for the next dialog. There’s a lot of options here (continues below the area in this screenshot), but in this case I’m interested just in the issues logged for my Blackjack Twitterbot project, so I selected this specific project:

In my Task List view I can now see a list of my assigned tasks, open and competed:

Double-clicking any of these opens the ticket in Eclipse:

This is pretty typical of any issue ticket tracking support in Eclipse. At this point I can edit and update the tickets.

Next up, I’ll look at Jira and GitHub integration.

June 19, 2018

Building a Card Playing Twitter Bot: when AWS Lambdas timeout

node.js AWS Lambdas have a default timeout of 3 seconds when you create a new Lambda via the AWS Console. This can be increased with the slider in the Console, but if you keep the default and you get a timeout, you’ll see this error in CloudWatch:

2018-06-14T01:16:56.953Z 9f86... Task timed out after 3.00 seconds

From looking at the logs, my Blackjack card playing Twitter bot Lambda typically executes in around 2 secs, so the default of 3 secs probably doesn’t have enough wiggle room for unexpected slowdowns, and while testing I’ve seen this error a couple of times. When it does time out, it seems the execution just stops in the middle of whatever it was currently doing at that point, which leads to some unexpected results. In the case of this Twitter bot, the execution is being triggered via a 5 minute Cloudwatch event, so the only way I know it’s failed is to look at the logs (it’s not called via a webpage, so there’s no error being returned to the caller). Increasing the timeout to 10 secs seems to work fine for this particular Lambda.

Remember you are billed per GB-s of execution time, so keep an eye on how long your Lambdas execute for. If you’re aiming for low cost, then quicker execution times are obviously better.