Kev’s Blog – Page 102 – Kev's Development Toolbox

October 15, 2018October 15, 2018

Building bots on Twitter with AWS Lambdas (and other stuff)

I’ve built a few different bots on Twitter and written several articles describing how I built them. Some of these were a few months back – once they’re up and running it’s easy to forget they’re up and running (thanks to the free tier on AWS Lambda which means you can run scheduled Tweets well within the free tier limits). This is a summary of the bots I’ve developed so far.

Looking at where I got started, my first bot was to build an integration between Amateur Radio 2m Packet, retweeting packets received locally to Twitter. This was my first experience working with the Twitter REST apis and the OAUTH authentication, so I lot of what I learned here I reapplied to the following bots too:

For my next project, I was inspired by articles by researcher Janelle Shane who has been training ML models to produce some hilarious results, such as weird recipes, college course names and many others. I was curious what content a ML model would generate if I extracted all of my past 4000+ Tweets from Twitter and trained a model with the content. I had many questions, such as would the content be similar in style, and is 4000 Tweets enough text to train a model? You can follow my progress in these posts:

https://www.kevinhooke.com/2018/04/08/generating-tweets-using-a-recurrent-neural-net-torch-rnn/

This then led to repeating the experiment with over 10 years of my blog articles and posts collected here, which you can follow in these posts:

https://www.kevinhooke.com/2018/04/23/retraining-my-recurrent-neural-net-with-content-from-this-blog/

Next, what would it take to train my model in the cloud using AWS Sagemaker, and run using AWS Lambdas?

You can follow this bot on Twitter here: @kevinhookebot

I had fun developing @kevinhookebot – it evolved over time to support a few features, not just to retweet content from the trained ML model. Additional features added:

an additional Lambda that consumes the Twitter API ‘mentions’ timeline and replies with one of a number of canned responses (not generated, they’re just hard coded phrases). If you reply to any of it’s tweets or Tweet @ the bot it will reply to you every 5 minutes when it sees a new tweet in the mentions timeline
another Lambda that responds to @ mentions to the bot as if it is a text-base adventure game. Tweet ‘@kevinhookebot go north’ (or east/west/south) and the bot will respond with some generated text in the style of an adventure game. There’s no actual game to play and it doesn’t track your state, but each response is generated using @GalaxyKate ‘s Tracery library to generate the text using a simple grammar that defines the structure of each reply.

After having fun with the adventure text reply generator, I also used the Tracey library for another AWS Lambda bot that generates product/project names and tweets every 6 hours. I think it’s rather amusing, you can check it out here: @ProductNameBot :

My most recent creation I upped the ante slightly and wondered what it would take to develop a Twitter bot that playeda card game. This introduced some interesting problems that I hadn’t thought about yet, like how to track the game state for each player. I captured the development in these posts here:

I have some other ideas for something I might put together soon. Stay posted for more details 🙂

October 14, 2018October 14, 2018

MacOS Mojave spaces gestures stopped working after taking screenshot with Grab

I just took a screenshot with the updated Grab app in MacOS Mojave and my 4-finger swipe to switch Spaces desktops stopped working. I found this post which talks about something similar with multitouch gestures which sounds like it’s a randomly occurring issue with multitouch gestures. A reboot fixed it for me, but it was surprising how much I rely on gestures to switch desktops/apps, and when it’s not working how to you switch?! (Ctrl-Left/Right is also works as a keyboard shortcut).

October 5, 2018October 5, 2018

Building Redis from source on Ubuntu server 18.04

After downloading redis source and attempting to make on Ubuntu server 28.04, it looks like I’ve got some dependencies missing:

kev@ubuntu18-redis1:~/redis/redis-4.0.11$ make

cd src && make all

make[1]: Entering directory '/home/kev/redis/redis-4.0.11/src'

    CC Makefile.dep

    CC adlist.o

In file included from adlist.c:34:0:

zmalloc.h:50:10: fatal error: jemalloc/jemalloc.h: No such file or directory

 #include <jemalloc/jemalloc.h>

          ^~~~~~~~~~~~~~~~~~~~~

compilation terminated.

I found an identical post about this issue, with not building the dependencies. To fix:

cd deps
make hiredis jemalloc linenoise lua geohash-int
cd ..
make
sudo make install

That resulted in a clean install, ready to start redis-server !

September 30, 2018September 30, 2018

AWS VPCs and CIDR blocks (possibly more than you’ve ever wanted to know about IP addresses, networks, subnet masks and all that stuff)

If you’re interested in firing up some AWS EC2 instances, and some point you’re going to need to know about VPCs (Virtual Private Clouds) and CIDR blocks. If you’re studying for the AWS Solution Architect certification this is also an important topic covered in the exam. There’s a whole section in the AWS docs here that covers the topic of CIDR blocks.

Unless you’re an experienced network engineer or someone who works with configuring network topologies, this might be the first time you’ve come across this concept (I had seen the CIDR notation before but didn’t know what it meant). At first it might be easy to memorize a few of the common examples and remember which represents a smaller network and which is larger, but to understand why a /24 network has less available addresses than a /8 network you’ll need to dig a little deeper.

First, let’s look at typical IPv4 IP addresses, networks and subnets:

IPv4 IP addresses have 4 digits separated by ‘.’s. Each digit is 8 bits, and are referred to as ‘octets’.

A typical address on a home network like 192.168.1.16 has a subnet mask of 255.255.255.0. This means the first 3 octets are used to represent the network, in this case 192.168.1.0. Computers with this same IP prefix are therefore part of the same network, so

192.168.1.1
192.168.1.2
up to
192.168.1.254

… are all on the same network and assuming your router and each individual computer is setup ok, then each of the computers with these addresses on this same network can also see each other (without any additional routing between networks).

In this range there are also some reserved IPs for special purposes:

192.168.1.0 is referred to as the network itself
192.168.1.255 is a broadcast address for this network
this leaves 192.168.1.1 through 192.168.1.254 as usable addresses in the network

Originally networks were divided and categorized by 8 bit boundaries and were referred to as ‘class’ based networks:

Class C network:

subnet mask 255.255.255.0
first 3 octets are the network
the smallest network Class, with 256 available addresses
only the 4th octet is available for your host addresses, 0-255, 256 available addresses, (or 1-254 ignoring 0 and 255)
24 bits used for network, 8 for hosts

Class B network:

subnet mask 255.255.0.0
first 2 octets are the network
3rd and 4th octets available for host addresses
16 bits used for network, 16 bits for hosts

Class A network:

subnet mask 255.0.0.0
first octet is the network
the largest IPv4 network Class
2nd, 3rd and 4th octets available for hosts
8 bits used for network, 24 bits for hosts

Ok, so this summarizes Class based networks which are divided by 8 bit boundaries, of which we only have 3 options, A (largest), B and C (smallest). Now let’s look at Classless networks.

Instead of restricting to 8 bit boundaries, Classless networks can use any of the bits to represent the network and the remaining bits for the hosts. Now let’s first look at the Classless InterDomain Routing (CIDR) notation for the same Class based networks as the first examples:

/24 CIDR block is the same as Class C, with subnet mask 255.255.255.0
/16 CIDR block is the same as Class B, with 255.255.0.0
/8 CIDR block is the same as Class A, with subnet mask 255.0.0.0

You might have noticed that the number in the /xx CIDR block notation is referring to is the number of bits used for the network, and therefore implies the number of bits remaining for host addresses (from the 4 total octets, or 32 bits). This approach is not restricted to the 8 bit boundaries though, any number of bits can be used for combination of network and hosts, so /24, /23, /22 and any value from /32 to /1 are all valid (although /32 with all bits for the network is of little practical use, similarly for /31 as there’s only 1 bit remaining for a host address, but these are used for special purposes, e.g. for a single host route, or point to point network links).

Ok, so now let’s apply this to look at AWS VPCs and CIDR blocks.

For a /24 block, we already looked at x.x.x.0 address to refer to the network and x.x.x.255 for the broadcast address. AWS VPC subnets reserve a further 3 IP addresses (described here) for AWS usage, x.x.x.1, x.x.x.2, and x.x.x.3, so for each subnet there are 5 IP addressed unavailable for your own hosts.

Now we’ve looked at how the address blocks are comprised, it’s easy to calculate how many addresses are available in a VPC for any CIDR block. Taking /24 as an example:

32 – 24 = 8 bits for the host addresses
2^8 = 256
256 – 5 = 251

Knowing how IP addresses are structured, how CIDR blocks define the range of possible IPs for your hosts, the purpose of the .0 and .255 and with the additional 3 AWS reserved IPs, you can now calculate how many IP addresses are available to your VPS for any CDR block.