Kev's Development Toolbox

July 11, 2019July 11, 2019

Extracting (scraping) webpage content with Puppeteer

I’m working on a personal project to extract some content from a number of websites. This idea started out trying to use a REST api to a site but it turned out to be far more complicated involving far more steps that I was prepare to invest the time in to get working, so I realized I could just scrape their HTML content instead to get what I was looking for.

There are many HTML scraping and parsing libraries. I took a look at x-ray and a few others, but what I discovered is that many sites detect the fact that you’re not browsing the site in a real browser (presumably from checking your user-agent and cookies etc) and then force you into completing CAPTCHAs etc to prove that you’re not a robot.

I then stumbled across Apify, and more specifically Puppeteer. I think Apify does far more than I need for this little project. Instead, since Apify can also use Puppeteer under the covers, I found that just using Puppeteer directly does all that I need.

Here’s a small script for extracting some text from an example site:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://www.example-website.com');
    const extractedValue = await page.$eval('#element-id', el => el.innerHTML);
    console.log('extracted value: ' + extractedValue);

    await browser.close();
})();

June 28, 2019June 28, 2019

WSJT-X reports time difference > 2 seconds on Windows 10

All of the JT modes (JT65, JT9), and now the new mode FT8 are sensitive to correct system time within a couple of seconds. This is because the transmit/receive cycle starts and stops at a predetermined time, and the software of each transmitting station and each remote receiving station need to be in sync in order for each transmission to get successfully decoded.

FT8 seems to be forgiving within about 1.5 seconds of difference, but above this it flags the delta between stations in the DT column:

On a new laptop with Windows 10, even though the date and time is set to set automatically using a remote time server on the internet, almost every FT8 transmission received was consistenly > 1.5 seconds in difference.

To fix this I downloaded and installed Dimension 4 and set it to start automatically. Sync’d the time and now we’re all set – you can see the received calls below in the DT column are all now within a couple of 0.1s after the sync:

June 26, 2019June 26, 2019

Learning Golang (part 2)

It’s been several months since I started to look at Golang, but I’ve been picking it back up again. The design decisions in the language are starting to make me like this language more and more. It’s clear there were decisions made to address many pet peevs with other languages. Here’s a few I’ve noticed:

standard formatting and the ‘go fmt’ tool. In many other languages there’s endless debates about where your { and } should go, and honestly it really doesn’t matter. Go addresses this by requiring a single style where the opening { is syntactically required at the end of the line (and not allowed on the following line) in order for your code to compile. End of argument. Done
a single for loop for iteration. Other languages support for and while in a few different variations (while condition at the beginning of a block or at the end), but Go has a ‘for’ statement and that’s it. Keep it simple.
much time has been wasted in Java arguing about whether exceptions should be checked or unchecked. Go’s approach, no exceptions, no exception handling. You have an ‘error’ type which you can chose to return as the last return value from a function if needed

Here’s my notes following on from my earlier Part 1.

Imports

Imports are each on a separate line, with the package in quotes:

import "fmt"
import "unicode/utf8"

Variables

Variable types are defined after the variable name, both in variable declaration and function parameters:

func ExampleFunc(x int, y int) int {
  var example int
  var example2 string
  ...
  return 1
}

This function takes 2 params, both ints, and returns an int.

Unlike other languages where you can pass many params but only return a single result, Go functions allows you to return multiple results, e.g.

func ExampleFunc(x int, y int) (int, int) {
  var example int
  var example2 string
  ...
  return a, b
}

To call this function and receive the multiple results:

x, y := ExampleFunc(1, 2)

If you’re only interested in one of the return values you can use the blank identified to ignore one of the returned values:

_, y := ExampleFunc(1, 2)

Functions with a capital first letter are exported so they can be imported elsewhere, functions with a lowercase first letter are not.