Tech Notes – Page 8 – Kev's Development Toolbox

July 11, 2019July 11, 2019

Extracting (scraping) webpage content with Puppeteer

I’m working on a personal project to extract some content from a number of websites. This idea started out trying to use a REST api to a site but it turned out to be far more complicated involving far more steps that I was prepare to invest the time in to get working, so I realized I could just scrape their HTML content instead to get what I was looking for.

There are many HTML scraping and parsing libraries. I took a look at x-ray and a few others, but what I discovered is that many sites detect the fact that you’re not browsing the site in a real browser (presumably from checking your user-agent and cookies etc) and then force you into completing CAPTCHAs etc to prove that you’re not a robot.

I then stumbled across Apify, and more specifically Puppeteer. I think Apify does far more than I need for this little project. Instead, since Apify can also use Puppeteer under the covers, I found that just using Puppeteer directly does all that I need.

Here’s a small script for extracting some text from an example site:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto('https://www.example-website.com');
    const extractedValue = await page.$eval('#element-id', el => el.innerHTML);
    console.log('extracted value: ' + extractedValue);

    await browser.close();
})();

June 26, 2019June 26, 2019

Learning Golang (part 2)

It’s been several months since I started to look at Golang, but I’ve been picking it back up again. The design decisions in the language are starting to make me like this language more and more. It’s clear there were decisions made to address many pet peevs with other languages. Here’s a few I’ve noticed:

standard formatting and the ‘go fmt’ tool. In many other languages there’s endless debates about where your { and } should go, and honestly it really doesn’t matter. Go addresses this by requiring a single style where the opening { is syntactically required at the end of the line (and not allowed on the following line) in order for your code to compile. End of argument. Done
a single for loop for iteration. Other languages support for and while in a few different variations (while condition at the beginning of a block or at the end), but Go has a ‘for’ statement and that’s it. Keep it simple.
much time has been wasted in Java arguing about whether exceptions should be checked or unchecked. Go’s approach, no exceptions, no exception handling. You have an ‘error’ type which you can chose to return as the last return value from a function if needed

Here’s my notes following on from my earlier Part 1.

Imports

Imports are each on a separate line, with the package in quotes:

import "fmt"
import "unicode/utf8"

Variables

Variable types are defined after the variable name, both in variable declaration and function parameters:

func ExampleFunc(x int, y int) int {
  var example int
  var example2 string
  ...
  return 1
}

This function takes 2 params, both ints, and returns an int.

Unlike other languages where you can pass many params but only return a single result, Go functions allows you to return multiple results, e.g.

func ExampleFunc(x int, y int) (int, int) {
  var example int
  var example2 string
  ...
  return a, b
}

To call this function and receive the multiple results:

x, y := ExampleFunc(1, 2)

If you’re only interested in one of the return values you can use the blank identified to ignore one of the returned values:

_, y := ExampleFunc(1, 2)

Functions with a capital first letter are exported so they can be imported elsewhere, functions with a lowercase first letter are not.

June 5, 2019June 5, 2019

Solving a Quora programming homework question in ARM Assembly

It makes me sad when students ask the online communities to ‘please write me a program that does the following’. Not only is this flat out dishonest to ask someone else to do your homework for you, the opportunity for learning is in the process of solving the problem and writing the code yourself. If someone else writes the code, you’ve missed that opportunity. There’s little to be learned if someone else does the hard work and gives you the final result.

Most communities police these types of questions pretty well. Reviewers on Stackoverflow for example are quick to respond to these types of questions to help developers restructure a generic request for help into a specific question about a specific problem that the developer need help with. The guides that are usually referred to on suggestions to restructure these questions are actually very good advice and reminders for us all on how to ask good questions:

what types of question can you ask on Stackoverflow
how to ask a question

On other community sites, the community response goes in a different direction. This question for help on how to write a program was responded to be some of the funniest and bizarre approaches to solve the askers problem in all sorts of obscure language from Brainfuck to Whitespace and plenty of other weirdness inbetween.

Not to be left out but a little later to the party, I realized I hadn’t done any ARM assembly for a while, so here’s my solution in ARM Assembly that I developed on my Raspberry Pi:

.global main

main:
MOV R4, #3 @ init outer line counter =3 

_outerloop:
MOV R3, R4 @ init word loop counter with current value of outer counter

_wordloop:
MOV R7, #4  @ syscall 4: output to stdout
MOV R0, #1 @ stdout
MOV R2, #6 @ length of string
LDR R1, =output
SWI 0

SUB R3, R3, #1 @ decrement word loop counter
CMP R3, #0
BNE _wordloop

@print newline
MOV R7, #4  @ syscall 4: output to stdout
MOV R0, #1 @ stdout
MOV R2, #1 @ length of string
LDR R1, =eol
SWI 0
 
SUB R4, R4, #1 @ decrement outer counter
CMP R4, #0
BNE _outerloop 

_exit:
MOV R1, #0
MOV R7, #1
SWI 0
 
.data
output:
 .asciz "Smile!"
eol:
 .asciz "\n"