Upgrading an AngularJS 1.x app to use Webpack

I have an AngularJS 1.x app that I built a few years back, and as part of doing some updates, I thought it would be worthwhile to use Webpack to bundle and improve the deployment approach.

I first took a look at following this guide which was from a few years back

… but ran into this error:

Error: webpack.optimize.CommonsChunkPlugin has been removed, please use config.optimization.splitChunks instead.

This was my starting point for my webpack.config.js:

var webpack = require('webpack');
module.exports = {
context: __dirname + '/app',
entry: {
app: './js/SpotVizApp.js',
vendor: ['angular']
},
output: {
path: __dirname + '/js',
filename: 'app.bundle.js'
},
plugins: [
new webpack.optimize.CommonsChunkPlugin(/* chunkName= /"vendor", / filename= */"vendor.bundle.js")
]
};

It seems like some plugins used in earlier versions of Webpack are now included in base features of Webpack 4, so following this guide here it seems we can narrow this down and address the error earlier to this config:

var webpack = require('webpack');
const path = require('path');
module.exports = {
entry: './src/app/js/SpotVizApp.js',
output: {
path: path.resolve(__dirname, 'dist'),
filename: 'bundle.js'
}
};

When webpack starts reading dependencies from my index.html entry point, I now need to tell it where the other source dependencies are, so at the bottom of my app.js I also added:

require('./MapVizControllers');
require('./MapVizServices');
require('./SpotDataControllers');

… and at the bottom of each of these 3 files, exported the module like this:

module.exports = SpotVizControllers;

Now running webpack, it picks up each of the source .js files and bundled together into a single bundle:

$ npm run bundle
spotviz@1.0.0 bundle /Users/kev/develop/AmateurRadioCallsignSpotHistory/spotviz-angularjs-webpacktest
webpack --mode production
Hash: 3e2905ee6f48f56e35b7
Version: webpack 4.41.2
Time: 141ms
Built at: 10/27/2019 2:25:31 PM
Asset Size Chunks Chunk Names
bundle.js 10.4 KiB 0 [emitted] main
Entrypoint main = bundle.js
[0] ./src/app/js/SpotVizApp.js 2.28 KiB {0} [built]
[1] ./src/app/js/MapVizControllers.js 20.6 KiB {0} [built]
[2] ./src/app/js/MapVizServices.js 794 bytes {0} [built]
[3] ./src/app/js/SpotDataControllers.js 2.2 KiB {0} [built]

Ok, now things are looking good, but we’re missing other 3rd party module dependencies, other static files like .css and images, and we also need to update the index.html to refer to the new bundle file.

Adding a loader for CSS files per steps here, using a webpack module:

module: {
rules: [
{
test: /.css$/,
use: 'css-loader'
}
]
}

… and then add an import to my css in app.js so webpack can find it and include it in the bundle:

import '../css/callsignviz.css';

… at this point, the css is bundled but when the index.html loads, I get 404s on the .css files, and for that we also need to add the style loader, so now the config looks like this:

{
test: /.(s*)css$/,
use: [
{
// Adds CSS to the DOM by injecting a <style> tag
loader: 'style-loader'
},
{
// Interprets @import and url() like import/require() and will resolve them
loader: 'css-loader'
}
]
}

We need to generate a new index.html that is updated with new references to the bundled, files. following the steps here, I added this plugin:

plugins: [
new HtmlWebpackPlugin({template: './src/index.html'})
]

Ok, now I’ve got all my dependencies references as require(‘example.js’) in my main app.js, and now I’m moving on to resolving a 404 issue with AngularJS template files.

I added webpack config for the html templates:

{
test: /\.html$/, use: [
{
loader: 'html-loader'
}
],
}

Next, image files are getting 404:

ERROR in ./src/about.html
Module not found: Error: Can't resolve './images/wildfly_logo_200px.png' in '/Users/kev/develop/AmateurRadioCallsignSpotHistory/spotviz-angularjs-webpacktest/src'

So I add file-loader like this:

        {
test: /\.(png|svg|jpg|gif)$/,
use: [
{
loader: 'file-loader'
}
],
},

This next error was unusual one, to do with Angular’s bundled jqlite, in place of a full version of jQuery:

Error: [jqLite:nosel] Looking up elements via selectors is not supported by jqLite! See: http://docs.angularjs.org/api/angular.element
https://errors.angularjs.org/1.7.8/jqLite/nosel
at angular.js:138
at Object.JQLite as element

I think this is related to the need to load jQuery first before the AngularJS library (I’d see similar errors in an app prior to Webpack). This this post here, the solution was to force loading of jQuery with another Webpack plugin:

    new webpack.ProvidePlugin({'window.jQuery': 'jquery'}),

Although I didn’t know until later (see below re. the cal-heatmap library issues), I believe what ProvidePlugin is doing is allowing you to expose a library as a global, in this case by attaching jquery to window as jQuery.

This also showed up in at least 2 other areas in other libraries with a similar error for the same reason, saying jQuery is not defined but it’s just that it’s not visible in global scope. In this example a Bootstrap js library is expecting to find it defined:

Uncaught ReferenceError: jQuery is not defined
at Object../node_modules/bootstrap/js/transition.js (bundle.js:formatted:75970)
at webpack_require (bundle.js:formatted:37)

I ended up with 3 different configs to resolve the jQuery global related issues, like this:

new webpack.ProvidePlugin({
'window.jQuery': 'jquery',
$: "jquery",
jQuery: "jquery"
}),

Next, an issue with the moment library that I hadn’t seen before that required including the angular-moment library to provide the ‘angularMoment’ module, and then injecting it into each controller where needed.

Next problem at runtime, one of my pages uses ng-include directives, and on this one page I get 404s trying to load these templates in this file:

<div ng-include src="./historyPlaybackControls.html">
</div>


<div ng-include src="./visualizationPlayback.html">
</div>

Similar to how all other templates and static resource need to be referenced with an import or require and have their own Webpack loader to ensure they can be loaded at runtime from the bundle, these templates needed to be loaded by the ngtemplate-loader, configured like this:

        {
test: /historyPlaybackControls\.html$/,
use: [
{
loader: 'ngtemplate-loader'
}
],
},

And then changed in template to reference the 2 files like this:

<div ng-include src="'/src/app/mapviz/historyPlaybackControls.html'">
</div>

<div ng-include src="'/src/app/mapviz/visualizationPlayback.html'">
</div>

This working solution I pieced together from multiple articles, none completely gave me a working solution, but I got a working approach by combining the tips in each of these references, here, here, here, and here.

Next, the single remaining that I probably spent the most time looking into, before I found a description of what the problem actually was, and how it’s solved with Webpack’s ProvidePlugin (which I already used to solve the jQuery issues above, not understanding at the time what that plugin was doing on what issue it was solving).

The issue was with the cal-heatmap library, used by the angular-cal-heatmap-directive library, and looks like this:

Uncaught ReferenceError: CalHeatMap is not defined
at Module../src/app/js/SpotVizApp.js (bundle.js:formatted:135650)
at webpack_require (bundle.js:formatted:37)

Without understanding the problem, I went off on wild goose chase with loading angular-cal-heatmap-directive as a module using bower (because it doesn’t exist in npm), but the other 2 dependencies this library needs, d3 and cal-heatmap are available in npm. Initially I thought that including the libraries that had been installed locally with the different package managers was something to do with the issue (it’s not, the same .js exists as a file regardless of whether it was downloaded by bower or npm), so I tried to get the 3 of these all loaded by bower. This meant I needed to add this config:

resolve: {
modules: ["node_modules", "bower_components"],
descriptionFiles: ['package.json', 'bower.json'],
alias: {
'd3': path.resolve(__dirname, 'bower_components/d3/d3.js'),
'cal-heatmap': path.resolve(__dirname, 'bower_components/cal-heatmap/cal-heatmap.js'),
'angular-cal-heatmap': path.resolve(__dirname, 'bower_components/angular-cal-heatmap-directive/dist/1.3.0/calHeatmap.min.js')
}
},

… adding bower_components and it’s config file bower.json so Webpack knows to look there for modules, and the declaring aliases for d3, cal-heatmap and angular-cal-heatmap that each point to where their .js file needs to get loaded from.

This part is all good, but I’ve still got the “ReferenceError: CalHeatMap is not defined” error. At this point with some debugging I realized I could reference ‘new CalHeatMap()’ in the very top of my app.js, but it was not visible in any of my controllers., giving me the hint that this was a scope issue. After trying many (too many) random variations importing it with ‘import * as CalHeatMap from ‘cal-heatmap’ and other syntax combinations, some googling about weback and libraries with global scope vars led me to some tips about Webpack’s ProvidePlugin (here, and docs here). Adding one more config line to my ProvidePlugin like this:

CalHeatMap: "CalHeatMap"

… did the trick. It exposes CalHeatMap as a global var, so where the angular-cal-heatmap-directive library is looking to call ‘new CalHeatMap()’ it finds it in global scope as expected and resolves this issue.

One last error – after building my production minified build with webpack, at app startup I got this error:

Uncaught Error: [$injector:modulerr] Failed to instantiate module SpotVizApp due to:
Error: [$injector:unpr] Unknown provider: e

It seems this is to do with injected server names in AngularJS controllers getting minified and loosing reference, as described here. This also applied to how I configured my router, so I needed to use the same passing dependencies in an array approach, changing from this:

spotVizApp.config(function($stateProvider, $urlRouterProvider) {
...
});

to this:

spotVizApp.config([ '$stateProvider', '$urlRouterProvider', function($stateProvider, $urlRouterProvider) {
...
]]);

I’ve still got a couple of issues with the ng-include templates that look like they’re not getting bundled in my prod build, but at this point I think I’m pretty close.

GitLab CI Runner artifacts size error: “too large archive”

As part of putting together a GitLab CI pipeline to build a Python deployment for an AWS Lambda, I ran into an issue with the size of the build dir that I’m zipping up ready for deployment:

Uploading artifacts...
./build: found 2543 matching files 
ERROR: Uploading artifacts to coordinator... too large archive id=181 responseStatus=413 Request Entity Too Large status=413 Request Entity Too Large token=rtRUzgtp
FATAL: Too large

Hmm. Ok. A quick search found this post which says there’s a setting to increase the max build artifact size – it’s under /admin/application_settings, in the Continuous Integration setting – looks like the default is 100MB, so let’s bump that up and try again:

Initial thoughts with a new 15″ 2015 Macbook Pro

My 15″ 2012 MacBook Pro started to get unreliable in the past couple of months, with random kernel panic reboots and the ominous “Your computer restarted because of an issue“. Looking in the Console app at the error logs, there were a number of common errors, mainly with the Intel HD graphics. It’s clear something was starting to fail, as having this MacBook Pro as my daily driver for 5 years I’d gone from not rebooting for months (I’d just sleep it over night), to having a random hang then reboot once every couple of weeks, and it started to get more frequent.

Long story short, upgrading to High Sierra was the first OS X or MacOS upgrade that I’d experienced a failure during the install (I’ve gone through 10.5 Leopard, through 10.12 Sierra). I had two disks in this MBP, one SSD and one HDD. Installing High Sierra to my SSD, apparently High Sierra upgrades SSDs by default to the new APFS file system, which is only supported on High Sierra, and apparently is not reversible back to HFS+. So I ended up with a 5 year old machine with a failed upgrade and would boot to a black screen with a spinning progress circle and not get any further. Booting from my second drive which had El Cap on it, it couldn’t see the now APFS drive, and then also started crashing on startup where previously was fine. i spent far too long trying to reinstall a fresh install of El Cap and Sierra and couldn’t get back to a stable place. Even when I did get El Cap cleanly installed, it would crash after a couple of minutes after logging on, so clearly the time had come for this 5 year old machine to be retired.

So, new 15″ 2015 Macbook Pro. Externally it looks very similar to the 2012, with a few minor differences:

  • It’s slightly smaller
  • It’s thiner
  • Noticeably lighter
  • The front cutaway section when you insert your thumb to open the screen is not as deep and doesn’t have edges as sharp (which I always thought was weird on the the 2012)

More significant changes:

  • The Retina screen is INCREDIBLE. I don’t think I’ve ever seen a screen that is so sharp. Text is incredibly sharp and clear in all apps, and the level of detail even in the stock background images is mindblowing
  • The keyboard style is noticeably different. The travel on the keys is much shorter and softer, even squidgy. At first I’d describe the feel as what my 2012 keyboard feels like now it’s well worn after 5 years of daily usage, but after a few days of using it, it feels good for typing, as with less resistance and travel, there’s less need to really mash the keys. I’m not sure if the 2015 has the new butterfly style keys, but either way, a very noticeable change
  • It runs cold to the touch, even after being on for a few hours (my 2012 would get warm, and even hot if you were doing something intensive like editing a video)

The single most impressive change that actually prompted me to write this short review is the new touchpad. I could not put my finger on (pun intended) what was different with the touchpad and how it felt. At first I thought it was similar to the shorter, lighter travel of the keys, in that maybe the travel of the touchpad when you click it had been reduced. I did a quick search to read about what had changed, and then it clicked (ha!). The haptic feedback from the touchpad really does feel like it’s clicking, but without the physical movement it feels a bit odd. I’m not sure at this point if it’s better than before, but it’s definitely interesting and a very clever approach. I understand this was a design change to allow the thiner/lighter MacBooks and MacBook Airs to be even thinner without a touchpad that physically moved. Anyway, it was a lightbulb moment when I read about the haptic feedback.

So far, very impressed and pleased with my new machine. I hope that this one too will last another 5 years 🙂

Loading the Yelp dataset into MongoDB

In a previous post, I downloaded the Yelp dataset, 5.79GB of json data. My first thought (before I get to experimenting with Apache Spark), was how can I extract some basic stats from this dataset, basics like how many data items are there, and what do the records look like in each of the data files.

Using mongoimport and referring to the docs here, the syntax for the import is:

mongoimport -d database -c collection importfile.json

Here’s the Yelp dataset json file for importing, to get an idea of the size of each file:

kev@esxi-ubuntu-mongodb1:~/data/yelp$ ls -lS

total 5657960

-rwxrwxr-x 1 kev kev 3819730722 Oct 14 16:16 review.json

-rwxrwxr-x 1 kev kev 1572537048 Oct 14 16:22 user.json

-rwxrwxr-x 1 kev kev  184892583 Oct 14 16:16 tip.json

-rwxrwxr-x 1 kev kev  132272455 Oct 14 16:03 business.json

-rwxrwxr-x 1 kev kev   60098185 Oct 14 16:03 checkin.json

-rwxrwxr-x 1 kev kev   24195971 Oct 14 16:03 photos.json

 

So importing each of the datasets, one at a time:

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c checkin checkin.json

2017-10-14T16:49:35.566-0700 connected to: localhost

2017-10-14T16:49:38.564-0700 [#########……………] yelp.checkin 22.6MB/57.3MB (39.5%)

2017-10-14T16:49:44.474-0700 [########################] yelp.checkin 57.3MB/57.3MB (100.0%)

2017-10-14T16:49:44.475-0700 imported 135148 documents

 

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c business business.json

2017-10-14T16:49:59.593-0700 connected to: localhost

2017-10-14T16:50:02.592-0700 [#####……………….] yelp.business 27.9MB/126MB (22.1%)

2017-10-14T16:50:12.873-0700 [########################] yelp.business 126MB/126MB (100.0%)

2017-10-14T16:50:12.873-0700 imported 156639 documents

 

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c tip tip.json

2017-10-14T16:50:38.061-0700 connected to: localhost

2017-10-14T16:50:41.058-0700 [##………………….] yelp.tip 17.5MB/176MB (9.9%)

2017-10-14T16:51:07.381-0700 [########################] yelp.tip 176MB/176MB (100.0%)

2017-10-14T16:51:07.381-0700 imported 1028802 documents

 

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c user user.json

2017-10-14T16:51:28.648-0700 connected to: localhost

2017-10-14T16:51:31.648-0700 [……………………] yelp.user 36.9MB/1.46GB (2.5%)

2017-10-14T16:54:15.907-0700 [########################] yelp.user 1.46GB/1.46GB (100.0%)

2017-10-14T16:54:15.907-0700 imported 1183362 documents

 

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c review review.json

2017-10-14T16:57:01.018-0700 connected to: localhost

2017-10-14T16:57:04.016-0700 [……………………] yelp.review 34.9MB/3.56GB (1.0%)

2017-10-14T17:02:31.967-0700 [########################] yelp.review 3.56GB/3.56GB (100.0%)

2017-10-14T17:02:31.967-0700 imported 4736897 documents

 

Done! Almost 6GB of data imported to MongoDB. Now, time for some queries!