Loading the Yelp dataset into MongoDB

In a previous post, I downloaded the Yelp dataset, 5.79GB of json data. My first thought (before I get to experimenting with Apache Spark), was how can I extract some basic stats from this dataset, basics like how many data items are there, and what do the records look like in each of the data files.

Using mongoimport and referring to the docs here, the syntax for the import is:

mongoimport -d database -c collection importfile.json

Here’s the Yelp dataset json file for importing, to get an idea of the size of each file:

kev@esxi-ubuntu-mongodb1:~/data/yelp$ ls -lS

total 5657960

-rwxrwxr-x 1 kev kev 3819730722 Oct 14 16:16 review.json

-rwxrwxr-x 1 kev kev 1572537048 Oct 14 16:22 user.json

-rwxrwxr-x 1 kev kev  184892583 Oct 14 16:16 tip.json

-rwxrwxr-x 1 kev kev  132272455 Oct 14 16:03 business.json

-rwxrwxr-x 1 kev kev   60098185 Oct 14 16:03 checkin.json

-rwxrwxr-x 1 kev kev   24195971 Oct 14 16:03 photos.json

 

So importing each of the datasets, one at a time:

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c checkin checkin.json

2017-10-14T16:49:35.566-0700 connected to: localhost

2017-10-14T16:49:38.564-0700 [#########……………] yelp.checkin 22.6MB/57.3MB (39.5%)

2017-10-14T16:49:44.474-0700 [########################] yelp.checkin 57.3MB/57.3MB (100.0%)

2017-10-14T16:49:44.475-0700 imported 135148 documents

 

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c business business.json

2017-10-14T16:49:59.593-0700 connected to: localhost

2017-10-14T16:50:02.592-0700 [#####……………….] yelp.business 27.9MB/126MB (22.1%)

2017-10-14T16:50:12.873-0700 [########################] yelp.business 126MB/126MB (100.0%)

2017-10-14T16:50:12.873-0700 imported 156639 documents

 

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c tip tip.json

2017-10-14T16:50:38.061-0700 connected to: localhost

2017-10-14T16:50:41.058-0700 [##………………….] yelp.tip 17.5MB/176MB (9.9%)

2017-10-14T16:51:07.381-0700 [########################] yelp.tip 176MB/176MB (100.0%)

2017-10-14T16:51:07.381-0700 imported 1028802 documents

 

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c user user.json

2017-10-14T16:51:28.648-0700 connected to: localhost

2017-10-14T16:51:31.648-0700 [……………………] yelp.user 36.9MB/1.46GB (2.5%)

2017-10-14T16:54:15.907-0700 [########################] yelp.user 1.46GB/1.46GB (100.0%)

2017-10-14T16:54:15.907-0700 imported 1183362 documents

 

kev@esxi-ubuntu-mongodb1:~/data/yelp$ mongoimport -d yelp -c review review.json

2017-10-14T16:57:01.018-0700 connected to: localhost

2017-10-14T16:57:04.016-0700 [……………………] yelp.review 34.9MB/3.56GB (1.0%)

2017-10-14T17:02:31.967-0700 [########################] yelp.review 3.56GB/3.56GB (100.0%)

2017-10-14T17:02:31.967-0700 imported 4736897 documents

 

Done! Almost 6GB of data imported to MongoDB. Now, time for some queries!

Installing ESXi Guest Tools on CentOS 7

From here, edit /etc/yum.repos.d/vmware-tools.repo , add:

[vmware-tools]
name = VMware Tools
baseurl = http://packages.vmware.com/packages/rhel7/x86_64/
enabled = 1
gpgcheck = 1

To install:

sudo yum install open-vm-tools-deploypkg

On my freshly installed CentOS 7, this gave the error:

[kev@unknown000C2960F639 ~]$ sudo yum upgrade

Loaded plugins: fastestmirror

You have enabled checking of packages via GPG keys. This is a good thing. 

However, you do not have any GPG public keys installed. You need to download

the keys for packages you wish to install and install them.

You can do that by running the command:

    rpm --import public.gpg.key

Alternatively you can specify the url to the key you would like to use

for a repository in the 'gpgkey' option in a repository section and yum 

will install it for you.

For more information contact your distribution or package provider.

Problem repository: vmware-tools

It doesn’t say how to import the GPG keys, but this page mentions they are in this location:  /etc/pki/rpm-gpg

So to import,

sudo rpm --import  /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

Then I could update:

sudo yum update

Then install the vmware-tools:

sudo yum install open-vm-tools-deploypkg

Eclipse Oxygen.1 on MacOS fails at startup with Java 9 as system default

I’m a few days late to the Java 9 party, but installing Eclipse Oxygen after installing Java 9 on MacOS, I get this:

Looking in the .log file:

java.lang.NoClassDefFoundError: javax/annotation/PreDestroy
 at org.eclipse.e4.core.internal.di.InjectorImpl.disposed(InjectorImpl.java:450)
 at org.eclipse.e4.core.internal.di.Requestor.disposed(Requestor.java:156)
...snip...
Caused by: java.lang.ClassNotFoundException: javax.annotation.PreDestroy cannot be found by org.eclipse.e4.core.di_1.6.100.v20170421-1418
 at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:433)

Following instructions here, I added:

--add-modules=ALL-SYSTEM

Now Eclipse starts up, but adding Java 9 path on MacOS as /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home, Eclipse doesn’t want to see this as a valid jdk:

Continuing to follow the instructions, installing the Java 9 Support plugin from:

https://marketplace.eclipse.org/content/java-9-support-oxygen/

This still didn’t help but pressing the ‘Search’ button finds the Java 9 install and adds it to the list, which is odd because it points to exactly the same path:

At this point, things are still not good, as the System Libraries are not picked up:

It looks like official support is coming in the 4.7.1a release due on 10/11/17. Maybe at this point it’s best to wait for the official release.

Downloading Proxmox Container images

Before you can create a LXC container on a Proxmox virtualized environment, you need to download the template images first from an available list. You need to pre-download the images first befoe you can create new containers from them in the web ui.

From the docs here, there steps are (while ssh’d into your Proxmox server):

Update catalog of available templates:

pveam update

List the available templates:

root@pve:~# pveam available

system          alpine-3.3-default_20160427_amd64.tar.xz

system          alpine-3.4-default_20161206_amd64.tar.xz

system          alpine-3.5-default_20170504_amd64.tar.xz

system          archlinux-base_20170704-1_amd64.tar.gz

system          centos-6-default_20161207_amd64.tar.xz

system          centos-7-default_20170504_amd64.tar.xz

system          debian-6.0-standard_6.0-7_amd64.tar.gz

system          debian-7.0-standard_7.11-1_amd64.tar.gz

system          debian-8.0-standard_8.7-1_amd64.tar.gz

system          debian-9.0-standard_9.0-2_amd64.tar.gz

system          fedora-24-default_20161207_amd64.tar.xz

system          fedora-25-default_20170316_amd64.tar.xz

system          gentoo-current-default_20170503_amd64.tar.xz

system          opensuse-42.2-default_20170406_amd64.tar.xz

system          ubuntu-12.04-standard_12.04-1_amd64.tar.gz

system          ubuntu-14.04-standard_14.04-1_amd64.tar.gz

system          ubuntu-16.04-standard_16.04-1_amd64.tar.gz

system          ubuntu-16.10-standard_16.10-1_amd64.tar.gz

system          ubuntu-17.04-standard_17.04-1_amd64.tar.gz

For each of the templates you wish to use, download using for example for Ubuntu 14.04:

pveam download local ubuntu-14.04-standard_14.04-1_amd64.tar.gz

Now from the web ui, you should be able to click the ‘Create CT’ button and pick from your available templates: