2016년 12월 10일 토요일

How to create a mirror of the entire npm index including attachments using npm-fullfat-registry fullfat.js

In a previous post I attempted to create a local npm mirror by simply caching all packages I installed using npm install pkgname. Unfortunately, this approach is very slow, downloading only about 100 MB per hour. Considering that all the packages in npm take up more than 1.2 TB, this speed won't do. The recommended method for creating a local npm mirror is to use the nosql couchdb. In this method, packages from npm are stored directly in couchdb.

Step One

Install couchdb from your distro's package manager or download from the Apache CouchDB page and build from source. As of Dec 10, 2016 version 1.6 is available from the Fedora 24 official repos while version 2.0+ is available from the Archlinux repos.

Once couchdb is installed, start the couchdb service. On Fedora 24+ and Archlinux, you can do this with:

sudo systemctl start couchdb


Step Two

Access Fauxton, the web GUI for couchdb by navigating to:

http://localhost:5984/_utils

Follow the prompts to create an admin user and password. If you don't create an admin user, anyone connecting to localhost:5984 will be able to create and delete databases. Also create a new database by clicking on the gear icon at the top-left. The screenshot below shows the Fauxton UI for couchdb 1.6:




Step Three

Make sure that npm (nodejs package manager) is installed and then create a new directory into which you will install the npm package npm-fullfat-registry. Then from that directory, run as local user:

npm install npm-fullfat-registry

You will then find a sub-directory named node_modules and below that npm-fullfat-registry/bin.

cd node_modules/npm-fullfat-registry/bin

In this sub-directory you will find a single file named fullfat.js

This is the program you need to execute in order to create a local npm mirror, assuming you have already installed couchdb and have created a DB for this program to write to.

fullfat.js takes the following arguments:

-f or --fat : the url to the couchdb database for storing packages
-s or --skim : the url to the npm package index
--seq-file : file which keeps track of the current package being downloaded from npm
--missing-log : file which stores the names and sequence numbers of packages that cannot be found

To save myself the hassle of entering these parameters every time I want to invoke fullfat.js, I created a convenience script in Bash:

#!/bin/bash
# fullfat.sh
# Last Updated: 2016-11-08
# Jun Go
# Invokes fullfat.js for creating a local npm mirror containing
# npm index as well as attachments. This script is intended to
# be launched by 'npm-fullfat-helper.sh'
LOCALDB=http://user:pw@localhost:5984/registry
SKIMDB=https://skimdb.npmjs.com/registry
./fullfat.js -f $LOCALDB -s $SKIMDB --seq-file=registry.seq \
             --missing-log=missing.log

Of course you will need to edit the username and password for accessing Fauxton. The script above works, but it is not sufficient. fullfat.js crashes every so often so I created a monitor script to restart my fullfat.sh wrapper script whenever fullfat.js crashes. My monitor script is named npm-fullfat-helper.sh:

#!/bin/bash
# npm-fullfat-helper.sh
# Last Updated: 2016-11-08
# Jun Go

# During the mirroring process for npm, binary file attachments
# are saved into a local couchdb DB named 'registry', but sometimes
# downloading some packages fails or times out, which stops the
# entire process. If you manually resume with fullfat.js, you can
# start again where you left off. This script removes the need to
# do this manually.

until ./fullfat.sh; do
  printf "%s\n" "fullfat.js crashed with exit code $?. Respawning" >&2
  sleep 1
done

Using the script above, mirroring npm with fullfat.js becomes much more robust as it will be re-launched if the process returns anything other than exit code 0. But this is still not sufficient, because sometimes fullfat.js gets stuck while trying to download certain packages. No matter how many times it is restarted, certain packages (especially those with dozens of versions) never complete downloading, so you will be left with a lot of tar.gz files in temp directories but no final PUT command to couchdb. When this happens you have to manually edit the sequence file (which keeps track of which package is currently being downloaded). For example, if fullfat.js is stuck and registry.seq contains the number 864117, you must increment the number by 1 to 864117. Then if you relaunch the monitor script, fullfat.js should go on to the next package. If the package name is still unchanged, edit registry.seq once more by incrementing the new sequence by one.


Conclusion

Mirroring the npm index with file attachments using couchdb is much faster than simply caching packages installed through npm install foo. I get speeds of about 1 GB/hr. The problem is that manual intervention is required when fullfat.js gets stuck, i.e. you must manually change the sequence number stored in the sequence file (which I called registry.seq above) so that fullfat.js will skip a problematic package and go on to the next one. Another inconvenience is that as of Dec 12, 2016 the documentation for npm-fullfat-registry has not been updated. If you follow along with these old instructions, you will be told to invoke the following:

npm-fullfat-registry -f [url to local db] -s [url to npm package index]

But this won't work because the program you need to actually execute is fullfat.js. Simply replace npm-fullfat-registry above with fullfat.js and you'll be good to go. Keep in mind that the path to fullfat.js is node_modules/npm-fullfat-registry/bin in the directory into which you invoked npm install npm-fullfat-registry.

2016년 12월 1일 목요일

Using an external monitor with bumblebee and intel-virtual-output

I have an ASUS U36JC which uses both Intel integrated video and an Nvidia GeForce 310M GPU. On Windows, Nvidia has a solution called Optimus that will automatically switch between the Intel and Nvidia GPU's depending on the computing task. For example, when you are editing a document, the notebook will use the low-power integrated Intel GPU and when you are watching a movie or doing video editing, the notebook will use the power-hungry Nvidia GPU.

On Linux, however, this GPU-switching does not happen automatically. There are several different ways you can enable Nvidia GPU's on Linux, but I chose bumblebee-nvidia with the proprietary Nvidia driver as I explain in a previous post. To launch regular applications using the Nvidia GPU, you must launch your program by prefixing it with 'optirun', i.e.:

optirun [program name]

But in order to use multiple monitors on an Optimus-enabled notebook like the ASUS U36JC, you can't just invoke something like optirun startx. You need to edit a few Bumblebee X11 settings and make sure that bumblebeed (bumblebee daemon) is running to use the HDMI external monitor port (which is hard-wired to the discrete Nvidia GPU. The VGA port on the ASUS U36JC uses Intel integrated graphics, however, so bumblee is not required)

The required steps to use an external monitor through the HDMI port on Fedora 24 are as follows:

1. Make sure you have already installed and configured bumblebee.

2. sudo dnf install intel-gpu-tools (this pkg contains intel-virtual-output)

3. In /etc/bumblebee/xorg.conf.nvidia make the following changes:

  • Option      "AutoAddDevices" "true"
  • Option "UseEDID" "true"
  • #    Option "UseDisplayDevice" "none"
You can see my edited xorg.conf.nvidia file here:


4. Start bumblebee daemon: sudo systemctl start bumblebeed

5. Launch intel-virtual-output (the default is to launch as a daemon, but I will launch in foreground mode so I can easily kill the process with Ctrl-C when I am done using an external monitor). You can run this program as the local user (When executing as root, it doesn't work).

# intel-virtual-output -f

Now X windows should be able to detect an external monitor connected to the video ports wired to the Nvidia GPU. Running xfce4-display-settings shows the following:



The monitor named "VIRTUAL" is a path-through interface created by intel-virtual-output.
Arandr, the front-end to the Xorg screen layout/orientation tool xrandr, also detects multiple screens:


References:

https://wiki.archlinux.org/index.php/bumblebee#Output_wired_to_the_NVIDIA_chip
https://bugzilla.redhat.com/show_bug.cgi?id=1195962



2016년 11월 12일 토요일

Installing Nvidia driver for GeForce 310m on Fedora 24 with bumblebee-nvidia

My Asus U36JC notebook contains both Intel integrated graphics and an Nvidia GeForce 310M. Here are the results from lspci:

00:02.0 VGA compatible controller: Intel Corporation Core Processor Integrated Graphics Controller (rev 18)
...
01:00.0 VGA compatible controller: NVIDIA Corporation GT218M [GeForce 310M] (rev ff)

This notebook supports Nvidia's Optimus, which switches between integrated Intel video and the discrete Nvidia card depending on the computing task. For 3D-intensive workloads, the Nvidia card should be used. While this works OOTB on Windows 7+, on Linux you must do the switching manually. One project that makes this possible is Bumblebee. When you want to launch a certain program with the discrete Nvidia card, you simply prefix the program launch command with optirun.

I tried the open-source nouveau driver for my Nvidia GeForce 310M but I found the frame rate lacking when playing games so I switched to using Nvidia's proprietary driver. When you are searching for the Nvidia Linux driver for an older video card, DO NOT go to the main page for your video card and download drivers from there; if you do that, you will only be able to find old drivers for Linux kernel 2.4.X and 2.6.X, and these drivers are incompatible with Fedora 24 (which as of Nov. 13, 2016 is at 4.8.6.-201.fc24.x86_64)

Instead, to download the newest proprietary Nvidia Linux driver blobs, you must go to the following link and find your card:

http://www.nvidia.com/object/IO_32667.html

Then you can download the appropriate driver from the following link:

http://www.nvidia.com/object/unix.html

For my Nvidia GeForce 310M, the latest proprietary driver as of Nov. 13, 2016 is NVIDIA-Linux-x86_64-340.98 which can be downloaded from here:

http://www.nvidia.com/Download/driverResults.aspx/107868/en-us

Now it's time to install Bumblebee. I added the following repos for Bumblebee with Nvidia closed-source driver and unmanaged Nvidia Fedora repo:

dnf -y --nogpgcheck install http://install.linux.ncsu.edu/pub/yum/itecs/public/bumblebee/fedora24/noarch/bumblebee-release-1.2-1.noarch.rpm

dnf -y --nogpgcheck install http://install.linux.ncsu.edu/pub/yum/itecs/public/bumblebee-nonfree-unmanaged/fedora24/noarch/bumblebee-nonfree-unmanaged-release-1.2-1.noarch.rpm

I then installed the following:

dnf install bumblebee-nvidia bbswitch-dkms VirtualGL.x86_64 VirtualGL.i686 primus.x86_64 primus.i686 kernel-devel

Now copy the Nvidia binary blob which you downloaded earlier (it has the extension .run) into /etc/sysconfig/nvidia and then execute the following:

$ sudo bumblebee-nvidia --debug
[sudo] password for fedjun:
--debug mode selected.
Building NVIDIA video drivers: Creating directory NVIDIA-Linux-x86_64-340.98
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 340.98.................................................................................................................................................................................................................................
  [  OK  ]

You may be prompted to manually build bbswitch using DKMS during the installation. Note you will have to manually rebuild your Nvidia driver using the above command every time your kernel is upgraded. Once you build the driver, you may have to restart your machine to get bumblebeed to work.

Now make sure bumblebeed is running:

[fedjun@u36jcFedora nvidia]$ systemctl status bumblebeed
bumblebeed.service - Bumblebee C Daemon
   Loaded: loaded (/usr/lib/systemd/system/bumblebeed.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2016-11-12 09:21:47 KST; 24s ago
 Main PID: 20004 (bumblebeed)
    Tasks: 1 (limit: 512)
   CGroup: /system.slice/bumblebeed.service
           └─20004 /usr/sbin/bumblebeed

Nov 12 09:21:47 u36jcFedora systemd[1]: Started Bumblebee C Daemon.
Nov 12 09:21:47 u36jcFedora bumblebeed[20004]: [ 2077.225663] [INFO]/usr/sbin/bumblebeed 3.2.1 started

Finally, test that glxgears runs with your Nvidia card:

$ optirun glxgears

You should see an OpenGL window appear with 3 rotating gears. Press ESC to exit the window.

When I play Pillars of Eternity 3.03, I launch the game as follows:

$ optirun ./start.sh

I get much better frame rates when using the Nvidia discrete video card with this game.

References:

https://fedoraproject.org/wiki/Bumblebee

https://github.com/Bumblebee-Project/Bumblebee/issues/814

2016년 10월 29일 토요일

Slow transfer times when creating a local npm package mirror

In the current project I'm working on, the client wants developers to install packages from local mirrors of Ubuntu, PyPI, and npm. Creating local mirrors of Ubuntu default repositories is easy with apt-mirror, and PyPI can also be mirrored easily with Bandersnatch. Mirroring the npm repo, however, is not so easy. I followed guides which recommend creating a mirror with CouchDB, but I couldn't get this method to work. Instead, I set up Sinopia as an npm cache server so that every time I install an npm package locally it is also saved in Sinopia. The next time I try to install the same package, it will be installed from the Sinopia cache instead of from the npm site.

The problem with using Sinopia as a full npm mirror is that it will not automatically download packages. You must manually install npm packages with npm install pkgname. I got a list of all the packages in npm from http://skimdb.npmjs.com/registry/_all_docs and then parsed the file to get only package names. I then wrote the following script to download packages from npm (which will then be stored in Sinopia):


As of October 2016 when I mirrored PyPI using Bandersnatch, downloading 380GB took about 1.5 days on a 500 Mbit/s connection. Mirroring Ubuntu 14.04 and 16.04 repos requires 300GB each and takes almost one day for each. But using my bash script, I am only getting speeds of about 100MB per hour. Considering that npm is currently 1.2 TB in size, it would take me over 1200 days to create a complete npm mirror at the current download speed. Why is npm so slow compared to Ubuntu repos and PyPI?

2016년 10월 22일 토요일

Adding nodes to Openstack Mitaka with Packstack

In my initial Openstack Mitaka deployment on four Fujitsu servers, I used a customized answer file specifying Neutron OVS bridge interfaces and the slave interfaces for bridges. The relevant setting in the answer file is:

CONFIG_NEUTRON_OVS_BRIDGE_IFACES=br-ex:eno1,br-eno2:eno2

However, I want to add 2 additional compute nodes which are a different kind of server, namely IBM. The interfaces are named differently, so the slave interfaces for br-ex and br-eno2 will be different.

I copied my original answer file and renamed it mitaka-ibm.cfg

Inside this answer file, I simply edited the field

CONFIG_COMPUTE_HOSTS=10.10.10.6,10.10.10.7

and added the mgmt network IP's for the two new compute hosts and removed the IP's for the existing compute nodes.

I then edited

CONFIG_NEUTRON_OVS_BRIDGE_IFACES=br-ex:enp1s0,br-eno2:enp2s0

to change the slave interface names for the bridge interfaces on the 2 IBM servers.

Since I don't want Packstack to overwrite the config on existing Openstack nodes, I also added the mgmt network IP's as well as external network IP for Horizon to the following field:

EXCLUDE_SERVERS=10.10.10.2,10.10.10.3,10.10.10.4,10.10.10.5,192.168.4.51

However, the field

CONFIG_CONTROLLER_HOST=192.168.4.51

must be filled in with the external IP of the control node, otherwise the installation will fail when running the puppet file nova.pp

References:
https://www.rdoproject.org/install/adding-a-compute-node/

2016년 10월 15일 토요일

Boot Information Negotiation Layer (BINL) UDP 4011 must be opened for PXE

I wrote a script that opens ports in the firewalld dynamic firewall so I can run a PXE server, but I neglected to add one port -- UDP 4011 for the Boot Information Negotiation Layer. Once the kernel and initrd are sent by TFTP, I use http to send installation files from a mounted iso.

The ports I open are as follows:

UDP 69 (TFTP)
UDP 4011 (BINL)
UDP 67, 68 (DHCP)
TCP 5500 (VNC)

In the script I don't pass all port names explicitly; some can simply be passed as service names to firewall-cmd, which figures out which port numbers to open.



References:

http://www.configmgr.no/2012/03/21/ports-used-by-pxeosd/

2016년 10월 8일 토요일

Install Archlinux over UEFI PXE from an existing PXE server

When installing Linux over UEFI PXE, GRUB2 grub.cfg is used as the PXE image menu. Archlinux has some idiosyncratic PXE options which I will detail in this post.

As of Oct. 2016, the Archlinux wiki has a sample Legacy BIOS PXE menu entry but this format cannot be used verbatim for UEFI PXE.

First, download the latest Archlinux installation iso from the following link:

Mount the iso and note the following paths in the mounted image:

/arch/boot/x86_64
contains the linux kernel and initrd image for 64-bit
  • kernel: vmlinuz
  • initrd: archiso.img
/arch/boot
contains intel microcode and memtest
  • intel_ucode.img
  • memtest
In most other Linux distros, the initrd image on installation iso's is named initrd.img but Archlinux uses archiso.img

The Archlinux-specific Kernel boot parameters for PXE are as follows:
  • archisobaseddir=arch Specifies the root directory of the installation iso
  • archiso_http_srv=http://ip.ad.d.r:port/ Specifies installation file location over http (you can also use nbd and nfs instead of http)
  • ip=:::::eth0:dhcp Tells the Arch kernel to bring up the network iface (on the machine to be installed with Arch) and get an IP address via   DHCP. For predictability, the network iface in the Arch chroot environment is always named eth0
Keep in mind that eth0 is just a temporary name for your wired iface during installation. Once installation is complete and you exit the Arch install chroot and restart, your wired interface will come up with a systemd-style network device name.

My grub.cfg for UEFI PXE can be seen here:

The menu entry in grub.cfg for Archlinux UEFI should look like this if you are installing over http:

menuentry 'Archlinux iso 2016.09.03' --class arch --class gnu-linux --class gnu --class os {
        echo 'Loading kernel vmlinuz from installation ISO...'
        linuxefi images/archlinux/vmlinuz archisobaseddir=arch archiso_http_srv=http://192.168.95.97:8080/ ip=:::::eth0:dhcp
        echo 'Loading initial ramdisk ...'
        initrdefi images/archlinux/intel_ucode.img images/archlinux/archiso.img
}

For information on how to setup a PXE server that can install to both UEFI and Legacy BIOS machines, refer to my previous post on this topic:


If you don't have UEFI hardware lying around for testing, you can use KVM with OVMF Tianocore UEFI. I detail how to get started in the following post: