Nick Charlton

Prometheus Remote Write Examples

Wed, 22 Apr 2026 00:00:00 +0000

Last year, I worked on a project that sends telemetry data to Grafana Cloud. It was data from remote sensors (i.e.: internet of things) and my theory was that having the data end up in Prometheus would be ideal for data storage, software licensing and general support reasons as then it’d be possible to query the sensor data like any other application metrics.

In practice this worked out pretty well, and I’m really happy with how it worked out. But, with how prometheus_exporter ends up exposing the metrics, you end up with a situation where the metric data persists. Instead of the data being a point in time, the value stays constant between scrapes. Whilst this is inherent to the design, it means that unless the whole application is restarted (or goes down), you lose any gaps in the data which is itself a helpful piece of information that you can alert on. For a specific example: if temperature data is sent every 5 minutes at 21ºC, unless that value changes, the scraped data will always report 21ºC from that point onwards.

This can be solved by pushing rather than pulling metrics, but is something that Prometheus tries to steer you away from doing. There is a solution to this in the push gateway, but in this situation I couldn’t use it (without increasing the infrastructure complexity). Prometheus also supports remote write; intended for other scrapers to send data to another Prometheus.

There’s a specification for remote write is supposed to work, but if you look through the various client libraries, this isn’t implemented (usually deliberately to stop you using it) and I couldn’t find any helpful examples of how to use it. Whilst I never got time to implement it on the project it was intended for, I did end up writing some examples of implementing Remote Write, and I’ve collected these together in a repository in case it’s useful for others.

Using a Vodafone Mobile Broadband (ZTE K5161z) 4G USB dongle on Linux

Wed, 11 Mar 2026 00:00:00 +0000

There are plenty of these little devices around, and as of the start of 2026 they sell for £25 on Amazon unlocked for any network. I couldn’t figure out how to get the thing to actually work though on an otherwise failing boring Linux install.

It seems likely that the current glut of them is either because of customer returns from Vodafone, or just unused devices as 5G dongles are rolled out. But they work perfectly well still.

USB 4G devices like this are odd little beasts: this, and several other models known on the market, like some Huawei models, are effectively a stick shaped router which shows up as a combined mass storage device and a USB ethernet device. On insertion, dmesg tells us:

[Feb 5 14:53] usb 1-2: new high-speed USB device number 6 using xhci-hcd
[  +0.139388] usb 1-2: New USB device found, idVendor=19d2, idProduct=1225, bcdDevice=58.13
[  +0.000010] usb 1-2: New USB device strings: Mfr=2, Product=4, SerialNumber=5
[  +0.000003] usb 1-2: Product: Vodafone Mobile Broadband
[  +0.000003] usb 1-2: Manufacturer: Vodafone,Incorporated
[  +0.000003] usb 1-2: SerialNumber: 1234567890ABCDEF
[  +0.005580] usb-storage 1-2:1.0: USB Mass Storage device detected
[  +0.000188] usb-storage 1-2:1.0: Quirks match for vid 19d2 pid 1225: 1
[  +0.000042] scsi host0: usb-storage 1-2:1.0
[  +3.670038] usb 1-2: USB disconnect, device number 6
[  +0.380744] usb 1-2: new high-speed USB device number 7 using xhci-hcd
[  +0.143507] usb 1-2: New USB device found, idVendor=19d2, idProduct=1405, bcdDevice=58.13
[  +0.000007] usb 1-2: New USB device strings: Mfr=2, Product=4, SerialNumber=5
[  +0.000003] usb 1-2: Product: Vodafone Mobile Broadband
[  +0.000002] usb 1-2: Manufacturer: Vodafone,Incorporated
[  +0.000002] usb 1-2: SerialNumber: 1234567890ABCDEF
[  +0.201730] cdc_ether 1-2:1.0 eth1: register 'cdc_ether' at usb-xhci-hcd.0-2, ZTE CDC Ethernet Device, 34:4b:50:00:00:00
[  +0.000957] usb-storage 1-2:1.2: USB Mass Storage device detected
[  +0.000385] scsi host0: usb-storage 1-2:1.2
[  +1.010171] scsi 0:0:0:0: CD-ROM            Vodafon  USB SCSI CD-ROM  2.3  PQ: 0 ANSI: 2
[  +0.000567] sr 0:0:0:0: Power-on or device reset occurred
[  +0.002185] sr 0:0:0:0: [sr0] scsi-1 drive
[  +0.002111] sr 0:0:0:0: Attached scsi CD-ROM sr0
[  +0.000366] sr 0:0:0:0: Attached scsi generic sg0 type 5
[  +0.030690] scsi 0:0:0:1: Direct-Access     Vodafon  MMC Storage      2.3  PQ: 0 ANSI: 2
[  +0.003430] sd 0:0:0:1: Attached scsi generic sg1 type 0
[  +0.000233] sd 0:0:0:1: Power-on or device reset occurred
[  +0.000377] sd 0:0:0:1: [sda] Media removed, stopped polling
[  +0.000307] sd 0:0:0:1: [sda] Attached SCSI removable disk
[  +0.371026] ISO 9660 Extensions: Microsoft Joliet Level 1
[  +0.000250] ISOFS: changing to secondary root

The mass storage device provides a set of (unnecessary) drivers, and also supports the Micro SD card if you wanted to use it.

Assuming you’ve got a SIM card installed, it should just come up and provide internet access. The light starts off as red and once it gets a signal (this does take some time), will change to blue. If not, you might need to bring up the new interface, e.g.:

$ sudo ip link up eth1

The interface will then get an IP from the dongle. You can access the configuration interface by going to .1 on whatever the IP range you got assigned (maybe 192.168.0.*, maybe 192.168.6.*) and change some settings, read any SMS messages, etc. Once configured, I’ve found it comes back up when rebooting.

Vodafone USB LTE dongle interface

Hopefully this solves someone else spending ages going around in circles trying to figure out how you might use them, especially if like me, it didn’t come up automatically. Unfortunately it was much easier to find the wrong answer to how this particular device works than the correct one.

Working with Raspberry Pi images

Tue, 10 Mar 2026 00:00:00 +0000

I’ve found myself working with the Raspberry Pi frequently recently. It’s a nice platform, and I appreciate the software tooling for making it easy to get going and allowing you to focus on what you’re trying to make, rather than the embedded platform itself. But every guide points to using the Raspberry Pi Imager, which isn’t a satisfying answer for what the process actually is, which would help for figuring out how to automate the process.

`dd` is always our friend

Raspberry Pi OS, their Debian-based distribution, is available as a compressed image file, a .img.xz. Find the device (fdisk -l), then we can write the image:

$ unxz 2025-12-04-raspios-trixie-arm64-lite.img.xz
$ sudo dd if=2025-12-04-raspios-trixie-arm64-lite.img of=/dev/sda
5832704+0 records in
5832704+0 records out
2986344448 bytes (3.0 GB, 2.8 GiB) copied, 268.014 s, 11.1 MB/s

I’ve been skipping this step for some projects and using a full image by buying a pre-installed SD card, which is a helpful option.

Customisation

The disk image is made up of two partitions: a smaller FAT32 partition which is mounted at /boot and a larger Linux partition which holds the operating system image:

$ sudo fdisk -l
[ ... snip ... ]
Disk /dev/sda: 58.94 GiB, 63281561600 bytes, 123596800 sectors
Disk model: SD/MMC
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xffc763c1

Device     Boot   Start     End Sectors  Size Id Type
/dev/sda1         16384 1064959 1048576  512M  c W95 FAT32 (LBA)
/dev/sda2       1064960 5832703 4767744  2.3G 83 Linux

The /boot partition contains files like the config.txt, used to configure hardware options, plus the binaries and relevant device trees for booting the operating system. In addition, at the end of 2025, Raspberry Pi released cloud-init support. This allows us to drop a couple of YAML files onto the /boot partition of the image, which are read and processed on boot.

$ sudo mount /dev/sda1 /media/usb-drive
$ ls /media/usb-drive
bcm2710-rpi-2-b.dtb	  bcm2711-rpi-400.dtb	  bcm2712-rpi-500.dtb	      cmdline.txt   fixup.dat	     kernel8.img       start4.elf
bcm2710-rpi-3-b.dtb	  bcm2711-rpi-4-b.dtb	  bcm2712-rpi-5-b.dtb	      config.txt    fixup_db.dat     LICENCE.broadcom  start4x.elf
bcm2710-rpi-3-b-plus.dtb  bcm2711-rpi-cm4.dtb	  bcm2712-rpi-cm5-cm4io.dtb   fixup4cd.dat  fixup_x.dat      meta-data	       start_cd.elf
bcm2710-rpi-cm0.dtb	  bcm2711-rpi-cm4-io.dtb  bcm2712-rpi-cm5-cm5io.dtb   fixup4.dat    initramfs_2712   network-config    start_db.elf
bcm2710-rpi-cm3.dtb	  bcm2711-rpi-cm4s.dtb	  bcm2712-rpi-cm5l-cm4io.dtb  fixup4db.dat  initramfs8	     overlays	       start.elf
bcm2710-rpi-zero-2.dtb	  bcm2712d0-rpi-5-b.dtb   bcm2712-rpi-cm5l-cm5io.dtb  fixup4x.dat   issue.txt	     start4cd.elf      start_x.elf
bcm2710-rpi-zero-2-w.dtb  bcm2712-d-rpi-5-b.dtb   bootcode.bin		      fixup_cd.dat  kernel_2712.img  start4db.elf      user-data

The Pi can be configured like any other using cloud-init, but it’s also got a Raspberry Pi specific module which is handy for configuring hardware, including support for the fairly new USB gadget mode. Using cloud-init makes for a nice path for future automation.

I’m bootstrapping Pis with something that looks like this as the baseline for user-data (I’m leaving meta-data and network-config the same as the default behaviour is fine):

#cloud-config

hostname: test-pi
manage_etc_hosts: true

timezone: Europe/London

users:
  - name: pi
    groups: users,adm,dialout,audio,netdev,video,plugdev,cdrom,games,input,gpio,spi,i2c,render,sudo
    shell: /bin/bash
    lock_passwd: false
    passwd: $6$rounds=4096$MwcQMjo1p9YjlEBx$NkyMyUaGVa8hdD33aO/9XS/NoHdJ1a/ekfXOaF2QbYAWyt8eC7weNTAu9N2aS1Uk4zj1BNnD/tRr19nNoe/190
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFypAVlkyYVwdFAUrDnAcZqIM6Lkusv+9J3rRUn7qZzA
    sudo: ALL=(ALL) NOPASSWD:ALL

rpi:
  enable_usb_gadget: true

enable_ssh: true

The password used above is password, but generated using mkpasswd --method=SHA-512 --rounds=4096, as recommended by cloud-init. If you set a password, you also need to set lock_passwd: false, otherwise you can never login. enable_ssh: true is a Raspberry Pi-specific option, as otherwise SSH isn’t up when it first boots.

Combining secret values in files on Kubernetes

Mon, 09 Mar 2026 00:00:00 +0000

There’s a few applications which make handling their associated secrets and configuration particularly tricky to do on Kubernetes, because the main configuration file also has many secrets in it. We don’t want to end up in a position where the secrets are, or could easily be, left in a code repository, but we also don’t necessarily want to put all of our configuration into a secret that makes it hard to track changes to the configuration over time.

If we store our configuration in a ConfigMap, and our secrets in a Secret, we can have a file out of the configuration, and the secrets as environment variables. Unix has a solution to combining the two: envsubst.

I hadn’t seen anyone handle secrets like that on Kubernetes, so I thought I’d give it a go. It worked pretty well, but it is a little cursed.

The ConfigMap is fairly conventional:

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: config
  namespace: combined-secrets
  labels:
    app.kubernetes.io/name: config
data:
  config.yaml.template: |
    ---
    plain_value: Hello world!
    secret_value: $TOP_SECRET_VALUE

For secret_value, we have a placeholder which will be replaced by envsubst. Then we need a Secret, which here is directly with a file as an example:

---
apiVersion: v1
kind: Secret
metadata:
  name: secrets
  namespace: combined-secrets
  labels:
    app.kubernetes.io/name: config
type: Opaque
stringData:
  TOP_SECRET_VALUE: I am very secretive.

The values are in all caps, because all of the values will be directly mounted as environment variables later.

Then, our example Deployment:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment
  namespace: combined-secrets
  labels:
    app.kubernetes.io/name: deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: deployment
  template:
    metadata:
      labels:
        app.kubernetes.io/name: deployment
    spec:
      initContainers:
        - name: build-config
          command: ["/bin/sh"]
          args:
            - "-c"
            - >
              apt-get update -q; apt-get install -yq gettext-base;
              envsubst < /config.yaml.template > /data/config.yaml
          image: debian
          envFrom:
            - secretRef:
                name: secrets
          volumeMounts:
            - name: shared-files
              mountPath: /data
            - name: config-template
              mountPath: /config.yaml.template
              subPath: config.yaml.template

      containers:
        - image: debian
          name: debian
          command:
            - tail
            - "-f"
            - "/dev/null"
          volumeMounts:
            - name: shared-files
              mountPath: /data

      volumes:
        - name: shared-files
          emptyDir: {}
        - name: config-template
          configMap:
            name: config

This uses a couple of tricks to pull this off:

We use an initContainer to prepare our configuration before starting the main container,
To store the result, we use a volume that’s shared among the Pod as /data,
The Debian container image unfortunately doesn’t include envsubst, so we need to install that ourselves,
We also need to run the commands as a shell for the redirection to work,
To truly cement it’s cursed nature: we use > which doesn’t retain line breaks for what’s actually a one-liner,
Finally, for the main container we tail /dev/null which is just a trick to stop the container exiting whilst we use it.

If we connect to the deployment container, we can see the resulting combined file:

$ kubectl -n combined-secrets exec -it deployment-568b66968-z9t77 -- /bin/bash
Defaulted container "debian" out of: debian, build-config (init)
root@deployment-568b66968-z9t77:/# cat /data/config.yaml
---
plain_value: Hello world!
secret_value: I am very secretive.

It’s unfortunate that we need to install gettext for envsubst (it’s a good opportunity for a custom image), but this works well.

Static files with Nginx, Docker & Kubernetes

Sun, 18 May 2025 00:00:00 +0000

I wanted a way to serve up a set of .well-known paths, as easily as I could get away with, but keep it all alongside some other Kubernetes manifests.

I hoped it’d be something I could trick the Nginx Ingress controller into doing (as that already exists), perhaps with just a ConfigMap but it’s not really designed to do that. I didn’t think it should be necessary to build a custom container image either. But we could do it with a Deployment and an Nginx container. There’s a bit to it, but perhaps it’s helpful for others.

I’m going to use Matrix’s .well-known paths here, but the same principle could apply to anything static.

A quick experiment with Docker

To start with, we need something to serve:

# .well-known/matrix/server
{
  "m.server": "matrix.nickcharlton.net:443"
}

We can then serve it by mounting the current directory in the right place, using the Nginx Docker image:

docker run --rm -p 8080:80 -v .:/usr/share/nginx/html:ro nginx

And test it with curl:

$ curl http://localhost:8080/.well-known/matrix/server
{
  "m.server": "matrix.nickcharlton.net:443"
}

That works quite nicely, which is just enough experimenting to turn it into Kubernetes manifests.

Running on Kubernetes

To start with, a ConfigMap to hold the data:

# configmap-well-known.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: well-known
  namespace: default
  labels:
    app.kubernetes.io/name: well-known
data:
  server: |
    {
      "m.server": "matrix.nickcharlton.net:443"
    }

Then a Deployment to run Nginx:

# deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: well-known
  namespace: default
  labels:
    app.kubernetes.io/name: well-known
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: well-known
  template:
    metadata:
      labels:
        app.kubernetes.io/name: well-known
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html/.well-known/matrix/server
              subPath: server
      volumes:
        - name: data
          configMap:
            name: well-known

Finally, a Service and Ingress:

# service.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: well-known
  namespace: default
  labels:
    app.kubernetes.io/name: well-known
spec:
  # type: LoadBalancer
  clusterIP: None
  ports:
    - name: http
      port: 80
      targetPort: http
  selector:
    app.kubernetes.io/name: well-known

# ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: well-known
spec:
  ingressClassName: nginx
  rules:
    - host: nickcharlton.net
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: well-known
                port:
                  name: http

I use kube-vip on my clusters, so I could use a LoadBalancer service here to get an IP outside of the cluster and use that (a pattern you can use elsewhere too). But, it’s also possible to use a headless service with an Ingress, which is what’s done here.

We can test via the Ingress directly by forwarding a port:

$ kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80

$ curl --resolve nickcharlton.net:8080:127.0.0.1 http://nickcharlton.net:8080/.well-known/matrix/server
{
  "m.server": "matrix.nickcharlton.net:443"
}

Or, if we can access outside the cluster (assuming the domain resolves):

$ curl http://nickcharlton.net:8080/.well-known/matrix/server
{
  "m.server": "matrix.nickcharlton.net:443"
}

Two sides and four columns of A5: or a technique to help with too many projects

Fri, 10 Jan 2025 00:00:00 +0000

2024 sucked for various reasons, but when I’m not happy (over a long period of time especially), I find I start far more projects than I get close to finishing. I end up having far too many things in progress, where I’m (consciously, or unconsciously) trying to work on multiple things at the same time, continually thinking about what I’m not doing, being stuck on one project so others can’t proceed, or missing out on new and other interesting things because I’m too busy and stressed out with the amount of projects going on.

Towards the end of last year, I resolved to try and get this under control by trying out a new idea and review process: writing down all of the projects and their current state, then reviewing this list every two weeks. I’d ask myself these questions:

What projects are in progress?
What not-quite-a-project stuff is happening?
What should I kill?
Is this the right frequency to think about it?

Then I’d write down all of the projects in a fancy notepad my partner bought back for me from a trip to Japan which has two columns on it. This notebook sits to the left of my keyboard, so it’s always in sight.

The notebook sits to the left of my keyboard, crossing them out whilst they're done.

The first time I sat down and wrote down the projects, I ended up with two bits of paper and four whole columns on them of projects I’d started but not completed. Just doing this the first time really emphasised the state of things. As the time went on, I’d cross off each item (or a sub-item, until I could cross off the whole item).

Two weeks later, I returned to do it all again. Starting by looking at the state of the current list, I’d take a new sheet and write out the projects again. The crossed out ones would be gone, but any new projects would be added.

In the first review, I was surprised by how much I’d managed to get through. I’d spent the time getting rid of as many projects as possible. Not that excited about it? Archive it, and move on. Last thing to get it over the finish line? Do that first. In these first two weeks, I’d managed to get down to just two columns, on one sheet.

I continued this for several months (never getting it any smaller than two columns), gradually crossing items out, bit by bit. Adding new projects as they happened (but while being very selective about those, to avoid making the problem even worse and ending up back where I started). But, finally, as I publish this, I’m down to just one column. Hopefully I can keep it that way.

Maybe something similar can work for you.

Configuring a serial console on Debian 12

Thu, 09 Jan 2025 00:00:00 +0000

I’m a fan of serial consoles for out-of-band access, since having used them a lot for network equipment. Recently, I was trying to figure out a graphics incompatibility on a system using Wayland on Debian and wanted to set up a serial console to try to stop part of the debugging process being so painful.

Since systemd has rolled out to Debian though, there are a lot of outdated articles, and now it’s quite a bit easier to get going. According to the Debian Wiki, we can now start a service to do this on demand:

systemctl enable serial-getty@ttyS0.service
systemctl start serial-getty@ttyS0.service

But if we configure the bootloader (grub) instead, we can get serial console output from much earlier in the boot process.

Determining the correct serial device

Most of the time, you can likely assume it’s going to be ttyS0, but it’s worth checking:

$ sudo dmesg | grep tty
[    0.066243] printk: console [tty0] enabled
[    0.685110] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    0.686459] 00:02: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[   18.042511] systemd[1]: Created slice system-getty.slice - Slice /system/getty.

This motherboard has a built-in serial port, plus another is configured using a header cable.

Configure Grub

We open up /etc/default/grub, and modify GRUB_CMDLINE_LINUX to set a console on the serial port, followed by updating the live Grub configuration:

$ sudo vim /etc/default/grub

-GRUB_CMDLINE_LINUX=""
+GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"

$ sudo update-grub

By default, this will be 9600 baud. But we can configure that, for example, setting 115200:

GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8"

A Debian login screen in gtkterm

Fun aside: Firmware Console Redirection

Whilst I was configuring this board (a Supermicro X11SSH-LN4F), I noticed that it supports console redirect inside the firmware (BIOS). If we configure the same settings from the firmware through to grub, we can keep the same console session through the whole boot process. This also works with FreeDOS, which is handy if you’re doing a firmware upgrade.

We can configure the BIOS even inside a serial console

As long as you know the correct command to get into the firmware (delete, in this case) when the screen goes blank you can just keep hitting the combination to get into setup.

Of note, I found that less than 115200 baud made the firmware incredibly slow to redraw. It also doesn’t draw elements under the cursor, so you might need to move around a bunch to see what’s being selected.

We can configure COM1 (which matches to `ttyS0`) for console redirection.

We can also configure the speed, in this case, 115200, to match Debian once it's booted.

And get this fun ASCII art too

Installing Debian 12 on a Dell Wyse 3040 Thin Client

Fri, 03 Jan 2025 00:00:00 +0000

The Dell Wyse 3040 is a little thin client device that’s surprisingly powerful and also very low power. I picked one up last year, new in box, for about £30 for a project. Unfortunately, getting Debian running on it is not so straightforward because it has a buggy EFI implementation which means that whilst you can complete the install, you can’t boot it afterwards.

The Wyse 3040 only uses UEFI and doesn’t support a legacy BIOS mode. Usually, a boot variable is defined which points to the bootloader for each vendor’s operating system. In Debian’s case, this should be \EFI\debian\grubx64.efi. But, the Wyse 3040 firmware is missing this. Instead, we need to force the bootloader to exist at another known path — originally intended for removable media: \EFI\boot\bootx64.efi.

Fortunately, there’s an option in the Debian installer to do this which is much more robust than doing it ourselves (which the Debian wiki also describes) as it could be broken with updates.

I’ll be installing Debian on an otherwise empty USB flash drive to maintain the original install, and booting from another USB drive with the latest Debian netinst on. You can disable booting from the internal drive, if you want. Nothing else in the firmware needs changing from the defaults.

A screenshot of the EFI firmware, showing the internal drive disabled.

Some things which are helpful:

F2 for setup
F12 for the boot menu
The default password is “Fireport”

Installing Debian

Boot from the USB drive, once you’re at the Debian installer, go into Advanced options and select Expert install.

Debian installer ready to start an "Expert install"

You can then proceed through each section of the install. In the Expert install mode, you’re asked many more questions, but it’s broadly the same as the normal installer. You just have to select each stage of the installer yourself. I didn’t need to select any additional locales or components for my install and preselected values (for the NIC, kernel, drivers, etc) were all fine.

Eventually, you’ll proceed through the setup until you’re ready to select Install the GRUB boot loader:

Debian installer showing the next installation step

Then look out for Force extra installation to the EFI removable media path?. We’ll select Yes.

During the Debian install, we select "yes" to work around the buggy EFI implementation

I also selected No to Update NVRAM variables to automatically boot into Debian? to preserve the existing boot behaviour.

During the Debian install, we select "no" to not have Debian automatically boot.

Once this stage of the installation is finished, you’ll be prompted to Finish the installation and then to reboot.

Debian should then successfully boot.

Finally, we can boot into a login prompt.

You will see errors about an invalid parameter on boot, and also some Intel firmware warnings. For the former, Grub launches soon after and for the second this seems to be harmless (so far).

On the path to Administrate v1

Wed, 27 Nov 2024 00:00:00 +0000

Nearly 8 years after cutting my first release of Administrate, we’re nearly at the point of releasing v1, which has been my big open source focus all year!

We’re close to releasing Administrate v1. We’d love it if you could try it out!

See on Giant Robots

Automating barcode scanner tests with Capybara

Tue, 26 Nov 2024 00:00:00 +0000

Well over a year ago, we worked on a project that used barcode scanners. Knowing that they’re just human interface devices (keyboards) in most cases (and/or can be configured as such), it makes them quite easy to work with in a typical browser-based testing framework. But putting these two things together wasn’t very clear to most of my colleagues (and probably not to most people either!).

My colleague Silumesii took the time to write it up (we started working on this together, so I’ve got an author credit, but it’s really all of his work getting published!):

Implementing barcode scanner tests using Capybara.

See it on Giant Robots

diff-check is on GitHub Marketplace

Fri, 27 Sep 2024 00:00:00 +0000

Today, I cut the first release (v1.0.0) and published diff-check to GitHub Marketplace.

diff-check came out of seeing a common pattern of seeing side effects when another automated process would run, creating more work which wouldn’t be seen at the time. It was really common when I was working on React Native projects (if you upgrade an npm dependency, you’ll often get a change in the Podfile.lock that Dependabot wouldn’t know about), but less frequently in projects using Appraisal (another project I maintain). I previously wrote up how it works in the announcement blog post.

With some help from Oscar Gustafsson (thank you!), he helped see some bits I’d got wrong and also pointed out some small ways in which we could improve the output too. Several weeks ago, I’d also merged in a PR to use this on Administrate and so now I’m pretty happy it’s working.

I’d love to hear if it works out for you!

Generating Rails projects from Git patches

Wed, 11 Sep 2024 00:00:00 +0000

A couple of weeks ago, Joel asked:

Starting a new rails app is fun!

But after setting my bespoke set of preferred gems and config for the N-hundred’th time I think it may be time for some sort of template/generator/whatever.

Any tips?

So, I mentioned the approach I’ve used several times recently, using git’s format-patch feature, but it really needs walking through to understand how this is done.

Patches are how you sent commits over email, something still done on a lot of projects but gradually becoming more esoteric. But they’re very helpful. It’s a plain-text file with the commit information (author, time, message) and the change set from that commit. The key commands are format-patch to create the patches, and am which lets you apply them.

To do this, we start with an existing, sort of “template” project that’s setup how you’d like. But typically, I’ll fetch from the last one I took this approach with. The key bit is having well isolated and phrased commits, like this:

$ git show 020090b
commit 020090b47d75d3e77c9ee55518e5fcd9074d5c53
Author: Nick Charlton <nick@nickcharlton.net>
Date:   Mon Feb 26 14:31:59 2024 +0000

    Setup a Rails app (#1)

    rails new TemplateProject --database postgresql --skip-keeps \
          --skip-action-mailbox --skip-action-text --skip-active-storage \
          --skip-action-cable --skip-hotwire --skip-jbuilder

    Plus: RSpec, FactoryBot, lograge, rack-timeout, flashes and i18n test
    configuration.

Here’s the first few items of git history for a project I put together at the start the year to experiment with an idea, which served as the “template project”:

$ git log --oneline
1f26eed (HEAD -> main, origin/main, origin/HEAD) Enable UUID primary keys (#6)
4d8dd1c Accept connections from an external tunnel (#5)
96f4241 Setup Ruby linting with standardrb (#4)
a3d0e2a Setup Bundle Audit (#3)
5bce041 Setup GitHub Actions for running tests (#2)
020090b Setup a Rails app (#1)
dc277a8 Initial commit; add README

These commits are were themselves from a another project I’d thrown together to experiment with something completely different, and the one before that was when I sat down and figured out the bits I really cared about in Rails projects. But they’re all from around the same era — it’s all Rails 7.1.

We can generate patches up to the “initial commit” — the first commit in a Git repository is special, so we want to avoid that — and then apply them to another project:

$ git format-patch dc277a8
0001-Setup-a-Rails-app-1.patch
0002-Setup-GitHub-Actions-for-running-tests-2.patch
0003-Setup-Bundle-Audit-3.patch
0004-Setup-Ruby-linting-with-standardrb-4.patch
0005-Accept-connections-from-an-external-tunnel-5.patch
0006-Enable-UUID-primary-keys-6.patch

I’ll create a new branch, then apply the patch: git am 0001-Setup-a-Rails-app-1.patch. As long as you apply them in the order they were written, they’ll apply cleanly. If not, you can always modify the diff inside the patch. Then, I’ll go ahead and modify that commit until it’s in the state I want it to be:

Change any name references to the new project,
Run bin/setup and check it works,
Run bin/dev and check it works,
Update any gems that need updating, to save needing to do it immediately after,
Finally, before merging this branch, I’ll update the author/commit dates to avoid confusing myself in future (e.g.: git commit --amend --date=now)

In this example, the other patches generated are around linting, GitHub Actions for CI, Bundle Audit to catch vulnerable dependencies and some others like UUID primary keys. In the project I’m creating whilst writing this, I don’t care about UUID so I’ll just skip that but I’ll want the rest.

I’ll caveat this approach, however, with a you probably shouldn’t do it this way. From maintaining Administrate and having many ideas I’d like to try out, I create a lot of throw-away Rails projects.

For many years, I’d used Suspenders but it went through a period where using it was pretty unreliable as we were struggling to keep up with Rails changes. Now we’re at the end of 2024, we’ve fixed that, Suspenders is once again in a good place and there’s been a big boon in Rails Templates too. But now you know you can do it completely differently, if you fancied it.

Running PowerShell scripts locally with Packer

Thu, 15 Aug 2024 00:00:00 +0000

I maintain a set of virtual machine templates that use Packer. For VMware VMs, the workflow is to build them locally using the vmware-iso builder, then push to vSphere using the vsphere post-processor. The idea behind this approach is to be able to use the same templates for local VMs (which are exported as an ova archive) as those that end up on vSphere. Unfortunately, there’s some differences between VMware Workstation and vSphere that create some tricky problems like mismatched operating system versions.

Fortunately, PowerCLI exists which makes it fairly pleasant to work with vSphere from PowerShell. Packer can run local scripts, so I hatched the plan to run a PowerShell script (using the shell-local provisioner) that would adjust settings after the build had run. Alas, this is one of those problems that took many months of evenings to figure out how to do.

I’m using Packer v1.11.2 and PowerShell 7.4.4, and running everything on Linux. Here’s an example that works, and an explanation of how it got here below:

# hello-world.ps1
Write-Output "Hello world!"

Write-Output $env:MY_VAR
Write-Output $env:WITH_SPACES
Write-Output $env:PACKER_BUILDER_TYPE
Write-Output $env:PACKER_BUILD_NAME

Write-Output "and done!"

# example.pkr.hcl
source "null" "example" {
  communicator = "none"
}

build {
  source "source.null.example" {}

  provisioner "shell-local" {
    env                = {
      "MY_VAR": "hi",
      "WITH_SPACES": "and again"
    }

    execute_command    = ["pwsh", "-Command", "& {   }"]
    env_var_format     = "$env:%s=\"%s\"; "
    script             = "hello-world.ps1"
  }
}

$ packer build example.pkr.hcl
null.example: output will be in this color.

==> null.example: Running local shell script: hello-world.ps1
    null.example: Hello world!
    null.example: hi
    null.example: hello world
    null.example: null
    null.example: example
    null.example: and done!
Build 'null.example' finished after 450 milliseconds 889 microseconds.

==> Wait completed after 450 milliseconds 927 microseconds

==> Builds finished. The artifacts of successful builds are:
--> null.example: Did not export anything. This is the null builder

This example uses the “null” builder, as we just want to run a local file, it’s really helpful for testing,
Using “shell-local” we can execute a file on the local filesystem, which would usually default to a shell script (on a Unix),
But, this can be any command, and there’s a few options we can use to adjust how the command is put together,
Packer exposes the builder type and build name as environment variables or anything set in environment_vars or env, which will be key to providing things like secrets to the script later on,
Unfortunately, PowerShell doesn’t accept environment variables as arguments, and seems to fail when assigned ahead of the command (e.g.: ` bash ‘’` would usually work in an existing shell session),
Instead we can use a PowerShell block ({ }), and use & in front to execute immediately,
There’s two key things this is doing to make this work reliably, first, we provide a different template to env_var_format so that the output is how PowerShell expects it’s environment variables (and how it ends up in the block), secondly, we use env rather than environment_vars and provide a map of values as this means that the output is escaped correctly

thoughtbot Open Source Maintainers Sync notes is public

Tue, 02 Jul 2024 00:00:00 +0000

We’ve been working recently on coming together as maintainers at work, trying to share what we’ve been getting up to and seeing how we can help each other out.

Then we thought about sharing our notes in public, and now they’re available for everyone.

Looking for an opportunity to contribute to Open Source? Look no more. We’ve made our monthly internal Open Source Maintainers Sync notes public to facilitate getting more people into contributing to Open Source.

See it on Giant Robots

Installing Opnsense with virt-install on KVM

Fri, 07 Jun 2024 00:00:00 +0000

I was trying to setup a quick (…foreshadowing) VM to run a VPN on a host I use for a few VMs with KVM. But getting it setup was surprisingly painful until I tried the “nano” install image.

As the host is headless, I wanted a disk image I could import on the host with Opnsense already installed to avoid trying to redirect VNC over the network, or something similarly daft. A serial terminal was ideal (and how virt-install and the rest of libvirt works nicest). Alas, I couldn’t get a serial terminal configured correctly with the dvd image (it’d output boot logs, but when I got to the stage of interacting with the console it would stop outputting anything), the docs only mention using the Web UI which I couldn’t do at that stage of the install, and then the USB serial image just wouldn’t boot.

But the nano image provided the eventual solution: it’s designed to be written to a disk and then just booted where it expands to fill the new volume and serial is already configured. Perfect, but it wasn’t clear how to do it. So:

Fetch and extract the image from the download page:

wget https://mirror.ams1.nl.leaseweb.net/opnsense/releases/24.1/OPNsense-24.1-nano-amd64.img.bz2
bunzip2 OPNsense-24.1-nano-amd64.img.bz2

Convert to a qcow2 disk image: qemu-img convert -f raw -O qcow2 OPNsense-24.1-nano-amd64.img OPNsense-24.1-nano-amd64.qcow2
Resize to give us more space for logs, etc: qemu-img resize OPNsense-24.1-nano-amd64.qcow2 +8G
Rename/put the disk somewhere helpful: cp OPNsense-24.1-nano-amd64.qcow2 opnsense.qcow2
Use virt-install to import into a new VM:

virt-install --connect qemu:///system \
  --name opnsense \
  --os-variant freebsd12.2 \
  --memory 4096 \
  --disk opnsense.qcow2 \
  --network default \
  --graphics none \
  --console pty,target_type=serial \
  --import

Hopefully this saves someone some future bother!

diff-check: A GitHub Action which fails if anything changed

Sat, 17 Feb 2024 00:00:00 +0000

Last year, I started to see a few common workflows that would create more work by those unaware or open yourself up to making annoying mistakes if you did, that I thought could be caught with just a little more automation.

This was typically when Dependabot would go and bump a dependency that would have other implications: for example, if you bump a npm dependency in a React Native project, it might change the Podfile.lock in the iOS portion of the codebase, or if you bump a dependency used by Appraisal, one of it’s generated files could change. It gets pretty annoying for everyone else if you merge in that seemingly okay Dependabot PR and then someone (even if it’s yourself) has to go and add another commit to add in the forgotten lock file change.

To try and tackle this, I’ve put together a GitHub Action called “diff-check”. It runs a command and fails if anything subsequently changes. For example, you might create a workflow that looks like this:

---
name: diff-check
on: [push]

jobs:
  demo:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: nickcharlton/diff-check@main
        with:
          command: echo "hello world" >> README.md

It would then fail because README.md changed and report on the summary page:

The demo running and showing the job summary.

At its heart, it’s a fairly short shell script, that’s called by the action:

#!/bin/sh

set -e

if ! git diff-index --quiet HEAD; then
	printf 'These files changed when running the command:\n\n'

	git diff --name-only | while read -r n ; do
		echo "* $n"
	done

	exit 1
fi

But there’s enough logic in here that I really wanted some tests to make sure it was correct. I turned to Oli’s jet_black, which I’ve been meaning to try out for years and let me write some tests using RSpec. I’m pretty happy with how that turned out.

Trying to solve problems like these in such an abstract way is far harder than first solving a specific problem. I’d originally created the repo back in July of last year, and I really wanted the implementation to be able to suggest what needed to be done through a review suggestion, but this isn’t possible despite a long-standing feature request for it. In the end, this was the common thread through the situations I kept seeing. Hopefully it might help out some others too.

Fixing unsupported SFP+ modules/"no carrier" errors with Intel X520 cards on Debian & FreeBSD

Mon, 15 Jan 2024 00:00:00 +0000

I’ve been reconfiguring a router I built at the end of last year to be a transparent firewall (or filtering bridge) using Opnsense. On the router, I’ve got a Dell-branded Intel X520-DA2 NIC, along with some Flypro 10GBase-T SFP+ modules which I picked up on offer last year. But … they don’t work.

Intel — like a lot of other vendors — restricts the SFP+ modules to those which have been approved, regardless of whether they may pose a problem and so you can’t just use any SFP+ module which isn’t programmed for that vendor’s hardware. I’d not actually considered that this was going to be a problem (even knowing about the vendor restriction with some cards), so it derailed the project whilst I tried to debug what was going wrong, as at least in Opnsense this wasn’t that clear.

You might see disconnected interfaces which list “no carrier” even when modules are inserted, or just no interfaces at all (if you booted with them in, the driver won’t use them). The screenshot below shows when the interfaces do show up, but this required booting up with no SFP+ modules, then inserting them afterwards.

Screenshot from Opnsense's interface section, showing no carrier (even though there is).

After reconfiguring a few times, I noticed the interface was always disconnected, regardless of what was physically plugged in or what I’d set up. I’m not so familiar with Opnsense/FreeBSD, so booted up a Debian live image to try out something I was familiar with. I’d originally thought that maybe it was a platform driver issue, but booting Debian left me with the following error:

ixgbe 0000:04:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
ixgbe 0000:04:00.0: Reload the driver after installing a supported module.

(You can find it later in dmesg.)

Some third-party vendors supply tools to re-programme the boards, but these Flypro modules don’t support that (nor do I have one), but there are options.

Debian workaround

On Debian, we can disable the check in the driver module:

root@debian:~# rmmod ixgbe
root@debian:~# modprobe ixgbe allow_unsupported_sfp=1

Then bring the interfaces up (with a module in/remove and reinsert if already there):

ip link set dev ens2f0 up

If we wanted to persist, it we could do:

root@debian:~# echo "options ixgbe allow_unsupported_sfp=1" > /etc/modprobe.d/ixgbe.conf
root@debian:~# depmod -a
root@debian:~# update-initramfs -u -k `uname -r` # if you're not running a live system

We can then reload the module to bring up the new configuration:

root@debian:~# rmmod ixgbe
root@debian:~# modprobe ixgbe

FreeBSD workaround

On FreeBSD (and so Opnsense, pfSense, etc), we see a similar log message and can work around it similarly. On boot, we’ll see (and also in /var/log/dmesg.today):

ix2: Unsupported SFP+ module detected!

We can then set a tunable to allow unsupported modules:

echo "hw.ix.unsupported_sfp=1" > /boot/loader.conf.d/ix.conf

After a reboot, the interface should be usable. I did this on Opnsense and it persists between reboots, and also when resetting to factory defaults.

Disabling the check on the card

These workarounds were really helpful to isolate my problem to the restriction and not any other hardware or software configuration, but it’d be much better to just not have to think about this problem again, especially as this workaround is reportedly not possible with something like Windows or ESXi, both of which I use elsewhere.

Over on the ServeTheHome forums, NathanA figured out exactly how to do this after digging into various data sheets and support threads. Raymond Douglas then distilled this down into the steps required. It relies on using ethtool (I used a “standard” (without desktop environment) live CD image) to compare a bit (IXGBE_DEVICE_CAPS_ALLOW_ANY_SFP, in the Linux ixgbe driver) on the device’s EEPROM before writing a new version.

1. Identify the current value

root@debian:~# ethtool -e enp4s0f0 offset 0x58 length 1
Offset             Values
------             -------
0x0058:            fc

2. Write the new value

root@debian:~# ethtool -E enp4s0f0 magic 0x10fb8086 offset 0x58 value 0xfd

3. Verify by reading back

root@debian:~# ethtool -e enp4s0f0 offset 0x58 length 1
Offset             Values
------             -------
0x0058:            fd

After a reboot, the Debian live environment then used the new module without any further configuration.

I’d really recommend reading the original ServeTheHome post, as it explains in much more depth what’s going on, why this works and why you need to be careful not to modify or set incorrect values, as you risk bricking the card.

Always showing all GitHub Checks with a user style sheet

Thu, 11 Jan 2024 00:00:00 +0000

If you’ve worked with pull requests on GitHub for a while — especially with Actions — you’re probably used to having a long list of checks, that you find yourself scrolling every time to find the failing one:

The usual view of GitHub checks when you've got a lot, and one failing

Last year, Ben and I started working on a user style sheet to stop you needing to scroll and instead show the whole section:

After the user style sheet, with the scrolling area expanded to fit the amount of checks.

I use UserScripts in Safari, but I’ve heard good things of Stylus too. We came up with this:

/* ==UserStyle==
@name        Expand GitHub PR Statuses
@description Always fully expand the statuses on the GitHub PR page
@match       https://github.com/*
==/UserStyle== */

/* don't restrict the height of the open state */
.branch-action-item.open > .merge-status-list-wrapper > .merge-status-list, .branch-action-item.open > .merge-status-list {
  max-height: none;
}

/* always expand to fill the content of the above */
.merge-status-list { max-height: none; }

/* collapse the hidden state */
.merge-status-list.hide-closed-list { max-height: 0; }

I’ve used this since November and it works great!

Resolving ESXi 7.0 NIC connection issues on Supermicro X10SDV-4C-TLN2F motherboards

Fri, 17 Mar 2023 00:00:00 +0000

I have a few Supermicro X10SDV-4C-TLN2F motherboards that I’ve been setting up to run VMware ESXi on. The first board was fine, but when setting up another I could never get ESXi to acquire an IP over DHCP. The hardware was happy the connection was valid, and both Debian 11 and Windows Server 2019 were fine …but not ESXi:

A screenshot showing ESXi with 'waiting for DHCP'

A screenshot showing the adaptors as disconnected.

I’d been using a slightly older version of ESXi (7.0U3D/19482537), so I tried a newer version (7.0U3g/20328353) that I’ve upgraded to on one of the other boards, but this didn’t help. Both of my boards were running the same BIOS version (2.1) and build date (2019-11-22), so that seemed an unlikely cause.

After more head-scratching, I came across Workaround and fix for intermittent Intel X552/X557 10GbE/1GbE network link-down outages on Xeon D-1500 Series. I didn’t have the exact boards they were talking about in the start of the post nor the intermittent issues mentioned, but the one I have does have an Intel X552 so I thought this was promising. After going through the whole post (and most of the more recent comments), it seemed like it might be a NIC firmware difference between the two of my boards. I’m using a basic gigabit switch too, and not a 10GbE one.

Investigating the current NIC & firmware

Using the console, we can find out more about the hardware. When connected directly (or using IPMI), Alt+F1 will switch to the console.

$ esxcli network nic list

Name    PCI Device    Driver   Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------  ------------  -------  ------------  -----------  -----  ------  -----------------  ----  -----------
vmnic0  0000:03:00.0  ixgben   Up            Down             0  Half    ac:1f:6b:12:f8:6e  1500  Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
vmnic1  0000:03:00.1  ixgben   Up            Down             0  Half    ac:1f:6b:12:f8:6f  1500  Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T

$ esxcli network nic get -n vmnic0
  Advertised Auto Negotiation: true
  Advertised Link Modes: Auto, 1000BaseT/Full, 10000BaseT/Full
  Auto Negotiation: false
  Cable Type: Twisted Pair
  Current Message Level: -1
  Driver Info:
      Bus Info: 0000:03:00:0
      Driver: ixgben
      Firmware Version: 0x800005ad
      Version: 1.7.1.35
  Link Detected: true
  Link Status: Up
  Name: vmnice0
  PHYAddress: 0
  Pause Autonegotiate: true
  Pause RX: true
  Pause TX: true
  Supported Ports: TP
  Supports Auto Megotiation: true
  Supports Pause: true
  Supports Hakeon: false
  Transceiver:
  Virtual Address: 00:50:56:56:62:7
  Hakeon: None

On the board with issues, the firmware version is 0x800005ad. But the working board is 0x800003e7, which is curious.

A Temporary Solution

I then tried the suggestion from Bruno Zeidan, who had the same board:

esxcli network nic set --speed 1000 --duplex full -n vmnic0

This worked — and continues to do so after a reboot. Unfortunately, if the network cable is unplugged it won’t come back up again.

Upgrading the BIOS

My next thought was to try the new BIOS release. Supermicro put out 2.3 on 2021-06-04 and upgrading the BIOS is enough of a faff I didn’t bother previously.

I wrote a FreeDOS 1.3 LiveCD using Rufus to a USB drive and dropped in the BIOS Zip contents. I’d originally tried to use the FreeDOS bundled with Rufus, but this gave me an out of memory error (like mentioned in this blog post).

After booting back into ESXi after flashing completed, it did immediately get an IP, which was encouraging. Post upgrade, the firmware version was 0x800005ad still and unplugging the cable ended up with the same situation as before.

Trying to flash the NIC

Knowing that a difference between NIC firmware versions was the most likely explanation for the different behaviour on each board, I was curious to see what would happen if I tried to flash them. I also wasn’t convinced about the hex values of the version number — Intel didn’t have those listed anywhere.

I got a copy of the most recent driver CD and moved the Intel NIC BOOTUTIL over to the FreeDOS USB drive from earlier. Calvin Bui’s post about flashing the NIC firmware enabled me to figure out what was needed. From reading around, it seems like Supermicro likely have their own firmware specifically for the onboard NIC, so I stuck to the available (two) driver CDs.

On the problematic board, it turned out that the version was 2.3.58:

C:\BOOTUTIL\DOS\bootutil.exe

Intel(R) Ethernet Flash Firmware Utility
BootUtil version 1.6.40.1
Copyright (C) 2003-2017 Intel Corporation

Type BootUtil -? for help

Port Network Address Location Series  WOL Flash Firmware     Version
==== =============== ======== ======= === ================== =======
  1   AC1F6B12F86E     3:00.0 10GbE   YES UEFI,PXE Enabled   2.3.58
  1   AC1F6B12F86F     3:00.0 10GbE   YES UEFI,PXE Enabled   2.3.58

But the working board was 2.3.53:

C:\BOOTUTIL\DOS\bootutil.exe

Intel(R) Ethernet Flash Firmware Utility
BootUtil version 1.6.40.1
Copyright (C) 2003-2017 Intel Corporation

Type BootUtil -? for help

Port Network Address Location Series  WOL Flash Firmware     Version
==== =============== ======== ======= === ================== =======
  1   OCC47A95BE18     3:00.0 10GbE   YES UEFI,PXE Enabled   2.3.53
  1   OCC47A95BE19     3:00.0 10GbE   YES UEFI,PXE Enabled   2.3.53

NVMUpdate is the tool to flash the NIC firmware and the easiest way to run this was to create a Debian Live USB drive and copy over the versions from the driver ISO images.

Once you’re booted into Debian, you can:

Find the drive: sudo fdisk -l,
Create a mount point: sudo mkdir /media/usb-drive,
Then mount it, e.g.: sudo mount /dev/sdb1 /media/usb-drive)

But running ./nvmupdate64e would always give a device access error on all available Supermicro versions (and the most recent Intel version available from their site):

$ sudo ./nvmupdate64e

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.38.3.3
Copyright(C) 2013 - 2023 Intel Corporation.

WARNING: To avoid damage to your device, do not stop the update or reboot or
poweroff the system during this update.
Inventory in process. Please wait [...|******]

Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Connection          N/A(N/A)   15AD 00:003 Access error
    X552/X557-AT 10GBASE-T

Tool execution completed with the following status: Device not found.
Press any key to exit.

Now I was stumped. The next best option is to get in touch with Supermicro support, but I’d love to hear from anyone who gets further than me!

For now, I’m using the temporary fix mentioned above as I need to get on and get it setup.

Building a 1Password CLI Windows MSI Installer

Fri, 17 Mar 2023 00:00:00 +0000

1Password has a CLI tool that’s very helpful for integrating into things like setup scripts and handling development secrets more securely.

Unfortunately, on Windows, there’s no installer. Instead, you’re prompted to extract an archive, create the destination directory and add it to your PATH. This is annoying once, impractical to do multiple times and enough of a hurdle for others to be quite a pain, especially when you want to keep up to date with the most recent version.

I’m sure 1Password will release one eventually, but for now, I’ve put together a quick project to make installers that you can find on GitHub. On the releases page, you can find built versions for i386 and amd64.

I’ll keep these updated until 1Password releases their own, but please let me know if you see any problems (and if you find them useful too)! I would like to have these code signed eventually, but for now, that’s not something I can do.

This project initially came out of a frantic evening back in December 2022 where I got caught up in trying to assemble a basic installer with WiX Toolset, initially inspired by trying to resolve an installation issue where we’ve been side-loading an enterprise UWP app. Whilst there’s much more to do (and another blog post to go with that topic), assembling the missing installer for the 1Password CLI seemed like an excellent first step.

I elected to use WiX v3, instead of v4 as v3 has much better information about it. Using v3 meant I wasn’t also trying to understand how to build installers at the same time as trying to understand a whole new version of a tool along with the rest of the community, and there’s a lot of “TODO: WiX v4 documentation is under development.” in the docs, which wasn’t too inspiring.

As installer projects go, it’s pretty straightforward. There’s two build architectures defined, along with a set of platform-dependent variables (in Config.wxi):

<?xml version="1.0" encoding="utf-8"?>
 <Include>
 <?if $(var.Platform) = x64?>
 <?define PlatformProductName = "1Password CLI (x64)"?>
 <?define PlatformUpgradeCode = "D7D4A655-76AB-45C9-B50F-0A8C1009E8F5"?>
 <?define PlatformProgramFilesFolder = "ProgramFiles64Folder"?>
 <?else ?>
 <?define PlatformProductName = "1Password CLI (x86)"?>
 <?define PlatformUpgradeCode = "E21CFAE2-49F0-489C-A069-437374D61DA5"?>
 <?define PlatformProgramFilesFolder = "ProgramFilesFolder"?>
 <?endif ?>
 </Include>

In the main Product.wxs file, first, the target directory is setup:

<Directory Id="TARGETDIR" Name="SourceDir">
    <Directory Id="$(var.PlatformProgramFilesFolder)">
        <Directory Id="INSTALLFOLDER" Name="1Password CLI" />
    </Directory>
</Directory>

Then a ComponentGroup is defined that places the executable (kept at the root of the project) and sets up the environment variable:

<ComponentGroup Directory="INSTALLFOLDER" Id="ProductComponentGroup">
    <Component Id="cmp_op.exe" Guid="*">
        <?if $(var.Platform) = x64 ?>
        <File KeyPath="yes" Name="op.exe" Source="op_x64.exe" />
        <?else ?>
        <File KeyPath="yes" Name="op.exe" Source="op_x86.exe" />
        <?endif ?>
    </Component>
    <Component Id="cmp_EnvironmentVariable" Guid="46D0B3FB-60B3-4F08-A911-CC03F3907DC2" KeyPath="yes">
        <Environment Id="InstallPath"
            Name="Path"
            Value="[INSTALLFOLDER]"
            Action="set"
            System="yes"
            Part="last"
            Separator=";" />
    </Component>
    <Component Id="cmp_RegistryEntry" Guid="*">
        <RegistryKey Root='HKLM' Key='Software\[Manufacturer]\[ProductName]'>
            <RegistryValue KeyPath='yes' Type='string' Name='Install location' Value='[INSTALLFOLDER]' />
        </RegistryKey>
    </Component>
</ComponentGroup>

Compiling independently for each architecture builds an installer in 1PasswordCliInstaller\bin\Release with this setup.

kurtanr’s WiXInstallerExamples project was very helpful in trying to put this together, and many more examples if you find yourself trying to do something similar.

Setting up an Azure Site-to-Site VPN to a Ubiquiti EdgeRouter through NAT

Tue, 03 Jan 2023 00:00:00 +0000

I have a Home Lab with a couple of devices for experiments and testing. It’s set up on a separate physical network — using a Ubiquiti EdgeRouter X — but I wanted to connect this to the outside world and get access to public IPs I could proxy to local services.

A while ago, I started using Azure for other experiments (because I have a bunch of credits to use). I’ve also found the cost of the more basic virtual machines cheaper than other cloud providers. At the time of writing, a B1s is £3.19/m on pay-as-you-go and down to £1.20/m reserved for three years) and importantly, allowed me to allocate the most IPs to a single virtual machine.

On the Azure side, I’m using a VPN Gateway associated with a virtual network. On the Home Lab side, a Ubiquiti EdgeRouter X behind a Ubiquiti Security Gateway (which is doing NAT for my home network). I don’t have a static IP, but it changes infrequently, so this shouldn’t be too much of a problem.

To configure a Site-to-Site VPN, both sides of the connection need to know about each other. We can’t do this with NAT — as NAT hides the devices behind it — but we can work around this by forwarding the necessary ports for IPSec on the local side. In some ways, this is a bit of a cop-out solution, but otherwise, we’d need a Point-to-Site that isn’t supported on the lower-end VPN Gateway.

(I initially started this project by trying to build a custom VyOS image and do this using a VyOS VM on the Azure side and the EdgeRouter locally. After months of on-and-off effort, I got the VM working, but it’s challenging to configure when both sides are behind NAT. I also realised that the lowest-tier VPN Gateway worked out to be about £22/m, which is fine.)

For the rest of this article, I’ve based it on this Ubiquiti help guide, but using Terraform for the configuration. We’ll configure an IKEv1/IPSec VPN using a shared secret and end up being to connect our local network to virtual machines running in our Azure virtual network.

Local Network Adjustments

IPSec uses UDP ports 4500 and 500 over UDP. I configured that in the UniFi controller to point to the EdgeRouter.

A screenshot showing the UniFi controller, forwarding IPSec ports

Configuring the Azure Virtual Network Gateway using Terraform

We need to configure a few things to get this up and running (and this assumes you already have a resource group (named rg here):

resource "azurerm_local_network_gateway" "home_lab" {
  name                = "HomeLab"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  gateway_address     = "LOCAL_NETWORK_PUBLIC_IP"
  address_space       = ["10.1.0.0/24"] # local network subnet
}

resource "azurerm_subnet" "gateway" {
  name                 = "GatewaySubnet"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.core.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_public_ip" "gateway" {
  name                = "Gateway"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  allocation_method   = "Dynamic"
}

resource "azurerm_virtual_network_gateway" "gateway" {
  name                = "Gateway"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  type                = "Vpn"
  vpn_type            = "PolicyBased"
  active_active       = false
  enable_bgp          = false
  sku                 = "Basic"

  ip_configuration {
    name                          = "Gateway"
    public_ip_address_id          = azurerm_public_ip.gateway.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.gateway.id
  }
}

resource "azurerm_virtual_network_gateway_connection" "home_lab" {
  name                = "HomeLab"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  type                       = "IPsec"
  virtual_network_gateway_id = azurerm_virtual_network_gateway.gateway.id
  local_network_gateway_id   = azurerm_local_network_gateway.home_lab.id
  shared_key                 = "SHARED_KEY"
}

In the azurerm_local_network_gateway resource, we define the local network: the IP we’ll be configuring the VPN to use and what the local network looks like. azurerm_virtual_network_gateway represents the primary resource, and then azurerm_virtual_network_gateway_connection builds the connection to the local network.

I made up the shared key using openssl rand -base64 20, removing non-alphanumeric characters as these caused the connection to fail to authenticate. I couldn’t tell which side had a problem with it. You probably want to keep the secret in a Key Vault.

The Virtual Network Gateway will take a decent amount of time to be operational — it took about 25 minutes most times.

Configuring the EdgeRouter

Connect over SSH and then in configuration mode:

configure

Followed by configuring the firewall rules, IKE/ESP groups, then setup the connection on both sides:

set vpn ipsec auto-firewall-nat-exclude enable

set vpn ipsec ike-group FOO0 key-exchange ikev1
set vpn ipsec ike-group FOO0 lifetime 28800
set vpn ipsec ike-group FOO0 proposal 1 dh-group 2
set vpn ipsec ike-group FOO0 proposal 1 encryption aes256
set vpn ipsec ike-group FOO0 proposal 1 hash sha1

set vpn ipsec esp-group FOO0 lifetime 3600
set vpn ipsec esp-group FOO0 pfs disable
set vpn ipsec esp-group FOO0 proposal 1 encryption aes256
set vpn ipsec esp-group FOO0 proposal 1 hash sha1

set vpn ipsec site-to-site peer AZURE_GATEWAY_IP authentication mode pre-shared-secret
set vpn ipsec site-to-site peer AZURE_GATEWAY_IP authentication pre-shared-secret SHARED_KEY
set vpn ipsec site-to-site peer AZURE_GATEWAY_IP connection-type respond
set vpn ipsec site-to-site peer AZURE_GATEWAY_IP description ipsec
set vpn ipsec site-to-site peer AZURE_GATEWAY_IP local-address 0.0.0.0

set vpn ipsec site-to-site peer AZURE_GATEWAY_IP ike-group FOO0
set vpn ipsec site-to-site peer AZURE_GATEWAY_IP tunnel 1 esp-group FOO0
set vpn ipsec site-to-site peer AZURE_GATEWAY_IP tunnel 1 local prefix 10.1.0.0/24
set vpn ipsec site-to-site peer AZURE_GATEWAY_IP tunnel 1 remote prefix 10.0.0.0/22

Then:

commit ; save

Setting the EdgeRouter to listen on 0.0.0.0 rather than the public IP allows us to work around the public IP not being configured — and helpfully means we wouldn’t need to reconfigure the EdgeRouter when the public IP changes.

Testing the Connection

After some time, the connection should hopefully be established. Azure will report back if it doesn’t work, and it’s possible to view the connection status on the EdgeRouter by using

show vpn status
show vpn sa

You can see the connection state in the Azure portal, too:

A screenshot showing the Azure portal, showing the connection status

VPNs can be notoriously difficult to debug when they go wrong. I’d start by checking the configuration on either side, then randomly changing things until it works, if it doesn’t eventually come up. Good luck.

On to Mastodon

Sun, 20 Nov 2022 00:00:00 +0000

I’m now using Mastodon: @nick@nickcharlton.net.

Some time ago, I stopped contributing on Twitter as much as possible. I was still obsessively scrolling and using it to keep up with stuff going on, but I was finding people elsewhere, ideally through subscribing to people’s blogs via RSS. I limited myself to occasionally liking tweets (I’d always used this as bookmarking for later and the occasional DM for meeting up with people at events, too).

As Twitter seemed to be starting a terminal decline, I thought this was great, and I’d not bother trying to replace it with something else. Maybe finally I’d lose the thing I find myself obsessively scrolling every day.

But two weeks ago, I changed my mind. I always thought Mastodon was quite neat, but if not enough people I’m following on Twitter started to use it, it was always a bit moot. Over the last few weeks, this changed and lots of people started joining.

I feel strongly about retaining control over the things you produce. For example, this site uses Netlify. Still, I have control over the domain and moving it elsewhere is relatively trivial, if not just a little annoying if I have to in a hurry. But if I did, nothing significant would change for those visiting or reading many of the things I have written over the years. If I could run Mastodon myself, it was probably worth the effort if it was going to take off.

None of this was happening in isolation. With Heroku’s slow death and then the removal of their free plan, I was already migrating off the old and onto the new. Some of my projects have gone to Fly, but I have access to a bunch of Azure credits, and I was keen to try and spin up a Kubernetes cluster there as it’s been a long time since I tried to do that. Mastodon is a Rails app, after all, and I’m quite familiar with those.

I plan to write up a lot of the configuration (it did take me about two weeks to get to this point, fitting in bits here and there), but I make a lot of notes as I go along. But join me on Mastodon, maybe?

Site-wide configuration with Administrate

Thu, 17 Nov 2022 00:00:00 +0000

I haven’t been doing too much Rails recently, but I did do this and it’s worked out really well. It was one of those situations where you can go, “oh right! We can do this…” and put together an example in less than an hour which are always very satisfying.

Adding site-wide configuration with a Rails model can be quite easy to do with Administrate.

See on Giant Robots

Self-updating GitHub README

Mon, 10 Oct 2022 00:00:00 +0000

Back in 2020, GitHub added profile READMEs. I jumped on it and created mine when they silently launched the feature but then did …nothing with it.

As the profile README is implemented as a special repository which gets rendered on your profile page, I’d wanted to build something that would auto-update and show contributions after reading that Simon Willison had done the same. I was never going to remember to update these myself, but as it’s just a GitHub repo, we can use GitHub Actions to make it possible.

GitHub README with contributions and blog posts

Fetching recent cross-GitHub activity

I often make contributions across various different repos and I thought it’d be interesting to show these off. I might be working on something quite interesting at work and usually when I do so, I end up opening issues or contributing PRs to various things. I started off by trying to use the GitHub GraphQL API, but the complexity was a lot for a query which was better off calling the Search API.

I already have a search I use to check what open issues and PRs I have that I should follow up with, so that was where I started:

is:open author:nickcharlton archived:false sort:updated-desc

We want archived items (to see closed or merged issues/PRs), but otherwise that’s what we pass through to the API:

headers = {
  "User-Agent" => "Recent GitHub Contributions (#{username})",
  "Accept" => "application/vnd.github+json",
}

client = Excon.new("https://api.github.com/search/issues", headers: headers)

response = client.get(query: {
  "q" => "author:#{username} sort:updated-desc is:public", per_page: count
})

From here, I’m repacking as a Contribution object (which is just a PORO with the keys below) so that it’s a little easier to work with later on:

data = JSON.parse(response.body, symbolize_names: true)

data[:items].map do |item|
  Contribution.new(
    id: item[:id],
    title: item[:title],
    url: item[:html_url],
    state: item[:state],
    type: item.has_key?(:pull_request) ? "pull_request" : "issue",
    created_at: item[:created_at],
    updated_at: item[:updated_at]
  )
end

One deliberate decision was to remove the GITHUB_TOKEN from the request entirely to avoid the possibility of leaking private information. I have access to a lot of repos and I wouldn’t want to accidentally leak something that shouldn’t be. Fortunately, GitHub allows enough public access without a token.

You can see the full implementation on GitHub.

Pulling in an RSS feed

This one is relatively straightforward. My blog has an Atom feed and we can pull that in and fetch the top five posts. I used Feedjira, which seems to be a well maintained RSS/Atom library which is able to handle malformed feeds well to avoid other problems:

client = Excon.new("https://nickcharlton.net/atom.xml")

response = client.get
feed = Feedjira.parse(response.body)

feed.entries.first(count).map do |entry|
  Contribution.new(
  id: entry.id,
  title: entry.title,
  url: entry.links.first,
  state: nil,
  type: nil,
  created_at: entry.published,
  updated_at: entry.updated,
  )
end

I figured re-using the Contribution object would be good enough — maybe it’d be nice to make the GitHub contribution specific fields default to nil but this will do.

You can see the full implementation on GitHub.

Assembling the README

This was much more fun to put together. In Simon’s original, he used an HTML comment to surround a block to replace and I did the same. So we’ll have the following in the README.md:

<!-- contributions starts -->
<!-- contributions ends -->

Elsewhere, we’ll assemble the line from the list of contributions and then we can replace the text:

replacement = <<~REPLACEMENT
  <!-- #{marker} starts -->
  #{content}
  <!-- #{marker} ends -->
REPLACEMENT

document.gsub(
  /<!\-\- #{marker} starts \-\->.*<!\-\- #{marker} ends \-\->/m,
  replacement.chomp
)

We’re able to use the marker to use the same code to update multiple sections, which is nice. You can see the full implementation on GitHub.

Building using GitHub Actions

GitHub Actions has a bunch of nice features we can use here. To start with, Actions are able to make changes to the repository that they’re running on, so we can use the git commands without configuring more than the name and email address. Next, we can use the schedule event to regularly trigger an update. I chose daily at 11am — I don’t make enough contributions for any more to be beneficial. I also configured workflow_dispatch so that I could manually trigger it if I so desired.

I had a bit of a chicken and egg problem in doing this. I wanted to test the Action on push to make sure it was working, but didn’t want the README changes to get picked up when merging. I ended up making sure it works, then deleting that commit, removing the push trigger and running it manually after. Anyway, here’s how it works:

---
name: Update README

on:
  workflow_dispatch:
  schedule:
    - cron: '0 11 * * *'

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Ruby
      uses: ruby/setup-ruby@v1
      with:
        bundler-cache: true
    - name: Install dependencies
      run: bundle install
    - name: Update README
      run: |-
        bin/update_readme
        cat README.md
    - name: Commit and push if changed
      run: |-
        git diff
        git config --global user.email "actions@users.noreply.github.com"
        git config --global user.name "README Bot"
        git add -A
        git commit -m "Updated content" || exit 0
        git push

So now I have an auto-updating GitHub README which shows up on my profile. Go take a look!

Flashing the firmware in an IKEA BEKANT to add position memory

Sun, 09 Oct 2022 00:00:00 +0000

As I started to work from home at the beginning of the pandemic, I got an IKEA BEKANT sit/stand desk. It’s been great for my posture and managing the fatigue of being at a desk for a long time every day. My main frustration has always been that it doesn’t have a memory — you can only hold the up or down buttons to move it and so trying to get it to the same position is annoying enough to avoid doing it.

Fortunately, someone reverse engineered the firmware and published a replacement which can be flashed on the existing hardware.

The BEKANT controller is a PIC microcontroller, so you’ll need a PICkit programmer to flash it. I got one of these, which was about £20 a few months before doing it. You’ll also need the Microchip environment, MPLAB X.

Preparing the controller

First, I set the desk to the right sitting height. If it went wrong, it was much better to have it lower. You’re going to want to do something like:

Power down the desk,
Unscrew the controller from underneath,
Open up the back

IKEA BEKANT Controller: Opening

It’s quite difficult to open the back. It’s glued down and made me have some grumpy thoughts about right to repair. I first tried to use a knife to get some space, but this didn’t work too well. From seeing photographs of others doing this, I realised there was space unused in the case where I should be safe to drill into. I drilled three holes into the corner and then used some pliers and snips to break it open.

You’ll see something that looks like this.

IKEA BEKANT Controller: Back Cover Removed

Configuring MPLAB

Open the MPLAB IPE,
Configure for the PICkit 3.5
- Settings → Advanced Mode (the default password is microchip),
- Set the device to PIC16LF1938 and click “Apply”,
- Under “Power”, select “Power target circuit from PICkit3” and set the voltage level to “3.25”,
- Under “Production”, select “Allow Export Hex”,
- Then logout from Advanced mode

Connecting the programmer

You’re going to need to set the pins correctly, this diagram from the docs should be enough to figure it out:

Pin connection between the controller and PICkit

Connecting the pins correctly

Then, Click “Connect” in the main UI, and the programmer will attempt to connect to the chip.

MPLAB: Connecting

I had some problems in ensuring there was a good connection. Leaning my finger on the connector was good enough to solve that.

My finger pushing the PICkit pins to ensure a good connection

Reading off the current firmware

Next, read off the existing firmware in case it goes horribly wrong.

Screenshot of MPLAB reading the existing firmware

You can find the firmware from mine here.

Flashing the new firmware

Fetch the firmware from the releases page,
Load the firmware (“Hex File” → “Browse),
Program it: Click “Program”,
Verify it: Click “Verify”

It should program and then verify successfully.

MPLAB: Flashing and verify the new firmware

Testing

To test it out, I first followed the advice in the docs and cleared enough space for safety and then wired everything up in a way I could quick disconnect it if things went wrong.

Then I followed the rest of the guide which involved testing out the default buttons, followed by the memory the firmware has by default. Everything worked out great! I can now press Up whilst pressing Down to move to the higher position and Down whilst pressing Up to move down. The original commands all work too.

All that was left to do was to mount the control unit back to the desk (this time without the top cover).

Using HTTPListener to build a HTTP Server in C#

Fri, 29 Apr 2022 00:00:00 +0000

I needed an HTTP server to test out a software integration on Windows. Nothing was quite basic enough to just echo out the responses, so I wrote one.

See on Giant Robots

Week Notes #31

Wed, 02 Mar 2022 00:00:00 +0000

I’ve been jumping back into open source recently, plus doing a lot of technical writing for work,
I released v0.17.0 of Administrate the other week, which catches up on a whole load of contributions after nearly a year,
Next is going to be a big breaking change, as we remove a lot of dependencies (like datetime_picker_rails) and make the biggest change so far to how assets are handled. We’ll likely be removing Sass completely and moving to standard CSS, which I’m quite excited about, but the big news is that it’ll work much nicer with all of the Rails asset changes which have happened recently,
Technical writing is hard — I’ve been doing a bunch of it, and trying to distill down a lot of the way receipt printers work so that others’ can read one guide. I’m hoping this will turn into a series of blog posts, but that’s going to take much longer!
I enjoyed the latest Speedshop newsletter on different styles of open source contribution,
Finally, last week I went to see Wolf Alice. Standing in a crowd of people listening to music is great.

Converting Unix Shell Aliases to PowerShell

Thu, 10 Feb 2022 00:00:00 +0000

I’ve been using Windows a lot recently and have been quite enjoying it. Some things are new, but most are old and familiar. But after using a lot of shell aliases over the last decade of using Linux & macOS, I needed them to come over with me. I’ve been using PowerShell 7, inside Windows Terminal (and mostly VS Code as the editor), and so far have pulled over my Git aliases. If you’re trying to bring over your own, this should be enough to get you going.

PowerShell has an equivalent profile configuration file which can be found in:

C:\Users\<username>\Documents\PowerShell\Microsoft.PowerShell_profile.ps1

…and available as $profile. From here, you can configure aliases and functions which are loaded across each session.

There’s a couple of nuances in converting existing aliases to PowerShell though, as an example I’m converting the following which covers all of the cases I’ve seen so far:

In Unix Shell:

alias gws='git status -sb'
alias gl='gll -20'
alias gll="git log --pretty='format:%C(yellow)%h %C(blue)%ad %C(reset)%s%C(red)%d %C(green)%an%C(reset), %C(cyan)%ar' --date=short"

In PowerShell:

function Get-GitStatus { git status -sb $args }
function Get-GitLog([string] $limit)  {
  git log --pretty='format:%C(yellow)%h %C(blue)%ad %C(reset)%s%C(red)%d %C(green)%an%C(reset), %C(cyan)%ar' --date=short $limit
}

function Get-GitLogShort {
  Get-GitLog '-20'
}

Set-Alias gws Get-GitStatus
Set-Alias gl Get-GitLogShort
Set-Alias gll Get-GitLog

Notably, aliases don’t expand in place, which means you can’t automatically chain additional arguments. To do this, we need to use a function to wrap what we like to do and then use $args (a special argument equivalent to $@) to capture anything further. This is also the case if we want to be explicit about requiring further arguments, too.

By convention, functions are named Verb-Noun. I’m using a separate call to Set-Alias to replicate the original pattern I’m copying, but there is a way to define these directly on functions.

This is my current $profile, which is not without the occasional problem, but is working well so far:

function Get-Git {
  ($isGit = git symbolic-ref HEAD) > $null

  if ($isGit) {
    $time_since_last_commit = git log -1 --pretty=format:"%ar"
    Write-Output "Last commit: ${time_since_last_commit}"
    git status --short --branch $args
  }
}

function Get-GitStatus { git status -sb $args }
function Get-GitBranch { git branch -vv $args }
function New-GitCommit { git commit --verbose $args }
function New-GitCommitAll { git commit --verbose --all $args }
function New-GitFixupCommit { git commit --fixup HEAD $args }
function New-GitCheckout { git checkout $args }
function New-GitAddition { git add $args }
function New-GitPush { git push $args }
function New-GitForcePush { git push --force-with-lease $args }
function New-GitRebase { git rebase $args }
function New-GitStash { git stash $args }
function Get-GitDiff { git diff --no-ext-diff $args }
function Get-GitLog([string] $limit)  {
  git log --pretty='format:%C(yellow)%h %C(blue)%ad %C(reset)%s%C(red)%d %C(green)%an%C(reset), %C(cyan)%ar' --date=short $limit
}

function Get-GitLogShort {
  Get-GitLog '-20'
}

Set-Alias g Get-Git
Set-Alias gws Get-GitStatus
Set-Alias gb Get-GitBranch
Set-Alias -Name gc -Value New-GitCommit -Force
Set-Alias gca New-GitCommitAll
Set-Alias gcf New-GitFixupCommit
Set-Alias gco New-GitCheckout
Set-Alias gia New-GitAddition
Set-Alias -Name gp -Value New-GitPush -Force
Set-Alias gpf New-GitForcePush
Set-Alias gr New-GitRebase
Set-Alias gs New-GitStash
Set-Alias gwd Get-GitDiff
Set-Alias -Name gl -Value Get-GitLogShort -Force
Set-Alias gll Get-GitLog

Week Notes #30

Sun, 09 Jan 2022 00:00:00 +0000

A New Year always brings a new set of optimism, plans and possibilities for me and this year is no different,
Last year was another rough one (who knew living through a pandemic would be extremely difficult, even if you never got sick), but we made it,
A lovely Christmas and New Year, then back where I was previously; trying to get my head around Windows, C# and a lot of receipt printer minutiae,
Later this year, I’m doing the Frontier 300, but I need to get my fitness well up from where it is currently so I got a turbo trainer, a Wahoo Kickr Snap which I started using this weekend, it’s excellent,
My plan is to use it through winter to get my cycling fitness back up to where it was, and then hopefully, way ahead of wherever I managed to get it to previously, probably through Zwift. My training plan starts tomorrow!
I note that this is the thirtieth one of these I’ve done (and with it being a new year) feels like a bit of a milestone. I enjoy writing these, even if I’ve skipped a lot recently, onwards!

Week Notes #29

Sun, 21 Nov 2021 00:00:00 +0000

A busy couple of months or so, which so easily ends up breaking the habit of writing something regularly,
It’s mostly been work, my current project — which involves building lots of different bits of software and some hardware integration — has been fairly intense recently, but I’ve also been learning to drive (which I picked up again after a 10 year hiatus),
This has all lead to my previously delightfully minimalist one computer, one monitor and tidy desk setup to have another monitor, another computer and various bits of hardware spread everywhere (two receipt printers, three laptops with my own counted and an industrial Windows tablet). It’s been the most interesting and challenging thing I’ve ever worked on,
I’ve had more opportunities to write TypeScript, and most recently C# which is not something I’ve done since University. I’m a fan; especially when tied with Visual Studio,
The most noticeable thing is strongly typed languages when it comes to refactoring. The initial process of getting going might well be slower, but the middle of a project and onwards is so much easier,
I’ve also ended up reading a bunch of technical books lately: Domain Driven Design Distilled (I’d recommend it), Building Event-Driven Microservices (I’m about half way through, and using DDD to think about breaking up services through events seems to fit so nicely), and C# In Depth (although this was a bit too …in depth so far),
I’m extremely far behind in all of my open source notifications, partly through having much less time at thoughtbot whilst we try out a short Friday experiment, so if you’re waiting on something, sorry about that,
Finally, I got around to upgrading my Mac to Big Sur (a year after it was released) and thought sod-it and immediately jumped onto the Monterey beta with the RC. It’s been great so far, so 🤞

Switching to FeedBin and NetNewsWire

Sun, 21 Nov 2021 00:00:00 +0000

I’ve been using FeedWrangler and Reeder since Google Reader shutdown. All of these years they’ve continued to work really well. But I’ve wanted to move back to NetNewsWire since it was taken back over by Brent Simmons and made Open Source.

NetNewsWire was always more of a “Mac app” than Reeder ever intended to be, which played around with new UI ideas and helped push along some of the more innovative patterns we see today. But two things always bugged me:

It has no feed organisation, you just have one long list of unread items,
The swipe gesture and I never really got along as I’d accidentally jump around whilst reading,
The keyboard shortcuts always surprised me, being always slightly different

NetNewsWire solves all of these. But it doesn’t support FeedWrangler. FeedWrangler has been great; I jumped on it in the days when Google Reader announced it was going to shutdown and the frenzy of RSS syncing services started up. But since those days, it’s broadly stayed the same and is now in a deliberate stagnant mode.

I use multiple devices and operating systems, so to switch to NetNewsWire, I needed a different syncing service. I was keen on something that:

Restored the web reading experience from the days of Google Reader,
Something that would be around for a long time,
Supported NetNewsWire (…obviously)

I ended up picking Feedbin. It’s open source so if I really needed, I could host it myself. It’s been around for ages and I could pay for it with a reasonable monthly cost, so I felt reasonably confident it’d be around for much longer. Plus, the web UI looked great.

But the most compelling feature to me was Actions. I use RSS in two ways: to keep up with news, and to follow people’s blogs. People’s blogs are often management or technical in nature and so if I read them a few months later it’s not a big deal. News isn’t like that, and there’s often regular stories I immediately mark as read. With Feedbin’s Actions, I could automate marking stuff as read. That’s wonderful.

An example of Feedbin's Actions auto-mark as read feature

Migrating was a bit of a challenge. I don’t keep anything like RSS reader inbox zero. Mostly recently, I’ve been avoiding it too, as it hasn’t been working well for me and reading email newsletters much more instead. At the start of the migration, I had just about 600 unread items. These went back to 2018, but most were quite recent. It was broadly a case of:

Skim the headlines and mark everything I didn’t care enough about as read,
Push longer items to Instapaper for reading later,
Open shorter items, videos and everything I wanted to read properly in the browser,
Skim back through and mark more as unread,
End up with some short items to skim read

From here, I could export my list of feeds from FeedWrangler as an OPML file and import that into Feedbin. Then install NetNewsWire everywhere.

I did this a few months ago and found myself getting excited about RSS again and starting to stop the newsletters which end up in my email inbox get back over to RSS!

Week Notes #28

Mon, 21 Jun 2021 00:00:00 +0000

I’ve spent the last few weekends doing a lot of woodworking. I set myself the goal this year of building much more for the garden and — mostly because of the weather, but also because everything takes far longer than you’d expect — I ended up finishing one of the three planters I’d wanted to build,
This first one (pictured below) is for cucumbers: it’s a box with a built-in trellis (which is thick twine) which they grow up. Now I’m writing this, they’ve all reached the top and are full of fruit!
Unfortunately for my plans for woodworking, the weather changed and it’s been raining for the past week. But that gave me an empty weekend to chill out …and make some time to ship a bunch of site changes I started a year ago,
Since this time last year, I’ve had a branch called redesign-2020 which I’ve repeatedly picked up and put back down again since. But in the spirit of designing in the open (and also, honestly, making sure I ship something), I tidied up the branch and just merged it. I’m going to keep fiddling with small bits from now on to keep it going. I have a long list of bits to fiddle with and improve and so these will happen gradually over time,
Plus, some long running blog posts have some more substance to them now. I’ll be finishing those up and posting them soon!

Cucumber planter with plants

Filtering Jekyll Posts by Tag

Sun, 20 Jun 2021 00:00:00 +0000

I’m doing a bit of spring cleaning around here, and I wanted to split the list of posts on the index into Week Notes and everything else. I differentiate Week Notes posts from others by tagging them with week-notes and filtering by these seemed easily enough. But I was wrong.

In Jekyll, posts exist in a collection called site.posts, which you could pipe into a filter to change the list returned, so you could get a list of Week Notes posts by doing:

{% assign posts = site.posts | where_exp: "item", "item.tags contains 'week-notes'" %}
{% for post in posts limit:5 %}
  {{ post.title }}
{% endfor %}

Unfortunately, you can’t negate the expression as it’s not been implemented and doing so seems like it would break some established patterns. You can do something like unless post.tags contains 'week-notes' inside the loop, but this would mean we can’t limit the amount of posts we try to render which is both awkward and inefficient.

Fortunately, it’s not too difficult to build your own filter and so I came up with:

{% assign posts = site.posts | filter_posts: "tags", "include 'week-notes'" %}

…which is similar to where_exp, but more specific to filtering posts; I didn’t want to get too deep into parsing expressions, so I used a regular expression and a bit of meta programming to get something which works nicely:

module Jekyll
  module FilterPosts
    def filter_posts(posts, attribute, expression)
      method_name, key = expression.scan(/(\w*)\s?'([\w-]*)?'/).first

      method = case method_name
               when "includes"
                 :select
               when "excludes"
                 :reject
               else
                 nil
               end

      return [] unless method

      posts.send(method) { |post| post.data[attribute].include?(key) }
    end
  end
end

Liquid::Template.register_filter(Jekyll::FilterPosts)

RSpec.describe Jekyll::FilterPosts do
  include Jekyll::FilterPosts

  describe "tags" do
    it "can filter by presence of tags" do
      document1 = double("Jekyll::Document", title: "Post 1",
                                             data: { "tags" => ["week-notes"] })
      document2 = double("Jekyll::Document", title: "Post 2",
                                             data: { "tags" => ["projects"] })

      posts = filter_posts([document1, document2],
                           "tags",
                           "includes 'week-notes'")

      expect(posts).to match_array([document1])
    end

    it "can filter by exclusion of tags" do
      document1 = double("Jekyll::Document", title: "Post 1",
                                             data: { "tags" => ["week-notes"] })
      document2 = double("Jekyll::Document", title: "Post 2",
                                             data: { "tags" => ["projects"] })

      posts = filter_posts([document1, document2],
                           "tags",
                           "excludes 'week-notes'")

      expect(posts).to match_array([document2])
    end

    it "is empty if the filter method is invalid" do
      document1 = double("Jekyll::Document", title: "Post 1",
                                             data: { "tags" => ["week-notes"] })
      document2 = double("Jekyll::Document", title: "Post 2",
                                             data: { "tags" => ["projects"] })

      posts = filter_posts([document1, document2],
                           "tags",
                           "something 'week-notes'")

      expect(posts).to match_array([])
    end
  end
end

A Jekyll::Document has a data hash which we can ask for information about the post, so attribute here is calling that. I didn’t test for it, but you could presumably filter for other things as well as tags. For testing, you’d usually want to test the output of the filter, but in this case it’s a collection and so it seemed much easier to do that directly.

You can see it all tied together in the PR which added it.

Week Notes #27

Tue, 04 May 2021 00:00:00 +0000

A long weekend which didn’t quite go to plan: a slow (but necessary) Saturday, followed by a bad start to some wood working projects (nothing that bad though!),
In the UK we had a public holiday and so this weekend was three days long. I usually like to use them for starting projects; in three days you can make some good progress. I’d planned to replace some table legs and build an indoor cucumber planter: a big box with a trellis built in for the plants to grow up,
But I didn’t get very far into the table legs. I’m bad at getting accurate cuts with a handsaw and so the initial cuts were close, but not good enough. I tried to sand the ends down but using a random orbital sander isn’t the right tool for it and so I had to give up. I did plan out the cucumber planter, but next weekend I’m going to build a jig for my (soon to arrive!) circular saw which I’m hoping will solve all my problems. I’ll report back,
A work though, a new project has me on batman duties (jumping around between team members helping things move through so lots of pairing, code review and testing). The intention is twofold: one to help me with burn out (although, this role has it’s own issues in that regard) and to help things move through faster. It’s hard to tell concretely how improved things have been, but the general feeling is positive,
Finally, I’m trying to stop my endless open tab addiction by starting to use Instapaper again. For a long time, I’ve opened a lot of browser tabs of articles during the week, which I’d usually not get through and so would build up and up. I’d found that Instapaper would become another inbox, but I’m thinking that by doing a quick weekly review should keep the list down somewhat. I’ll report back on how well it works!

Week Notes #26

Tue, 20 Apr 2021 00:00:00 +0000

Quite a gap since I last wrote one of these; I recently rotated off a tough project which was taking up all of my time. But now I’m on something new and then had a lovely week off!
The Food Almanac via John that’s a collection of seasonal recipes and stories, which in turn lead to picking up An Everlasting Meal which I last tried in 2013 and didn’t really get on with, but was reminded of again because of Alicia Kennedy’s newsletter which I always look forward to reading. As I read it this second time, I’m realising I’ve (accidentally?) grown as a cook to do a lot of the topics covered, but I’m still learning a lot and I enjoy the little bits of recipes spread throughout,
I’ve been focused on wrapping up stuff I’d started, and a few workflow optimisations the past few weeks,
On my dotfiles, I’ve been to removing some dependencies which have been causing some extra complexity and also trying to make them more portable. This has meant removing base16, which I was using for colours and moving to something which relies more on the shell itself,
Plus I’ve been trying to finish a very long in progress blog post around my desk and workplace setup which I started last year,
I’ve also been planning out wood working projects for this year: replacing some table legs we use as a kitchen island, building several new planters. I’ve also been thinking a lot about building outside furniture, after being inspired by this Laura Kampf video. I don’t quite have the amount of tools though!

Week Notes #25

Tue, 09 Mar 2021 00:00:00 +0000

I just finished four sessions of physio: I’d been limping everywhere since before Christmas, but now I’m all back to running which after so much pain is excellent,
Plus, starting the longer rides again as I start to build up the fitness for cycling again, but mostly so I can spend some time away from the concrete of London!
It’s been very satisfying to get back into writing — more to come here,
I’ve been working a bunch on converting some projects to GitHub Actions for CI: a few of them (particularly Administrate) are fairly slow, which is partly the speed of tests itself, but mostly because we just keep sitting in queues to run the jobs,
Second beer, this time a “California Common”, which is close to a lager-style ale and the first one which really needed the Brew Fridge to make work.

Making Drafts' Action Bar fit my workflow

Mon, 08 Mar 2021 00:00:00 +0000

I use Drafts for capturing text (everything from quick notes to this blog post started there). I’ve been using Drafts for a year, and over that time, it’s become integral to my workflow. But I’d not done much to customise it to fit what I regularly do.

I typically have a view that looks like this:

Drafts Main window, showing the editor area with Action Bar at the bottom

I use Flags to keep many notes at the top of the pile and Workspaces as a context (but typically show everything in the Inbox). As I have Drafts take up a quarter of my display, I always have the Actions sidebar hidden to maximise note space. But the exciting bit is what I ended up doing with the Actions Bar, which shows just below the note itself.

Outside of just being an excellent editor for notes, the real power of Drafts is in Actions. I use very few in practice, but the ones I have, I find myself using a lot. After having this workflow for a year, I’ve finally gotten a good idea of what to do and made the Actions Bar reflect it.

I started by creating a new group — I just called mine “Default” as it was the first appropriate word that came to mind — and then moved the most common set of actions I remember myself using to end up with a list which is:

Tasks (which turns a line into a checkbox: - [ ] Item),
Insert Date,
Markdown Reference Link,
Markdown Table (generates a table, which I can never remember how to do),
Sort (which sorts alphanumerically),
Copy as Rich Text,
Copy as HTML,
Public Gist (this and the next is the Post Gist to GitHub Action),
Private Gist,
Preview (which pops open an HTML preview of the note),
Selection to Draft (which takes the current selection and makes a new Draft from it; great for breaking out notes),
To Things (which creates an item in Things with the title and description)
Move Line Up (Opt-↑)
Move Line Down (Opt-↓)
Indent Line (⌘-])
Outdent Line (⌘-[)

The ones without links are already part of Drafts. To do this, I moved the originals from their groups into the new one. It didn’t seem worth doing anything else more complex.

The final four I mostly use when outlining or making notes which might cover lots of topics but which don’t need to be expanded out too much. 1-on-1s are a good case of these.

The real power of tools like Drafts is in the way you can customise and then mix-and-match scripts to do stuff for you. But also, this is perhaps your general reminder to go back to evaluate what you use regularly and optimise it for how you use it now!

Building a Brew Fridge

Wed, 24 Feb 2021 00:00:00 +0000

I started brewing beer about three years ago and got to the stage where I was doing full-grain brewing and was really into it. But then I moved house and …just stopped.

Over those past few years I’ve been living in two (broadly identical) draughty old Victorian houses with nowhere appropriate to leave a fermenter in. With these kind of houses, the temperature fluctuates massively throughout the day and so the first brew was a bit of a disaster. It fermented fine but just didn’t taste very good and eventually most of it went down the drain.

I wanted to take this environmental restriction out of the equation and had read a lot about people converting fridges and freezers to control fermentation temperature. Did it solve the problem? Yes! I’m now two successful brews in. Here’s what I did…

Scope

Given a fridge and a heating element, the idea is to provide a temperature regulated environment to store your fermenter.

This whole idea started with discovering the BrewPi way back when I started brewing all of those years ago. The BrewPi allows you to control and visualise the temperature with the combination of a bunch of relays and thermocouples. It has a few guides and is presented as a nice looking product. It’s a very neat device, but it’s also quite expensive starting at about €170.

I needed to reduce the cost, luckily the brewing community has shared lots of alternative methods. This lead to searching for an off-the-shelf temperature controller which would do a good enough job of maintaining the temperature in a generic fridge. I didn’t really need an extremely precise amount of temperature control (not so much 0.1º, but 1º) because most recipes are in a wide enough recommended temperature range and that visualising what was going on would be nice, but not really needed.

This lead me to finding the ITC-1000 and can be had for about £15. The downside with the ITC-1000 is that they still need a bunch of wiring work but some more research lead to the Inkbird ITC-308 which didn’t. You plug in your heater in one socket and the fridge into the other. Perfect.

Choosing a Fridge

This left the biggest expense: a fridge.

I first looked at fridges back in 2018. My main concern was finding a model that would definitely be big enough, as once you take into account not being able to use the space around the hump (either because we need to put a heater in there or because it takes up too much space) the space is quite restricted. On top of this, it’s quite hard to find fridges with as few features as possible: similarly, we don’t want an ice box as that too takes up valuable space.

My first instinct was to try and reuse one, but not being able to pick one up myself meant that buying a new one was cheaper. The modern world sucks sometimes.

Old Fridge Research

In the end I settled on a Currys Essentials CUL55W20, which at the time worked out to be £139.99 with free delivery. This wasn’t quite the cheapest possible, but just about one up. The main difference here is the width, which means a 30L fermentation bucket fits with room to spare, which with a slimline one might not have quite worked out.

As I’d come back to this project a while after first starting, I’d realised that the model number breaks down to be 55W, meaning 55cm wide and 20 for the model year. (If you look at the photos above, the model year is 18). I think UL is likely “undercounter larder”.

I’d always traditionally used bubbler airlocks, but these won’t fit with the amount of vertical space available. I picked up a few cylindrical ones instead which do fit.

Fitting a Heater and Temperature Controller

Now I had a fridge, it was time to work out what else I’d need. A big decision I’d made early on was to make sure I could use the fridge as it was originally intended if necessary (over Christmas 2020, it was full of potatoes and then the turkey!) and so I didn’t want cables just coming through the fridge wall or to have to remove any material inside the fridge which would stop the shelves being able to go back in.

Through a long-lost forum thread, I was inspired to install a waterproof socket which would leave a surface mount on the back wall and also allow the heater to be removed if I wanted to.

For the heater, I went with a basic 60W tubular one — typically used for outside bathrooms or greenhouses. These come with a bare set of wires, as they’re designed to go into a fixed fused spur at 3A. I achieved the same thing by installing a socket on one end to the same amperage.

The most difficult part of the whole project was working out where to drill the hole for the socket. I previously wrote about hiring a thermal camera and this project is really what it was for. For a less technical route, there’s a commonly cited method of vodka and corn flour (the idea being that the vodka evaporates quickly, leaving dry cornflour to highlight the coolant lines) but I didn’t feel particularly confident with it. So I took a couple of thermal images to confirm my suspicions of where might be good to drill a hole.

The inside of a switched on fridge through a thermal camera

From looking at some product diagrams of fridges and then from looking at the fridge itself, I figured that my best bet was to use the hump at the bottom. This is there to provide a space for the compressor, pump and electronics for the fridge without it sticking out the back. In addition to this, from looking at the coolant lines which are visible, it was possible to see these go up into the back of the fridge (and so above where I wanted to drill). You can see this happening in action on the thermal image below:

Thermal image showing the hump at the back of a fridge giving off heat

This left drilling the actual hole. First, I drilled a 5mm pilot hole through the outer layer of plastic, then pushed through a long piece of stiff wire to break through all of the insulation. This ensured I wasn’t about to hit something I didn’t want to and that the exit hole also wouldn’t hit the compressor. After this, I used a step drill to go through wide enough to get the 23mm hole required by the socket. I found the insulation to be much thicker than you’d expect (which I suppose is not surprising!), but it’s easy to get through as it’s just foam. You can broadly see the process in the photos below.

Pilot Hole, with wire to breach the insulation

Drilling the main hole

Passing through the heater wire

Socket installed

The socket is from the Elkay waterproof range which I got from Rapid Electronics, I paired this with some 0.75mm² flex which is the same as used on the heater. The heater itself is relatively generic and just big enough to cover most of the area at the bottom of the fridge at about 50cm.

The tubular heater installed in the fridge

Then the fridge and heater are plugged into the temperature controller. Some advice I’d seen on the Amazon listing for the controller was to set the thresholds to 0.5º for the fridge and 1.0º for the heater. The thermal mass of the fermenting bucket will slow down any temperature changes, but this should stop the fridge cycling on and off too often and shortening the life of it.

Finally, I built a platform out of some bits of old wood. Cut wide enough and then the edges thinned to slot into the place where the glass originally was, the fermenter sits on top of these.

The fridge with everything installed and setup

Overall, the project (including specific tools) came to £236.89, broken down into:

Item	Cost
Currys Essentials CUL55W20 Undercounter Fridge	£139.99
Inkbird ITC-308 plug-in temperature controller	£35.98
60W 1’ Tubular Heater	£20.99
Reel of 0.75mm² flex (you really only need about 2M!)	£13.49
13A Plug	£0.93
Pack of 3A fuses	£1.90
Step Drill Set	£11.99
Elkay 3 Pole Waterproof Socket	£5.52
Elkay 3 Pole Waterpoof Plug	£6.18

Future Expansion

For the first brew, I just taped the temperature probe to the outside of the fermenter. I wasn’t brewing a beer which is very temperature sensitive (a bitter close to Fuller’s ESB) and so I didn’t need to worry too much about the extra degrees given off by the fermenting process.

But for the second, I installed a thermowell — a device which allows the probe to be isolated from — but still inside — the bucket. I got this one from The Malt Miller, which whilst a bit pricier (at £18) did actually have everything required in one purchase and so I didn’t need to hunting around for washers or the right nut to fit.

For now, this is the last of the modifications. It works great as it is and if I wanted to do multi-day temperature changes that’s absolutely possible. In the future, it might be interesting to add a BrewPi Spark but this is definitely Good Enough™.

Thanks to Luke Mitchell for looking at a draft

Week Notes #24

Mon, 25 Jan 2021 00:00:00 +0000

I’ve started a new habit this year of tidying up my desk before I leave at the end of a Friday, now when I return on a Monday morning it’s nice and tidy and ready to go, it’s very nice!
I finished How To Do Nothing last week. It’s an excellent read and I’d definitely recommend it, but it is one of those books you might just have to read at least twice,
Then moved onto Louis Theroux’s Gotta Get Theroux This, plus the 99% Invisible City, both of which have been good,
I bottled the first beer I mentioned previously yesterday, so far spot on on the expected alcohol percentage so I’m excited to try this in two weeks’ time,
Otherwise, lots of writing (which takes ages) in the background, plus a bit of a site redesign I’ve been working on for a few months.

A weekend with a Flir ONE

Fri, 15 Jan 2021 00:00:00 +0000

I spent a couple of weekends back playing around with a Flir ONE, a thermal camera which attaches to a phone. I’d been working on a project which involved some gnarly drilling and was concerned I’d hit a pipe — it turned out as a good excuse to give one a go.

The Flir ONE device itself

It’s a really neat device which worked well for my needs but is let down by some fairly average software. The model I used was a few years old (the newer ones have a much higher resolution) but this wasn’t a significant problem for my intended use-case. As an example of the things I found: the device would often disconnect whilst in use, sometimes during filming something which required stopping, resetting and starting again. The view would “flash out” completely; I wondered if this was a regular recalibration, but I couldn’t work out a consistent behaviour for it. Finally, the software just didn’t really follow iOS design affordances: lots of bits of UI which had no feedback or touch controls which were far too small.

Whilst I had it, I ended up taking a bunch of photos and some short videos of various things:

A row of houses, showing where windows leak so much heat

A mug of tea

A selfie

My 3D printer bed heating up

I rented it for a weekend from someone locally using Fat Llama which worked out really well. I picked it up on Saturday morning and dropped it off on the Monday for £30. It was great fun to have for a weekend, but with some of the annoyances I don’t think I’d buy one.

Week Notes #23

Wed, 13 Jan 2021 00:00:00 +0000

First Week Notes of 2021!
It was a busy old time at the end of the year, where I both did nothing quite worth writing up about and a lot of things I couldn’t,
The big news is that now I’m fully remote and won’t be going back to the office when the pandemic restrictions lift (maybe for some Friday’s if others do too, but not at all like I once did); this is both exciting and scary as it opens up many options which previously weren’t possible,
On the book front, I just finished English Pastoral: An Inheritance, James Rebanks’ new book about inheriting his family’s farm and being a farmer in the modern world. I enjoyed it a lot. In a few ways it pairs well with George Monbiot’s Feral: Rewilding the Land, Sea and Human Life that I read last year and provided a more nuanced view on what the countryside is for, how we produce food and what it should be (specifically, I’m a big fan of the idea of rewilding ecosystems but the details are quite nuanced; for example, we’ve been doing organised farming for thousands of years and so the natural world has evolved around us and so I don’t think it’s reasonable to expect to return to vast amounts of temperate rainforest),
I’m now reading How To Do Nothing: Resisting the Attention Economy which is excellent. It’s the book that formed this lovely talk from 2017, and (not quite what I was expecting) has similar themes from the prior two books,
Before Christmas, I also read What Got You Here Won’t Get You There, which came by recommendation from Alex Ellis. It’s about habits that might have been fine so far, but are now holding you back (like being a jerk to people without realising it, or shutting down conversations when you don’t mean to). I’d recommend it,
Outside of all of the cooking I did (I stayed in London, for the first time), I built a Brew Fridge and brewed my first beer in couple of years. I’d started brewing beer about four years ago, but in the old Victorian houses I’ve been in since it’s been very difficult to have the right place to ferment (over a period of two weeks and getting a stable temperature). So I built something to ferment in. Longer post to follow, but it was much easier to do than I expected!

Week Notes #22

Mon, 02 Nov 2020 00:00:00 +0000

It’s been the end of the first part of Review Season this week, a time where I spend an unreasonable amount of time staring blankly at a text editor trying to come up with anything to say. It’s that perfect combination of dreadfully difficult but also concerningly meaningful which leads me to deep writers block. vvvvroooommmmmmm the deadline goes, as it flies by …and then suddenly words happen and manic typing begins. Except this time, with one illusive answer to the question of what can x improve on? to leave me stuck once more,
Because just sitting at a blinking cursor never really helps once you’ve been doing it for several hours, I tried to beat the rain by cleaning my two bikes on Friday afternoon in preparation for servicing them. I got most of my way through until it started raining right towards the end and then hastily hurrying everything in after having been working under torchlight for about half a hour. But now, a new chain, new brakes and two shiny looking bikes all ready to go,
As another lockdown is due to start on Thursday, it was time to pick up new running shoes which, hopefully, will be the end of the bad series of injuries I’ve had this year,
Otherwise, that’s been just about it. Much of this week has been working down long issue lists and trying to review more Open Source PRs, slow but gradual progress!

Week Notes #21

Mon, 26 Oct 2020 00:00:00 +0000

Back to work last week and it was a good one: started off with helping out with a database migration (DigitalOcean’s managed database solution is pretty nice!) and then a new project starting with the second phase of a project that’s just Rails. It’s good to be back in the land of Rails, a place where I can comfortably test-drive new features in just a few hours and really focus about the product we’re delivering rather than the minutiae of the code,
I somehow ended up leaving a year (to the month!) in getting back to reading Practical Object Oriented Design in Ruby, but I finished the final chapter this week. It’s good and well recommended,
After just short of two weeks using Rectangle, I’ve gotten used to it now and it’s working great. I generally use four quadrants: Slack, Drafts, Mail (although I only open this once in the morning), then often Chrome (which I use for work) on one half of the screen. Occasionally I’ve been using a two-thirds/one-third split with Chrome and Drafts for calls, as Jira can’t quite fit otherwise, my terminal session always fullscreen,
This weekend saw me tear down the greenhouse that I’ve had setup for the last two years; it was just a plastic tent and in the sun they don’t last forever. I ended up picking the last two aubergines and the last cucumber of the season,
Then frame three prints that I’ve been meaning to do for years: one a Guinness poster which came in a rotting frame and I finally replaced, an early-run of Cameron Moll’s Brooklyn Bridge letterpress print and finally a print of the design proof of the BR logo which came out of Wallace Henning’s work on the Corporate Identity Manual. It’s only taken me about 8 years to start doing this!

Week Notes #20

Sun, 18 Oct 2020 00:00:00 +0000

A second week of doing my own stuff; much of the same but a relaxing way to spend a week as I get around to some things I’ve wanted to do for months,
I started off by spending a day doing some workflow improvements:
- I switched from Magnet to Rectangle for macOS window management,
- Started the move to kitty, with a hurdle around first removing base16 which I’d previously been using to handle colours. It worked really well, but now I’ve learned more about ANSI colours I’ve found I can do the bits I want manually …but I didn’t quite start on this yet,
More Tailwind this week, finishing up the menu for the side project I’ve been working on and then doing a little “accounts” view so there’s something to see once you’re logged in,
In implementing this accounts view (it’s a list of bank accounts a user has), I used this as the opportunity to try out ViewComponent. I found that it fit really nicely between being able to test drive a view and not write a heavyweight feature/system test with the added advantage of knowing that reusing components would be much easier in future,
I switched gears at the end of the week to pick up some maintenance of a project I started about four years ago for notifying you on library releases and I never launched. It’s kinda not necessary now as GitHub does have a native feature for it, but I’d wanted something to build on to track a few other things so I picked it back up. I got it working again (I’d broken the background jobs at some point), replaced Neat with CSS Grid and then came up a with a short list of things to do to actually launch it,
In my continuing discovery of Linux on the Desktop (with a barebones Debian and i3), I learned about UEFI SecureBoot and signed a kernel module, plus made both audio and the backlight keys work]. As I write this, I still can’t quite copy and paste how I’d like,
I always enjoy The Margins, a newsletter by Can Durak and Ranjan Roy, but this weeks’ on how to consume news was particularly good,
I’ve also been enjoying reading the Tailscale blog and this on how they traverse NAT was really interesting.

Week Notes #19

Sun, 11 Oct 2020 00:00:00 +0000

Last week was the end of a — long for me — nine-month project. It was not the easiest nor was it working in a particularly exciting to me technology (React and later React Native) but I learned a lot. And spent a lot of time in Zoom calls,
Unfortunately, the client has been badly affected by COVID-19 so the project ended up finishing much earlier than we’d hoped for (it’s nice to actually finish the work you start!) and a little abruptly: the last week was wrapping up everything we could, leaving notes and fixing a load of small annoyances I’d wanted to for months,
But that did open up an opportunity to take some time off, so this week (and also next) I’ve been working on my own stuff. A mix of fixing things which I’d wanted to, but were much more involved than I could find the time for plus trying out a few new things,
I started off by profiling my zsh configuration; it’d gotten horribly slow. I started off with 1.33 minutes (!), and got it down to 0.08 seconds by switching nvm (the main culprit) and jabba to lazy-load. It was a lot easier to do than I expected, I started off by profiling as explained in this article and then trying out a few lazy loading approaches,
I spent Monday and Tuesday on Administrate and closed 74 issues and PRs! I enjoy working on this project (when I get the time), but it’s a lot of work to keep on top of and I usually find I drop behind for a few months at a time and then the backlog becomes somewhat overwhelming. My goal this year has been get below 50 open issues and 30 open PRs and as I write this we’re on 65 and 31 respectively, so it looks like I might actually make it,
For a long time, I’ve been working on a tool for pulling my bank data into YNAB (an application which has been fairly transformative for me), it’s just for me so it works out as a good platform for experimenting on. It was feature complete about a year ago until the service I was using to pull UK bank data was shutdown and since I’ve been building out a UI so that you could log in to it. This is is needed because the (relatively new) Open Banking APIs in the UK require the end user to click through to confirm — previously it was a Rails app which just had Sidekiq to run some background jobs for syncing,
Anyway, I’ve been working with Tailwind on and off for a while plus using Tailwind UI, and this week I added Stimulus to do some interaction. After spending most of the year working on a huge React app, Stimulus is very satisfying: much closer to the way the browser actually works so you’re not fighting with someone else’s interpretation of how it should be and so little unexpected magic (which I always felt with Redux) and the docs are relatively tiny,
Like many people, I have a bit of an aversion to the huge amount of classes you end up putting in a “component” and my first instinct was to extract these out (in the original version of this project, that’s exactly what I did). But it took hours of working out naming and revisiting CSS conventions. I’m building a menu that I’m very likely to never touch agin, what’s the point? I think the individual component solution is going to be ViewComponent, which I’m going to try out soon,
Finally, I picked up a Thinkpad T470s. I have — for about the last three years or so — been thinking about trying to run Debian and the i3 tiling window manager and see how I go with it. It’s been good so far, but my expectations are to use it for messing about with, rather than anything more; can you imagine having to piss about with xrandr to get a presentation to work in a client meeting? nightmare-ish.

Week Notes #18

Tue, 15 Sep 2020 00:00:00 +0000

Unfortunately, as I write this, I have another — hopefully not so long lasting — running-related injury as I’ve pulled a muscle or perhaps sprained my right lower leg/ankle. Queue a day of hobbling around the house and cursing having stairs,
My recent injuries have been caused by a combination of things I’ve been gradually understanding over the past year: spending the last decade plus generally sitting has caused a hip imbalance (where the muscles at the front are much weaker), which leads to bad feet placement (pronation) which in turn can lead to various injuries (if you don’t have the right sort of support),
Ironically, standing with the new desk over the past month has likely contributed too!
tl;dr: I need new shoes and to work on improving my gait and strength again. This video has some really good background on it,
In other news, I moved my Weekly Review (this is something I’ve been meaning for months to write up) forward from Monday to Sunday. I find there’s enough going on that I just can’t get a good deep bit of reflection on Monday and so I’ve been skipping through far too quickly. I’m trying this now on a Sunday afternoon to see how this goes,
I’ve been continuing to setup the Synology I mentioned last week, mostly through moving data off ancient, disparate hard drives (including 2TB of old backups) into a folder structure I spent far too much time thinking through,
One of the motivations behind getting the Synology has been to decommission the Hetzner setup (which I’ve written about previously) I have and set something else up for experiments; this week a Ubiquiti EdgeRouter X arrived which will be part of the network for that. The plan is broadly: move any experiments there locally and setup a tunnel to make it accessible.

Week Notes #17

Tue, 08 Sep 2020 00:00:00 +0000

I’m now three weeks into having a standing desk and I’ve been doing a 50/50 split of sitting and standing ever since. The most interesting thing so far — and also one of the things I was very excited about — has been that I’m no longer so tired after work. I’ve been finding that standing keeps me going much longer (or, maybe, consistently?) which has always been something I’ve struggled with,
Another thing has been that I think I’ve split the past feeling of tiredness and hunger at the end of the day: I can stop working, go and do something else and then have dinner a bit later instead of needing to go eat right at the point I finish. It’s much nicer.
But I just had a week off where I did as little as possible apart from sorting out this and that, a spot of gardening and visiting my Mum for the first time eight months. It was particularly noticeable after a few stressful weeks!
In other news, I bought a NAS (network attached storage device). On the plan it was an investment, I got a Synology DS1520+ which will fit five drives (it is a little excessive!) plus the first 4TB HDD. I’ll be getting more drives over the next six months. More to write up here on my plans for it, but all the obvious stuff: backups, making some media accessible and a few utility VMs.

Week Notes #16

Mon, 17 Aug 2020 00:00:00 +0000

I made an example of using Pronto with standardrb that’ll run on GitHub Actions. It worked out to be one of those things which (mostly) just slotted together and worked, which was nice,
A warm cycle around London last week, which mostly reminded me how poorly thought through much of the cycle infrastructure is in certain boroughs: coming off a road into a “dual use” pedestrian/cycling area in a busy shopping area is a dreadful bit of street design,
Over the past few weeks I’ve been struggling with a mix of back and (eventually) shoulder pain which is part stress but also part my desk setup,
Step 1 was to replace the old IKEA chair I had (this is a good chair; just not for my needs) with one from the office: I now have an Aeron and this made a huge difference,
Step 2 was to get a standing desk — something I’ve wanted for years — and so I now have one of those (there’s a picture below),
After a long day, it does seem to have helped a lot. My big hope was to get to the end of it and not feel so tired and so far, that’s worked out!

Standing desk with monitor setup

Week Notes #15

Wed, 29 Jul 2020 00:00:00 +0000

I haven’t written these for a few weeks as pretty much every week has become the same thing and then you realise it’s been quite a long time…
A few weekends ago, I had my first actual pint in a pub and it was a strange experience and not quite the full experience I’d hoped for. But it’s a start!
I’ve been out doing long-cycles again in the much nicer weather and I have new gravel tyres which should be much better on that sort of terrain that I’ve been getting into more so now,
I’ve also picked up a frame bag (something I wanted already for bike packing), but has now become a permanent fixture on the bike holding a spare tube, pump, battery, tools and some snacks,
I made a Drafts action to extract a selection from one Draft into another. I had a long list of 1-on-1 notes in one long draft, which was starting to be unmanageable, so I’ve now split them into lots of smaller drafts quite easily,
We started a new project with Makers Academy, having three people who recently graduated work on Administrate for a few months. It’s been going well so far,
Finally, after being unable to for the past three-months, I’m able to run again and that’s been such a significant one for me to miss.

Week Notes #14

Tue, 26 May 2020 00:00:00 +0000

Week Notes #14

Since my last post, I published Setting Jenkins Credentials with Groovy. I’ve been using some of the extra time I’ve had from lockdown to wrap up some things I started years ago and this is part of the same thread. There’s a few more posts to come related to this before I get onto something a bit more interesting,
But now that the weather has gotten much hotter, I’ve been spending more time outside. I grew a bunch of plants last year and I’m doing the same this year. To start off with, I’ve been trying to make everything a bit easier than it was last year (I had too many plants and so watering took ages!),
One of the goals this year was to make the lawn look better and so last weekend, I sowed grass seed. To avoid having all of the seed eaten, I built twelve stakes to make a platform for mounting netting over. I also did a temporary extension of the garden tap to make access to water much less of a pain,
This weekend, I started building some window boxes & planters out of some scrap wood I’d collected over the past few years. There’s a photo below. There’s something really nice about working with scrap wood; this is all from old shipping pallets which shouldn’t be burned (they contain chemicals to protect the wood) so it’s really cool to see them get more life.

Planters and Window Boxes from old shipping pallets

Setting Jenkins Credentials with Groovy

Fri, 08 May 2020 00:00:00 +0000

I’ve been building up a nice pattern for bootstrapping Jenkins’ secrets through init.groovy.d, storing the secrets themselves inside configuration management. So far, this has been the simplest way to get a working configuration without additional moving parts beyond Jenkins and a configuration management tool.

The Jenkins Credentials plugin supports a few different secret types: “secret text” (which can be used as an environment variable), username & password, files from the file system and a few others. We can handle SSH private keys using the SSH Credentials plugin. These can be made available globally (i.e.: across multiple build nodes), or just on specific build nodes but here we’re just going to treat them as global.

As `init.groovy.d` scripts

I’ve used a few sources to understand how to do this in the past, especially this Gist. But I wanted to do this incrementally and be cautious on which dependencies were being imported. So, here’s how to implement each:

Secret Text

#!/usr/bin/env groovy

import jenkins.model.Jenkins
import com.cloudbees.plugins.credentials.domains.Domain
import org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl
import com.cloudbees.plugins.credentials.CredentialsScope
import hudson.util.Secret

instance = Jenkins.instance
domain = Domain.global()
store = instance.getExtensionList(
  "com.cloudbees.plugins.credentials.SystemCredentialsProvider")[0].getStore()

secretText = new StringCredentialsImpl(
  CredentialsScope.GLOBAL,
  "SECRET_NAME",
  "SECRET_DESCRIPTION",
  Secret.fromString("SECRET_TEXT")
)

store.addCredentials(domain, secretText)

Username & Password

#!/usr/bin/env groovy

import jenkins.model.Jenkins
import com.cloudbees.plugins.credentials.domains.Domain
import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl
import com.cloudbees.plugins.credentials.CredentialsScope

instance = Jenkins.instance
domain = Domain.global()
store = instance.getExtensionList(
  "com.cloudbees.plugins.credentials.SystemCredentialsProvider")[0].getStore()

usernameAndPassword = new UsernamePasswordCredentialsImpl(
  CredentialsScope.GLOBAL,
  "SECRET_NAME",
  "SECRET_DESCRIPTION",
  "USERNAME",
  "PASSWORD"
)

store.addCredentials(domain, usernameAndPassword)

The source for the implementation contains some additional options.

SSH Private Key

#!/usr/bin/env groovy

import jenkins.model.Jenkins
import com.cloudbees.plugins.credentials.domains.Domain
import com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey
import com.cloudbees.plugins.credentials.CredentialsScope

instance = Jenkins.instance
domain = Domain.global()
store = instance.getExtensionList(
  "com.cloudbees.plugins.credentials.SystemCredentialsProvider")[0].getStore()

privateKey = new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(
  '''
PRIVATE_KEY_TEXT
  '''
)

sshKey = new BasicSSHUserPrivateKey(
  CredentialsScope.GLOBAL,
  "SECRET_TEXT",
  "PRIVATE_KEY_USERNAME",
  privateKey,
  "PRIVATE_KEY_PASSPHRASE",
  "SECRET_DESCRIPTION"
)

store.addCredentials(domain, sshKey)

This was more difficult to figure out than the others, with the implementation of BasicSSHUserPrivateKey.java invaluable. The indentation of PRIVATE_KEY_TEXT is missing so that extraneous whitespace doesn’t cause problems. I found that where Jenkins reads the key in itself, indentation is not significant (like when connecting to build nodes) but where it’s reused elsewhere (like the Git plugin) would fail to connect to the repository.

It supports multiple different types of SSH keys: entering directly (what we’re doing here) but also reading off the disk. We need to use ''' to have a multi-line string as the private key will span a few lines. You’ll need the ssh-credentials plugin installed, too.

Configuring with Ansible

I’m using Ansible (with secrets stored in Ansible Vault), based around Jeff Geerling’s Ansible role for Jenkins, but this pattern could be replicated elsewhere.

Ansible Vault arranges secrets by encrypting variables which are accessible when playbooks are run. My original idea was to have a single variable name which when written would reflect on the type you set, for example:

---
jenkins_global_secrets:
  - name: EXAMPLE_SECRET_TEXT
    description: An example secret text value
    value: a_very_important_secret
    type: :secret_text

But this caused quite a bit complexity when trying to work out where the logic should be. To bridge the secrets from Ansible Vault to the provisioned machine, they’re written out from a template of the Groovy file and so splitting some of the logic between the template stage (which Ansible does) and the Groovy file (which is executed on runtime) felt misguided.

A much nicer approach seemed to be to use many top level variables, and instead you end up with this:

---
jenkins_secret_text_credentials:
  - name: EXAMPLE_SECRET_TEXT
    description: An example secret text value
    secret: a_very_important_secret

Which is very similar to how the Jenkins Chef cookbook solves this problem. From here, we can use this in a template like this (configure_jenkins_credentials.groovy.j2):

#!/usr/bin/env groovy

import jenkins.model.Jenkins
import com.cloudbees.plugins.credentials.domains.Domain
import org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl
import com.cloudbees.plugins.credentials.CredentialsScope
import hudson.util.Secret

instance = Jenkins.instance
domain = Domain.global()
store = instance.getExtensionList(
  "com.cloudbees.plugins.credentials.SystemCredentialsProvider")[0].getStore()

{% for secret in jenkins_global_secrets %}
secretText = new StringCredentialsImpl(
  CredentialsScope.GLOBAL,
  "{{ secret['name'] }}",
  "{{ secret['description'] }}",
  Secret.fromString("{{ secret['text'] }}")
)

store.addCredentials(domain, secretText)
{% endfor %}

In Ansible, this would then be written to the right place with something like this:

- name: Place the Jenkins Credentials Groovy script
  template:
    src: "configure_jenkins_credentials.groovy.j2"
    dest: "{{ jenkins_home }}/init.groovy.d/configure_jenkins_credentials.groovy"

With multiple top-level variables like this, the final result ends up being:

#!/usr/bin/env groovy

import jenkins.model.Jenkins
import com.cloudbees.plugins.credentials.domains.Domain
import org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl
import com.cloudbees.plugins.credentials.CredentialsScope
import hudson.util.Secret
import com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey
import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl

instance = Jenkins.instance
domain = Domain.global()
store = instance.getExtensionList(
  "com.cloudbees.plugins.credentials.SystemCredentialsProvider")[0].getStore()

{% for credential in jenkins_secret_text_credentials %}
secretText = new StringCredentialsImpl(
  CredentialsScope.GLOBAL,
  "{{ credential['name'] }}",
  "{{ credential['description'] }}",
  Secret.fromString("{{ credential['text'] }}")
)

store.addCredentials(domain, secretText)
{% endfor %}

{% for credential in jenkins_ssh_credentials %}
privateKey = new BasicSSHUserPrivateKey.DirectEntryPrivateKeySource(
  '''
{{ credential['private_key'] }}
  '''
)

sshKey = new BasicSSHUserPrivateKey(CredentialsScope.GLOBAL,
                                    "{{ credential['name'] }}",
                                    "{{ credential['username'] }}",
                                    privateKey,
                                    "{{ credential['passphrase'] }}",
                                    "{{ credential['description'] }}"
)

store.addCredentials(domain, sshKey)
{% endfor %}

{% for credential in jenkins_username_password_credentials %}
usernameAndPassword = new UsernamePasswordCredentialsImpl(
  CredentialsScope.GLOBAL,
  "{{ credential['name'] }}",
  "{{ credential['description'] }}",
  "{{ credential['username'] }}",
  "{{ credential['password'] }}"
)

store.addCredentials(domain, usernameAndPassword)
{% endfor %}

…and then how the secrets would be structured:

---
jenkins_secret_text_credentials:
  - name: SECRET_TEXT
    description: SECRET_DESCRIPTION
    text: SECRET_TEXT
jenkins_ssh_credentials:
  - name: SSH_KEY
    username: PRIVATE_KEY_USERNAME
    private_key: |
      -----BEGIN OPENSSH PRIVATE KEY-----
      ...
      -----END OPENSSH PRIVATE KEY-----
    passphrase: PRIVATE_KEY_PASSPHRASE
    description: SECRET_DESCRIPTION
jenkins_username_password_credentials:
  - name: SECRET_USERNAME
    description: SECRET_DESCRIPTION
    username: SECRET_USERNAME
    password: SECRET_PASSWORD

Week Notes #13

Tue, 05 May 2020 00:00:00 +0000

I got a fun new toy this week: The new iPad Pro. I went for the bigger screen this time (I’ve had a few over the years and developed a few iPad apps for them) and also got the new keyboard which has a trackpad built in. The trackpad is curious, I still mostly tap the screen and don’t feel I’ve gotten much out of it yet (the cursor changing dynamically is a cool UI effect though). Having a good keyboard will change things more, I’m aiming to write some of the workflow adjustments when I get more used to it,
Part of this has been a plan to try and use my laptop less often, which has become more important during lockdown,
My friend Miles released Keep Touch, which prompts you on an interval you set to check in with those you love. We paired a little on some of the time calculations which was fun but also quite difficult to think about,
Planted out some seeds this weekend, a mix of tomatoes, cucumbers, chillis and peas. I did quite a lot last year which worked out to be far too much, so slimmed it down this year,
I made a point of making quite a few notes on what was difficult or just generally a pain to do. Watering was one of those, so I’m going to automate some of it to reduce the workload,
Open source activity — as seen through my GitHub notifications — is peaky, which I find continually curious. In the UK we had a long weekend over Easter and it’s taken me until this week to get through it all. But as we start this week it’s gotten all quiet. You’d expect activity of others to jump once mine does, but this isn’t all of it. I think it’d be an interesting thing to research.

Week Notes #12

Tue, 28 Apr 2020 00:00:00 +0000

Just after the last Week Notes post, I published a short one on configuring Jenkins’ email-ext plugin. Jenkins, as much as a challenge it is to work with, still works the best for running arbitrary jobs and I’ve been working on some infrastructure experiments again,
Ubuntu 20.04 came out too, and it has a new installer. This lead to some experiments with trying out the new automated installer method and eventually this post once I made it all work. I’m now tracking an issue on Packer, an issue on Chef’s bento project, and another one too, as we try and make the new installer work everywhere,
The feedback loop for debugging these things is possibly the worst I’ve ever worked on: the only way to test if a theory works is to go through much of the process repeatedly, which means waiting for 5-10 minutes every time. You end up reading a lot whilst you wait,
In more open source news, I’m trying out having Administrate “Office Hours” by allocating two hours every other Friday for people to set a time to come with an issue or pair on what I’m looking at. I’m starting this internally only for now, but eventually I’d like to expand this out to everyone. So, if you’re reading this, this is an invitation to ask!
With an injured leg for the past few weeks (caused by, I think, my old running shoes being overdue for replacement), I’ve been off running for a while now,
But, I’ve been cycling around an empty London instead, which is eerily fascinating.

Automating Ubuntu 20.04 installs with Packer

Sun, 26 Apr 2020 00:00:00 +0000

Ubuntu 20.04 — which was released few days ago (23rd April) — brings with it a new installer, replacing the previous Debian installer with subiquity. This means that any of the previous approaches for automated/unattended installs no longer work and need to be replaced.

No one seemed to have documented doing this successfully yet with Packer, so I set out to figure it out. But first, here’s a working unattended configuration:

ubuntu-2004.json:

{
  "builders": [
    {
      "name": "ubuntu-2004",
      "type": "vmware-iso",
      "guest_os_type": "ubuntu-64",
      "headless": false,

      "iso_url": "http://releases.ubuntu.com/20.04/ubuntu-20.04-live-server-amd64.iso",
      "iso_checksum": "caf3fd69c77c439f162e2ba6040e9c320c4ff0d69aad1340a514319a9264df9f",
      "iso_checksum_type": "sha256",

      "ssh_username": "ubuntu",
      "ssh_password": "ubuntu",
      "ssh_handshake_attempts": "20",

      "http_directory": "http",

      "memory": 1024,

      "boot_wait": "5s",
      "boot_command": [
        "<enter><enter><f6><esc><wait> ",
        "autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
        "<enter>"
      ]
    }
  ],

  "provisioners": [
    {
      "type": "shell",
      "inline": ["ls /"]
    }
  ]
}

user-data:

#cloud-config
autoinstall:
  version: 1
  identity:
    hostname: ubuntu-server
    password: '$6$wdAcoXrU039hKYPd$508Qvbe7ObUnxoj15DRCkzC3qO7edjH0VV7BPNRDYK4QR8ofJaEEF2heacn0QgD.f8pO8SNp83XNdWG6tocBM1'
    username: ubuntu
  network:
    network:
      version: 2
      ethernets:
        ens33:
          dhcp4: true
          dhcp-identifier: mac
  ssh:
    install-server: true
  late-commands:
    - sed -i 's/^#*\(send dhcp-client-identifier\).*$/\1 = hardware;/' /target/etc/dhcp/dhclient.conf
    - 'sed -i "s/dhcp4: true/&\n      dhcp-identifier: mac/" /target/etc/netplan/00-installer-config.yaml'

We also need the presence of the file meta-data: touch meta-data in the directory available to the Packer HTTP server. The password is ubuntu. Whilst this uses the VMware builder, the relevant configuration options exist among the other builders, too.

I’ve also created an example Github repo with the working configuration in.

The new unattended installation is well documented and I started from the Quick Start guide.

There were four things that need solving on top of the basic configuration:

Ensure that there’s enough memory to run the installer with. 512MB caused a kernel panic, 1GB works fine,
Install an SSH server,
Allow the SSH handshake to fail for longer, because the installer has it’s own SSH server which will cause “Waiting for SSH…” to connect to the wrong thing and the build to fail,
Ensure that we have a persistent IP address after the installation is completed.

The typical way to do this is to restore the DHCP identifier used back to the MAC address of the device, rather than the device identifier which is now common. This is the same problem as I’ve seen previously with Debian Buster (10), and I’ve reused the late command here to set the DHCP client to do that. We also seem to need to do this with netplan too, to reliably get the same IP back that Packer is expecting.

Interestingly, subiquity does not seem to support dhcp-identifier and so we need to do this ourselves after the installation is completed. We need to quote the sed line, as otherwise we fall into a trap when loading cloud-init configuration, as it seems to think it’s YAML.

This took quite a bit of time to get working right. To solve issues along the way, I:

Used Alt + F2 to get a working console when the installer failed or got stuck,
Read the output of /var/log/installer/subiquity-debug.log to get the network configuration correct,
…and /var/log/syslog to debug the YAML parsing issues around the late-commands.

Week Notes #11

Mon, 20 Apr 2020 00:00:00 +0000

A little while ago, thoughtbot added a new team lead role and I started that a week ago when we rolled that out into our London office. It’s exciting to start doing something a bit different formally,
Read The Coaching Habit; this was recommended to me by a friend a month or so back and it’s all about giving less advice and asking more questions. I ended up reading it on a sunny bank holiday afternoon in the garden and I’d recommend it,
Less successfully, I spent some time on a bit of a folly of trying to write a script to rebuild all gems; I have a lot of Rubies installed from jumping around different projects and I think Homebrew broke things by tidying up OpenSSL. The main problem I faced is the shell just hanging, which is next to impossible to debug. I ended up doing it all manually again and it took hours,
Over the long weekend, I solved an issue I’d opened way back in November 2018, which is my way of saying: yeah, I really do get around to things eventually,
One of the problems of long weekends is that open source notifications continue along: from a good state of being on top of them, as I write this I’m far behind where I’d like to be,
This past Sunday (19th Apr), I went for a cycle around what’s now a very empty London, through a 30km loop from Hyde Park, up to Paddington and back through Oxford St it’s very strange seeing areas which I’m used to being so busy just being devoid of people,

Configuring the Jenkins email-ext plugin with Groovy

Mon, 20 Apr 2020 00:00:00 +0000

The default email notifications provided by Jenkins are a little inflexible, but the email-ext plugin adds a whole host of configuration options. I use this to provide the build log in some sync jobs I have, but I wanted to configure this via a script in Jenkins’ init.groovy.d directory to automate the configuration.

I’ve leaned this article, this Gist about setting up users and this other Gist about configuring credentials as I’ve done this before, but I couldn’t find anything which pulled together how you’d configure email-ext. But I did come across this now implemented issue which exposes the important bits of what we’ll need to configure, so a combination of the other examples lead me to come up with the following:

import jenkins.model.Jenkins

def inst = Jenkins.getInstance()
def emailExt = instance.getDescriptor(
  "hudson.plugins.emailext.ExtendedEmailPublisher")

emailExt.setSmtpAuth("username",
                     "password")
emailExt.setDefaultReplyTo("jenkins@example.com")
emailExt.setSmtpServer("smtp.example.com")
emailExt.setUseSsl(true)
emailExt.setSmtpPort("587")
emailExt.setCharset("utf-8")
emailExt.setDefaultRecipients("someone@example.com")

emailExt.save()

…which allows you to configure the basics of the account setup. This is loading the current instant of ExtendedEmailPublisherDescriptor, so any of the setters in that class can be called this way.

Week Notes #10

Mon, 06 Apr 2020 00:00:00 +0000

Double digits! I’ve written ten of these now. It’s gradually becoming a habit, especially once I’ve noted at least one thing down during the week,
This week I moved everything over to Drafts. Previously I was using Bear (which is a great app), and then sort of using both tools whilst I figured this out,
But the interesting bit isn’t the tool, it’s the habit formation: I write nearly everything first in Drafts before moving it elsewhere (be it Basecamp messages, GitHub issue replies or emails), keep notes on everything during the day and use it to write out my thoughts before saying them,
We’re now three weeks into full working from home because of COVID-19. I’m used to it now, even though I was struggling with the lack of my usual level of exercise and then not being able to sleep very well,
I join more remote calls from my desk now, so now I’m wondering how to make remote calls better, should I upgrade the video, or the audio, or both?
We started a new planning approach on Administrate by starting to use GitHub projects. The intention here is to highlight work-in-progress, which was previously hidden, and to set expectations for if something is soon likely to make it into a release,
I use Octobox for keeping on top of my GitHub notifications and for the first time in years, I got to inbox zero! For the last few months, I’d been hovering around 25 items much of the time, but was never able to get it any lower. As I write this it’s 3 things.

Week Notes #9

Mon, 23 Mar 2020 00:00:00 +0000

This was the first full week of working from home for me in years. I used to do this when I freelanced, which whilst a while ago often seems to help in surprising ways,
I am seeing meetings which were a mix of colocated and remote run better and gradually finding work becoming more asynchronous, which is all great to see. These are things that’ll make everything better in the long run, once the immediacy is over.
But, never before has writing “take care” in an email to a relative stranger actually meant something,
Because of COVID-19, The Focus on Time course opened up for free and I jumped right onto that. I’m finding I have more time now (no commute!), but I’d already not been using my time quite how I’d wanted. I’m not too far in yet, but it’s good so far!
Finally, I switched to using coc.nvim, which brings support for IntelliSense over from VS Code. I’d been struggling with “modern” JavaScript tooling anyway, but decided to take the leap. I’m pleased I didn’t need to configure much.

Week Notes #8

Sun, 15 Mar 2020 00:00:00 +0000

This was the week where Coronavirus/COVID-19 mitigations really stepped up, especially for people (like me!) working for predominantly US companies. In the absence of stronger advice and organisations where working remotely is relatively easy, this is a good move,
On that, we moved to being fully remote for the foreseeable future. I’ve got a pretty good setup in this regard as I already do this once a week. Ergonomics are important to me; if I just work on my laptop I get tired faster and my back aches horribly,
In other news, I published an Administrate security vulnerability this week. This is the first time I’ve gone through this process, which started a couple of weeks back with Benoit Côté-Jodoin from Shopify emailing us to let us know. I then ended up using an interview to pair on the potential solution and using GitHub’s new private fork functionality. All in all, the most difficult part of this process was writing up the advisory itself, making something clear and not accidentally withhold something important.

Week Notes #7

Mon, 02 Mar 2020 00:00:00 +0000

Last week, I released a new version of administrate-field-nested_has_many,
These last two weeks have been interesting in that there’s been lots going on …and none of that I can publish. After doing this seven times, I’m pleased about how much I really do operate in the open,
Tailwind UI came out this week, I’ve been following along through the Art of Product podcast and spent some time playing around with it this weekend. I’m still on the fence with Tailwind. It’s a utility-first CSS framework and so to work with it you add classes to build up what you need. For prototyping this means you’ll write a lot of classes, but these can then be extracted out into components. I haven’t quite done enough with it yet, but I’m undeniably productive with it and so I’m holding back my old-school don’t-mix-content-with-presentation ethos,
Finally, I switched to Jetbrains Mono as my terminal font. Ligatures finally make error handling in Go and the awkward bits of JavaScript look nice!

Week Notes #6

Sun, 16 Feb 2020 00:00:00 +0000

With Storm Ciara bringing wind gusts of up to 60mph, I ended up not cycling all week. There’s not much fun (and safe) cycling to be had in those sorts of wind speeds,
I spent a good chunk of time this week working on Appraisal, and I’m really happy about that. It’s been falling behind with everything since Bundler 2.0 was released, and my approach of ignoring it wasn’t going to last forever. I closed a bunch of PRs, opened a few more related to Ruby versions and accepting paths (which should speed up builds on Administrate),
For many months, I’ve been working on replacing our office internet and network and this week, because we’re also building a new meeting room it stepped up a gear into planning out what we should do. It’s quite fun doing this stuff again (I did this sort of thing early in my career). But perhaps I’ll think differently after rewiring a forty-two port patch panel,
All of this is in contrast to spending the main part of this week regularly stuck on what I’m supposed to be working on. Some parts through architectural complexity, some through people not being around and some from less-than-ideal prior planning. Here’s to hoping we can solve that next week.
I enjoyed reading about The Wall of Technical Debt this week. Unlike a lot of articles, it’s correct around what technical debt really is (a set of trade-offs) and (most importantly), has some nice suggestions around exposing the costs to everyone on the team,
And also Agile as Trauma and this associated Twitter thread, which talks about the position the Agile manifesto comes from and some of the drawbacks today. I’ve seen a lot of the points mentioned.

Week Notes #5

Sun, 09 Feb 2020 00:00:00 +0000

Some things are a blessing in disguise: we started the week blocked on our current piece of work, which after the hectic week previously really helped,
This was because we were blocked on everything as the feature we’re working on depends on an API, which depends on the release of a service rewrite and the next (and final) bit of work is the same as we’ve already done. We could certainly do more but also achieve less, it’s a tough one but was the right call,
Tuesday morning was entertaining: the combination of a strong headwind and last weekend meant cycling was the worst it’s been in a very long time and I barely could get up the hills,
I finished the first pass at some tooling I’m working on to improve my maintainer workflow. It allows you to check out a contributors’ pull request in your local clone, whilst maintaining your ability to push changes back to their branch. I need to write it all up yet and give it a spin for a couple of weeks to check it works well but so far I’m pretty pleased with it,
This week, I started on an bike interval training regime as I try and build up my bike fitness. Unfortunately, this means I won’t be running again for quite a while (trying to fit both in is …not practical),
On Thursday, I overzealous in merging some work I’d been working on without getting it QA’ed. It did, of course, break. With it being both paired on and test-driven, I came at it with a lot of confidence that it’d be fine. Our bit was, sadly the requests it makes goes to a service not quite as reliable,
I enjoyed reading Andrew Hao’s High Output Management for (Non-managing) Tech Leads this week. I find a lot of leadership/management articles useful in a consulting context (although, often with a bit of a leap and with the passage of time). As a consultant, I’m always on the look out for work of high leverage (especially when this means ignoring fun but not-valuable work). Similarly, the questions to ask around output indicators are helpful (e.g.: “What do you care about seeing?”, “What worries you the most about our team?”)

Week Notes #4

Sun, 02 Feb 2020 00:00:00 +0000

I decided to skip last week’s. Nothing of interest happened, outside of the weekend which fits just as well here.

Spent last weekend with my Mum, which was lovely. We started off a batch of homemade wine, which was great fun,
Started using Drafts on inspiration from Luke. I’d previously been using Bear (which I’ve really liked), but I’ve been curious about Draft’s actions since starting these Week Notes and automating what steps there are (manually breaking out long lines with links is a tedious process),
I saw 1917 on Tuesday. It’s a fascinating movie. The single shot style really draws you in and puts you right there with how bleak the WW1 battlefield must have been. It’s been a while since a film has stuck with me for several days, too,
This weekend was spent in mid-Wales, cycling a loop from Brecon to Moel Prysgau bothy and then back to Brecon again. It was a very fun trip, starting off on the road, then some gravel forestry tracks towards the bothy. On the Sunday, we decided to go off-road and ended up fording four or five rivers working our way through. After that, we got back on the road to build up some momentum, eventually heading up through a very windy Sennybridge ranges on the way back to Brecon,
This trip was helpful in making me realise where the shortcomings of my cycling fitness are. I need to do some long rides to improve my endurance and some time doing some hill-training wouldn’t hurt either,
It also made me realise how bad it is if you don’t give it enough time to pack before leaving. I forgot a bunch of things (a mug, fork, etc.) and could’ve done with some spare shoes (my feet got very wet), which wouldn’t have been a problem if I’d been able to take my time,
Anyway, for various reasons that ended up being a very busy and hectic week. Let’s hope the next is easier…

Week Notes #3

Sun, 19 Jan 2020 00:00:00 +0000

New office, new cycle route for the first half of the week. This always throws out the routine for a little bit, but you get used to it soon enough,
With this project I’m trying out a new tactic: pairing by default. I’m starting from a position of not being confident in the technologies we’re using, so that’s helping me. Plus I’m trying to solve a problem I regularly have, which is to fall deeply into a specific problem at the expense of the overall goal. I’m hoping by instead asking, “shall we pair on this?” we ask “why shouldn’t we be pairing on this?”,
This week I really enjoyed Cadence by William Van Hecke, who previously worked on OmniFocus. It fits well into what I’ve been starting to do recently: focusing on building habits (something I see come up more often), thinking about the long-scale things I want to do and reviewing it regularly. But also: keeping loads of notes (mostly with Notion, some on paper), using the crappiest notebook I could get myself to like and staying no to things that don’t fit in the bigger plan,
Related to that, this week I planned out three bikepacking trips and a weekend to brew some beer (something which without planning is impossible to make happen), but also decided to not do another handful of things. It’s a weird feeling.
Tortoise did a good series on Trees; about planting, to capture CO2 and the science behind it. I’d recommend starting with Plant a Trillion Trees.
And related to that, this week I started (and then finished!) reading Getting Started in Your Own Wood. Owning a wood is one of those ideas that comes back to me every so often, did so again recently, and this time I decided to go a bit deeper into working out what that might be like. Still much, much more to look into.

Week Notes #2

Sun, 12 Jan 2020 00:00:00 +0000

Started on a new project this week. I can’t tell you about it, other than I’ll be writing React. I’ve traditionally struggled to do enough for the knowledge to end up sticking, so I’m hoping this will help,
I completed my goals planning for the year. Every year I do this it gets easier, and this year I’m especially pleased with how specific and actionable they’ve ended up,
Two days into cycling to work ended up with a puncture on the way home. Turned out to likely be something I missed previously: I use Continental Gatorskins, which has a wire mesh and I think I ran over a pin back in December, and the mesh was exposed on the inside. I tidied that up, applied some gorilla tape as precaution and was back on the go. Let’s see how long it lasts this time!
I spent a bunch of time this week trying to improve my GitHub pull request reviewing workflow (I often do very minor edits to people’s contributions before merging to help get them over the line) and ended up instead blowing away someone’s changes. Not the most enjoyable thing to do on a Friday afternoon! I’ll write this all up once I eventually figure out the best workflow.
Finally, I got my main bike back (it’d been in storage at my parents’ since doing the London to Brighton last year), so long rides are an option again.

Week Notes #1

Sun, 05 Jan 2020 00:00:00 +0000

Several people I follow post #weeknotes, which I’ve always found a particularly interesting insight into what people are doing. I thought I’d give it a go.

I plan to do a mix of work (when I can post about it), open source, side projects and fitness & adventure topics, which I hope will keep it interesting. Right. Here we go…

I’ve been spending the past week diving deep into Administrate’s issues and open PRs, looking for trends and building a taxonomy of labels,
To do this, I ended up printing 204 index cards!
So far, I’ve closed around 40 issues and PRs (most of which were out of date or duplicates),
Dan and I entered into the Frontier 300, a coast to coast 300km off-road cycle race from the Irish Sea to the North Sea in June,
We then planned a weekend of bikepacking in the New Forest for March,
And spent a long time talking about kit,
I finished reading The Man Who Cycled The World, I enjoyed it a lot,
At the start of the year, I always spend some time setting goals and thinking about what I want to do. I’ve done so again this year, with plans for many more long cycling trips, a side project I want to launch, getting a driving license (which I’ve said I’d do for years!) and a few more things. I didn’t quite yet.

Tailwind CSS with Rails 6 and Webpacker

Sun, 10 Nov 2019 00:00:00 +0000

I don’t do many things frontend these days, but I’ve wanted to try out Tailwind for a while, and I finally had the opportunity. Alas, it was a Rails app which had no frontend at all (apart from administrate), so I need to start from the very beginning. Here’s how I did it:

Webpacker

I needed to add Webpacker, as I’d initially not done this when generating the application. You should be able to skip this if you already have a working setup. First, I added webpacker to the Gemfile and then ran the generator:

bundle exec rails webpacker:install

This command configures much of what’s needed. The Webpacker directory layout ends up looking like this:

app/javascript
├── packs
│   └── application.js
└── src

packs contain the collection of functionality processed with Webpacker to build the package of JavaScript (and CSS!) that serves your application. The default one being application.js.

In app/javascript/packs/application.js, I have the following:

require("@rails/ujs").start()
require("turbolinks").start()

And then, in the application layout, we’ll add the two hooks required to get JavaScript and styles included:

<%= stylesheet_pack_tag "application", media: "all", "data-turbolinks-track": "reload" %>
<%= javascript_pack_tag "application", "data-turbolinks-track": "reload" %>

Which loads both the JavaScript application pack and the separated stylesheet. These replace any existing stylesheet_link_tag/javascript_link_tag usage.

Tailwind CSS

Then, I added tailwindcss to the package.json using Yarn:

yarn add tailwindcss --dev

And generated a Tailwind CSS config file:

npx tailwindcss init app/javascript/src/tailwind.config.js

The configuration file starts empty and looks like this:

module.exports = {
  theme: {},
  variants: {},
  plugins: [],
}

To setup Tailwind, I’m importing the suggested imports in app/javascript/css/application.css:

@import "tailwindcss/base";
@import "tailwindcss/components";
@import "tailwindcss/utilities";

Webpacker is using postcss-import, so here I’m not using @tailwind to pull in the dependency as suggested in the documentation.

To configure PostCSS, I needed to add Tailwind in the right place (and overwrite the default location it loads the config file from):

module.exports = {
  plugins: [
    require('autoprefixer'),
    require('postcss-import'),
    require('tailwindcss')('./app/javascript/src/tailwind.config.js'),
    require('postcss-flexbugs-fixes'),
    require('postcss-preset-env')({
      autoprefixer: {
        flexbox: 'no-2009',
      },
      stage: 3,
    }),
  ],
};

Finally, I needed to configure Webpacker to expose a standalone CSS file for every environment in the default section of config/webpacker.yml:

# Extract and emit a css file
extract_css: true

(You can also to delete the same line overridden in production.)

Asset management in Rails is gradually moving towards Webpacker, and this is probably a good thing — standardising around the most common tools used in the JavaScript community. But it’s still a bit confusing if this isn’t something do you often.

Token Authentication with Rails

Fri, 08 Nov 2019 00:00:00 +0000

How can you do token authentication with Rails out of the box?

See on Giant Robots

Some resources for learning Go

Sun, 20 Oct 2019 00:00:00 +0000

I’m starting on a Go project for the first time in a while, and a few people have asked me how to starting writing Go. The last time I spent a significant amount of time writing Go was over a year ago, so while I also needed to refresh my knowledge, there have been a few new things released since.

I learn best in two phases: first reading a lot about a subject and then building something with it. The project itself is the last bit, but going into it I want to be as ready as I can to hit the ground running. That all said, what I read is essential. I don’t find programming books which walk through things like type primitives in great depth remotely engaging — I want to know just enough to be productive.

I’m assuming that you’re coming from the position of already being confident with a programming language and testing. I don’t think I’d start with learning Go from the beginning.

Learn Go with Tests

Learn Go with Tests provides a lovely introduction to the basics of Go while taking you through test-driving what you’re learning. This is the first time I’ve seen this approach, and I’ve found it’s helped me get back into the language itself.

Go in Practice

Once you’ve got an understanding of the basics of Go and have written a little, I’d recommend Go in Practice. It’ll run through some common things you’ll need to do and expose you to some patterns as you go. You might well find you can use the examples mostly as-is, which I find helps in making it feel like you’re making good progress in building something.

I read this cover to cover, but you don’t need to do that. I find once I have, it’s much easier for me to pick the book up and refer to something I then need.

Ultimate Go Programming Course

Once I’ve built a few simple projects, I then want to go into some more depth on how to structure more complex applications. I find this sort of depth doesn’t help until much later on and watching a video course helps keep me engaged.

I’ve been finding Bill Kennedy’s Ultimate Go Programming fit this well, he’s an engaging teacher and the depth is just right.

If you have an O’Reilly subscription, you can find it there.

And notes and slides on GitHub.

Go by Example

Finally, something I refer to regularly that’ll help fill in some gaps. I jump to Go by Example when I broadly know what I’m trying to do, but can’t quite figure out how to do it in Go.

It’s how I solve something like: “I need to read a file …how do I do that?”

Structuring Terraform with Remote State

Fri, 11 May 2018 00:00:00 +0000

I’ve been using Terraform for just about four years at this point, but outside working with other organisations’ configuration, I’ve not sat down and built something from scratch since the very beginning. Then, I’d only had a single environment that was configuring DNS for all of my domains or was putting together examples for other projects.

But most recently, I’ve wanted to pull this configuration together as I’ve been experimenting with Kubernetes on Google Cloud and to have something to point to when recommending to folks just starting to build out their configurations.

A common problem I’d been seeing is mistakes in one environment breaking others (nothing like building test cluster and nearly bring down production!) and when splitting by environment, struggling to share common configuration between them. When collaborating across teams, all of these problems are exacerbated.

I’d wanted to figure out an approach which would either solve or mitigate these issues and that would be a foundation to build upon. In coming up with this, I’ve taken a lot from Charity Major’s writing, The Packer Book (which filled in many gaps in my understanding when it came to how remote state could all fit together) and Travis CI’s open source configuration. I recommend exploring all of these.

Understanding Remote State

The solution to some of these problems (splitting environments, collaboration, etc.) lies in using remote state. This wasn’t a feature back when I started using Terraform but opens up some interesting patterns.

Terraform uses the state to build a map between your configuration files, the real-world resources which back them, and how the dependencies slot together. By default, this is held locally.

This works out okay for small projects, but as your configuration and those trying to collaborate on grow this begins to become unwieldy. You risk overwriting someone else’s changes or suffering through particularly horrible merge conflicts.

Remote state helps mitigate these. The backend can provide locking so that people’s changes don’t override another’s (for example, S3 with a DynamoDB table or Consul) and data can be referred to across different environments which brings with it the opportunity for smart separation of concerns when it comes to Terraform configuration.

Directory Structure

Something I’d struggled with when reading other things was understanding the bigger picture of how everything could fit together, and some limitations imposed by how Terraform works with directories. Before going over the details of each directory, here’s the overall directory structure that I’ve been working with:

modules/ # shared modules
  remote_state/
    interface.tf
    main.tf
state/ # "environment" for bootstrapping state buckets
  main.tf
  terraform.tfstate
global/ # everything which doesn't fit inside a specific environment
  main.tf
production/
  main.tf

This focuses on each of the environments, one of which is for handling the state (and the only one with a local file .tfstate), one for things like DNS or IAM details which don’t fit into a specific environment and then the first true environment, production. modules allow us to share between environments.

Terraform doesn’t descend any further than the current directory when looking for *.tf files. This means that it’s not possible to use directories for grouping (e.g.: you can’t do global/dns/main.tf and global/iam/main.tf, for example). Instead, this encourages us to push configuration into reusable modules and simplify the depth of the directory tree.

Depending on needs, it might make sense to annotate regions inside the environment name, such as europe-production, us-production, etc.

Workflow

Initially, you’ll want to work inside state to build up the resources required for storing state for the rest of your environments. Depending on your circumstances, you may find this is better implemented independently from the main Terraform configuration, but I’ve found it’s fine in the same repository. It might look something like this:

# state/main.tf
provider "aws" {
  version = "~> 1.15"
  region  = "eu-west-1"
}

module "remote_state_global" {
  source = "../modules/remote_state"

  environment = "global"
}

module "remote_state_production" {
  source = "../modules/remote_state"

  environment = "production"
}

# modules/remote_state/interface.tf
variable "prefix" {
  default     = "ng-test"
  description = "Organisation Name to prefix buckets with"
}

variable "environment" {
  default     = "development"
  description = "Environment Name"
}

output "s3_bucket_id" {
  value = "${aws_s3_bucket.remote_state.id}"
}

# modules/remote_state/main.tf
resource "aws_s3_bucket" "remote_state" {
  bucket = "${var.prefix}-remote-state-${var.environment}"
  acl    = "authenticated-read"

  versioning {
    enabled = true
  }

  tags {
    Name        = "${var.prefix}-remote-state-${var.environment}"
    Environment = "${var.environment}"
  }
}

After this, you can start building out one of the environments. I found thinking about the global environment the easiest place to start; it holds many prerequisites for the other environments, like DNS, S3 buckets and IAM details. It might look something like this:

# global/main.tf
terraform {
  backend "s3" {
    bucket = "ng-test-remote-state-global"
    key    = "terraform.tfstate"
    region = "eu-west-1"
  }
}

provider "aws" {
  version = "~> 1.15"
  region  = "eu-west-1"
}

resource "aws_s3_bucket" "example" {
  bucket = "ng-test-example"
  acl    = "private"

  tags {
    Name        = "NG: Test Example Bucket"
    Environment = "ng-test"
  }
}

As you start developing infrastructure, patterns will start to emerge. This can be build up into Terraform Modules and shared. A public registry exists which can help speed up understanding what you can do.

Kubernetes with Terraform on Google Cloud

Mon, 07 May 2018 00:00:00 +0000

I’ve been playing around with Kubernetes a bunch recently, especially with Google Kubernetes Engine (GKE). Google Cloud, and especially, their managed Kubernetes solution works really well. The best tool for configuring this sort of thing is Terraform, but the few examples I came across had lots of extra complexity which I felt distracted from what really needed to be there. This starts with a very basic implementation to bring up a cluster, through to some useful configuration for nodes which you can build on.

Setup

Firstly, if you haven’t already, make sure you have an account with Google Cloud, have installed Terraform and the Google Cloud SDK. Then, there are some configuration prerequisites to walk through.

Google Cloud organises resources under a “project”. Here, I’m going to be using terraform-gke as an example project to work with. You can set this up in the Google Cloud Console.

Next, we need to set up a few things to have access via the API. First, enable the GKE API in the Google Developer’s Console. Then, we’ll need service account credentials to use the API. In the “Credentials” section, choose “Create Credentials” and then “Service account key”.

You should then be asked to select which account to use. If GKE API access is setup correctly, you’ll see “Compute Engine default service account”. That’ll do fine for our requirements, so select that and “JSON” as the type.

Now, create a file for the provider, e.g.: google.tf:

provider "google" {
  credentials = "${file("account.json")}"
  project     = "terraform-gke"
  region      = "europe-west2"
}

The account.json file is what we’d previously downloaded. Setting the project here allows us to avoid setting it elsewhere, and the same with the region.

Finally, terraform init to have Terraform download the required provider.

Basic Cluster

resource "google_container_cluster" "primary" {
  name               = "gke-example"
  zone               = "europe-west2-a"
  initial_node_count = 2
}

This is the most basic, but workable cluster I could come up with. It has two nodes, exists in a single availability zone and uses the smallest standard machine type available (which defaults to n1-standard-1). It’s named gke-example. You could place this in the same google.tf file, but any .tf would do.

If you run terraform apply, this will bring the cluster up.

To use kubectl, the credentials can be fetched using gcloud:

gcloud config set project terraform-gke
gcloud container clusters get-credentials gke-example

…which sets the correct project, and then fetches the credentials for our cluster. Now, kubectl cluster-info should tell you something about the cluster which was just created.

Managing the Default Node Pool

GKE uses the idea of “node pools” to segregate specific types of nodes in a cluster. You might have a set of default size nodes which are applicable to most job requirements, but some high-memory nodes for larger batch jobs or perhaps some with attached GPUs for some machine learning work.

By default, creating a cluster (either through Terraform, or manually with the gcloud client will create a default node pool which you’re not able to manage yourself. This means that if you want to enable auto-scaling, custom machine types or any of the management options, this isn’t available to you through Terraform.

The solution to this is to delete the initial node pool and provide our own.

We can do this like so:

resource "google_container_cluster" "primary" {
  name                     = "gke-example"
  zone                     = "europe-west2-a"
  remove_default_node_pool = true

  node_pool {
    name = "default-pool"
  }
}

resource "google_container_node_pool" "primary_pool" {
  name       = "primary-pool"
  cluster    = "${google_container_cluster.primary.name}"
  zone       = "europe-west2-a"
  node_count = "2"

  node_config {
    machine_type = "n1-standard-1"
  }
}

We need to either provide an initial_node_count or a node_pool block, so here we just name the default node pool. This is the node which will be removed after initial cluster creation. We then go onto configuring the “primary pool” with two n1-standard-1 nodes.

Improving the Initial Nodes

One of the great things about running on a managed Kubernetes provider is having the sort of the upgrade and repair functionality you’d only get with a legion of staff in a traditional hosting environment.

For auto-upgrades, this helps ensure you’re running a recent stable build of Kubernetes across your cluster and for [auto-repair][], it ensures that every node in the cluster is functioning correctly. If a node is behaving incorrectly, the provisioned Pods will be migrated off and the node replaced with a new one.

These two features can be enabled with the management block inside the node pool:

management {
  auto_repair  = true
  auto_upgrade = true
}

GKE also supports another great feature: Auto-scaling. If the required load on the cluster can’t be handled by the amount of currently provisioned nodes, GKE will spin up more nodes (up to a limit). In the reverse, if there’s ample capacity on the cluster, GKE will remove nodes from it.

Much like the management functionality, this can be enabled inside the node pool block:

autoscaling {
  min_node_count = 2
  max_node_count = 5
}

Whilst this is something that’ll depend much on your use case, it seems wise to keep the auto-scaling max just above the peak load you’d expect to see in normal circumstances. But, of course, your mileage will vary.

Additional Node Pools & Preemptable Nodes

Not all of your workloads might warrant the same set of underlying nodes. If you have a few background jobs which require much more memory but otherwise have a workload which is primarily web requests, you might save money by adding an additional node pool for the background jobs that are configured with a different machine type.

Some of your workloads might be fault-tolerant enough to use Preemptable VMs, which are priced significantly cheaper than normal instances but only exist for 24hrs (before being re-provisioned) and might be terminated on short notice.

You can use node taints inside Kubernetes to make sure the right jobs right on the right nodes. In Terraform, these are configured just like our primary-pool above.

Book Review: How To Have Your Cake And Eat It Too: An Introduction to Service Design

Sat, 03 Mar 2018 00:00:00 +0000

Whilst I do spend most of my time writing code, I’m also a consultant. Every year I work with a bunch of small companies, founders and those who are running teams in much larger organisations. These are people who do care about the quality of the code I might produce, how well tested it is and that decisions have been made for the long term, but that’s all a bit moot if we’re not building the right thing for their customers.

Over the last few years, government and large enterprises have gradually started to come around to what’s made new upstarts so successful: focusing on the user and their needs.

For the new upstarts, some of this has been easy. They don’t have decades worth of processes and organisational structure which stands in the way of delivering on their goals. For successful new companies, they can be arranged from the customers’ perspectives from the start. If they’re not, they’re young and small enough to iterate towards this quickly (or simply just not make it). For existing organisations, this is rarely the case.

Service Design is a collection of tools which allow you to find out what customers actually value in a given service or product.

Who are your customers?
What journey does a customer take in using your service?
What are they trying to achieve?
What are their pain points?

These can be answered by building personas, drawing out customer journey maps and then doing customer interviews to fill in the gaps.

The consultant is a product of the tools in their toolbox. I’ve participated in several, and reasonably recently run a design sprint. I’ve built products using various flavours (and in various levels of dysfunction) of agile. But I’d been hunting for something out of those which is missing, and a step into this “service design” thing I’d been hearing about regularly.

J. Margus Klaar presents us with a brief overview of all of the above. At 106 pages, it’s short enough to read and digest in a couple of hours and present you with another collection of tools to have at your disposal.

They discuss how consumer actions have changed over the past fifty years and how in reaction to the world changing, people’s behaviour has changed. They discuss cases where looking at customer pain points have driven innovative solutions (like Apple and short mouse cables because they put a USB hub in the keyboard) and cases where customer relationships have completely changed from where the organisation once believed they were (e.g.: young people and not caring about their banks).

Then they go on to outline the common tools for delivering this customer focus. User journeys, personas, empathy maps, interviews, touch-points.

The book covers these quite well but doesn’t go into how to use them in detail or run them in a session or workshop. For example, a good chunk of the book discusses talking to users but seems to avoid providing example questions someone might ask. This is probably partly a requirement (the nature of those questions would emerge from the process) and also helpful in not prescribing how to use them.

One of the most important parts of this process is that it is visual and these are shown with a collection of illustrations. I can see myself copying many of them out onto a whiteboard in the future…

Broadly, I found the examples easy to understand and extract something from. But I would have liked to see some more in-depth case studies showing how changes had successfully been bought out using these tools.

If you’re looking for one resource to point to, that covers the basics and is concise, I’d recommend this.

You can find it on Amazon.

Setting up CircleCI 2.0 for Rails

Wed, 03 Jan 2018 00:00:00 +0000

CircleCI 2.0 brings some great new features, but it’s not so straightforward to get started with. Let’s walk through how to get going.

See on Giant Robots

2017

Tue, 02 Jan 2018 00:00:00 +0000

2017 was a tough one.

This year was the start of the implementation of the bad things which came about in 2016. Brexit, Trump and a further, significant, continuation of the expansion of social, racial and economic inequality. Many of these things go back a very long time and so if you’re surprised by them, you’ve not been paying enough attention.

For me, 2017 was exhausting and stressful. But also a time where I got to the stage where I started finding myself and doing the sorts of things I’ve been wanting to do.

At the end of 2016, I took over being the maintainer of administrate, a popular open source project for building admin dashboards with Rails. It’d been abandoned for much of 2016 and seeing a shitty issue irritated me enough to take over maintaining it.

I started with a big issue and pull request backlog and since this has been a relentless stream of work. With the help of my some of my amazing colleagues, Tyson Gach , Pablo Brasero (to highlight two) and a wonderful collection of external contributors (Joel, Carlos, Ben, Rich and many more) we’ve been able to remove a large number of dependencies (including by implementing our own design/grid system), add a much-wanted feature in namespace support, kept track of Rails releases and have worked hard on our handling of internationalisation.

I was able to help get // TODO London started by providing their initial event space (something we’ll be doing in 2018, too). This is possibly the best London meetup and I’m very happy that this worked out so well.

At thoughtbot, we’ve run three events of our own. One about how developers grow, one about design sprints, and finally about our company playbook. I helped with some of the filming of those events, and along with Bruno Antunes and Luke Mitchell put together a significantly updated version of our playbook …and got it printed into a real book!

I started co-hosting Build Phase, a podcast about iOS development (a thing I once did a lot of, and now sadly do a lot less) …but then successfully allowed that to tail off completely. I also joined Chad Pytel for an episode of Giant Robots as a reflection of WWDC 2017.

I was able to expand my interest in Kubernetes, Go, Docker and virtualisation, all of which I’m planning to write about in 2018.

Somewhere in here, I also managed to:

start brewing (full grain) beer and found my favourite recipe which produces a not-too-heavy dark beer (much like a mild),
view the 2017 eclipse in Wyoming with two great friends, followed by road-tripping from Colorado to San Fransisco …then getting robbed when our car was broken into,
got to go down an abandoned part of the tube and an old air-raid shelter,
moved house from North London to Docklands sort-of-area with three friends who somehow are capable of putting up with me,
and probably reading the largest amount of books I ever have before.

As I began writing this, I was unsure of what 2017 had been (apart from seeing it as a struggle). But as is often the case with reflecting on a year, it turns out to have a lot more in there than you’d realised.

Sadly, I end 2017 only-mildly avoiding a serious burnout. If it’s not the burden of maintaining open-source projects (both those through work and those of my own), trying to assist with sales and marketing efforts with work, or just work in general, it’s been a tough one.

For 2018, I’m focusing on documenting and self-management. For the former: a combination of writing and at least one meetup talk. For the latter, aiming to manage my time and well-being better to try and avoid burning out.

I want 2018 to be a year where I communicate better and mentor more.

On Giant Robots for WWDC

Wed, 07 Jun 2017 00:00:00 +0000

If starting to co-host Build Phase wasn’t enough, Chad Pytel and I did a WWDC cross-over episode of Giant Robots, the main company podcast.

We ran through the news out of the main keynote and parts of the Platforms State of the Union (the developer keynote) and what we thought about in just about an hour. It was good fun and a bit different to talk about something which is all tech news. I hope we’ll do it again!

You can find future episodes of Giant Robots at giantrobots.fm, on iTunes or where-ever you like to listen to podcasts.

Co-hosting Build Phase

Wed, 07 Jun 2017 00:00:00 +0000

I now co-host a podcast!

A few weeks ago, I agreed to take over co-hosting Build Phase, our podcast about mobile (mostly iOS) development. I’m joining Jack Nutting, who used to work with us in Stockholm.

Podcasting is something I’ve wanted to do for a long time, but without someone prompting me, I was going to keep avoiding it. Fortunately, Jack knows what he’s doing and helped make the first episode pretty painless. We’ll be releasing new episodes every other week from here on out.

In the first episode, I talk a little about my first day at thoughtbot (which was New Bamboo at the time … but not for very much longer), building Hypermedia APIs, managing client expectations, a bit about Swift evolution, and how we’re still having to deal with FTP.

You can find future episodes at buildphase.fm, on Twitter as @buildphase, on iTunes or where-ever you like to listen to podcasts.

Automating macOS using Ansible

Tue, 06 Dec 2016 00:00:00 +0000

I have a collection of macOS machines, some of which are used for services like Jenkins and others which are used for testing. These all have a consistent base configuration which I’ve been using Ansible to configure.

Ansible is being used to to drive a set of utilities that have emerged to help with some of the rougher edges of configuring macOS. In places where this abstraction isn’t available, I’m calling out to defaults write to set settings directly. There’s a few examples and a site dedicated to figuring out what’s possible.

I’m usually doing this in VMs, so I start by creating a new one using a set of Packer templates which create a base machine using a recent installer, runs updates and installs an ssh key. From here, I’ll then run an Ansible playbook that will:

Install Homebrew,
Set the dock position, size and contents,
Show devices on the desktop, align files to the grid, show hidden files, etc.

It ends up looking like this:

Ansible Configured macOS VM

Some of these settings are the same on my own Macs, but most of these (like the dock and it’s contents) are all about making it as easy as possible to interact with it over a screen sharing session and having something consistent across the board.

To achieve these, I do this:

---
- name: Base OS X configuration
  hosts: all
  roles:
    - geerlingguy.homebrew
    - base

This is the base playbook. It’s held in a central ansible repo, but here we’re installing Homebrew (using Jeff Geerling’s Homebrew Role) and then executing the base role. The base role looks like this:

---
- name: Install base utilities
  homebrew:
    name: "{{ item }}"
  with_items:
    - m-cli
    - dockutil

- name: Remove all items from the Dock
  shell: /usr/local/bin/dockutil --remove all

- name: Set the default Dock items
  shell: "/usr/local/bin/dockutil --add {{ item }} --no-restart"
  with_items:
    - /Applications/Safari.app
    - "\"/Applications/App Store.app\""
    - "\"/Applications/System Preferences.app\""
    - /Applications/Utilities/Terminal.app
    - /Applications/Utilities/Console.app
    - "/Applications --section others"
    - "~/Downloads --section others"

- name: Reduce the size of the Dock to 30 points
  shell: defaults write com.apple.dock tilesize -int 30

- name: Show the Dock on the left-hand side
  shell: /usr/local/bin/m dock position LEFT

- name: Disable the Screensaver
  shell: defaults write com.apple.screensaver idleTime 0

- name: Arrange Files by Kind
  shell: |
    /usr/libexec/PlistBuddy -c "Set :DesktopViewSettings:IconViewSettings:arrangeBy kind" ~/Library/Preferences/com.apple.finder.plist
    /usr/libexec/PlistBuddy -c "Set :StandardViewSettings:IconViewSettings:arrangeBy kind" ~/Library/Preferences/com.apple.finder.plist

- name: Set the Grid Spacing for Files
  shell: |
    /usr/libexec/PlistBuddy -c "Set :DesktopViewSettings:IconViewSettings:gridSpacing 54" ~/Library/Preferences/com.apple.finder.plist
    /usr/libexec/PlistBuddy -c "Set :StandardViewSettings:IconViewSettings:gridSpacing 30" ~/Library/Preferences/com.apple.finder.plist

- name: Use Smaller Icons
  shell: |
    /usr/libexec/PlistBuddy -c "Set :DesktopViewSettings:IconViewSettings:iconSize 48" ~/Library/Preferences/com.apple.finder.plist
    /usr/libexec/PlistBuddy -c "Set :StandardViewSettings:IconViewSettings:iconSize 64" ~/Library/Preferences/com.apple.finder.plist

- name: Show ~/Library
  shell: chflags nohidden ~/Library

- name: Show Drives on the Desktop
  shell: defaults write com.apple.finder ShowHardDrivesOnDesktop -bool true

- name: Show External Drives on the Desktop
  shell: defaults write com.apple.finder ShowExternalHardDrivesOnDesktop -bool true

- name: Show Removable Media on the Desktop
  shell: defaults write com.apple.finder ShowRemovableMediaOnDesktop -bool true

- name: Show Hidden Files
  shell: defaults write com.apple.finder AppleShowAllFiles -bool true

- name: Show File Extensions
  shell: defaults write NSGlobalDomain AppleShowAllExtensions -bool true

- name: Show the Status Bar in Finder
  shell: defaults write com.apple.finder ShowStatusBar -bool true

- name: Show the Path Bar in Finder
  shell: defaults write com.apple.finder ShowPathbar -bool true

- name: Restart Finder
  shell: killall Finder

It relies on m-cli and dockutil to reconfigure the dock and then relies on an Ansible form of some of my existing dotfiles defaults.sh script. These commands need to be run inside a shell, so they’re executed through shell (only through command will cause them to fail).

Of note, I found that I now need to set the ordering of Finder items and their size before showing additional devices. This wasn’t the case before OS X 10.11. (We need to use PlistBuddy because it’s nested, but this doesn’t seem to be the reason for why.)

Building Virtual Machines with Packer on ESXi 6

Sat, 03 Dec 2016 00:00:00 +0000

Over the past couple of weeks, I’ve been building up a box running ESXi 6 to host a bunch of virtual machines. I documented the initial configuration in a previous post, but this goes further and attempts to automate much of this using Packer. My end goal is to have the complete configuration held in a repository to make rebuilding (or adding new boxes) as painless as possible and also to provide an example for others implementing something similar.

I’ve put together a (public) repository which contains templates for set of base images which can be cloned to speed up provisioning new VMs. These templates assume you’re able to acquire an IP via DHCP.

Configuring the ESXi Host

Our ESXi host needs a little bit of configuration to allow Packer to work. Packer communicates over SSH, so first we need to open that. Secondly, we’ll enable an option to discover Guest IPs from the Host and then finally allow VNC connections remotely.

Enable SSH

Inside the web UI, navigate to “Manage”, then the “Services” tab. Find the entry called: “TSM-SSH”, and enable it.

You may wish to enable it to start up with the host by default. You can do this inside the “Actions” dropdown (it’s nested inside “Policy”).

Enable “Guest IP Hack”

Run the following command on the ESXi host:

esxcli system settings advanced set -o /Net/GuestIPHack -i 1

This allows Packer to infer the guest IP from ESXi, without the VM needing to report it itself.

Open VNC Ports on the Firewall

Packer connects to the VM using VNC, so we’ll open a range of ports to allow it to connect to it.

First, ensure we can edit the firewall configuration:

chmod 644 /etc/vmware/firewall/service.xml
chmod +t /etc/vmware/firewall/service.xml

Then append the range we want to open to the end of the file:

<service id="1000">
  <id>packer-vnc</id>
  <rule id="0000">
    <direction>inbound</direction>
    <protocol>tcp</protocol>
    <porttype>dst</porttype>
    <port>
      <begin>5900</begin>
      <end>6000</end>
    </port>
  </rule>
  <enabled>true</enabled>
  <required>true</required>
</service>

Finally, restore the permissions and reload the firewall:

chmod 444 /etc/vmware/firewall/service.xml
esxcli network firewall refresh

If you’d like to, these changes can be persisted between reboots by modifying /etc/rc.local.d/local.sh, as described in this VMware knowledge base article. (Thanks to JP Toto who pointed this out!)

Running Packer

A Packer template for running on ESXi is very similar to one which runs locally, but we’re not able to use the built-in HTTP server and we specify the host on which it runs. Here’s an example using Ubuntu 16.04, with a preseed.cfg provided by mounting a floppy:

variables.json:

{
  "esxi_host": "",
  "esxi_datastore": "primary",
  "esxi_username": "",
  "esxi_password": ""
}

ubuntu-1604-base.json:

{
  "builders": [{
    "name": "ubuntu-1604-base",
    "vm_name": "ubuntu-1604-base",
    "type": "vmware-iso",
    "guest_os_type": "ubuntu-64",
    "tools_upload_flavor": "linux",
    "headless": false,

    "iso_url": "http://releases.ubuntu.com/xenial/ubuntu-16.04-server-amd64.iso",
    "iso_checksum": "b8b172cbdf04f5ff8adc8c2c1b4007ccf66f00fc6a324a6da6eba67de71746f6",
    "iso_checksum_type": "sha256",

    "ssh_username": "nullgrid",
    "ssh_password": "nullgrid",
    "ssh_timeout": "15m",

    "disk_type_id": "thin",

    "floppy_files": [
      "preseed/ubuntu.cfg"
    ],

    "boot_command": [
      "<enter><wait><f6><esc><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
      "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
      "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
      "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
      "/install/vmlinuz noapic ",
      "preseed/file=/floppy/ubuntu.cfg ",
      "debian-installer=en_US auto locale=en_US kbd-chooser/method=us ",
      "hostname={{ .Name }} ",
      "fb=false debconf/frontend=noninteractive ",
      "keyboard-configuration/modelcode=SKIP keyboard-configuration/layout=USA ",
      "keyboard-configuration/variant=USA console-setup/ask_detect=false ",
      "grub-installer/bootdev=/dev/sda ",
      "initrd=/install/initrd.gz -- <enter>"
    ],

    "shutdown_command": "echo 'shutdown -P now' > shutdown.sh; echo 'nullgrid'|sudo -S sh 'shutdown.sh'",

    "remote_type": "esx5",
    "remote_host": "{{user `esxi_host`}}",
    "remote_datastore": "{{user `esxi_datastore`}}",
    "remote_username": "{{user `esxi_username`}}",
    "remote_password": "{{user `esxi_password`}}",
    "keep_registered": true,

    "vmx_data": {
      "ethernet0.networkName": "VM Network"
    }
  }]
}

preseed.cfg:

#
# Based upon: https://help.ubuntu.com/12.04/installation-guide/example-preseed.txt
#

# localisation
d-i debian-installer/locale string en_US.utf8
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/layoutcode string us

# networking
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string unassigned-hostname
d-i netcfg/get_domain string unassigned-domain
d-i netcfg/wireless_wep string

# apt mirrors
d-i mirror/country string manual
d-i mirror/http/hostname string archive.ubuntu.com
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string

# clock and time zone
d-i clock-setup/utc boolean true
d-i time/zone string GMT
d-i clock-setup/ntp boolean true

# partitioning
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
# fix: http://serverfault.com/questions/189328/ubuntu-kickstart-installation-using-lvm-waits-for-input
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# users
d-i passwd/user-fullname string Null Grid
d-i passwd/username string nullgrid
d-i passwd/user-password password nullgrid
d-i passwd/user-password-again password nullgrid
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false

# packages
tasksel tasksel/first multiselect standard, ubuntu-server
d-i pkgsel/install-language-support boolean false
d-i pkgsel/include string openssh-server nfs-common curl git-core
d-i pkgsel/upgrade select full-upgrade
d-i pkgsel/update-policy select none
postfix postfix/main_mailer_type select No configuration

# boot loader
d-i grub-installer/only_debian boolean true

# hide the shutdown notice
d-i finish-install/reboot_in_progress note

It can be run like so:

packer build -var-file variables.json ubuntu-1604-base.json

The packer-esxi repo contains a much more comprehensive example, which I’d suggest cloning and using. The Debian example foregoes the floppy and instead fetches a file from Amazon S3 as the installer no longer supports fetching the preseed from a floppy.

I’m using these templates to provide a base image, which I clone and configure to their needs. This is pretty similar to cloud providers like AWS EC2 or Digital Ocean.

Persistent SSH Keys with ESXi 6

Wed, 30 Nov 2016 00:00:00 +0000

ESXi is a funny beast when it comes to SSH keys, and there’s a lot of misinformation about on how to configure them persistently, not to mention security FUD.

ESXi uses an in-memory only filesystem for the bootable portion of it, so following the published guide will only get you so far. You’ll find that after rebooting, your changes will no longer be there.

You can work around this limitation by rebuilding the keys on boot using /etc/rc.local.d/local.sh. Somewhere before the exit 0, add a modified version of the following:

mkdir -p /etc/ssh/keys-<username>
echo "ssh-rsa AAAAB3..." > /etc/ssh/keys-<username>/authorized_keys
chmod 700 -R /etc/ssh/keys-<username> && chmod 600 /etc/ssh/keys-<username>/authorized_keys
chown -R <username> /etc/ssh/keys-<username>

where <username> and the value of the public key are replaced with actual values.

Once you’ve done this, ensure it’s backed up by running: /sbin/auto-backup.sh.

auto-backup.sh runs every hour and backs up certain known files which are marked with the “sticky bit” (chmod +t file). These are then restored on reboot. auto-backup.sh only knows about certain files, so we’re adding this to one it knows about.

This pattern can used in a similar way for other settings which need to stay around after reboots.

VMware seem to not recommend enabling SSH by default, and that’s fine. If you disagree with that, you can enable it inside the Web UI by going to “Manage”, “Services” finding “TM-SSH” and selecting “Actions”, “Policy”, enable “Start and stop with host”.

Configuring VMware ESXi 6 on Hetzner

Mon, 28 Nov 2016 00:00:00 +0000

Hetzner are a relatively low-cost hosting provider based out of Germany. They provide a range of powerful but cheap dedicated servers which make a good platform for experimenting with VMWare ESXi or other virtualisation software.

However, their networking is based around statically routing MAC addresses which makes it both a bit different to more common VLAN setups and harder to get up and running. They do have a canonical guide to configuring this, but I found my knowledge of networking a lacking from what it assumes and so I found it a little hard to follow and caught myself in a few traps, so I thought I’d write up some notes.

Prerequisites

After you’ve ordered your dedicated server, go into Robot and order your desired subnet. When doing this, you’ll first want to request a standalone “additional IP” and then have the subnet (of any size you’ll use) statically routed to it. You may wish to mention it’s for ESXi here to make this clear.

ESXi doesn’t support routing out of the box, so we’ll use the additional IP on a router VM to provide a gateway for our subnet.

Installing ESXi 6

If you’ve already done the IP configuration, or whilst you wait you can proceed to install (or jump to the next section if you’ve done this).

Using the LARA console, mount the most recent available version of ESXi that you can. I used VMware-VMvisor-Installer-6.0.0.update02-3620759.x86_64.iso, which is available under bootimages/vmware

I’d recommend avoiding passwords with special characters as these didn’t seem to pass through on my first attempt and I needed to re-install to get root access.

Configuring Networking

IP Addressing

We should now have two standalone IP addresses and a subnet. Let’s use these as an example:

Main IP: 140.201.25.139
Additional IP: 140.201.300.61
Subnet: 140.201.2.32/28

The subnet has usable addresses from: 140.201.2.33 to 140.201.2.47, so .32 defines the network and .47 is the broadcast address.

I’ve used the first usable IP as the gateway for all of the VMs (the second IP configured below in the router). Partly this is because this IP will be pre-filled out by the Debian installer.

We’ll then be able to create 12 VMs with public IP addresses, once we account for those we’ll need to create to get it working.

Router VM

When you first login to ESXi, you’ll find there are two networks. One is the “Management Network” and the other is the “VM Network”. The “Management Network” relates to the initial IP that was configured with your server. The “VM Network” is the network available to VMs on this. We’ll not quite be able to use this directly.

To use our subnet, we’ll want to create a new vSwitch and Port Group. Under “Networking”, create a new vSwitch (I called mine subnet0) and a Port Group to go with it (I called mine Subnet 0). We’ll create a router VM which acts as a gateway between this network and the “VM Network”.

Subnet 0 Network Topology

To do this, create a new VM (I’m using Debian 8, configured with 512MB RAM and a 32GB disk image) using the “VM Network” which came by default. You should manually set the MAC address to the one provided inside Robot for the “additional IP”. DHCP will work and so the installer should autoconfigure networking.

Once this is up and running, shut it down and add a new network adapter. This should be connected to the subnet you created above. All other settings can be the default. Bring the VM back up and configure /etc/networks/interfaces to look something like this:

auto eth0
iface eth0 inet static
  address 140.201.300.61
  gateway 140.201.300.57
  netmask 255.255.255.248
  dns-nameservers 213.133.98.98 213.133.99.99

auto eth1
iface eth1 inet static
  address 140.201.2.33
  netmask 255.255.255.240

The eth1 network needs neither a gateway (because we’re the gateway for the network) or nameservers (we’ll be passing packets over without much interest).

Finally for the router VM, configure IP forwarding so that packets will travel through the router:

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

You can persist this by adding:

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

to ` /etc/sysctl.conf`.

Now you can create a VM using one of your subnet IPs. Specify the IP address and netmask, then the gateway IP as the one on the router connected to our subnet vSwitch (140.201.2.33 in this case). Your configuration will look something like this:

auto eth0
iface eth0 inet static
  address 140.201.2.34
  gateway 140.201.2.33
  netmask 255.255.255.240
  dns-nameservers 213.133.98.98 213.133.99.99

Further Subnets

This isn’t something I’ve yet tried, but if you wanted to configure a second (or third, etc.) subnet you’d go about this in a similar manner.

Get the new subnet routed onto the “additional IP”.
Create another vSwitch and Port Group (perhaps, subnet1).
Add another network adapter to the router VM.
Connect this new network adapter to the new vSwitch.
Configure VMs in the same way.

You should now be able to bring up additional VMs and use the rest of your assigned subnet!

Resize all Safari windows with AppleScript

Mon, 30 May 2016 00:00:00 +0000

I have a particular preference for the size of my browser window, and I like them all to be consistent. I divide my time between a 13” MacBook Pro and the same device attached to a 27” Thunderbolt Display. Usually this works out great, but over time, my browser windows will all end up being slightly different. Some will be a bit shorter, some wider, some off centre.

This is a pain to maintain manually, so in drops some AppleScript. I can reset all visible windows to be the same size (across all Spaces) and be centred on the screen. This is my goal:

Resized Safari Window

Here’s the AppleScript:

set height to 575
set width to 1212

tell application "Finder"
    set screen_resolution to bounds of window of desktop
end tell

set screen_width to item 3 of screen_resolution
set screen_height to item 4 of screen_resolution

tell application "Safari"
    activate
    reopen
    set y to (screen_height - height) / 2 as integer
    set x to (screen_hidth - width) / 2 as integer
    set the bounds of every window to {x, y, width + x, height + y}
end tell

The first two variables are set the size I’d like the window to be (it’s in points). Over time, I’ll adjust this, so it’s best pulled out at the top.

The script then goes on to calculate the x and y positions according to the current screen resolution (this ensures it’s centred) and then applies this to every window.

This is a reasonably simple script, but it took a while to figure out. Such is the nature of AppleScript.

Note: This doesn’t support multiple displays. The reported bounds returned for the desktop include all of the displays combined together. We’d need to figure out which windows belonged to which display and then calculate it that way. This is a reasonable shortcoming for my usage.

A script is great, but not the most convenient. For this, I turned to Alfred and put together a quick workflow to invoke it with: “Reset Safari Windows”:

Resized Safari Window Workflow

You can download the workflow here.

Rails: Importing Local Data Dumps

Mon, 11 Apr 2016 00:00:00 +0000

I regularly work with production data (obfuscated when necessary) with local Rails apps. It’s much better to work with real-world data when building out projects as it allows you to make decisions according to what will really happen.

But, importing them can be a bit of a pain. I’d previously been writing out the command for pg_import manually. So, I wrote a quick Rake task to do this for me:

namespace :db do
  desc "Import a given file into the database"
  task :import, [:path] => :environment do |_t, args|
    dump_path = args.path
    connection_config = ActiveRecord::Base.connection_config

    case connection_config[:adapter]
    when "postgresql"
      system("PGPASSWORD=#{connection_config[:password]} pg_restore " \
        "--verbose --clean --no-acl --no-owner " \
        "--username=#{connection_config[:username]} " \
        "-d #{connection_config[:database]} #{dump_path}")
    when "mysql", "mysql2"
      system("mysql -u #{connection_config[:username]} " \
        "-p#{connection_config[:password]} " \
        "#{connection_config[:database]} < #{dump_path}")
    else
      raise NotImplementedError, "An importer hasn't been implemented for: " \
        "#{connection_config[:adapter]}"
    end
  end
end

This uses the database configuration for the current Rails environment to invoke the standard tools for importing into Postgres and MySQL. If a different database adaptor is being used, it’ll safely fail with a message.

Postgres doesn’t provide a command line manner in which to provide the password, so we’re using the (not-recommended) approach of using an environment variable. In our case, this shouldn’t be an issue as this is intending to be imported into a local development database.

Rake’s argument handling can appear a little strange (because you can run multiple commands at once, using OptParser becomes a little bit more complex), but it’s executed like so:

bundle exec rake db:import[latest.dump]

…where latest.dump is a file in the same directory as you’re executing the command.

This version supports just Postgres and MySQL as I don’t come across others so regularly, more database adaptors are left as an exercise for the reader.

Docker via Homebrew

Sun, 17 Jan 2016 00:00:00 +0000

On the site, Docker recommend using the Docker Toolbox to get up and running with it. I’m personally not much of a fan of these “platform” installers; in attempting to provide a common solution for most, people like me end up fighting with it. Fortunately, this can all be done standlone through Homebrew. Here’s how to do it:

brew install docker docker-machine

This pulls in the docker executable and the new(er) way to managing machines (as you can’t use Docker directly on OS X): docker-machine.

Creating a Local Docker Machine

This will pull in a recent boot2docker image (platform dependent). You’ll then need to set the environment variables so that Docker knows to use it.

docker-machine create main
eval $(docker-machine env main)

The eval will set the following bits of configuration (although it’d vary for you):

export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.237.131:2376"
export DOCKER_CERT_PATH="/Users/nickcharlton/.docker/machine/machines/main"
export DOCKER_MACHINE_NAME="main"

…with VMware Fusion

But, I prefer to use VMware Fusion for running VMs, so I do it this way instead:

docker-machine create --driver vmwarefusion main
eval $(docker-machine env main)

There we go. Docker that can be done nicely as with any other tool through Homebrew.

Mirroring Bazaar Repositories with Git

Tue, 21 Jul 2015 00:00:00 +0000

Ubuntu uses bazaar as it’s source control system, with Launchpad as the hosting service. This is fine, apart from when you want to include something which is maintained in Git or otherwise doesn’t have good enough support for it. In my case, I wanted to mirror a repository on GitHub so that it could be included elsewhere as a submodule.

Bazaar provides dpush for this, but it wasn’t so obvious at first sight how to work with it. (Replace example with the appropriate name):

1. Ensure You’ve Got the Git Plugin

The Git plugin provides the interoperability you’ll need. Some package repositories will have as bzr-git (like Ubuntu does). You may need to install it manually.

Something like this, perhaps:

cd ~/.bazaar/plugins
bzr branch lp:bzr-git git

(You may need to create the plugins directory.)

You’ll probably also have to install dulwich: pip install dulwich.

1. Clone the Bazaar Repository

bzr branch lp:example example

2. Create a Git Repository

You can create this where ever you usually create your hosted repositories.

All you’ll need is the URL given to you to create the repo, in this case:

ssh://git@github.com/nickcharlton/example.git

I was unable to use this directly, so instead gave bzr a hint by adjusting it to:

git+ssh://git@github.com/nickcharlton/example.git

4. Push the Clone to the Remote

The final step is to push the repository, including using the conventional branch:

bzr dpush -v git+ssh://git@github.com/nickcharlton/example.git,branch=master

With Python installed via brew, I needed to ensure $PYTHONPATH was set correctly, to the following:

export PYTHONPATH="/usr/local/lib/python2.7/site-packages"

You should now have a full copy of all of the original bazaar history, but now all configured with git.

Terraform: AWS VPC with Private and Public Subnets

Wed, 08 Jul 2015 00:00:00 +0000

I’m currently in the process of designing out the architecture for a project which is soon to be hosted on AWS. My aim has been to isolate groups of components (like Redis and/or Postgres instances) from other groups (like web application servers) as much as possible to restrict access.

AWS provides VPC (Virtual Private Cloud) to do such a thing, but it’s quite fiddly to get going. This is where Terraform steps in. Terraform is a tool that allows you to automate your interactions with services like AWS (and indeed others) and record the state, giving you the ability to place your infrastructure in source control.

In the AWS documentation, this is given as an example: “Scenario 2” and this post will show how this can be replicated using Terraform. I’d recommend you familiarise yourself with the AWS documentation’s version first and also get a good understanding of Terraform from it’s documentation. You can find a repository with the full example over on GitHub.

Terraform, Briefly

Terraform abstracts out the interaction with various infrastructure services (AWS, Digital Ocean, OpenStack, etc) and provides a unified configuration format for it.

As the Terraform docs point out, the best way to show it is through examples, but a few important points:

.tf files are all combined to provide the full configuration.
- This gives us a handy way to break the configuration up into thematic sections.
.tfstate and .tfstate.backup holds the last-known state of the infrastructure, you’ll want to store this, too.
.tfvars contain values for the declared variables, typically called: terraform.tfvars.

To see what Terraform will do: terraform plan -var-file terraform.tfvars

To bring up the infrastructure we’ll run: terraform apply -var-file terraform.tfvars

And to destroy it, we’ll run: terraform destroy -var-file terraform.tfvars

Infrastructure Aims

AWS's Scenario 2 Network Diagram

The network diagram above is the best demonstration of what’ll be implemented:

The private subnet is inaccessible to the internet (both in and out).
The public subnet is accessible; just dependent on the configuration of the security groups. Elastic IPs can be associated with instances in here.
Instances in the public subnet can access instances in the private subnet (also dependent on security groups) because they’re in the same VPC (this is enabled by the route tables).
Routing is handled like this:
- Private subnet is routed through the NAT instance.
- Public subnet is routed directly to the internet gateway.

Implementing in Terraform

In my implementation, I’ve opted to split the configuration into files broken by what they do. I feel this is the one of the great strengths of the Terraform configuration format, and, I find makes it much easier to track changes.

Additionally, I’ve liberalised some of the security groups compared to the original (allowing connections via SSH from anywhere and allowing ICMP packets). This is also specified to use Ubuntu 14.04 instances in the eu-west-1 (Ireland) region

`variables.tf`

This file is typically called variables.tf by convention. It’s typically full of environment specific configuration, like which ami to use and which credentials to use.

variable "aws_access_key" {}
variable "aws_secret_key" {}
variable "aws_key_path" {}
variable "aws_key_name" {}

variable "aws_region" {
    description = "EC2 Region for the VPC"
    default = "eu-west-1"
}

variable "amis" {
    description = "AMIs by region"
    default = {
        eu-west-1 = "ami-f1810f86" # ubuntu 14.04 LTS
    }
}

variable "vpc_cidr" {
    description = "CIDR for the whole VPC"
    default = "10.0.0.0/16"
}

variable "public_subnet_cidr" {
    description = "CIDR for the Public Subnet"
    default = "10.0.0.0/24"
}

variable "private_subnet_cidr" {
    description = "CIDR for the Private Subnet"
    default = "10.0.1.0/24"
}

Outside of the authentication credentials, here we’ve configured the default AMI and the default region to use. Then the CIDR blocks for the VPC overall and the two subnets contained within it.

`terraform.tfvars`

This is the file which is passed into each command, and provides the “secrets” and more specific values.

aws_access_key = ""
aws_secret_key = ""
aws_key_path = "~/.ssh/aws.pem"
aws_key_name = "aws"

(The key should be already configured with AWS.)

`aws.tf`

This configures the provider, note how it uses string interpolation to pull out the variables from previously:

provider "aws" {
    access_key = "${var.aws_access_key}"
    secret_key = "${var.aws_secret_key}"
    region = "${var.aws_region}"
}

`vpc.tf`

This is the largest of the lot and configures both the VPC, the NAT instance, the two subnets and the relevant security groups.

resource "aws_vpc" "default" {
    cidr_block = "${var.vpc_cidr}"
    enable_dns_hostnames = true
    tags {
        Name = "terraform-aws-vpc"
    }
}

resource "aws_internet_gateway" "default" {
    vpc_id = "${aws_vpc.default.id}"
}

/*
  NAT Instance
*/
resource "aws_security_group" "nat" {
    name = "vpc_nat"
    description = "Allow traffic to pass from the private subnet to the internet"

    ingress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["${var.private_subnet_cidr}"]
    }
    ingress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["${var.private_subnet_cidr}"]
    }
    ingress {
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
        from_port = -1
        to_port = -1
        protocol = "icmp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    egress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["${var.vpc_cidr}"]
    }
    egress {
        from_port = -1
        to_port = -1
        protocol = "icmp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    vpc_id = "${aws_vpc.default.id}"

    tags {
        Name = "NATSG"
    }
}

resource "aws_instance" "nat" {
    ami = "ami-30913f47" # this is a special ami preconfigured to do NAT
    availability_zone = "eu-west-1a"
    instance_type = "m1.small"
    key_name = "${var.aws_key_name}"
    vpc_security_group_ids = ["${aws_security_group.nat.id}"]
    subnet_id = "${aws_subnet.eu-west-1a-public.id}"
    associate_public_ip_address = true
    source_dest_check = false

    tags {
        Name = "VPC NAT"
    }
}

resource "aws_eip" "nat" {
    instance = "${aws_instance.nat.id}"
    vpc = true
}

/*
  Public Subnet
*/
resource "aws_subnet" "eu-west-1a-public" {
    vpc_id = "${aws_vpc.default.id}"

    cidr_block = "${var.public_subnet_cidr}"
    availability_zone = "eu-west-1a"

    tags {
        Name = "Public Subnet"
    }
}

resource "aws_route_table" "eu-west-1a-public" {
    vpc_id = "${aws_vpc.default.id}"

    route {
        cidr_block = "0.0.0.0/0"
        gateway_id = "${aws_internet_gateway.default.id}"
    }

    tags {
        Name = "Public Subnet"
    }
}

resource "aws_route_table_association" "eu-west-1a-public" {
    subnet_id = "${aws_subnet.eu-west-1a-public.id}"
    route_table_id = "${aws_route_table.eu-west-1a-public.id}"
}

/*
  Private Subnet
*/
resource "aws_subnet" "eu-west-1a-private" {
    vpc_id = "${aws_vpc.default.id}"

    cidr_block = "${var.private_subnet_cidr}"
    availability_zone = "eu-west-1a"

    tags {
        Name = "Private Subnet"
    }
}

resource "aws_route_table" "eu-west-1a-private" {
    vpc_id = "${aws_vpc.default.id}"

    route {
        cidr_block = "0.0.0.0/0"
        instance_id = "${aws_instance.nat.id}"
    }

    tags {
        Name = "Private Subnet"
    }
}

resource "aws_route_table_association" "eu-west-1a-private" {
    subnet_id = "${aws_subnet.eu-west-1a-private.id}"
    route_table_id = "${aws_route_table.eu-west-1a-private.id}"
}

The NAT instance is a special Amazon Linux AMI which handles the routing correctly. It can be found on the AWS Marketplace using the aws command line tool by doing something like this:

aws ec2 describe-images --filter Name="owner-alias",Values="amazon" --filter
Name="name",Values="amzn-ami-vpc-nat*"

Like all AMIs, there’s specific images for each region.

`public.tf`

This file (and the next) is where I’d split the file structure into more specific sections (like a web.tf or application.tf) instead of just the single public.tf in a larger implementation, as this is where much of the detail will be described. It works good enough here, as we’ll just be implementing a single instance and security group in each subnet.

/*
  Web Servers
*/
resource "aws_security_group" "web" {
    name = "vpc_web"
    description = "Allow incoming HTTP connections."

    ingress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
        from_port = -1
        to_port = -1
        protocol = "icmp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    egress { # SQL Server
        from_port = 1433
        to_port = 1433
        protocol = "tcp"
        cidr_blocks = ["${var.private_subnet_cidr}"]
    }
    egress { # MySQL
        from_port = 3306
        to_port = 3306
        protocol = "tcp"
        cidr_blocks = ["${var.private_subnet_cidr}"]
    }

    vpc_id = "${aws_vpc.default.id}"

    tags {
        Name = "WebServerSG"
    }
}

resource "aws_instance" "web-1" {
    ami = "${lookup(var.amis, var.aws_region)}"
    availability_zone = "eu-west-1a"
    instance_type = "m1.small"
    key_name = "${var.aws_key_name}"
    vpc_security_group_ids = ["${aws_security_group.web.id}"]
    subnet_id = "${aws_subnet.eu-west-1a-public.id}"
    associate_public_ip_address = true
    source_dest_check = false


    tags {
        Name = "Web Server 1"
    }
}

resource "aws_eip" "web-1" {
    instance = "${aws_instance.web-1.id}"
    vpc = true
}

`private.tf`

/*
  Database Servers
*/
resource "aws_security_group" "db" {
    name = "vpc_db"
    description = "Allow incoming database connections."

    ingress { # SQL Server
        from_port = 1433
        to_port = 1433
        protocol = "tcp"
        security_groups = ["${aws_security_group.web.id}"]
    }
    ingress { # MySQL
        from_port = 3306
        to_port = 3306
        protocol = "tcp"
        security_groups = ["${aws_security_group.web.id}"]
    }

    ingress {
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["${var.vpc_cidr}"]
    }
    ingress {
        from_port = -1
        to_port = -1
        protocol = "icmp"
        cidr_blocks = ["${var.vpc_cidr}"]
    }

    egress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    vpc_id = "${aws_vpc.default.id}"

    tags {
        Name = "DBServerSG"
    }
}

resource "aws_instance" "db-1" {
    ami = "${lookup(var.amis, var.aws_region)}"
    availability_zone = "eu-west-1a"
    instance_type = "m1.small"
    key_name = "${var.aws_key_name}"
    vpc_security_group_ids = ["${aws_security_group.db.id}"]
    subnet_id = "${aws_subnet.eu-west-1a-private.id}"
    source_dest_check = false

    tags {
        Name = "DB Server 1"
    }
}

Bring it up & Testing

Once you’ve defined the environment configuration files, it’s time to bring it up. You can do that by firing off:

terraform apply -var-file terraform.tfvars

That’ll bring up the VPC, all of the security groups, the NAT instance and finally the Web and DB instances. You’ll then have a set of machines like so:

Instance Name	Private IP	Public IP
VPC NAT	10.0.0.210	52.16.161.59
Web Server 1	10.0.0.37	52.16.185.18
DB Server 1	10.0.1.22	n/a

(Obviously, with different values).

Our security groups allow us to connect through the NAT intuance to the other instances inside the private subnet. I’d recommend configuring a separate instance for this, which is typically known as a “bastion”. But, to keep it simple, adjusting the security groups is enough for this case.

Connect to the NAT instance, also using agent forwarding so we can reuse the session:

ssh -i ~/.ssh/key_name.pem -A ec2-user@52.16.161.59

And then inside the NAT instance, you’ll be able to connect to either of the other two instances:

# Web Server 1
ssh ubuntu@10.0.0.37
# DB Server 1
ssh ubuntu@10.0.1.22

The nuance of AWS security groups are out of scope for this article, but in general, you’ll need to define a “egress” route for packets leaving an instance and an “ingress” to define them going into (another) one. This gives you a pretty powerful way to lock down access to specific instance groups.

But, that’s it. You’ve now defined the state of an infrastructure in a group of files, bought it all up in a single command and can track future changes with it. All following Amazon’s example “Scenario 2”.

Installing Jenkins on OS X Yosemite

Tue, 07 Jul 2015 00:00:00 +0000

Jenkins is a continuous integration (CI) server written in Java. It’s a pretty common solution for self-hosted CI servers.

A lot of the documentation for installing on OS X is a little old (OS X has changed a lot when it comes to say, Java, in the last few years) and it seemed a good plan to write up something a bit newer.

I host a Jenkins instance on a hosted Mac mini with Macminicolo. In addition to Yosemite, it’s also got the OS X Server package installed, Open Directory (which is Apple’s name for LDAP) configured and a few other tools. It’s typically used for boxes builds.

I’m assuming you’ll be installing Jenkins via Homebrew as the Jenkins installer does some odd things around how the user is handled. I’m also assuming you’re connected via VNC/Screen Sharing, as often it requires a GUI.

1. Install Java

You likely don’t have Java installed yet, so open a terminal and enter java to request the install. Follow the instructions.

2. Create a user for Jenkins

It’s best to run Jenkins as it’s own user (it can then be limited in the permissions it has), and you’ll want to create a standard (full) user for it.

You can do this through System Preferences, the Server Manager or the command line.

For a local user:

# create an applications group
dseditgroup -o create -n . -u username -p -r ‘Applications’ applications
# get the id for that group
sudo dscl . -read /Groups/applications
# find a unique identifier to give the user
sudo dscl . -list /Users UniqueID
# create the jenkins user
sudo dscl . -create /Users/jenkins
sudo dscl . -create /Users/jenkins PrimaryGroupID 505
sudo dscl . -create /Users/jenkins UniqueID 1026
sudo dscl . -create /Users/jenkins UserShell /bin/bash
sudo dscl . -create /Users/jenkins RealName "Jenkins"
sudo dscl . -create /Users/jenkins NFSHomeDirectory /Users/jenkins
sudo dscl . -passwd /Users/jenkins
# create and set the owner of the home directory
sudo mkdir /Users/jenkins
sudo chown -R jenkins /Users/jenkins

(Replace username with the username of an admin user).

For an Open Directory user (replace the IP with the location of the relevant OD tree):

# create an applications group
dseditgroup -o create -n /LDAPv3/127.0.0.1 -u diradmin -p -r ‘Applications’ applications
# get the id for that group
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -read /Groups/applications
# find a unique identifier to give the user
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -list /Users UniqueID
# create the jenkins user
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -create /Users/jenkins
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -create /Users/jenkins PrimaryGroupID 505
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -create /Users/jenkins UniqueID 1026
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -create /Users/jenkins UserShell
/bin/bash
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -create /Users/jenkins RealName "Jenkins"
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -create /Users/jenkins NFSHomeDirectory /Users/jenkins
sudo dscl -u diradmin -p /LDAPv3/127.0.0.1 -passwd /Users/jenkins
# create and set the owner of the home directory
sudo mkdir /Users/jenkins sudo chown -R jenkins /Users/jenkins

You’ll now able to login as the Jenkins user by doing something like: sudo -u jenkins -i (this will login as Jenkins with a full user session).

3. Install Jenkins

Run brew install jenkins as the user you’d normally use brew with.

4. Configure the Launch Item

OS X handles services using launchd and has a few different types for where they should be placed:

Type	Context
LaunchDaemon	System
LaunchAgent	User

More detail can be found in the Daemons and Services Programming Guide.

In our case, we want to run Jenkins as a LaunchDaemon as our newly created jenkins user, so create a plist file as /Library/LaunchDaemons/homebrew.mxcl.jenkins.plist with:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>homebrew.mxcl.jenkins</string>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/bin/java</string>
      <string>-Dmail.smtp.starttls.enable=true</string>
      <string>-jar</string>
      <string>/usr/local/opt/jenkins/libexec/jenkins.war</string>
      <string>--httpListenAddress=127.0.0.1</string>
      <string>--httpPort=8080</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>UserName</key>
    <string>jenkins</string>
  </dict>
</plist>

And then load it: sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.jenkins.plist.

(I’m sticking with the convention for naming as recommended by Hombrew.)

5. Test it out

Now you’ll be able to go to http://127.0.0.1:8080 (locally) and see Jenkins. You’ll want to verify that it’s launched with the correct user, which can be found under “System Info”.

The Launch Daemon specifies that it’ll only listen on 127.0.0.1, so it’s not possible to access it outside the local machine.

(I’d recommend keeping Jenkins inside an internal network, as it’s executing code on the local machine. Maybe by restricting it like below…)

6. (Optional) Reverse Proxying Apache & Restricting by Networks

It’s likely that you’ll want to access Jenkins at a specific domain and not have to use a separate port. This can be accomplished in lots of ways, but here I’m going to explain Apache as that’s what OS X Server uses by default. If you’re not using OS X Server, either install Apache or another Web Server such as nginx. The steps below configure this using SSL (using this guide), you can cut out much of this if you’d prefer not to.

First, enable “Websites” in Server. It doesn’t need any special configuration for what we’ll be doing.

Next, create a new site from within Server. This will ensure it’s all working in the manner it expects. Here, I’ve restricted it to internal only IPs:

Jenkins Website Configuration

The configuration for “Websites” is held in: /Library/Server/Web/Config/apache2. There’s a README which might be helpful to read.

Next, reconfigure the configuration file for the new site (which is an Apache virtualhost):

# /Library/Server/Web/Config/apache2/sites/0000_10.0.0.1_443_jenkins.example.com.conf
<VirtualHost 10.0.0.1:443>
    ServerName jenkins.example.com
    ServerAdmin admin@example.com
    DocumentRoot "/Library/Server/Web/Data/Sites/jenkins.example.com"
    DirectoryIndex index.html index.php /wiki/ /xcode/ default.html
    CustomLog /var/log/apache2/access_log combinedvhost
    ErrorLog /var/log/apache2/error_log
    <IfModule mod_ssl.c>
        SSLEngine On
        SSLCipherSuite "ALL:!aNULL:!ADH:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM"
        SSLProtocol -ALL +TLSv1
        SSLProxyEngine On
        SSLCertificateFile "/etc/certificates/certificate_name.cert.pem"
        SSLCertificateKeyFile "/etc/certificates/certificate_name.key.pem"
        SSLCertificateChainFile "/etc/certificates/certificate_name.chain.pem"
        SSLProxyProtocol -ALL +TLSv1
        SSLProxyCheckPeerCN off
        SSLProxyCheckPeerName off
    </IfModule>
    <Directory "/Library/Server/Web/Data/Sites/jenkins.example.com">
        Options All -Indexes -ExecCGI -Includes +MultiViews
        AllowOverride None
        <IfModule mod_dav.c>
            DAV Off
        </IfModule>
        <IfDefine !WEBSERVICE_ON>
            Require all denied
            ErrorDocument 403 /customerror/websitesoff403.html
        </IfDefine>
    </Directory>

    <proxy>
        Order Deny,Allow
        Deny from all
        Allow from 127.0.0.1
        Allow from 10.0.0.1/8
        Allow from 199.19.86.77
    </proxy>

    ProxyPass / http://127.0.0.1:8080/ nocanon
    ProxyPassReverse / http://127.0.0.1:8080/
    ProxyRequests Off
    AllowEncodedSlashes NoDecode
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Port "443"
</VirtualHost>

Much of this is the default configuration, with the bottom two additions being the important bit:

<proxy>
    Order Deny,Allow
    Deny from all
    Allow from 127.0.0.1
    Allow from 10.0.0.1/8
</proxy>

ProxyPass / http://127.0.0.1:8080/ nocanon
ProxyPassReverse / http://127.0.0.1:8080/
ProxyRequests Off
AllowEncodedSlashes NoDecode
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"

…which configures it as a reverse proxy.

Next, restart Apache:

sudo /Applications/Server.app/Contents/ServerRoot/usr/sbin/serveradmin stop web
sudo /Applications/Server.app/Contents/ServerRoot/usr/sbin/serveradmin start web

7. (Optional) Configure Users with LDAP

The final step I took was to configure Jenkins’ user support against OS X Server’s Open Directory. This is just LDAP with a different name, and it’s easy to get working.

Under “Manage Jenkins” → “Configure Global Security”, configure the server to the correct name and then fill in the root DN like so:

dc=server,dc=example,dc=com

…where server, example, com make up the hostname of the configured LDAP tree (server.example.com). This screenshot might help:

Jenkins LDAP Configuration

Jenkins will be able to configure the rest itself.

8. (Optional) Manage the Jenkins Process with `brew services`

The default way to manage processes on OS X is using launchctl, but the syntax isn’t the easiest to use. For example, restarting Jenkins (something you’ll want to do each time it’s upgrade):

sudo launchctl unload /Library/LaunchDaemons/homebrew.mxcl.jenkins.plist
sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.jenkins.plist

The alternative is to use brew services, which can make this much easier:

# first, install it:
brew tap homebrew/services

# then you can view the services:
sudo brew services list

# restarting:
sudo brew services restart jenkins

Closing Steps

Before configuring your first set of builds, you’ll likely want to install a set of plugins. I use:

Git, to support Git as a SCM type
GitHub, for deeper integration
AnsiColor, to support colours in terminal output
Ruby, to support Ruby as a script type
Python, to support Python as a script type

Providing Internal DNS with OS X Server

Mon, 06 Jul 2015 00:00:00 +0000

Picture the scene: You’ve got a hosted Mac somewhere on the internet, you connect to it via a VPN to access the services you host on it, but you’d like to use domains to refer to these and have them look up correctly.

The solution to this (and to a bunch of similar problems) is to configure a DNS server internally to handle look ups for you. It’ll return internal-relevant IPs for domains (which could in this case be anything).

This is basically a much more specific version of Apple’s “Provide DNS service” article.

During these steps, you’ll need to:

Configure a forwarding server to route requests outside of the local network.
Set the lookup behavior to serve the local network.
Configure the VPN to provide the DNS server in it’s configuration.
Add a relevant zone.
Add the names and aliases needed to that zone.

Configure Forwarding Servers

It’s assumed you’ll configure this with whichever the upstream ISP provides. You can do this in Server.app’s DNS section under forwarding servers. They might well already be pre-populated.

Configure Lookup Behavior

The lookup behavior defines which networks the DNS server will be available to. You’ll want to configure both “The server itself” and “Clients on the local network” as the screenshot below shows:

DNS Server Lookup Behavior

Configure the VPN DNS Settings

Next, switch to the VPN service and configure the local DNS server:

VPN DNS Settings

(Assuming 10.0.0.1 is the local server IP.)

Create a Zone & Configure it

Switch back to the DNS Service. First, check “Show All Records” in the cog dropdown. Then, “Add Primary Zone”:

Adding the Primary Zone

Next, add an A record by selecting “Add Machine Record”:

Adding an A record

Testing

Finally, you can test that this all worked by using dig. On a device connected to the VPN you should get a result like the following:

$ dig test.example.com

; <<>> DiG 9.8.3-P1 <<>> test.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42105
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;test.example.com.      IN  A

;; ANSWER SECTION:
test.example.com.   10800   IN  A   10.0.0.1

;; AUTHORITY SECTION:
example.com.        10800   IN  NS  test.example.com.

;; Query time: 153 msec
;; SERVER: 10.0.0.1#53(10.0.0.1)
;; WHEN: Mon Jul  6 10:34:46 2015
;; MSG SIZE  rcvd: 64

If you get no result (or it forwards up to the external example.com zone, you won’t see your internal IP listed. This would suggest that the DNS server isn’t specified correctly, or one of the other steps didn’t quite work right.

Fixing Problems with OS X Yosemite Server

Mon, 06 Jul 2015 00:00:00 +0000

I have a hosted Mac mini with Macminicolo that I use for a range of things, but predominantly as a build server for boxes and other projects (which require something like Xcode). However, there’s two problems which are slightly outside of the GUI which crop up. Here’s a few notes on fixing those…

Configure Server Manager to use a Custom SSL Certificate

If you’ve configured a custom SSL certificate, Server Manager will apply it to all of the services managed by it (Websites, Mail, etc). I’ve tended to use a wildcard one as it can then be reused across different places.

Unfortunately, Server Manager doesn’t set the certificate for itself and so you’ll end up with a warning when connecting from a remote machine. Like this:

Server.app Certificate Failure

This is easily fixed in the system keychain where the certificate is defined.

1. Open Keychain Access

Open Keychain Access (in /Applications/Utilities) and select the System Keychain on the right-hand side.

2. Find the `com.apple.servermgrd` Identity Preference

This is the setting that defines which certificate is used for Server.app.

From the “Preferred Certificate” dropdown, select the correct one. It’ll need to match the domain on which you’re connecting to the server on (probably it’s hostname, e.g.: server.example.com).

Keychain Identity Preference

By default, a new user created on an Open Directory tree isn’t able to use Screen Sharing. This can be a bit of a problem if you’ve created a new admin user via OD and then expect to be able to login again. There’s a few posts around explaining this, but they’re all rather old (like this one on the Apple Discussion boards).

After an extensive amount of digging, I cornered everything down to two separate steps.

1. Create Groups for Apple Remote Desktop

The first part comes from Apple’s Remote Desktop documentation (see Chapter 5, Page 63) which details the grid of permissions it relies upon.

You’ll want to create all four groups (ard_admin, ard_reports, ard_manage and ard_interact) and then add the relevant users to them. This can be done on the command line like so (where username is the relevant user):

dseditgroup -o create -n /LDAPv3/127.0.0.1 -u diradmin -p -r 'ard_admin' ard_admin

dseditgroup -o edit -n /LDAPv3/127.0.0.1 -u diradmin -p -a username -t user ard_admin

Repeat the above whilst replacing ard_admin with the other four groups.

2. Enable Directory Users

The next step is to configure Apple Remote Desktop to allow directory logins. This can also be done by installing Apple Remote Desktop (locally) and building a custom client installer (there’s a step midway through to enable Directory Logins). But the command line is easier:

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -configure -clientopts -setdirlogins -dirlogins yes

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -restart -agent -console

And now, you should be able to login over VNC using a directory user.

Static Sites with Rack and Sass

Tue, 14 Apr 2015 00:00:00 +0000

Rack is the (excellent) common denominator web library for Ruby. Sass also happens to be written in Ruby. Combining the two can be the perfect solution to building Living Styleguides, especially if you’re providing them as an Gem.

I did this on a recent project, but the documented combination of the two was a bit lacking.

Static Sites with Rack

Rack is very barebones (it’s usually used behind the scenes in Sinatra or Rails), but it does provide Rack::Static which provides much of what we’ll want to do.

Following typical Ruby conventions, we’ll end up with a directory structure which looks like:

.
├── Gemfile
├── config.ru
└── public
    ├── index.html
    └── stylesheets

# Gemfile
source 'https://rubygems.org'

gem 'rack', '~> 1.6'

# config.ru
require 'rack'

use Rack::Static, urls: ['/stylesheets'], root: 'public'

run lambda { |_env|
  [
    200,
    {
      'Content-Type' => 'text/html',
      'Cache-Control' => 'public, max-age=86400'
    },
    File.open('public/index.html', File::RDONLY)
  ]
}

And then it can run with: bundle exec rackup.

This will serve public/index.html when called at: http://localhost:9292 and also serve the files in public/stylesheets. You can replicate that for other directories (e.g.: for javascript).

Adding Sass Support

Sass comes with a Rack plugin that makes this easy. The difficulty is in handling the paths for the source data. Adjust the above to look something like this:

# Gemfile
source 'https://rubygems.org'

gem 'rack', '~> 1.6'
gem 'sass', '~> 3.4'

# config.ru
require 'rack'
require 'sass/plugin/rack'

use Sass::Plugin::Rack

use Rack::Static, urls: ['/stylesheets'], root: 'public'

run lambda { |_env|
  [
    200,
    {
      'Content-Type' => 'text/html',
      'Cache-Control' => 'public, max-age=86400'
    },
    File.open('public/index.html', File::RDONLY)
  ]
}

The Sass plugin will assume it’s stylesheets are in public/stylesheets/sass and compile them to public/stylesheets.

In my case, I was overriding this by doing (which is the current recommended way):

Sass::Plugin.add_template_location('app/assets/stylesheets')
use Sass::Plugin::Rack

This adjust the source directory for the Sass files, and in this case follows along with the standard Rails convention for them. This is because it’s structured to behave as a Rails Engine.

Setting up Transparent Proxying VMs for mitmproxy

Sat, 28 Mar 2015 00:00:00 +0000

I seem to do a lot with Virtual Machines and this was no different. I’d started out trying to reverse engineer an otherwise undocumented section of a client for a hosting service I use and I was keen on configuring an isolated environment for it.

mitmproxy is a tool for intercepting HTTP and HTTPS traffic and then allowing you to easily inspect it. In transparent proxy mode, it can sit at the network level and intercept everything without any other configuration.

There’s a few steps to it, and it seemed worth documenting:

1. Configure Two VMs

The first thing to do is to configure two VMs. I used a Ubuntu 14.04 LTS install for the server and an Xubuntu (also 14.04 LTS) for the client. I wanted a GUI on the client (for a web browser) and I had an Xubuntu ISO lying around.

You’ll want to install the virtual machine tools, too. (boxes has a script which might help). I named the server proxy-server and the client proxy-client. Otherwise they’re very standard configurations.

2. Setup the Proxy Server

The proxy server will need two network interfaces, one to the outside world (the default eth0) and another for clients to connect on (eth1).

You’ll need to add a second network interface to the VM itself, with the new one configured to be “internal only”. This sets up an isolated network on the host machine which the VMs are able to communicate through.

Once the virtual adaptor is added, we’ll configure that with a static IP:

Edit /etc/network/interfaces:

# Proxy Server network interface
auto eth1
iface eth1 inet static
address 192.168.3.1
netmask 255.255.255.0
gateway 0.0.0.0

Then bring it up: sudo ifup eth1. You can verify it worked by checking the response of ifconfig -a. To understand what’s going on, you might find the Ubuntu Documentation article on Network Configuration helpful.

The next step is to configure dnsmasq to provide us with DHCP and DNS on our internal network. First install it: sudo apt-get install dnsmasq

Then replace /etc/dnsmasq.conf with:

# Listen for DNS requests on the internal network
interface=eth1
# Act as a DHCP server, assign IP addresses to clients
dhcp-range=192.168.3.10,192.168.3.100,96h
# Broadcast gateway and dns server information
dhcp-option=option:router,192.168.3.1
dhcp-option=option:dns-server,192.168.3.1

The final step is to configure iptables to forward incoming traffic on ports 80 and 443 to our mitmproxy instance:

iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 \
    -j REDIRECT --to-port 8080
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 \
    -j REDIRECT --to-port 8080

Then we can install and open mitmproxy in transparent mode:

$ mitmproxy -T --host

(-T enables transparent proxy mode, --host infers the hostname of the request and displays that instead of the IP.)

3. Configure the Client

The client will need pointing to the correct network, and then the mitmproxy certificates installed. Technically, installing the root CA for mitmproxy is optional, but without it you’ll get a lot of SSL warnings you need to jump through.

First, the network:

Reconfigure the client’s network adaptor to be “internal only”.
Set the network configuration to look like (through either the GUI or a similar method to above):

Address: 192.168.3.10
Subnet: 255.255.255.0
Gateway: 192.168.3.1
DNS: 192.168.3.1

You’ll then want to test it all works. A non-HTTPS web page is likely the easiest.

Finally, add the generated mitmproxy CA root certificate. mitmproxy generates these on first run, so you’ll want to take these from the ~/.mitmproxy directory on the host. You’re looking for mitmproxy-ca-cert.cer.

Take this from the proxy server and place it on the client. You’ll then want to make the certificate known to the OS:

sudo mitmproxy-ca-cert.cer /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
sudo update-ca-certifcates

(Note, as part of this, it’s renamed to have more usual crt extension, which update-certificates will pick up.)

In the response you should see that a certificate was added. You can test this all worked by doing something like: wget https://nickcharlton.net. The response should show up in your mitmproxy window.

Firefox maintains it’s own certificate store and so you’ll want to add this in in Preferences → Advanced → Certificates. Just add the mitmproxy-ca-cert.cer file as an authority.

4. Begin Making Requests

You’ll now be able to make requests, both through a terminal or in a browser.

For example, wget https://nickcharlton.net should give you something that looks like the next two screenshots:

mitmproxy Host List.

mitmproxy Response.

Some of the details of this post comes from the mitmproxy documentation, but threaded out with a bit more detail that I’d needed to understand to get it all working. Now hopefully you’ll be able to replicate a similar setup.

Conditionally Chaining ActiveRecord Queries

Wed, 11 Mar 2015 00:00:00 +0000

Sometimes, ActiveRecord queries can get pretty complex, especially if you’re implementing a feature like search over a typical “index” page that also has pagination and the term itself is optional. Fortunately, ActiveRecord queries can be chained in a few ways to make this a little bit nicer.

The most common is like:

user = User.where(name: 'Nick Charlton').limit(1)

Which you’ll see often. But you can also do something like this, which works really well for more complex queries:

user = User.where(name: 'Nick Charlton')
user = user.where(email: 'nick@nickcharlton.net')

user #=> <User id: 1, name: 'Nick Charlton'>

The result isn’t evaluated until you use the resulting object and so the queries will be combined for you. This is because it returns an ActiveRecord::Relation object, and not the fully evaluated query.

This can be a much cleaner solution to conditional filtering of records, like you might wish to do with a reasonably complex search interface:

articles = Article.where(published: true)
articles = articles.search('thing') if params[:term]

I’m following this pattern on a few projects where I’d previously branched on the parameters given, but this is pretty unwieldy and leads to you duplicating a bunch of code. This is much better.

Using Postmark with Sinatra

Mon, 15 Dec 2014 00:00:00 +0000

I usually use Postmark for outgoing transactional email, I find this to be better than expecting the underlying system to have a correctly configured sendmail and it helps with deliverability. Sinatra, though, doesn’t have a convention for handling email. The Sinatra FAQ lists an example using Pony, so going from there, here’s an example of using the Postmark Gem with Sinatra:

require 'sinatra'
require 'postmark'

configure do
  set :mailer, Postmark::ApiClient.new('')
end

get '/send_mail' do
  settings.mailer.deliver(from: 'example@example.com',
                          to: 'example@example.com',
                          subject: 'A Test Email',
                          text_body: 'A simple plain text test email.')
end

I’m just using the standard settings handling to keep hold of the Postmark client here. You could do it in any way. You can also render views (like you’d typically do with ActionMailer), like so:

get '/send_html_mail' do
  settings.mailer.deliver(from: 'example@example.com',
                          to: 'example@example.com',
                          subject: 'An HTML Test Email',
                          html_body: erb(:email))
end

…where, :email is email.erb in views/:

<p>A simple <i>HTML</i> test email.</p>

And then you have simple Postmark email support. The rest of the Postmark Gem documentation details everything else you’d be able to do with it.

Structuring Sinatra Applications

Sat, 29 Nov 2014 00:00:00 +0000

Sinatra is a cool little framework for building web tools in Ruby. I’ve been using it for years to build out little examples, tools like moviesapi, experimenting with REST and Hypermedia API design and now I’m using it to build a relatively complex web service that provides the core of a new project I’m part of.

But I’ve never written down how (and why) I structure projects in the way I do. Being such a tiny framework, Sinatra doesn’t impose any pattern on you (both a blessing and a curse) and after several false starts (especially on the larger projects), I’ve found I’ve settled into a pattern which works very well.

Small Ones

The little applications are easy, it’s just one file called app.rb. This is most often a ‘classic’ application, but sometimes it can be a ‘modular’ one instead. For example:

# app.rb
require 'sinatra'

get '/' do
  'Hello World!'
end

The directory structure will usually end up looking something like this:

.
├── Rakefile
├── app.rb
├── config.ru
└── spec
    ├── app_spec.rb
    └── spec_helper.rb

Directories like models or lib fit quite well into the top level like spec does. moviesapi provides a good working example of this.

Big Ones

The structure for small applications doesn’t scale out very well because of the focus on one single file. Once I’ve reached (or expect to reach) the stage where I have several well grouped set of routes, I start to follow a Rails-like MVC pattern. This splits the routes into separate classes, almost exactly how Rails handles things. Some of this originally came from Sinatra: Up and Running which gives a nice overview.

(Sometimes, using Rails is still a better approach than replicating it’s structure like this. I keep with Sinatra for machine facing projects like APIs once they get this big, anything else would be Rails.)

This structure looks a lot like this:

.
├── Rakefile
├── app
│   ├── controllers
│   │   ├── application_controller.rb
│   │   └── example_controller.rb
│   ├── helpers
│   │   └── application_helper.rb
│   └── views
│       ├── example.erb
│       ├── layout.erb
│       └── not_found.erb
├── config.ru
└── spec
    ├── controllers
    │   ├── application_controller_spec.rb
    │   └── example_controller_spec.rb
    ├── helpers
    │   └── application_helper_spec.rb
    └── spec_helper.rb

Here, app.rb has been broken out into a directory called app instead. Inside this are the controllers, helpers, views (and probably also models, too) which make up the application. In practice, each of the controllers inherit from Sinatra::Base and so we keep the simple route call feature of the original style and by inheriting from a common application_controller we can provide a few common configuration settings all around.

The ApplicationController subclass could be as simple as just: class ApplicationController; end, which should keep each of our controller subclasses easy to reuse and test.

The ApplicationController and ExampleController look like this:

# application_controller.rb
class ApplicationController < Sinatra::Base
  helpers ApplicationHelper

  # set folder for templates to ../views, but make the path absolute
  set :views, File.expand_path('../../views', __FILE__)

  # don't enable logging when running tests
  configure :production, :development do
    enable :logging
  end
end

# example_controller.rb
class ExampleController < ApplicationController
  get '/' do
    'Example!'
  end
end

config.ru is used to bring everything up together and this looks like this:

# config.ru
require 'sinatra/base'

# pull in the helpers and controllers
Dir.glob('./app/{helpers,controllers}/*.rb').each { |file| require file }

# map the controllers to routes
map('/example') { run ExampleController }
map('/') { run ApplicationController }

My biggest use of this structure has been on a recent project with a lot of routes that have been exposed. It’s worked really well so far. That said, a lot of my projects are of the much smaller type — the kind of projects where Sinatra has always worked really well.

Reserve: Caching with Expiring Keys and Redis

Sun, 12 Oct 2014 00:00:00 +0000

I’ve just released a new RubyGem which makes caching objects (and allowing them to expire) in Redis easy. It’s called reserve and the source is on GitHub.

This came from a desire to easily wrap common but slow tasks in a block that would give me the advantages of caching but not increase the code complexity significantly. We can do this by assuming that the output of most operations could be serialised as JSON and then be thrown into a Redis instance. That means that the simplest implementation could look like:

reserve = Reserve.new(Redis.new)

item = reserve.store :item do
  { value: 'this is item' }
end

Here, item will be the hash object { value: 'this is item' } that will be generated and stored on the first call, then subsequent calls will be cached in redis until they expire. By default, the expiry time is set to 10800 seconds (3 hours). After it expires, it’ll regenerate by executing the block again.

The need behind this project was in caching responses from commonly used screen scraping tools. I wanted to be able to wrap the code that was already being used for the scraping so that on subsequent requests this was reused for a set period of time. Redis works great for this because we can easily store data with SET, and the expiry is also handled by Redis with EXPIRE. But there’s lots of other operations which work well with this sort of time-based caching.

With Reserve, the goal was to take those two Redis commands and have a very thin layer between the application code and Redis. It should be possible to “wrap” a variable assignment to add this sort of caching and not require all that much more.

I also wanted to be able to support a wide range of Redis drivers and keep the rest of the dependancies as low as possible. I think it fits these requirements quite well.

Two places where this will be used are UrbanScraper and moviesapi, which I’ve been maintaining for a while.

You can browse the code on GitHub, view on RubyGems.org and also read the docs at RubyDoc.info.

Site v4

Wed, 20 Aug 2014 00:00:00 +0000

If you’re reading this, it means that I successfully rolled-out the fourth¹ version of this site. It’s a small set of changes from before, but there is a new style which accompanies a simpler design.

On the design side, I wanted to deemphasize the latest set of posts (they’re not, and never have been time dependent) and instead have it as a short representation of me. Thus the blurb on the homepage. I also wanted to remove the dedicated pages of static content (About, Projects, etc) which sets up for having large amounts of text that doesn’t get updated (and, I suspect, people don’t really want to read through).

The project page does stay, but presented in a simpler manner and refreshed with more recent ones. But the experiment with having a “links” section has gone (those posts have been pushed into the posts).

On the implementation, I’ve chopped out SASS because it’s mostly unnecessary with so few styles and the amount of templates could also go down. But, I did have to implement a few changes to add support for HTML definition lists.

And so, the site is updated for another year.

Sadly I’m not sure what the actual version should be, so I’m going with how long I’ve had a “modern” iteration of it. I’ve been reworking this every year for the past four and so this seemed to fit well. ↩

Custom Pandoc Options with Hakyll 4

Wed, 20 Aug 2014 00:00:00 +0000

Pandoc has a huge set of extensions which are not enabled by default and whilst Hakyll does enable quite a few in its own options (like footnotes), I wanted to add support for definition lists.

In Hakyll, the best way to do this seems to be to implement a custom content compiler, as this can then be used everywhere you render content. This is exactly what I did.

The first thing was to look up the reference to the default compilers and see what was already there. Hakyll provides:

pandocCompiler
pandocCompilerWith
pandocCompilerWithTransform

The second of these allows you to pass in options (ReaderOptions and WriterOptions) and is actually called by the first but with only the default options. The implementation of pandocCompiler looks like:

pandocCompiler :: Compiler (Item String)
pandocCompiler =
    pandocCompilerWith defaultHakyllReaderOptions defaultHakyllWriterOptions

Which gives us both the pattern to copy and the type declaration. The next step is to add on custom options to the defaults. Pandoc gives a list of the options that can be passed to it in it’s documentation.

The defaultHakyllWriterOptions are defined as a set, so we’ll need to add to that set and we’ll also need to be able to access the Pandoc options, so first import those:

import qualified Data.Set as S
import           Text.Pandoc.Options

Then, we need to build up the custom compiler. That looks like this:

customPandocCompiler :: Compiler (Item String)
customPandocCompiler =
    let customExtensions = [Ext_definition_lists]
        defaultExtensions = writerExtensions defaultHakyllWriterOptions
        newExtensions = foldr S.insert defaultExtensions customExtensions
        writerOptions = defaultHakyllWriterOptions {
                          writerExtensions = newExtensions
                        }
    in pandocCompilerWith defaultHakyllReaderOptions writerOptions

(To see it in place, you can see the version in the repository.)

To walk through the compiler:

We set the list of extensions we wish to apply.
Compute the default set of extensions.
Insert our list of extensions into the set.
Initialise the writer options.
Pass that over to the pandocCompilerWith compiler.

Subsequently, we can use customCompiler everywhere we were previously using pandocCompiler, and without having to repeat our options.

Debugging SenTestingKit to XCTest Linker Errors in Upgraded Xcode Projects

Mon, 18 Aug 2014 00:00:00 +0000

There’s a couple of (client) projects which I maintain which have been in existence for quite a while (one still has full iOS 5 support and hopefully we’ll be able to drop that soon), but they’re still well maintained and have reasonable test suites that have followed them through rather well. Sadly, Xcode can be a bit difficult and this time was no exception — although it did take a long time before the issue was seen.

In this project, I had a situation where the original SenTestingKit tests had been upgraded to use XCTest, but now they were failing to build. It looked to be a configuration error, so to come to the correct settings, I compared a new project (so, created straight out of Xcode with nothing else) and another project which was as new, but had Cocoapods attached. This allowed me to get the correct (well, current) configuration and apply it back to the old project.

In the end, it appeared to have been caused by Xcode’s automatic upgrader (which I’d run back when I upgraded the project) having mangled the build settings. Here’s how I fixed it:

Update the Framework Search Paths

The first set of errors related to the imported headers being missing, and looked like:

ld: framework not found -XCTest

In the test target under Xcode 5, the Header Search Paths listing should have looked like:

$(inherited)
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include

Switch the linker flag from SenTestingKit to XCTest

The next set of error messages related to the testing framework being missing and looked like:

Undefined symbols for architecture i386 "_OBJC_CLASS_$_XCTestCase"

This was caused by the old SenTestingKit still being linked aginst, and the solution to this was to switch the “Other Linker Flags” setting to:

-Objc -framework XCTest

Of course, by the time someone else comes across the same issue (or a similar) one, the ideal configuration has likely changed somewhat. The trick to finding the solution is to build new Xcode projects from the templates and compare your configuration.

Ruby Subprocesses with stdout and stderr Streams

Wed, 08 Jan 2014 00:00:00 +0000

I’ve been doing a few things with Ruby which involve controlling and responding to long-running processes, where the Ruby-based ‘wrapper’ takes the task of automating something which is otherwise quite complex. Perhaps the best example is boxes, which uses a collection of Rake tasks to generate Vagrant boxes using Packer –– each build takes somewhere in the region of twenty minutes to complete.

But, I wanted to be able to more closely control the output (hiding much of it from view) and react to events like build failures, which wasn’t possible by using system(). This needed to be able to handle the output as it came line by line without blocking (handling them as a stream), be able to handle stdout and stderr independently, and allow me to collect all of the output from a subprocess (for providing as a sort-of stack trace).

I went through many different solutions (using Open3.popen3, PTY and others), before coming across this hybrid solution using popen3 and separate threads for each output stream in this Stack Overflow post which met most of my requirements.

This gave me a basic solution which looks like this:

require 'open3'

cmd = './packer_mock.sh'
data = {:out => [], :err => []}

# see: http://stackoverflow.com/a/1162850/83386
Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
  # read each stream from a new thread
  { :out => stdout, :err => stderr }.each do |key, stream|
    Thread.new do
      until (raw_line = stream.gets).nil? do
        parsed_line = Hash[:timestamp => Time.now, :line => "#{raw_line}"]
        # append new lines
        data[key].push parsed_line
        
        puts "#{key}: #{parsed_line}"
      end
    end
  end

  thread.join # don't exit until the external process is done
end

Line 3 pecifies the command that will be run. This is just a shell script which prints a multitude of characters for testing. Line 4 defines the final data structure; a Hash with two Arrays for stdout and stderr.

The next interesting bits are Lines 9 and 10 which create a Thread for handling the stdout and stderr streams seperately. Inside this the until block reads from the given stream, structures it and stores it. I’m adding a Time object here to aid my presentation of it later.

Line 16 would be replaced by a conditional depending on the amount of verbosity the user desired. Finally, Line 21 joins the thread once it has finished executing.

This, then, allows me to continue handling long processes as a stream, but handle each line individually. But the interface is a little awkward to use. Providing a simpler command a single block could simplify this, something like:

Utils::Subprocess.new './packer_mock.sh' do |stdout, stderr, thread|
  puts "stdout: #{stdout}" # => "simple output"
  puts "stderr: #{stderr}" # => "error: an error happened"
  puts "pid: #{thread.pid}" # => 12345
end

Which could be implemented like this rather impressively nested bit of code:

require 'open3'

module Utils
  class Subprocess
    def initialize(cmd, &block)
      # see: http://stackoverflow.com/a/1162850/83386
      Open3.popen3(cmd) do |stdin, stdout, stderr, thread|
        # read each stream from a new thread
        { :out => stdout, :err => stderr }.each do |key, stream|
          Thread.new do
            until (line = stream.gets).nil? do
              # yield the block depending on the stream
              if key == :out
                yield line, nil, thread if block_given?
              else
                yield nil, line, thread if block_given?
              end
            end
          end
        end

        thread.join # don't exit until the external process is done
      end
    end
  end
end

Unlike the first approach, this just passes back the lines as strings which is nil if there’s no value. The final argument to the block is the thread the subprocess is run as. thread.pid will give the PID.

For now, this works pretty well for boxes and will allow me to throw it into something like Jenkins without a ridiculous amount of logs to parse to see which ones build successfully.

Annual Review 2013

Tue, 07 Jan 2014 00:00:00 +0000

Introduction

For at least the past three years, I’ve been reading Chris Guillebeau’s Annual Review series. He publishes a set of blog posts in December each year running through what he (and his business) did, what he thought of it and what he’d like to do in the upcoming year.

I’ve been doing something similar myself for about the same time (I also used to come up with a plan for summers back when I had long expanses of free-time; usually a collection of things I wished to learn), but I never published them. Much of the concept of the “Annual Review” is in doing it, but without publishing, it’s all too easy to forget you did it and there’s a certain feeling of accountability when you’ve published a block of self-criticism and plans for the year ahead. Especially when they all seem to pass by so quickly.

This year was my final year of University, so that’s a general trend throughout. The final push after several years of studying seems to throw out much of the other things you want to do, and so the latter few months are much more interesting.

But anyway, let’s begin…

Projects

Nearly all of my projects (big and small) are something to do with programming, most recently these seem to have refocused on Ruby. But for much of University, I spent my time using Python (especially when graphs where involved), but after some niggling annoyances (no closing of blocks, pip, time and date handling) I found myself being pushed me back over to Ruby.

Outside of Ruby, I’ve been spending it working with iOS, with a little bit of Mac development interspersed. The introduction of iOS 7 has made for some interesting new challenges.

I continue to maintain several projects which are hosted on GitHub and of these I’ve been able to impart a stronger focus on automated tests. Several of them have been configured with Travis CI, but so far many of them have poor coverage, especially the ones with a web component, or the needs of a complex environment. Next year, I expect to keep going down this path.

There’s still a hardware project or two which I’ve not gotten around to thinking much about since the end of University. The new year may (or may not) bring me to coming back to them.

I’m quite pleased with all of these smaller projects and I get quite a lot of satisfaction out of building small well-tested libraries (before going away for Christmas, I also released keypath-ruby –– an approach for accessing nested Ruby collections).

One of the things I’ve been less pleased with is the amount of time I’ve spent working with servers and infrastructure. Whilst the work I’ve done with Packer has been going rather well (this bought about boxes), automating infrastructure using Chef has been a slow and quite painful process. I find it to be a frustrating tool to work with, but I’m also not hugely convinced by the alternatives.

Sadly, this has kept me from working on some of the things I’ve really wanted to like Predict the Sky, and a few others.

But overall, this is probably the most pleasing section. I’m pleased with both the odds and ends I’ve spent my time working on, and the bigger ones too.

Writing

This year, I published 16 blog posts, most of them in August. I also added a link/commentry section to the site and published 23 links to (mostly long) articles I’ve come across. Over the year, the amount of visitors to this site have gone up quite markedly. It’s nice to be read.

After August, I seem to have come to a bit of a halt as I took on a much bigger project. But I do wish I’d been writing along the way; instead I seem to have lots of drafts.

So, as a plan, I aim to write and publish more. I’m already quite good with keeping a journal.

Reading

Books. I love them. But I don’t read as many as I’d like. I’ve set myself a plan to read 50 of them in 2014 and I’ll be interested to see how well I do.

I’m currently reading Morrissey’s Autobiography, which turned out to be a surprisingly gripping dive into his mind, from his early days in school, to a discovery of music and poetry to the Smiths, legal battles and his solo career. It’s a bit unfocused, but interesting. I’m most of the way through now.

I’ve also ended up with subscriptions to the New Statesman, GQ and The New Yorker, which regularly fill me with lots of different (and notably offline) things to read.

Travel

With my final year of University, I didn’t travel much at all this year (apart from between Plymouth and Surrey¹ which doesn’t really count). I hope to fix this next year, with at least a trip or two to mainland Europe.

Studying

Graduating is a curious thing. Suddenly you are propelled forth into the world, expected to have some sort of plan. For most of us, this certainly isn’t the case.

But after four years of University (one was on placement), I’m now the owner of a Bachelors Degree in Computer Science.

Work

After taking a few months off to recover from University, I started freelancing in June. First with some smaller projects, but then from a lucky bit of timing picking up something much larger.

Next year I look forward to expanding the client base, and mixing up the projects a little. Some of this is already in motion.

Health & Fitness

University seemed to put this into quite a sharp decline. I used to cycle daily when I was on placement, but I stopped on moving back to Plymouth. Now, I don’t do anywhere near what I’d like to be doing.

I intend to start running in 2014. But I’ve said this (mostly to people) before. I’ll see how I go in a year.

Into 2014

2013 was interesting, and 2014 will mark my first complete year out of some sort of education system. I’m really looking forward to it.

I’m still seem to be on an academic calendar, so what most would consider some sort of “new years resolution” or some such has been in progress since September. Be it balancing work with other projects, learning German through Duolingo or one of a few other things.

But in 2014, I want to finish learning to drive and acquire a car, get running regularly (I would like to be capable of running a 5k), keep going on my desire to dress much better, and chipping away at all of those other things that I do which irritate me.

It’s going to be a fun year.

My Mum’s. ↩

Switching Season Report, 2013 Edition

Wed, 28 Aug 2013 00:00:00 +0000

Alex Payne looks at the alternatives to his current setup and draws some conclusions.

In his article, he’s put words to things I’ve so far been unable to describe to people myself:

“the Galaxy S4 is uninspired but good”

“does not feel like a premium product. It is plasticky and sort of embarrassing to carry around, though its cheapness does lend it a sense of devil-may-care durability.”

And, then:

Let’s not even speak of the system fonts, ugly but ignorable on a phone and downright offensive on a tablet. No, while much improved from several years ago, one does not use Android in 2013 for its looks.

On Linux as a deskop:

“Trying to compute like a normal with Linux is, after all these years, still an exercise in masochism.”

Yep.

I sometimes get to play around with Android devices (it’s interesting to see how they’re changing), and recently I acquired a ZTE One to have a look at Firefox OS. I’m intending to write a review about that soon.

Alfred Workflow: Paste Cleanly

Tue, 27 Aug 2013 00:00:00 +0000

There is nothing more annoying than seeing, or ending up sharing URLs that look like this (it’s split so it doesn’t look completely terrible…):

http://www.nytimes.com/2013/08/25/opinion/sunday/ \
im-thinking-please-be-quiet.html? \
ref=opinion&_r=3&utm_source=buffer&utm_campaign=Buffer
&utm_content=buffer8fcfe&utm_medium=twitter&

It’s not so much the efforts of marketing people to track how their URLs spread around the web that annoys me, but more that it’s so damn ugly and far too long. But it is also happens to be the case that I don’t particularly care about marketing people’s feelings by me removing them.

I’m already an avid user of Alfred, where I use one of the example workflows to allow me to use “Cmd + Shift + V” to paste as plain text. I figured the addition of a little “cleaning” script based on a regex would be a nice way to implement it, so I did¹.

Jump to the end if you’re just looking for the workflow.

Regex and Test Pattern

The Google Analytics arguments all start with “utm_” (presumably standing for the original Urchin product name). This is quite easy to match:

(?i)(?:utm_+)[a-zA-Z]*=[a-zA-Z0-9]*(&)?

This searches for “utm_”, a collection of other characters until an =, then another collection of characters until either the end is reached, or it walks into an “&”. It also does this case insensitively.

This will correctly match/remove the offending string from all of the URLs below:

http://example.com/slug?utm_content=test
http://example.com/slug?another=yep&utm_content=test
http://example.com/slug?utm_content=test&required=true

I was originally also matching either a ? or & at the start and removing that, too. But, with the last example (which, actually, I haven’t seen in the wild), this could potentially break the URL. Instead, I’m checking for a lost ? on the end of the URL before passing it back.

And so, the workflow’s regex implementation:

input = '{query}'

# remove any matching Google tracking strings
input.gsub!(/(?i)(?:utm_+)[a-zA-Z]*=[a-zA-Z0-9]*(&)?/, '')

# if there's a trailing ?, remove it
if input.end_with? '?'
  input.chop!
end

# pass it back, without the newline
print input.strip

Alfred Workflow

The workflow is based on the “Paste as plain text from hotkey” example, which I already had mapped to “Cmd + Shift + V”. I’m just slotting through a “Run Script” action before it is pasted.

The Paste Cleanly Workflow.

You can download an archive of the workflow here.

And so, there we go. As I get offended by more web-based atrocities[^strong], I’ll likely add to the script more.

Something similar might already exist. It was quicker for me to build it myself, than search the forums to see if someone else had done it already. But, I suppose, this is rendered a bit moot now I’ve written a blog post on it. [^strong]: Yeah, I know. That is a bit strong. ↩

Mocking Web Requests with VCR and MiniTest

Thu, 15 Aug 2013 00:00:00 +0000

I just released moviesapi. In the post I introduced it, I mentioned wanting to be able to add reliable tests. Ben Keeping responsed suggesting that I have a look at VCR. So I did.

It’s a Ruby library that records the web requests that your application depends upon and saves it down to disk. On subsequent test runs, it reuses (“replays”) the previously saved data, vastly increasing the speed. For screenscraping tools like moviesapi or UrbanScraper, I can verify that my code is behaving correctly, even if the source has changed (this is another problem) and without constantly hitting the remote web service.

Tutorials covering both VCR and MiniTest were a little thin on the ground, so I thought I’d write one. As an introduction to MiniTest, I’d suggest Matt Sears’ Quick Reference post. I’d also suggest giving the VCR README at least a skim read.

The overall application I’m testing is a Sinatra one, but here, I’m more interested in testing the class that handles the screenscraping. Some of the MiniTest suggestions came from the Sinatra Recipes site.

Gemfile

Firstly, I added development and tests groups to my Gemfile, like so:

group :development, :test do
  gem 'minitest', '~> 5.0.6'
  gem 'webmock', '~> 1.12.0'
  gem 'vcr', '~> 2.5.0'
end

VCR is a high-level wrapper around a group of different web mocking libraries, webmock is just one of those supported. We’ll need this too.

MiniTest is actually already in the standard library from Ruby 1.9, but I’m keeping a reference for clarity (and a slightly newer version).

Spec Structure

I’m writing specs here, so everything is in a directory named “spec”:

spec/
    cassettes/
    support/vcr_setup.rb
    spec_helper.rb
    movie_spec.rb

cassettes holds the recorded requests. vcr_setup.rb contains VCR configuration (it’s loaded by spec_helper.rb). spec_helper.rb sets up the tests and provides any common configuration. Finally, movie_spec.rb is the spec I’ll be running. It’s from moviesapi.

`vcr_setup.rb`

require 'vcr'

VCR.configure do |c|
  c.cassette_library_dir = 'spec/cassettes'
  c.hook_into :webmock
end

This specifies where to find the cassettes — we assume everything is run from the root of the project. Then it specifices which mocking library to use, in this case, webmock.

`spec_helper.rb`

This contains common code to all of the specs (or tests). It’s typically used to load in the application and run any common load configuration (like setting ENV to test) and is then included in each spec (or test) to make it available on test run.

require 'minitest/autorun'
require 'minitest/pride'

# pull in the VCR setup
require File.expand_path './support/vcr_setup.rb', __dir__

# pull in the code to test
require File.expand_path '../movies.rb', __dir__

`Rakefile`

Finally, these additions to the Rakefile will allow your tests to be running according to typical Ruby conventions. It will also run the test suite as the default rake task:

require 'rake/testtask'

Rake::TestTask.new :spec do |t|
  t.test_files = Dir['spec/*_spec.rb']
end

task :default => :spec

For all of these helper files, they have been slimmed down a little. You may find the ones in the repo more helpful. (Also, these will work with Travis CI.)

Writing Specs that use VCR

A typical MiniTest spec looks a bit like this:

describe 'Something' do
  before do
    # something that should be done before a test starts
  end

  after do
    # something that should be done after a test ends
  end

  it 'does something' do
    # test things
  end
end

The MiniTest DSL provides several blocks that make up the spec. The describe block defines the behaviour you are specifying. The it block defines the test case. Then, inside here, “matchers” are used to confirm the output. MiniTest provides a reasonable collection of these in it’s docs, but you can also define your own¹.

When using VCR with MiniTest, there are two approaches to work with. The first is to manually specify a cassette to encapsulate the test run. This is described in the VCR Getting Started Guide and looks a bit like this:

VCR.use_cassette('cassete name') do
  # the test
end

The second approach is to use the before and after blocks along with some runtime metadata that MiniTest provides. That looks a bit like this:

describe 'Movies' do
  before do
    VCR.insert_cassette name
  end

  after do
    VCR.eject_cassette
  end

  it 'fetches a list of cinemas' do
    # the test
  end
end

before and after are executed around each it block. So here, name is “fetches a list of cinemas”. If you were to have multiple it blocks, cassettes would be defined for each. The cassettes are then saved in spec/cassettes, in this case it is: test_0001_fetches_a_list_of_cinemas.yml. This is quite a nice approach for having a cassette dynamically defined for each spec.

For testing the result of the screenscraping, I have been checking the contents of the eventual data structure. Unlike with a typical web service, I can’t check the data that is contained within. Similarly, the cassettes are commited to the repository because I am checking for the behavioural correctness of my code — not that the web service/site has changed and broken it. For testing the behaviour of code that depends upon this, mocking will fit perfectly.

MiniTest combines with VCR quite nicely — especially once you work out the conventions to follow for structuring the test suite. If not for anything else, mocking out the web requests like this saves a signficant amount of time when testing web service interaction.

This gist by Jared Ning contains a good set of examples of defining your own. ↩

Young Rewired State 2013

Tue, 13 Aug 2013 00:00:00 +0000

And so, sadly, that is the end of Young Rewired State for another year. Like last year, I was a mentor down in Plymouth, where we had five participants and (overall) three mentors.

Much like last year, this year was also incredible. After going through all of the introductory material, then talking about sources of open data and technology options, we stepped back to give them some time to brainstorm. By the end of the first day, they all had an idea of a project that they wanted to work on together.

This turned into Things2Do, a simple search driven service which collates a combination of different data sources (for places, weather forecasts, movies, etc) and combines these to give a smart result based on your location (and what you wish to do). It looks like this:

Things2Do Homepage and Results

For now, the area and some of the data is limited to just Birmingham (this was just for testing), but I’m sure they’ll fix that soon. It’s also responsive and well tested across a range of different devices (although, there is a few bugs here and there). I’m really impressed with how far they came in just a week. They have some interesting plans for it in the future, so I’ll be following along closely to see how it develops.

For the whole week, we were almost completely hands off. We did help with some debugging, cross-browser issues, suggestions on approaches & direction, and I got to indulge myself in building a screenscraping tool and API to provide some data that otherwise was unavailable to them, but the outcome was all of the team’s effort.

The Festival of Code was great, too. We arrived at about three in the afternoon after a — long and standing room only — train journey. In the evening, Sam was on stage along with TheraminHero and Pixelh8. Then on Saturday morning, we had the first stage of judging, the “Heats” where the teams from different centres around the country presented their projects. Our group’s and all of the others went extremely well, with only a few technical hitches, it was great.

Unlike last year, the final was kept to the Sunday. This meant everything was far less rushed and as we had the heats in the morning, a free afternoon. Come Sunday, we filed into “Space 2”, ready to see who had made it into the final. Sadly, our team didn’t make it but they seemed more releaved in not having to do another demo! Seeing all of the other projects is always fascinating …the PiBike seemed to be a favourite for many people with whom I spoke with and by the end, it had won the award for Best in Show.

I was hugely impressed by the judging, as I was last year. It’s all well and good having a panel of interesting people handling the judging process, but if they don’t ask the right kind of questions, debunk common fallacies¹ and give the presenters the space to get out what they mean to say, it is a huge disservice to the people presenting. The judges handled it marvelously.

After a little bit of time running over, the event wrapped up and we headed for our train. It was an interesting (and hilariously overcrowded) journey back after a great week.

Over the weekend, Chris, Sam and I talked about what we’d like to do with YRS in the South West in the future. We’d like to see more people, over a much greater area of Devon and Cornwall. Currently, the closest centre to us is Bristol — which isn’t really the South West when you are used to being in Plymouth or Exeter. We did hear of people wishing to come but finding Plymouth (in addition to getting to Birmingham) all too expensive for the travel. We’ll be aiming to work out all of these for next year — we have several ideas for each of them.

Anyway, if you’re interested, you can already sign up for next year (as a mentor or as a partipant).

Like Aral did with the rather common: “but we’re not designers!”. ↩

Tweetbot Mute Filters

Tue, 13 Aug 2013 00:00:00 +0000

The other day, I tweeted a link to a regex-based mute filter for Tweetbot. The full URL looks like:

tweetbot:///mute/keyword?regex=1&text=(%3Fi)%23%3Fbreaking(%3F%3D%20%3Fbad)

It uses Tweetbot’s URL scheme¹, to provide a predefined mute filter. This way, a filter can be shared without copy and pasting it, which is quite nice. In trying to work out how to do this, I ended up browsing around Twitter trying to find an example of how to pass along a regex in the URL, as I had seen it done before. As I imagine I’ll do this again, I thought I’d write it up.

Regex

Before it is encoded, the regex looks like this:

(?i)#?breaking(?= ?bad)

It’s designed to match;

“breaking bad”
“breakingbad”
“#breakingbad”
“Breaking Bad”

and similar variations, but not;

“breaking”
“bad”

when used alone.

The first bit, (?i) sets the regex to be case insensitive. #? means it may or may not start with a hash (matching hashtags or not). Then, the rest comprises of a lookahead assertion.

A lookahead assertion tests to see if a given set of characters are followed by another set. It’s considered an assertion because it doesn’t consume these characters, it will only match them. For testing regular expressions, I use Patterns, so you get something that looks like this:

Testing in Patterns

You will see that whilst this matches the appropriate lines, it doesn’t match the whole term. In this situation, this is fine (Tweetbot filters any tweet that would match). So, breaking(?= ?bad) looks for the word “breaking” followed by “bad”, with or without a space between them. The lookahead assertion is the bit in brackets, (?=).

Assembling the URL

The next bit is to make it valid inside a URL. I cheated and used Eric Meyer’s URL Decoder/Encoder, but the invalid characters are below:

Character	Replacement
?	`%3F`
#	`%23`
=	`%3D`
Space	`%20`

There are obviously many tools to help do that bit.

So now, the next time I want to avoid spoilers to the finale of a pretty good TV show, I’ll be save in the knowledge that once, I wrote down how it did it.

Many other iOS/Mac apps have a similar thing, it’s a bit of a hack to have inter-application communication. Macstories has a bunch of posts on them. ↩

The Market for Used Mac minis

Mon, 12 Aug 2013 00:00:00 +0000

Writing on the Macminicolo blog:

“you could have purchased an original G4 Mac mini for $500, used it as a server for 8 years, and sold it today for $100”

This is nuts.

iOS 7: Watershed Moment

Mon, 12 Aug 2013 00:00:00 +0000

Gedeon Maheux talks about iOS 7 and new versions of apps:

“Taking full advantage of new APIs, designing new interactions and more can represent a healthy investment, time is money after all. At what point in the update process does a developer decide she needs to charge for it? How many users will be alienated by charging again? Will these users be offset by the huge influx of new people Apple brings to the table with the launch of the new OS?”

For me, iOS 7 is absolutely a watershed moment. All of the advantages of an all-new iOS release (marketing, new APIs, much improved personal experience) has thrust me into working on new versions of Where’s Next? and thinking about how the Predict the Sky app will turn out.

I’d be a fool not to want to run with the opportunity and release new versions — especially of Where’s Next? — and get a boost in the store and, looking at Craig Hockenberry’s recent post on App Updates for iOS 7 it doesn’t look like I’m alone.

moviesapi: A Simple API for UK Cinema Listings

Thu, 08 Aug 2013 00:00:00 +0000

This week has been Young Rewired State week. At YRS i-DAT in Plymouth, we’ve had a team of five working on an tool to give you suggestions on what to do if you were bored, all based upon your location.

Among lots of other data sources, they wanted a way to get nearby cinemas and the times of films which were showing there. After a quick hunt around for different sources, it looked a bit bleak. Back in 2009, Yahoo had a movies API. Google had one too as part of iGoogle, but that went when iGoogle was shut down.

Then I came across Find any Film. The first reference I could find for it was a 2009 Guardian article, explaining how the UK Film Council had launched the site, and it mentioned that: “an API will also be rolled out that will allow developers to build applications around this unique and rich data set. ‘We’ll be thinking carefully about the best way of doing that.’”. Sadly, it looks like this never happened.

So, I took it upon myself to screenscrape Find any Film and spit out JSON formatted data that could be reused by our YRS team. That became moviesapi.

It’s a small collection of Ruby, with the parsing handled by Nokogiri and the API implemented using Sinatra. It’s very similar to my last screenscraping project, UrbanScraper.

As it is now, it implements the ability to find cinemas based on a postcode and to find out the show times today at a given cinema. This was just enough to get it working for the team, but I’ll probably add more functionality later on.

At some point, I need to add some tests¹, and it would be nice to run it through Travis too. Additionally, some caching (probably using Redis) would be good — the requests are quite slow and don’t change all that often.

The legal status of this data is a little complex. As far as I can understand, any user (including me, and then you using the API) would be bound by the Find any Film Terms and Conditions which says that you cannot use the data commercially along with a bunch of other things. It’d be nice if people would just share this kind of data, but that’s another complex argument. I’m of the opinion that I’ll continue to run such a thing (doing as much as I can to stop Find any Film being abused by caching at my end) until I’m told otherwise.

Anyway, the source is on GitHub and the live API is hosted on Heroku.

I’m not sure how to test screenscraping tools (UrbanScraper is in the same boat and also needs them). My thoughts fell along testing the existence of returned data and their types. ↩

Automated Backups with backup and Rsync.net

Fri, 02 Aug 2013 00:00:00 +0000

For the past 18 months, I’ve been meaning to review the way I go about doing backups. Whilst I’m quite settled on backing up workstations, until about two weeks ago I had a half-configured, non-recoverable solution which was only setup on one out of several servers. That’s not very useful.

As well as the obvious requirement for something automated, I also wanted something which could allow me to reuse much of the backup definitions across multiple servers but also allowed me to avoid writing bash scripts. I’d come across the backup Ruby gem some time ago and after some quick testing, this seemed to work quite well.

Then, I needed some sort of remote host to backup to. I had heard of Rsync.net through the prgmr mailing list. They provide a reasonably cheap remote filesystem with which you can access over ssh (and so, scp and rsync work). On the machine being backed up, this is great as I don’t need any special tools to upload backups and for restoring the same applies.

Backup Strategy

The backup gem uses the concepts of “models” to define a “backup”. This consists of any directories or databases you may wish to backup, along with any processing you may wish to do with it (compressing, encrypting, etc) and where to store it.

My approach to the models is to create different ones for the different “roles” a system performs. By default, this includes a “base” role which archives /etc, /home and /var/log (in a multi-user system, I’d probably split out /home to it’s own model). Then, additional models are defined for more specific roles. So, to backup a Wordpress site, a “site” model would first backup the associated database and then collect up the data in /var/www/site/ into the one archive.

Each model is set to be compressed using gzip, upload to Rsync.net and then notify me if any errors occur. It also keeps the last 5 versions of the backup.

backup handles organising the subsequent archives on the remote quite well. For each server, I define a directory to collect them in and then backup sorts each run by a directory named after the timestamp. Each model is then collected inside. Much like this:

server_name/
    base/
        2013.07.29.21.54.01/
            base.tar
        2013.07.30.00.30.07/
            base.tar

Finally, a cron job is used to automate it to run daily.

rsync.net

Rsync.net is refreshingly simple (a bit like prgmr is, I suppose). After signing up, you’ll get sent the details needed to access the relevant box, where you can ssh in, and do the basics (change the password, check your quota usage, move files, create directories, etc.).

I opted for a geographically distributed plan (it’s slightly more expensive, but it is the primary backup method I’m using) and the lowest plan — the amount of data is tiny as it’s mostly text files.

And that’s essentially it. I paid for a year so I’ll be reminded about it next year to go about renewing it.

`backup`

The backup Ruby gem gives you a command line tool which will help generate the models and run them. But you should just install it as a gem, rather than using Bundler or anything else.

I did all of this under a specific backup user. This is configured to allow it to use tar through sudo without asking for a password and not much else. It expects the backup models and configuration files to exist in ~/Backup/, so this seemed the best approach.

gem install backup

The documentation suggests using the model generator to get started and that’s pretty much what I did:

backup generate:model -t base --archives --storages='scp' --compressors='gzip' --notifiers='mail'

This will give a rather detailed and well commented example. I started with this and stripped it down to the bare essentials. If you don’t have one already, it will also create a template config.rb, which will contain a similar set of examples. config.rb can be used to set defaults for the models, so I opted to fill this with as much as possible.

But, some Real World™ examples are much more useful:

`base.rb`

##
# Base: Basic Linux backup model.
# Archives and compresses: /etc, /var/log, /home, /mail.
# Uploads to rsync.net.
#
# $ backup perform -t base
##
Backup::Model.new(:base, 'Basic Linux Model') do
  # archive
  archive :etc do |archive|
    archive.use_sudo
    archive.add "/etc/"
  end

  archive :logs do |archive|
    archive.use_sudo
    archive.tar_options '--warning=no-file-changed'

    archive.add '/var/log'
  end

  archive :home do |archive|
    archive.use_sudo
    archive.add '/home'
    # don't backup up the backup data.
    archive.exclude '/home/backup/Backup/.tmp/'
  end

  archive :mail do |archive|
    archive.use_sudo
    archive.tar_options '--warning=no-file-changed'

    archive.add '/var/mail'
  end
  
  # compressor
  compress_with Gzip

  # storage
  store_with SCP

  # notifier
  notify_by Mail
end

The archive blocks are just an abstration over tar, so you can pass through options. In this case, I’ve ignored file change warnings for areas which are likely to not harmfully change whilst the backup is running.

Both the storage and notifier lines assume the configuration has already been made. If you didn’t have these in config.rb, it wouldn’t work and you’d need to expand the line into a block.

`config.rb`

##
# SCP Storage Type Defaults
##
Backup::Storage::SCP.defaults do |server|
  server.username   = ""
  server.ip         = ""
  server.port       = 22
  server.path       = "~/server_name/"
  server.keep       = 5
end

##
# Notifier Defaults
##
Backup::Notifier::Mail.defaults do |mail|
  mail.from                 = ""
  mail.to                   = ""
  mail.address              = "smtp.gmail.com"
  mail.port                 = 587
  mail.domain               = ""
  mail.user_name            = ""
  mail.password             = ""
  mail.authentication       = "plain"
  mail.encryption           = :starttls
end

# * * * * * * * * * * * * * * * * * * * *
#        Do Not Edit Below Here.
# All Configuration Should Be Made Above.

##
# Load all models from the models directory.
Dir[File.join(File.dirname(Config.config_file), "models", "*.rb")].each do |model|
  instance_eval(File.read(model))
end

The scp block contains the details for Rsync.net. The mail defaults are currently set to the values for Gmail’s SMTP server (you’ll need to fill in all of the other relevant bits). By default, this will notify about all events (successful, with warnings or a failure). I kept this like this for about two weeks to confirm it was running correctly.

Aside: Criticism

The configuration files would be more appropriately stored in /etc, given it’s designed for Unix-like systems.
The timestamps are annoying. Unix timestamps or ISO 8601 is far more appropriate than defining another bloody date format.

Automation

To automate it, I just have a crontab entry under the backup user. This then runs at 0130 every morning and emails me if necessary.

30 01 * * * backup perform -t base,www >/dev/null

(By default, it writes a lot to stdout, I’d rather not fill up an unmonitored inbox with successes…)

I configured this about two weeks before writing up this blog post. It’s been working well since then, and I’ve deployed a similar configuration across at least two other boxes.

I now need to investigate the Chef cookbook and modify it to work similarly to this…

Debian/Ubuntu: Dynamic MOTD

Thu, 01 Aug 2013 00:00:00 +0000

In doing some client work recently, I noticed that Ubuntu now has a dynamically generated MOTD (Message of the Day) — the message shown on login, through SSH or locally.

It turns out that this has existed for a quite a while. It works by a hook in PAM (Linux’s authentication system), that runs a set of scripts to generate the MOTD which is then passed along to the client. (Before this, it was generated using a cron job that ran every 10 or so minutes.) Because it is part of PAM (pam_motd.so, specifically) it also has worked with Debian since Squeeze.

I had tried to do something similar before. Back in mid-2009, I posted something about configuring the “SSH Banner”. I had tried to do something dynamic before posting that, but without a cronjob it wasn’t possible to do and constantly regenerating a text file for a not-often seen login banner seemed silly.

Out of the box with Ubuntu, a set of provided scripts exist in /etc/update-motd.d/, which are run in ascending order to produce a static /etc/motd file. As the functionality is handled by PAM, on Debian, we just need to create the directory and populate with a set of scripts.

Like most config.d/ style configurations, the files are executed in ascending order, and so prefixing each script with 00-99 will arrange the order of the final output. Also, the scripts could be in any language but these are mostly shell apart from where it becomes complex enough to be better off in Python.

Final Result

 _          _          _      _
| | ___   _| |__  _ __(_) ___| | __
| |/ / | | | '_ \| '__| |/ __| |/ /
|   <| |_| | |_) | |  | | (__|   <
|_|\_\\__,_|_.__/|_|  |_|\___|_|\_\

Welcome to Debian 7.0 (3.2.0-3-amd64).

System information as of Mon Jul 29 22:13:06 UTC 2013:

System load:  0.0       Memory usage: 20.0%
Usage of /:   1.6%      Swap usage:   0.0%
Local users:  1

34 updates to install.
0 are security updates.

No mail.
Last login: Fri Jul 26 12:05:11 2013 from localhost

My goal here was to have something that could quickly tell me which machine I was using and some specifics of it, an understanding of the current state and any actions that should be taken. But, at the same time it should be as concise as is practical and, importantly be fast to execute.

Another addition I’ll make soon is to add configuration manager status, so in my case it will show the Chef last run, environment and roles applied. (This will also help remind me which boxes are using Chef before I change things which will automatically be reverted.)

Header (`00-header`)

The first part is the header. This comprises of the ASCII art hostname and the “Welcome…” text. I’m dynamically generating the ASCII art using the short version of the hostname, but you may wish to hard code it (obviously, this will need figlet installed.)

#!/bin/sh
#
#    00-header - create the header of the MOTD
#    Copyright (c) 2013 Nick Charlton
#    Copyright (c) 2009-2010 Canonical Ltd.
#
#    Authors: Nick Charlton <hello@nickcharlton.net>
#             Dustin Kirkland <kirkland@canonical.com>
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License along
#    with this program; if not, write to the Free Software Foundation, Inc.,
#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

[ -r /etc/lsb-release ] && . /etc/lsb-release

if [ -z "$DISTRIB_DESCRIPTION" ] && [ -x /usr/bin/lsb_release ]; then
        # Fall back to using the very slow lsb_release utility
        DISTRIB_DESCRIPTION=$(lsb_release -s -d)
fi

figlet $(hostname)
printf "\n"

printf "Welcome to %s (%s).\n" "$DISTRIB_DESCRIPTION" "$(uname -r)"
printf "\n"

System Information (`10-sysinfo`)

This is slightly more complicated. Using a mix of standard utilities, /proc and some text parsing, it’s quite easy to assemble the system information section. As I’m only targetting Debian/Ubuntu, it’s quite easy to get this working.

In the original implementation, Canonical use their “Landscape” product to generate the statistics which provides a bit more functionality than this does.

#!/bin/bash
#
#    10-sysinfo - generate the system information
#    Copyright (c) 2013 Nick Charlton
#
#    Authors: Nick Charlton <hello@nickcharlton.net>
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License along
#    with this program; if not, write to the Free Software Foundation, Inc.,
#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

date=`date`
load=`cat /proc/loadavg | awk '{print $1}'`
root_usage=`df -h / | awk '/\// {print $(NF-1)}'`
memory_usage=`free -m | awk '/Mem/ { printf("%3.1f%%", $3/$2*100) }'`
swap_usage=`free -m | awk '/Swap/ { printf("%3.1f%%", $3/$2*100) }'`
users=`users | wc -w`

echo "System information as of: $date"
echo
printf "System load:\t%s\tMemory usage:\t%s\n" $load $memory_usage
printf "Usage on /:\t%s\tSwap usage:\t%s\n" $root_usage $swap_usage
printf "Local users:\t%s\n" $users
echo

Updates (`20-updates`)

This is much more complicated than the other sections. Canonical achieves this in Ubuntu by using the same mechanism the GUI software updater works (the notifer, not synaptic). This implements the same requirement, but using the upstream dependency that — python-apt — which allows us to interact with apt’s internals.

This is quite closely based upon the version in Ubuntu, but is simplified².

#!/usr/bin/python
#
#   20-updates - create the system updates section of the MOTD
#   Copyright (c) 2013 Nick Charlton
#
#   Authors: Nick Charlton <hello@nickcharlton.net>
#   Based upon prior work by Dustin Kirkland and Michael Vogt.
#
#   This program is free software; you can redistribute it and/or modify
#   it under the terms of the GNU General Public License as published by
#   the Free Software Foundation; either version 2 of the License, or
#   (at your option) any later version.
#
#   This program is distributed in the hope that it will be useful,
#   but WITHOUT ANY WARRANTY; without even the implied warranty of
#   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#   GNU General Public License for more details.
#
#   You should have received a copy of the GNU General Public License along
#   with this program; if not, write to the Free Software Foundation, Inc.,
#   51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

import sys
import subprocess
import apt_pkg

DISTRO = subprocess.Popen(["lsb_release", "-c", "-s"],
                          stdout=subprocess.PIPE).communicate()[0].strip()

class OpNullProgress(object):
    '''apt progress handler which supresses any output.'''
    def update(self):
        pass
    def done(self):
        pass

def is_security_upgrade(pkg):
    '''
    Checks to see if a package comes from a DISTRO-security source.
    '''
    security_package_sources = [("Ubuntu", "%s-security" % DISTRO),
                               ("Debian", "%s-security" % DISTRO)]

    for (file, index) in pkg.file_list:
        for origin, archive in security_package_sources:
            if (file.archive == archive and file.origin == origin):
                return True
    return False

# init apt and config
apt_pkg.init()

# open the apt cache
try:
    cache = apt_pkg.Cache(OpNullProgress())
except SystemError, e:
    sys.stderr.write("Error: Opening the cache (%s)" % e)
    sys.exit(-1)

# setup a DepCache instance to interact with the repo
depcache = apt_pkg.DepCache(cache)

# take into account apt policies
depcache.read_pinfile()

# initialise it
depcache.init()

# give up if packages are broken
if depcache.broken_count > 0:
    sys.stderr.write("Error: Broken packages exist.")
    sys.exit(-1)

# mark possible packages
try:
    # run distro-upgrade
    depcache.upgrade(True)
    # reset if packages get marked as deleted -> we don't want to break anything
    if depcache.del_count > 0:
        depcache.init()

    # then a standard upgrade
    depcache.upgrade()
except SystemError, e:
    sys.stderr.write("Error: Couldn't mark the upgrade (%s)" % e)
    sys.exit(-1)

# run around the packages
upgrades = 0
security_upgrades = 0
for pkg in cache.packages:
    candidate = depcache.get_candidate_ver(pkg)
    current = pkg.current_ver

    # skip packages not marked as upgraded/installed
    if not (depcache.marked_install(pkg) or depcache.marked_upgrade(pkg)):
        continue

    # increment the upgrade counter
    upgrades += 1

    # keep another count for security upgrades
    if is_security_upgrade(candidate):
        security_upgrades += 1

    # double check for security upgrades masked by another package
    for version in pkg.version_list:
        if (current and apt_pkg.version_compare(version.ver_str, current.ver_str) <= 0):
            continue
        if is_security_upgrade(version):
            security_upgrades += 1
            break

print "%d updates to install." % upgrades
print "%d are security updates." % security_upgrades
print "" # leave a trailing blank line

This is reasonably well commented, if you wished to delve into the implementation, but at a higher level it:

Opens up the apt cache.
Does the equivalent of apt-get dist-upgrade.
Then the equivalent of apt-get upgrade.
Counts the possible packages which are marked to be installed or upgraded.
Reports, closes the cache and exits.

It’s important to note that whilst it does get a list of packages, it doesn’t commit any changes and so no state will change (and I’m pretty sure I didn’t do anything stupid).

I tested this on a Ubuntu 12.04 (Precise) install, Debian Squeeze and then Wheezy. That’s enough for me to be confident that it works well enough, but I couldn’t test all possible package and so it’s quite possible that the reported number will be incorrect.

Also, Ubuntu has the ability to show distro upgrades. I couldn’t work out how to replicate this. It’s less important with Debian — the major releases are hardly a common event.

You could do something similar with just bash and some text processing:

apt-get -s -o Debug::NoLocking=true upgrade | grep ^Inst | wc -l

But, this would lose the ‘security updates’ section. It’s also slightly slower, my Python came in at 0.732s, versus 0.762s on the shell command above. Not that that’s a very significant difference.

Footer (`99-footer`)

The default behaviour of motd is to present a combination of /etc/motd (which is regenerated on reboot) and /etc/motd.tail which is designed to append a sys admin message or similar. This just replicates the original functionality:

#!/bin/sh
#
#    99-footer - write the admin's footer to the MOTD
#    Copyright (c) 2013 Nick Charlton
#    Copyright (c) 2009-2010 Canonical Ltd.
#
#    Authors: Nick Charlton <hello@nickcharlton.net>
#             Dustin Kirkland <kirkland@canonical.com>
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License along
#    with this program; if not, write to the Free Software Foundation, Inc.,
#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

[ -f /etc/motd.tail ] && cat /etc/motd.tail || true

Configuration

All of these files (my filenames are in brackets around the section title) should be placed in /etc/update-motd.d/ and then made executable (chmod +x /etc/update-motd.d/). When you login, PAM will regenerate the motd file, which on Debian is located in /var/run/motd. The file /etc/motd is just a symlink to this location.

Finally, to replicate the manner in which Ubuntu presents the motd, you need to set the following in /etc/ssh/sshd_config:

PrintMotd no
PrintLastLog yes

#Banner /etc/issue

It seems conterintuitive, but PAM will post the motd anyway, so it should be turned off here. PrintLastLog gives you the last login host and time. Then, the banner can be commented out because our dynamically generated motd includes the same information.

Now I just need to turn it into a Chef cookbook…

It also, sadly, means often there’s more license text than actual code. But, I know people will blindly copy/paste the contents of the text boxes and it seems wrong to not include it. ↩
To be more specific, this is based upon apt-check.py, which can be found in /usr/lib/update-notifier/ in recent Ubuntu releases. ↩

7 Agile Best Practices that You Don't Need to Follow

Sun, 07 Jul 2013 00:00:00 +0000

Jim Bird pulls apart some of the bits of Agile.

Unlike most articles on the human side of software development, this brings up some interesting research on agile principles In the Real World™.

Notably:

Tests first or tests after isn’t important. Just that they’re there.
Pair programming as a default option seems to be counterproductive.
Remote pair programming (across locations, cultures, etc) doesn’t seem to work at all.
But, as a tool for certain situations (bring in new developers to a codebase or solving a specific problem, for example) it works rather well.
Daily stand up meetings a bit pointless taken as a constant.

Of course, like a tools, agile can look just like a fancy hammer.

via Status Code

Multiple Displays (and OpenGL)

Sat, 06 Jul 2013 00:00:00 +0000

Something I’ve not seen anyone comment on is the reasoning behind why Apple have put off Multiple Displays — like the way Mavericks implements them — for so long.

My hunch is OpenGL. Apple has a custom implementation of OpenGL which extends well into the way in which Cocoa and Cocoa Touch are implemented. On the Mac and on iOS everything is OpenGL. Mavericks finally brings us OpenGL 4.1, thus bringing the Mac up to speed with both the Linux and Windows implementations.

I assume that this was part of the reason. I imagine Mavericks brings us a much improved graphics stack and that this significant change (which bought in OpenGL 4.1) is also the reason why Apple decided to make a big change to multiple display support.

Of course, all of this is because of the impending new Mac Pro — there’s little point shipping a cool GPU configuration if we’re still writing graphics code from 2009.

But, as Thomas Brand says:.

How long we have waited.

Indeed.

Regarding the new Mac Pro, I, like Guy English, can also see cool new applications of the hardware configuration.

What's the mobile app market up to, then?

Sat, 06 Jul 2013 00:00:00 +0000

Graham Lee talks about the way the app market has ended up going.

I’ve seen his conclusion myself. The leads I’ve had recently have been for either building a specific user experience, or an interface to a wider system through some sort of API. The thing is, these projects are far, far more interesting than the crap that proceeded it.

But, it’s not necessarily good for those of us who wish we had a more early-2000’s Mac software environment — the likes where Panic, Red Sweater Software and many more can shine through with dedicated end-user customers.

How Stripe Buids Software: Interview with Greg Brockman

Sat, 06 Jul 2013 00:00:00 +0000

This interview with Greg Brockman of Stripe is quite interesting..

It touches on the way they manage their development teams at Stripe, breaking into teams for parts of the different projects with one of the programmers leading. It’s a good demonstration of how you can function a company around the way Open Source projects are run and using various different tools (in this case GitHub, Asana , etc) to glue things together.

via Status Code.

The Makers of Things

Fri, 05 Jul 2013 00:00:00 +0000

Anne Hollowday put together this fantastic series about members of the Society for Model and Experimental Engineers.

You should watch it.

Experiments with Android, a IOIO board and Heart Rate Monitoring

Thu, 06 Jun 2013 00:00:00 +0000

A few weeks back now, I worked on a project for iDAT. I came in at the end to fix a few bugs on a Android/hardware integration project. The idea was to use an Android phone, along with a IOIO board and a Polar heart rate monitor breakout board to collect data on a person’s movements, heart rate (and a few other things).

This was split into two distinct sections; the first was the hardware itself and secondly the software which would poll data from the IOIO board and do something to visualise it.

For the hardware, I started off first trying to get some data from the HMRI (the shorter name of the breakout board) using an Arduino. This worked quite quickly. The board uses I²C to communicate and so this is only a few pins. Whilst Dan Julio provides some example code for the Arduino, this was a bit old and so I needed to update this. You can find the updated code in my GitHub project, hmri_arduino. It should work fine with Arduino 1.0 and greater.

To interface the HMRI with the IOIO board, the process was much the same as the Arudino, but I needed to provide two 4.7kΩ pull up resistors to get a signal (the Arduino provides these for you). I also needed to desolder the SJ1 contacts and solder the OP0 contacts for an I²C connection, a section from the manual is listed below.

Figure 1: I²C Schematic from the manual

After this, I needed to interface the HMRI, through the IOIO board on the Android device. The IOIO project provides a library to do this — the interface is Arduino inspired — and once you get the correct version, it’s quite easy to get working.

I was using a Samsung Galaxy SII (with a slightly older version of Android) and the older style IOIO board. I matched up the board type with the current release as listed on the IOIO downloads page. This ended up stumbling me for quite a while; the board refuses to connect without precisely the right firmware/library combination.

For the implementation itself, I used a singleton¹ which is started up with the main activity and is then shutdown on completion. The idea behind this was to create a connection to the hardware and keep one thread which would handleit (the singleton starts a thread for interacting with the hardware, and then provides a thread-safe way in which to interface with it). Sadly, this didn’t quite work with the way Android is designed and so doesn’t quite work how I had expected. Instead, the hardware connection is started and stopped when the main activity run-loop is either started or stopped. Due to the way Android implements it’s run-loop this is all that is possible (coming from predominantly iOS, this such an odd implementation and quite annoying).

I never resolved the issue before shipping; as far as I understand, it’s impossible to fully resolve it because of Android’s stupid implementation.

Update: Ytai Ben-Tsvi (the developer behind IOIO) emailed me to say that I should have been using Android’s “Services” functionality, which I didn’t know existed. This would solve all of my state issues.

And finally, the code for the whole project is up on GitHub (which should give you the best way to see it in action).

Many consider the Singleton to be a programming anti-pattern, but I disagree for situations such as this. Yes, it does introduce global scope, but when you’re accessing hardware, or a data model (where I’ve used singletons before) it works quite well. You only want to instantiate it once, anyway. In the case of hardware it’s preferable to keep the same connection open. ↩

Side Projects

Tue, 04 Jun 2013 00:00:00 +0000

Much to my chagrin about Medium in general, there’s been some nice posts recently. This set on Side Projects fits exactly where I feel about them;

I like this idea, and as I redo mine (I’ve technically graduated now¹) I’m adjusting mine to focus on these equally with everything else.

I’m not looking for a job, as I’m intending on doing a Masters. However, I’m open to taking on freelance projects, if you have them. ↩

Wired writes something sensible about the Internet of Things