Guix Package, Part 1
Coming from Arch (btw I use Arch), I was very used to being able to pacman -Ss <anything> and find the program I was looking for. Less mainstream distributions like Guix, especially ones that have to maintain their own package format, often don’t offer the same convenience. This post follows my journey to build a Guix package for hugo, the tool I currently use to write this blog.
Anatonomy of a Guix package
The example in the Guix manual serves as a good start. Almost all the fields have an obvious replacement, though I don’t know how to get a hash for the origin or what a build-system is. The latter comes up pretty quickly in the manual, and Guix already has a build system for go. Some searching indicates I could run guix download to get the hash, but it doesn’t support git fetches. Other results suggest the system will guide you to the answer if you fake it at first. With that in mind:
(use-modules
(guix build-system go)
(guix git-download)
(guix licenses)
(guix packages))
(define-public hugo
(package
(name "go-github-com-gohugoio-hugo")
(version "v0.71.0")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/gohugoio/hugo")
(commit version)))
(file-name (git-file-name name version))
(sha256 (base32 "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"))))
(build-system go-build-system)
(arguments
`(#:import-path "github.com/gohugoio/hugo"))
(synopsis "The world’s fastest framework for building websites.")
(description "Hugo is one of the most popular open-source static site generators. With its amazing speed and flexibility, Hugo makes building websites fun again.")
(home-page "https://gohugo.io/")
(license asl2.0)))
Ok that’s not so bad, but what do I do with this file?
$ guix package -f hugo.scm
guix package: error: cannot install non-package object: #<unspecified>
Ah, I need to return the package
$ echo "\nhugo" > hugo.scm
$ guix package -f hugo.scm
The following package will be installed:
go-github-com-gohugoio-hugo v0.71.0
The following derivations will be built:
/gnu/store/jl15j2v4m3ljbwnmxg1z48g7ly9xfwkr-profile.drv
/gnu/store/623s1l3bzgll2g038ig32svivpsrv5ww-go-github-com-gohugoio-hugo-v0.71.0.drv
/gnu/store/xmzp4br0km4i6niydrs5bqcydmsz9jil-go-github-com-gohugoio-hugo-v0.71.0-checkout.drv
The following profile hooks will be built:
/gnu/store/171i08l07q637w5ppxmzi8zp4ipd2vgi-ca-certificate-bundle.drv
/gnu/store/icm1bi87klh1qkgfmcrjb45lfbh2lfhl-fonts-dir.drv
/gnu/store/pway1ljd6cqaz7z9ghjbpz6c6nc7ibzp-info-dir.drv
/gnu/store/v1sdkxp6fsyv56979h4j52wlghy76cnr-manual-database.drv
building /gnu/store/x298xzys1w5llnpryayh40p7y7yilj9p-go-github-com-gohugoio-hugo-v0.71.0-checkout.drv...
/r:sha256 hash mismatch for /gnu/store/9ihmg7z55w1r1m3fn3yg2s3h8kmpk0sw-go-github-com-gohugoio-hugo-v0.71.0-checkout:
expected hash: 0aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
actual hash: 1cjf69y8wvjq7kpncvqzmhc03pmf21nfxqpdnh64v34d14gpa1aa
hash mismatch for store item '/gnu/store/kvnww449q9ingada03v4gwqrslp8j8h0-go-github-com-gohugoio-hugo-v0.71.0-checkout'
build of /gnu/store/xmzp4br0km4i6niydrs5bqcydmsz9jil-go-github-com-gohugoio-hugo-v0.71.0-checkout.drv failed
View build log at '/var/log/guix/drvs/xm/zp4br0km4i6niydrs5bqcydmsz9jil-go-github-com-gohugoio-hugo-v0.71.0-checkout.drv.bz2'.
cannot build derivation `/gnu/store/623s1l3bzgll2g038ig32svivpsrv5ww-go-github-com-gohugoio-hugo-v0.71.0.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/jl15j2v4m3ljbwnmxg1z48g7ly9xfwkr-profile.drv': 1 dependencies couldn't be built
guix package: error: build of `/gnu/store/jl15j2v4m3ljbwnmxg1z48g7ly9xfwkr-profile.drv' failed
Sweet! Let’s try that again.
$ guix package -f hugo.scm
...
building /gnu/store/229yvb5yrhpyi3gqd75h9i5xb2mamaaz-go-github-com-gohugoio-hugo-v0.71.0.drv...
\ 'build' phasebuilder for `/gnu/store/229yvb5yrhpyi3gqd75h9i5xb2mamaaz-go-github-com-gohugoio-hugo-v0.71.0.drv' failed with exit code 1
build of /gnu/store/229yvb5yrhpyi3gqd75h9i5xb2mamaaz-go-github-com-gohugoio-hugo-v0.71.0.drv failed
View build log at '/var/log/guix/drvs/22/9yvb5yrhpyi3gqd75h9i5xb2mamaaz-go-github-com-gohugoio-hugo-v0.71.0.drv.bz2'.
cannot build derivation `/gnu/store/vjbvx8vqh3bxl18l5dapaj64ndy8z94s-profile.drv': 1 dependencies couldn't be built
guix package: error: build of `/gnu/store/vjbvx8vqh3bxl18l5dapaj64ndy8z94s-profile.drv' failed
Ok, so what does the log file say? A lot, it turns out, but near the bottom:
$ bzcat /var/log/guix/drvs/22/9yvb5yrhpyi3gqd75h9i5xb2mamaaz-go-github-com-gohugoio-hugo-v0.71.0.drv.bz2 | less
...
starting phase `build'
src/github.com/gohugoio/hugo/cache/filecache/filecache.go:31:2: cannot find package "github.com/BurntSushi/locker" in any of:
/gnu/store/w8gjhcw6a16rk1dvxa97bz2znal5fihm-go-1.13.9/src/github.com/BurntSushi/locker (from $GOROOT)
/tmp/guix-build-go-github-com-gohugoio-hugo-v0.71.0.drv-0/src/github.com/BurntSushi/locker (from $GOPATH)
src/github.com/gohugoio/hugo/parser/metadecoders/decoder.go:26:2: cannot find package "github.com/BurntSushi/toml" in any of:
/gnu/store/w8gjhcw6a16rk1dvxa97bz2znal5fihm-go-1.13.9/src/github.com/BurntSushi/toml (from $GOROOT)
/tmp/guix-build-go-github-com-gohugoio-hugo-v0.71.0.drv-0/src/github.com/BurntSushi/toml (from $GOPATH)
...
This is certainly unexpected, with Modules enabled, go is supposed to download dependencies automatically during go build or go install. So why isn’t it? Looking at the logs again, I see some debug output:
Here are the results of `go env`:
GO111MODULE="off"
...
GOMOD=""
That’s certainly suspicious. Looking at the docs:
GOMOD The absolute path to the go.mod of the main module. If module-aware mode is enabled, but there is no go.mod, GOMOD will be os.DevNull ("/dev/null" on Unix-like systems, “NUL” on Windows). If module-aware mode is disabled, GOMOD will be the empty string.
Sure enough, there’s a TODO comment in the go-build-system source. Well at least I know for certain what’s going on, and now I have a choice to make: either modify go-build-system to support modules or package all the dependencies. I can already tell this is going to be way more work than I had hoped :D
Hacking on the build system
You can tell from the section heading where this is going. There is an explicit call (setenv "GO111MODULE" "off") in the setup-go-environment exported by guix/build/go-build-system.scm. Having read the packaging tutorial, I know I can modify the phases of a build system. But this is pretty unfamiliar territory, so I want to attempt to exercise my (severely unpracticed) lisp instincts of using the repl.
$ guix repl
GNU Guile 2.2.6
Copyright (C) 1995-2019 Free Software Foundation, Inc.
Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.
This program is free software, and you are welcome to redistribute it
under certain conditions; type `,show c' for details.
Enter `,help' for help.
scheme@(guix-user)> (@ (guix build go-build-system) %standard-phases)
$1 = ((set-SOURCE-DATE-EPOCH . #<procedure set-SOURCE-DATE-EPOCH _>) (set-paths . #<procedure set-paths (#:key target inputs native-inpu
ts search-paths native-search-paths)>) (install-locale . #<procedure install-locale (#:key locale locale-category)>) (setup-go-environme
nt . #<procedure setup-go-environment (#:key inputs outputs)>) (unpack . #<procedure unpack (#:key source import-path unpack-path)>)
...)
scheme@(guix-user)> (define* (setup-gomod-environment #:key inputs outputs #:allow-other-keys)
... (setenv "GOCACHE" "/tmp/go-cache")
... (setenv "GOBIN" (string-append (assoc-ref outputs "out") "/bin"))
... #t)
scheme@(guix-user)> ,use (guix build utils)
scheme@(guix-user)> (modify-phases (@ (guix build go-build-system) %standard-phases)
... (replace 'setup-go-environment setup-gomod-environment))
$2 = ((set-SOURCE-DATE-EPOCH . #<procedure set-SOURCE-DATE-EPOCH _>) (set-paths . #<procedure set-paths (#:key target inputs native-inpu
ts search-paths native-search-paths)>) (install-locale . #<procedure install-locale (#:key locale locale-category)>) (setup-go-environme
nt . #<procedure setup-gomod-environment (#:key inputs outputs)>) (unpack . #<procedure unpack (#:key source import-path unpack-path)>)
...)
scheme@(guix-user)>
That’s quite a bit of code, but fortunately I see the change to setup-gomod-environment in the middle. After translating this to my package definition file…
ERROR: In procedure %resolve-variable:
Unbound variable: setup-gomod-environment
Ack, I was defining a top-level procedure, but the symbol can’t be resolved by the calling module. Replacing it with a (lambda* ...) definition works just fine. On to the next problem:
Fetching https://github.com?go-get=1
https fetch failed: Get https://github.com?go-get=1: dial tcp: lookup github.com on [::1]:53: read udp [::1]:56334->[::1]:53: read: connection refused
build command-line-arguments: cannot load github.com/gohugoio/hugo/commands: cannot find module providing package github.com/gohugoio/hugo/commands
Building '' failed.
I will not be beaten! I have a fuzzy recollection from past experience that golang has a dependency on some files in /etc/ for doing DNS lookups. It shouldn’t be specific to golang, but I just remember reading this issue awhile back where it was fixed. That’s enought to set me looking in the right direction.
$ sudo strace -o log -p $(pidof guix-daemon) -f
$ ls -lh log
-rw-r--r-- 1 jordan users 21M May 25 11:51 log
That’s a lot of syscalls. Searching for the domain gets me to the problem area, where sure enough:
20118 openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC <unfinished ...>
20118 <... openat resumed>) = -1 ENOENT (No such file or directory)
20118 openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC <unfinished ...>
20118 <... openat resumed>) = -1 ENOENT (No such file or directory)
There’s more output below that shows golang trying to open a socket on localhost, which explains the error from the build log (and is expected). Luckily, I also see some references to chroot in the file path, e.g.
/gnu/store/r9yirmj96n4bs4zfbk2df4hjsy1fxqg7-go-github-com-gohugoio-hugo-v0.71.0.drv.chroot
And sure enough, there’s a chroot syscall shortly after that directory is created. Ok, so it seems like the build environment is missing some critical files. It turns out that this is purposeful. And fortunately I’m familiar with this problem - I once got to know a guy who maintained a lot of packages for Fedora. There’s a belief that the system package manager should express the dependencies for language modules too. In other words, I shouldn’t need to use go get ... or rely on the list in go.mod or go.sum, instead I ought declare them in the deb or rpm or package manifest for Guix. I don’t disagree in principle, but it does take a significant effort.
As an aside, I’m equally annoyed and impressed that the whole reproducible build thing is taken seriously.
Dependencies
I have returned to the prior fork in the road and will attempt to package all of hugo’s dependencies.
$ wc -l go.mod
73 go.mod
Big oof. There’s no way I’m going to write roughly 70 package definitions by hand. Hmmm, I wonder what scripting language I should use for this…
#!/run/current-system/profile/bin/guile \
-e main -s
# vim: set syntax=scheme:
!#
(define (main args)
(display args))
I won’t elaborate on the process of writing this script; it’s source code is available on my channel1. Suffice it to say I
- parse the
go.modfile, creating a list of(name . version)pairs - iterate through the list to create a bunch of
(define-public <name> (package ...))statements - use
(ice-9 pretty-print)to write these to a file, prefixed with a(define-module ...)header - took forever to do it
I can’t emphasize that last point enough. I really thought I could knock this out pretty quick despite never having used Guile before. In truth, Guile was not my stumbling block, though naturally I spent a little extra time searching its manual for standard library functions. The difficulty was two fold:
- Getting a hash
I don’t want to edit the package definitions by hand, so using the trick from earlier won’t work. I had to read through a handful of the Guix source before arriving at:
(define (calc-hash-git uri commit name)
(format (current-error-port) "Downloading source for ~a\n" name)
(let* ((path (with-store store
(call-with-temporary-directory
(lambda (tmp)
(if (build-git:git-fetch uri commit tmp)
(add-to-store store name #t "sha256" tmp)
#f)))))
(hash (and path (let-values (((port get-hash) (open-sha256-port)))
(write-file path port)
(force-output port)
(get-hash)))))
(and hash (bytevector->nix-base32-string hash))))
- Fetching a go module
That’s right! You can’t just slap https:// in front of a package path and expect to git clone it. No, instead you need to make a HTTP request with ?go-get=1 and parse the response HTML for a <meta name="go-import"> tag, which tells you where to fetch the source from. And there are other silly rules too, like repos that define multiple packages in subdirectories.
Results
Once the script was finally working, I added the packages to the (propagated-inputs) field so that their source would be available at build time. Despite that, go refuses to look in $GOPATH/src/ to resolve modules. I’m not saying I tried everything, but no combination of tweaking file paths, GOPROXY, or the working directory seemed to help. It was quite a frustrating process - everything the compiler needed existed, but it’s hatred for sanity rivalled the power of a thousand suns. My mind is set on completing this, so I’ll return to the problem after some time to think…