From 5847245638c343b9b89878daa09e14ef811c9221 Mon Sep 17 00:00:00 2001 From: Nikola Petrov Date: Fri, 2 Oct 2026 10:24:24 +0200 Subject: [PATCH] Bundle a static pdftotext into the binary The PDF parsers shell out to pdftotext, so every machine running money needed poppler-utils installed. scripts/build-bundled.sh now builds a deployable binary that carries its own: pdftotext is compiled in a container from a checksum-pinned poppler release as a fully static musl executable, then embedded with `go build -tags bundled`. The result is one file that runs on any Linux of that architecture with nothing installed alongside it. It is still the real pdftotext, run as a subprocess. Linking poppler through cgo would have cost the pure-Go build, and its C++ text API is not guaranteed to space columns the way pdftotext -layout does, which is what the parsers were tuned on. Only what text extraction needs is compiled in -- no fontconfig, cairo or image codecs -- and its output is byte-identical to a full distro build on the same PDF. At runtime the embedded copy is written to the user cache directory, not /tmp, which servers often mount noexec. It is named by content hash, so a newer build never runs an older copy, and verified before reuse, so a write cut short by a killed process is replaced rather than trusted. `money config` says which pdftotext is in use. The tag is opt-in: plain go build and go test never need the 5 MB executable, which is gitignored rather than committed. Building with the tag for anything but linux/amd64 or linux/arm64 fails with a message saying so. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 4 + CLAUDE.md | 11 +++ README.md | 28 +++++- cmd/money/main.go | 1 + internal/parser/bundled.go | 118 +++++++++++++++++++++++++ internal/parser/bundled_linux_amd64.go | 13 +++ internal/parser/bundled_linux_arm64.go | 13 +++ internal/parser/bundled_test.go | 79 +++++++++++++++++ internal/parser/bundled_unsupported.go | 8 ++ internal/parser/pdftext.go | 16 +++- scripts/build-bundled.sh | 40 +++++++++ scripts/pdftotext.Containerfile | 70 +++++++++++++++ 12 files changed, 398 insertions(+), 3 deletions(-) create mode 100644 internal/parser/bundled.go create mode 100644 internal/parser/bundled_linux_amd64.go create mode 100644 internal/parser/bundled_linux_arm64.go create mode 100644 internal/parser/bundled_test.go create mode 100644 internal/parser/bundled_unsupported.go create mode 100755 scripts/build-bundled.sh create mode 100644 scripts/pdftotext.Containerfile diff --git a/.gitignore b/.gitignore index c7325d9..56b2684 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,7 @@ # Never commit a data root that happens to live inside the repo. *.db + +# The static pdftotext and the binaries scripts/build-bundled.sh makes from it. +/internal/parser/bundled/ +/dist/ diff --git a/CLAUDE.md b/CLAUDE.md index a7facef..14c34a9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -261,6 +261,17 @@ Keep PDF text extraction separate from parsing: the bank parsers expose a pure a runtime dependency of `nlb` and `traderepublic`; no Go library reconstructs column layout as well. +It can be carried inside the binary instead: `scripts/build-bundled.sh` builds +a static musl `pdftotext` in a container and embeds it under `-tags bundled`, +and `pdftotextCommand` writes it to the user cache dir (named by content hash, +verified before reuse) and runs that. It is still the real executable run as a +subprocess, deliberately — not poppler linked in through cgo, which would cost +the pure-Go build, and not a different extraction API whose spacing might not +match what the parsers were tuned on. The tag is opt-in so `go build`/`go test` +never need the 5 MB file, which is gitignored rather than committed. Bumping +`POPPLER_VERSION` means updating its pinned SHA-256 too, and checking real +statements still parse — that output is what the parsers depend on. + Do not hardcode absolute column positions from a sample PDF. `pdftotext` compresses runs of spaces, so columns shift with font and page size — derive positions from the header line (`traderepublic`) or from the line being parsed diff --git a/README.md b/README.md index a158a91..5f20d9a 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,31 @@ Give it the package path, not `cmd/money/main.go`. Naming the file puts the go tool in file mode: it names the binary after the source file (`main`) and compiles only the files listed, which breaks the moment the package has two. +That build runs `pdftotext` from `PATH` for the PDF parsers, so the machine +needs poppler-utils. To deploy one file with nothing to install alongside it, +build with `pdftotext` inside: + +``` +scripts/build-bundled.sh # → dist/money-linux-amd64 +scripts/build-bundled.sh arm64 # → dist/money-linux-arm64 +``` + +It needs podman or docker: `pdftotext` is compiled in a container from a +checksum-pinned poppler release (`scripts/pdftotext.Containerfile`), as a fully +static musl executable with only what text extraction needs, then embedded with +`go build -tags bundled`. The result runs on any Linux of that architecture. +The first PDF import writes the embedded copy to `~/.cache/money/` (or +`$XDG_CACHE_HOME/money/`) and runs it from there; `money config` says which +`pdftotext` is in use. Only Linux on amd64 and arm64 is offered — elsewhere, +install poppler-utils. + +Building arm64 from an x86 machine runs the container under emulation, so the +host needs qemu-user-static and the first build is slow; the executable is then +cached under `internal/parser/bundled/` and later builds only rebuild money. + +Poppler is GPL, so a bundled binary is GPL-licensed as a whole. That matters +only if you hand the binary to someone else. + ## Usage ``` @@ -672,7 +697,8 @@ parser = "traderepublic" The two PDF parsers shell out to `pdftotext -layout` (poppler-utils), exactly as the Python versions did; its layout reconstruction is what makes the -column-based parsing work. +column-based parsing work. A build made with `scripts/build-bundled.sh` +carries its own copy instead (see Install). Amount parsing is shared and deliberately tolerant: `1.234,56`, `-45.20`, `45,20-`, `(45.20)` and `45.20 EUR` all work, whichever parser reads them. diff --git a/cmd/money/main.go b/cmd/money/main.go index 40c6c8d..e2937df 100644 --- a/cmd/money/main.go +++ b/cmd/money/main.go @@ -137,6 +137,7 @@ func cmdConfig(root string, source config.RootSource) error { fmt.Fprintf(w, "data root\t%s\n", root) fmt.Fprintf(w, "chosen by\t%s\n", source) fmt.Fprintf(w, "config file\t%s (%s)\n", path, exists) + fmt.Fprintf(w, "pdftotext\t%s\n", parser.PdftotextSource()) if _, err := os.Stat(root); err != nil { fmt.Fprintf(w, "status\tdoes not exist yet\n") } else { diff --git a/internal/parser/bundled.go b/internal/parser/bundled.go new file mode 100644 index 0000000..8b0072e --- /dev/null +++ b/internal/parser/bundled.go @@ -0,0 +1,118 @@ +package parser + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "fmt" + "io" + "os" + "path/filepath" + "sync" +) + +// bundledPdftotext is a static pdftotext built into this binary, or empty +// when it was built without one. Only `go build -tags bundled` fills it (see +// scripts/build-bundled.sh), so an ordinary build stays pure Go and needs no +// 5 MB executable checked out next to it. +var bundledPdftotext []byte + +var ( + bundledOnce sync.Once + bundledPath string + bundledErr error +) + +// pdftotextCommand names the pdftotext to run: the bundled copy when there is +// one, otherwise whatever is on PATH. The bundled copy wins because it is the +// version the binary was built and checked with, which is the point of +// carrying it rather than trusting the host's. +func pdftotextCommand() (string, error) { + if len(bundledPdftotext) == 0 { + return "pdftotext", nil + } + bundledOnce.Do(func() { bundledPath, bundledErr = installBundled(bundledPdftotext) }) + return bundledPath, bundledErr +} + +// PdftotextSource says which pdftotext the PDF parsers will run, for +// `money config`: a deployment that lacks one should find out before an +// import fails on it. +func PdftotextSource() string { + if len(bundledPdftotext) == 0 { + return "pdftotext from PATH (not bundled into this build)" + } + path, err := pdftotextCommand() + if err != nil { + return fmt.Sprintf("bundled, but it cannot be installed: %v", err) + } + return "bundled, run from " + path +} + +// installBundled writes the executable to the user's cache directory, where it +// can be exec'd, and returns its path. The name carries the content hash, so a +// newer build never runs an older build's copy, and a file already there is +// only reused once its contents check out — a truncated write from a killed +// process must not become the pdftotext every later run trusts. +func installBundled(bin []byte) (string, error) { + sum := sha256.Sum256(bin) + name := "pdftotext-" + hex.EncodeToString(sum[:8]) + + // The cache directory rather than /tmp: /tmp is often mounted noexec on + // servers, and the cache survives reboots, so this happens once per build. + dir, err := os.UserCacheDir() + if err != nil { + dir = os.TempDir() + } + dir = filepath.Join(dir, "money") + path := filepath.Join(dir, name) + + if same, err := hasContents(path, sum); err != nil { + return "", err + } else if same { + return path, nil + } + + if err := os.MkdirAll(dir, 0o755); err != nil { + return "", fmt.Errorf("install bundled pdftotext: %w", err) + } + // Written beside the target and renamed into place, so a concurrent run + // never execs a half-written file. + tmp, err := os.CreateTemp(dir, name+".*") + if err != nil { + return "", fmt.Errorf("install bundled pdftotext: %w", err) + } + defer os.Remove(tmp.Name()) // a no-op once renamed + if _, err := tmp.Write(bin); err != nil { + tmp.Close() + return "", fmt.Errorf("install bundled pdftotext to %s: %w", dir, err) + } + if err := tmp.Chmod(0o755); err != nil { + tmp.Close() + return "", fmt.Errorf("install bundled pdftotext to %s: %w", dir, err) + } + if err := tmp.Close(); err != nil { + return "", fmt.Errorf("install bundled pdftotext to %s: %w", dir, err) + } + if err := os.Rename(tmp.Name(), path); err != nil { + return "", fmt.Errorf("install bundled pdftotext to %s: %w", dir, err) + } + return path, nil +} + +// hasContents reports whether the file at path exists and hashes to sum. +func hasContents(path string, sum [sha256.Size]byte) (bool, error) { + f, err := os.Open(path) + if os.IsNotExist(err) { + return false, nil + } + if err != nil { + return false, fmt.Errorf("check bundled pdftotext: %w", err) + } + defer f.Close() + h := sha256.New() + if _, err := io.Copy(h, f); err != nil { + return false, fmt.Errorf("check bundled pdftotext: %w", err) + } + return bytes.Equal(h.Sum(nil), sum[:]), nil +} diff --git a/internal/parser/bundled_linux_amd64.go b/internal/parser/bundled_linux_amd64.go new file mode 100644 index 0000000..14a2b0e --- /dev/null +++ b/internal/parser/bundled_linux_amd64.go @@ -0,0 +1,13 @@ +//go:build bundled + +package parser + +import _ "embed" + +// Built by scripts/build-bundled.sh amd64, which also runs the go build that +// needs it. The file is not checked in, so this only compiles after that. +// +//go:embed bundled/pdftotext-linux-amd64 +var embeddedPdftotext []byte + +func init() { bundledPdftotext = embeddedPdftotext } diff --git a/internal/parser/bundled_linux_arm64.go b/internal/parser/bundled_linux_arm64.go new file mode 100644 index 0000000..6b4dc20 --- /dev/null +++ b/internal/parser/bundled_linux_arm64.go @@ -0,0 +1,13 @@ +//go:build bundled + +package parser + +import _ "embed" + +// Built by scripts/build-bundled.sh arm64, which also runs the go build that +// needs it. The file is not checked in, so this only compiles after that. +// +//go:embed bundled/pdftotext-linux-arm64 +var embeddedPdftotext []byte + +func init() { bundledPdftotext = embeddedPdftotext } diff --git a/internal/parser/bundled_test.go b/internal/parser/bundled_test.go new file mode 100644 index 0000000..f268fbe --- /dev/null +++ b/internal/parser/bundled_test.go @@ -0,0 +1,79 @@ +package parser + +import ( + "os" + "os/exec" + "path/filepath" + "testing" +) + +// fakePdftotext stands in for the real executable: what matters here is how it +// is put on disk, not what it does once there. +var fakePdftotext = []byte("#!/bin/sh\necho bundled\n") + +// The bundled copy has to land somewhere it can be exec'd, and run. +func TestInstallBundledRuns(t *testing.T) { + t.Setenv("XDG_CACHE_HOME", t.TempDir()) + path, err := installBundled(fakePdftotext) + if err != nil { + t.Fatal(err) + } + out, err := exec.Command(path).Output() + if err != nil { + t.Fatalf("run %s: %v", path, err) + } + if string(out) != "bundled\n" { + t.Errorf("output %q", out) + } +} + +// A copy already in place is reused, but only once its contents check out: a +// write cut short by a killed process must not become what every later run +// trusts. +func TestInstallBundledReplacesADamagedCopy(t *testing.T) { + t.Setenv("XDG_CACHE_HOME", t.TempDir()) + path, err := installBundled(fakePdftotext) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, fakePdftotext[:5], 0o755); err != nil { + t.Fatal(err) + } + again, err := installBundled(fakePdftotext) + if err != nil { + t.Fatal(err) + } + if again != path { + t.Errorf("reinstalled to %s, want %s", again, path) + } + if got, _ := os.ReadFile(path); string(got) != string(fakePdftotext) { + t.Errorf("damaged copy kept: %q", got) + } +} + +// Each build's copy has its own name, so a newer binary never runs an older +// one's pdftotext left in the same cache. +func TestInstallBundledNamesByContent(t *testing.T) { + t.Setenv("XDG_CACHE_HOME", t.TempDir()) + a, err := installBundled(fakePdftotext) + if err != nil { + t.Fatal(err) + } + b, err := installBundled(append([]byte(nil), "#!/bin/sh\necho newer\n"...)) + if err != nil { + t.Fatal(err) + } + if a == b || filepath.Dir(a) != filepath.Dir(b) { + t.Errorf("paths %s and %s, want two names in one directory", a, b) + } +} + +// Without -tags bundled, nothing is embedded and PATH decides, as before. +func TestUnbundledUsesPath(t *testing.T) { + if len(bundledPdftotext) != 0 { + t.Skip("built with -tags bundled") + } + if cmd, err := pdftotextCommand(); err != nil || cmd != "pdftotext" { + t.Errorf("command = %q, %v; want pdftotext from PATH", cmd, err) + } +} diff --git a/internal/parser/bundled_unsupported.go b/internal/parser/bundled_unsupported.go new file mode 100644 index 0000000..11942db --- /dev/null +++ b/internal/parser/bundled_unsupported.go @@ -0,0 +1,8 @@ +//go:build bundled && !(linux && (amd64 || arm64)) + +package parser + +// pdftotext is only built for Linux, as a static musl executable; there is +// nothing to embed for this platform. Build without -tags bundled and install +// poppler-utils instead. +var _ = bundled_pdftotext_is_only_available_for_linux_amd64_and_linux_arm64 diff --git a/internal/parser/pdftext.go b/internal/parser/pdftext.go index fe2ebe2..978e2f5 100644 --- a/internal/parser/pdftext.go +++ b/internal/parser/pdftext.go @@ -17,12 +17,18 @@ const pdfToTextTimeout = 2 * time.Minute // // This shells out to poppler's pdftotext rather than decoding the PDF in Go: // the layout reconstruction it does is the whole reason the column-based -// parsers work, and no Go library matches it. +// parsers work, and no Go library matches it. A build made with -tags bundled +// carries its own copy, so a server needs nothing installed. func pdfToText(path string) (string, error) { + bin, err := pdftotextCommand() + if err != nil { + return "", err + } + ctx, cancel := context.WithTimeout(context.Background(), pdfToTextTimeout) defer cancel() - cmd := exec.CommandContext(ctx, "pdftotext", "-layout", path, "-") + cmd := exec.CommandContext(ctx, bin, "-layout", path, "-") var stdout, stderr bytes.Buffer cmd.Stdout = &stdout cmd.Stderr = &stderr @@ -34,6 +40,12 @@ func pdfToText(path string) (string, error) { if errors := strings.TrimSpace(stderr.String()); errors != "" { return "", fmt.Errorf("pdftotext %s: %w: %s", path, err, errors) } + if bin != "pdftotext" { + // The copy was installed, so failing to start it is about where it + // was put: a cache directory on a noexec mount is the usual cause. + return "", fmt.Errorf("bundled pdftotext at %s did not run (is that directory mounted noexec? "+ + "point XDG_CACHE_HOME somewhere else): %w", bin, err) + } if _, lookErr := exec.LookPath("pdftotext"); lookErr != nil { return "", fmt.Errorf("pdftotext is not installed (it ships with poppler-utils): %w", lookErr) } diff --git a/scripts/build-bundled.sh b/scripts/build-bundled.sh new file mode 100755 index 0000000..dd60fce --- /dev/null +++ b/scripts/build-bundled.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# Builds money with a static pdftotext inside it: one file to copy to a server, +# with nothing to install there — not even poppler-utils. +# +# scripts/build-bundled.sh # linux, this machine's architecture +# scripts/build-bundled.sh arm64 # linux/arm64 (podman/docker need qemu) +# +# pdftotext is built in a container from a checksum-pinned poppler release +# (scripts/pdftotext.Containerfile) and cached under internal/parser/bundled/, +# so later runs only rebuild money. Delete that file to rebuild it. +set -eu +cd "$(dirname "$0")/.." + +arch=${1:-$(go env GOARCH)} +case $arch in +amd64 | arm64) ;; +*) + echo "build-bundled: no static pdftotext for linux/$arch; use amd64 or arm64" >&2 + exit 1 + ;; +esac + +bin=internal/parser/bundled/pdftotext-linux-$arch +if [ ! -f "$bin" ] || [ scripts/pdftotext.Containerfile -nt "$bin" ]; then + engine=$(command -v podman || command -v docker) || { + echo "build-bundled: building pdftotext needs podman or docker" >&2 + exit 1 + } + out=$(mktemp -d) + trap 'rm -rf "$out"' EXIT + "$engine" build --platform "linux/$arch" -f scripts/pdftotext.Containerfile \ + --output "type=local,dest=$out" scripts + mkdir -p internal/parser/bundled + mv "$out/pdftotext" "$bin" +fi + +mkdir -p dist +CGO_ENABLED=0 GOOS=linux GOARCH=$arch \ + go build -tags bundled -trimpath -o "dist/money-linux-$arch" ./cmd/money +echo "dist/money-linux-$arch" diff --git a/scripts/pdftotext.Containerfile b/scripts/pdftotext.Containerfile new file mode 100644 index 0000000..92f9561 --- /dev/null +++ b/scripts/pdftotext.Containerfile @@ -0,0 +1,70 @@ +# Builds a fully static pdftotext for `go build -tags bundled`, so the money +# binary can carry it rather than depend on poppler-utils on the host. +# +# Only what text extraction needs is compiled in. Fonts are never rendered, so +# fontconfig, cairo and the image codecs (JPEG, JPEG 2000, TIFF) are left out, +# as are colour management, HTTP and signature support. Freetype and zlib are +# the two dependencies poppler will not build without. +# +# Built against musl because glibc cannot be linked statically in any way that +# survives a different host. +# +# Run it through scripts/build-bundled.sh rather than by hand. + +FROM docker.io/library/alpine:3.22 AS build + +RUN apk add --no-cache build-base cmake samurai pkgconf \ + freetype-dev freetype-static zlib-dev zlib-static \ + libpng-dev libpng-static bzip2-static brotli-static + +# Pinned by checksum: this binary ends up inside money, so the source it is +# built from must be exactly the one that was reviewed. +ARG POPPLER_VERSION=26.09.0 +ARG POPPLER_SHA256=8059eadb6805340768f138c465b57f8164c92b4a0773c37ef031ea6c0d987b2e + +WORKDIR /src +RUN wget -q "https://poppler.freedesktop.org/poppler-${POPPLER_VERSION}.tar.xz" \ + && echo "${POPPLER_SHA256} poppler-${POPPLER_VERSION}.tar.xz" | sha256sum -c - \ + && tar xf "poppler-${POPPLER_VERSION}.tar.xz" --strip-components=1 \ + && rm "poppler-${POPPLER_VERSION}.tar.xz" + +# The find modules would otherwise settle on the shared libraries, and freetype's static +# archive does not carry its own dependencies (png, bzip2, brotli), so they are +# appended from what pkg-config says a static freetype needs. +RUN cmake -S . -B build -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_SHARED_LIBS=OFF \ + -DFREETYPE_LIBRARY_RELEASE=/usr/lib/libfreetype.a \ + -DPNG_LIBRARY_RELEASE=/usr/lib/libpng.a \ + -DZLIB_LIBRARY_RELEASE=/usr/lib/libz.a \ + -DCMAKE_EXE_LINKER_FLAGS=-static \ + -DCMAKE_CXX_STANDARD_LIBRARIES="$(pkg-config --static --libs freetype2)" \ + -DFONT_CONFIGURATION=generic \ + -DENABLE_UTILS=ON \ + -DENABLE_CPP=OFF \ + -DENABLE_GLIB=OFF \ + -DENABLE_GOBJECT_INTROSPECTION=OFF \ + -DENABLE_QT5=OFF \ + -DENABLE_QT6=OFF \ + -DENABLE_BOOST=OFF \ + -DENABLE_LIBOPENJPEG=OFF \ + -DENABLE_LIBJPEG=OFF \ + -DENABLE_LCMS=OFF \ + -DENABLE_LIBCURL=OFF \ + -DENABLE_LIBTIFF=OFF \ + -DENABLE_NSS3=OFF \ + -DENABLE_GPGME=OFF \ + -DENABLE_HARFBUZZ=OFF \ + -DBUILD_GTK_TESTS=OFF \ + -DBUILD_QT5_TESTS=OFF \ + -DBUILD_QT6_TESTS=OFF \ + -DBUILD_CPP_TESTS=OFF \ + -DBUILD_MANUAL_TESTS=OFF \ + && cmake --build build --target pdftotext \ + && strip build/utils/pdftotext \ + # Refuse to hand over anything that still wants a dynamic loader. + && ! readelf -l build/utils/pdftotext | grep -q INTERP \ + && build/utils/pdftotext -v + +FROM scratch +COPY --from=build /src/build/utils/pdftotext /pdftotext