Compare commits
26
Commits
2a2c0c1db9
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a086f9242d | ||
|
|
c271cf2c8f | ||
|
|
3d72a54be1 | ||
|
|
ba7e0e2b42 | ||
|
|
ed52be3f1d | ||
|
|
bd406a110d | ||
|
|
a5f7541980 | ||
|
|
9ad6c05f9a | ||
|
|
7f5b4846ef | ||
|
|
17428a4f51 | ||
|
|
7ec9976b80 | ||
|
|
ec8a844441 | ||
|
|
8e4df08f43 | ||
|
|
5847245638 | ||
|
|
6faf99719a | ||
|
|
7d4c3eba17 | ||
|
|
656e7e1de4 | ||
|
|
6477147988 | ||
|
|
442684be60 | ||
|
|
5289250400 | ||
|
|
ebf7770569 | ||
|
|
ef6a231d98 | ||
|
|
93491016cd | ||
|
|
02814d09e5 | ||
|
|
5ca6cff87b | ||
|
|
16c2585637 |
+4
-1
@@ -2,5 +2,8 @@
|
||||
/cmd/money/money
|
||||
|
||||
# Never commit a data root that happens to live inside the repo.
|
||||
.money/
|
||||
*.db
|
||||
|
||||
# The static pdftotext and the binaries scripts/build-bundled.sh makes from it.
|
||||
/internal/parser/bundled/
|
||||
/dist/
|
||||
|
||||
@@ -2,7 +2,8 @@
|
||||
|
||||
Statement-driven personal finance tracker. A data directory holds one folder per
|
||||
account; statements dropped into those folders are parsed into a rebuildable
|
||||
SQLite index, categorised by ordered glob rules, and browsed in a Bubble Tea TUI.
|
||||
SQLite index, categorised by ordered glob rules, and browsed in a web app
|
||||
(`money serve`, which is also the default command).
|
||||
|
||||
Read `README.md` first — it is the user-facing reference for every config key.
|
||||
This file covers what the code assumes and why.
|
||||
@@ -10,30 +11,109 @@ This file covers what the code assumes and why.
|
||||
## Layout
|
||||
|
||||
```
|
||||
cmd/money/main.go subcommands; the TUI is the default
|
||||
internal/config rules.toml, account.toml, XDG config, data-root resolution
|
||||
cmd/money/main.go subcommands; serve is the default
|
||||
internal/config rules.toml (rules + transfers), account.toml, XDG config, data-root resolution
|
||||
internal/model Account, Transaction, amount formatting, description normalisation
|
||||
internal/glob the `*` / `?` matcher used by rules (linear time, no backtracking)
|
||||
internal/parser Parser interface + registry; csv, cmd, nlb, revolut, traderepublic
|
||||
internal/glob the `*` / `?` matcher used by rules (two-pointer, no exponential blowup)
|
||||
internal/parser Parser interface + registry; nlb, revolut, traderepublic
|
||||
internal/store SQLite index (modernc.org/sqlite, no cgo)
|
||||
internal/importer directory walk, dedupe, balance checks
|
||||
internal/rules applies ordered rules to rule_* columns only
|
||||
internal/report per-tag aggregation, transfers excluded
|
||||
internal/tui Bubble Tea models
|
||||
internal/rules applies ordered rules, writing rule_tag
|
||||
internal/transfers pairs the two legs of a movement, writing the transfers table
|
||||
internal/report per-tag aggregation
|
||||
internal/web `money serve`: JSON API + embedded single page (static/)
|
||||
```
|
||||
|
||||
## Invariants
|
||||
|
||||
**Rule verdicts and manual edits never share a column.** `rule_tag` /
|
||||
`rule_transfer` are rewritten wholesale on every retag; `manual_tag` /
|
||||
`manual_transfer` are only ever written by the user. Effective values are
|
||||
`COALESCE(manual_*, rule_*)`. This is what makes `money retag` safe to run at
|
||||
any time, and it is the first thing to preserve when touching the schema or the
|
||||
rules engine. Covered by `TestManualTagSurvivesRetag`.
|
||||
**Every tag comes from rules.toml.** There is no way to tag a transaction by
|
||||
hand; `rule_tag` is derived state that `Engine.Retag` rewrites wholesale, which
|
||||
is what makes `money retag` safe to run at any time. Nothing may write a tag
|
||||
from anywhere else — the moment something does, the index stops being
|
||||
reproducible and retag stops being safe. Covered by `TestRetagRewritesEveryTag`.
|
||||
|
||||
**A transfer is a pair or it is nothing.** A `[[transfer]]` block names both
|
||||
legs (`from_account` + `from_desc`, `to_account` + `to_desc`), and only a
|
||||
matched pair is dropped from the report — both legs together, never one. This
|
||||
is the whole reason the earlier boolean `transfer` flag was removed and this
|
||||
replaced it: a one-sided verdict let half a movement vanish and left the report
|
||||
unbalanced. An unmatched leg therefore keeps counting, and is surfaced as a
|
||||
warning instead: `⚠` on the transfers screen, on stderr from `import`/`retag`.
|
||||
Nothing may start excluding a single leg on the strength of one side matching.
|
||||
|
||||
Legs pair within `transfers.WindowDays`, nearest date first. **Within one
|
||||
currency the amount is the evidence and must be the exact opposite**, unless
|
||||
the definition sets `tolerance_pct` — a per-definition opt-in for a route where
|
||||
the bank takes a fee on the way, so the two statements genuinely disagree. It
|
||||
defaults to zero and belongs on the one definition that charges: a global or
|
||||
default tolerance would loosen every route that does not. **Across currencies
|
||||
the amount is not checked at all**: there are no exchange rates here, so the two
|
||||
numbers are unrelated and the dates carry the pairing alone. A tolerance means
|
||||
nothing there and is ignored. That asymmetry is the design, not an oversight —
|
||||
do not "fix" the cross-currency case by inventing a rate, and do not turn the
|
||||
tolerance into the default. Like rules, the first definition claims a leg and a
|
||||
transaction belongs to at most one transfer.
|
||||
|
||||
**A tolerated mismatch is a fee, and a fee is money, so it is reported rather
|
||||
than forgiven.** That is the condition on the tolerance existing at all: the
|
||||
pair leaves the report entirely, so a difference swallowed inside one would be
|
||||
spending that never appears anywhere. `Pair.Fee` is what left less what
|
||||
arrived, and `report.Excluded` carries it out per currency alongside the legs —
|
||||
named on the `money report` line and on its own row under the web report.
|
||||
Nothing may pair on a mismatch without that difference reaching `Excluded`.
|
||||
It is a fee and not a tag: no rule produces it, `report.ByTag` never sees it,
|
||||
and it must not be turned into a synthetic transaction to make the total
|
||||
reconcile. `Excluded` counts it only for pairs whose legs are *both* in view,
|
||||
for the same reason it counts legs and not pairs — half a pair cannot say what
|
||||
the other half received.
|
||||
|
||||
**`(transfer)` in a tag column is display only.** `Transaction.DisplayTag`
|
||||
falls back to `model.TransferTag` for a paired leg so the column does not read
|
||||
as a blank waiting for a rule, but nothing writes it: `rule_tag` stays what
|
||||
rules.toml made it, `store.Tags` never returns it, and no rule can be built
|
||||
from it. It is the same kind of label as `report.Untagged`. Do not "persist"
|
||||
it — that is precisely the second verdict this design exists to avoid.
|
||||
|
||||
**A paired leg is not untagged.** `store.Filter{Untagged: true}` means "no tag
|
||||
*and* no transfer", so a leg never turns up in `money ls --untagged`, the `u`
|
||||
view or the rule builder's preview asking to be tagged — it is spoken for, and
|
||||
the report drops it regardless. Covered by `TestPairedLegsAreNotUntagged`. An
|
||||
*unpaired* leg is untagged like anything else, which is how it gets noticed.
|
||||
|
||||
**The pairing is derived state, exactly like the tags.** `Engine.Link` rewrites
|
||||
the whole `transfers` table from rules.toml, so `money retag` re-derives both
|
||||
halves of what that file decides. It runs over the whole index deliberately —
|
||||
pairing within a filtered view would let a movement count as a transfer in one
|
||||
report and not in another. Note the asymmetry with a schema *column*: a new
|
||||
table is created by `CREATE TABLE IF NOT EXISTS` in `store.Open`, so adding one
|
||||
does not force an index rebuild.
|
||||
|
||||
**Rules and transfers share rules.toml, so textual deletion is block-aware.**
|
||||
`config.deleteBlocks` finds every `[[...]]` header, not only the kind being
|
||||
deleted, because a block ends where the *next* block of any kind begins —
|
||||
otherwise deleting a rule would swallow a transfer that follows it. Covered by
|
||||
`TestDeleteLeavesTheOtherKindAlone`.
|
||||
|
||||
**Statements are the source of truth; the index is disposable.** Deleting
|
||||
`.money/index.db` and re-importing must reproduce everything except manual tags
|
||||
and manual transfer marks.
|
||||
`index.db` at the root of the data directory and re-importing must reproduce
|
||||
everything, with nothing lost.
|
||||
|
||||
**The index is a cache, so there are no migrations.** `store.Open` applies
|
||||
`schema` and nothing else — no `ALTER TABLE`, no version column, no repair of
|
||||
an index an older build wrote. Changing the schema costs one line in the
|
||||
release note: delete `index.db` and import again. The two directions are not
|
||||
symmetric, which is worth knowing before you assume something is broken:
|
||||
|
||||
- *Removing* a column leaves an older index still working, carrying the dead
|
||||
column and its data unread.
|
||||
- *Adding* one the code reads breaks every command against an older index with
|
||||
`query transactions: SQL logic error: no such column: …` until it is deleted.
|
||||
|
||||
That asymmetry holds only because every statement names its columns —
|
||||
**no `SELECT *`, and nothing may depend on column order.** Keep it that way.
|
||||
Do not reintroduce migration machinery either; if re-parsing ever becomes too
|
||||
expensive to ask for, that is a decision to revisit deliberately rather than a
|
||||
helper to slip back in.
|
||||
|
||||
**Dedupe is by fingerprint**: `sha256(date | amount | normalised description |
|
||||
ordinal)`, where the ordinal distinguishes identical lines *within one
|
||||
@@ -45,28 +125,65 @@ milliseconds. That is the checksum skip working, not a failure.
|
||||
**Money is `int64` minor units**, never a float. Per-account currency, no
|
||||
conversion, and totals are never summed across currencies.
|
||||
|
||||
**First matching rule wins**, so transfer rules belong above general tag rules.
|
||||
A rule setting several of `match` / `counterparty` / `type` requires all of them.
|
||||
`config.AppendRule` therefore appends — never prepends — so saving from the
|
||||
rule builder cannot shadow a rule the user wrote by hand.
|
||||
**The most specific rule wins, not the topmost.** `rules.Engine` sorts the
|
||||
rules once in `New` and matches in that order: most literal characters first,
|
||||
then fewest `*`, then account-scoped over unscoped, with a *stable* sort so
|
||||
equally specific rules keep file order and the earlier one still wins. That is
|
||||
what lets `*NIKOLA*` carve an exception out of `*NIK*` from anywhere in the
|
||||
file, and a catch-all `*` sit wherever it reads best. A rule setting both
|
||||
`match` and `type` requires both of them, and both count towards its literals.
|
||||
|
||||
**The rule builder is a form, so the global keymap must not apply there.**
|
||||
`Update` routes to `updateRules` before `updateNormal` whenever the view is
|
||||
`viewRules`, or typing `q` would quit and `i` would start an import. Any new
|
||||
full-screen input needs the same treatment.
|
||||
The two orders must not be confused. `Engine.rules` stays in file order and
|
||||
`Rules()`, `Usage` and `MatchIndex` all speak in file positions, because that
|
||||
is what the rules screen numbers, what `config.DeleteRules` deletes by, and
|
||||
what the user can point at in rules.toml; only `Engine.order` is sorted.
|
||||
Anything new that reports a rule must report its file position too.
|
||||
|
||||
**In the rule builder, `tab` completes first and moves focus second.** The
|
||||
account and tag fields use `textinput.ShowSuggestions`, whose own `AcceptSuggestion`
|
||||
key is `tab` and whose `NextSuggestion`/`PrevSuggestion` are `up`/`down` — all
|
||||
three already meant something here. So `updateRules` intercepts `tab` and calls
|
||||
`acceptCompletion` before falling back to `setRuleFocus`, keeps `up`/`down` on
|
||||
field navigation, and lets `ctrl+n` / `ctrl+p` through to the input for cycling.
|
||||
`SetValue` does not re-match the suggestion list, so `acceptCompletion` re-sets
|
||||
it afterwards or `ctrl+n` would offer candidates that no longer fit the value.
|
||||
`config.AppendRule` still appends rather than prepends, but that now only
|
||||
settles ties: a saved rule cannot displace an equally specific one written by
|
||||
hand, while a narrower one is meant to take precedence and does.
|
||||
|
||||
**The report's period narrows the report and nothing else.** The screen opens on
|
||||
last month, and `←`/`→` step along `report.Periods` — the named windows, then
|
||||
the months the index holds. The report and the transaction list share a scope
|
||||
(account, search, untagged), which `web.filterFrom` reads and `state.filter`
|
||||
holds in the page; the period is read by the report handler alone and kept in
|
||||
`state.report`, so opening on last month cannot hide the rest of the index from
|
||||
the list. Nothing may put the period into that shared scope, which is exactly
|
||||
what would make it leak. Covered by `TestReportPeriodLeavesTransactionsAlone`.
|
||||
|
||||
The axis is built from `store.Months` over the *whole* index, not from the rows
|
||||
in view, or it would grow and shrink as the account or search filter changed and
|
||||
move under the cursor. It is rebuilt on every request (an import can reach
|
||||
further back), and the page names the period by its label rather than its
|
||||
position, so a grown axis keeps the window the user was on. The windows are
|
||||
relative to today, never to the newest statement: "last month" with nothing in
|
||||
it reports nothing and says so, because silently answering for a month nobody
|
||||
asked for is worse than an empty screen. That is also why an empty period and
|
||||
an empty index give different messages — one asks for another period, the
|
||||
other for an import.
|
||||
|
||||
**The report's sort rearranges rows; it never changes which rows there are.**
|
||||
`report.Order` is passed to `ByTag` and is deliberately *not* part of
|
||||
`store.Filter` or the shared scope — the period decides what is counted, the
|
||||
order only how it is listed, and merging the two would make a sort able to hide
|
||||
a tag. Currency stays the outer sort key under every order, because there are no
|
||||
exchange rates to compare two currencies by, and every order falls back to the
|
||||
tag so ties keep a fixed position instead of shuffling between reloads.
|
||||
`Order.Column` is what lets the page mark the sorted heading without keeping its
|
||||
own copy of that mapping; a new order needs a column of its own, which
|
||||
`TestOrderColumnsAreDistinct` checks.
|
||||
|
||||
**There are no keyboard shortcuts.** The page used to carry the TUI's keymap —
|
||||
screens on `1`–`8`, `u`, `/`, `i`, `r`, and chords inside the builders — and it
|
||||
was removed on purpose: every action is a link or a button. Do not add a global
|
||||
`keydown` handler back without being asked; a single-letter key fires while
|
||||
typing in a field unless every one of them is guarded, which is what the old one
|
||||
existed to do.
|
||||
|
||||
**A rule's usage count is how many transactions it wins, not how many its glob
|
||||
could match** — `Engine.Usage` counts by `MatchIndex`, so a rule shadowed by an
|
||||
earlier one correctly reports zero. That is what makes the rules screen able to
|
||||
could match** — `Engine.Usage` counts by `MatchIndex`, so a rule shadowed by a
|
||||
more specific one correctly reports zero. That is what makes the rules screen able to
|
||||
find dead rules at all.
|
||||
|
||||
**A rule's `note` is documentation that round-trips.** It is a TOML key rather
|
||||
@@ -74,12 +191,88 @@ than a `#` comment so `LoadRules` can return it, the builder can write it and
|
||||
the rules screen can show it. It never takes part in matching — `rules.Engine`
|
||||
does not look at it — and it must stay that way.
|
||||
|
||||
**`config.DeleteRules` edits rules.toml textually, never by re-serialising the
|
||||
parsed rules**, because comments and formatting are not recoverable from
|
||||
`[]Rule`. A rule owns the comment lines directly above it; a comment separated
|
||||
by a blank line is a heading for what follows and stays. Both writers go
|
||||
through `writeFileAtomic`, and deletion re-parses the result before replacing
|
||||
the file.
|
||||
**`config.DeleteRules` and `config.ReplaceRule` edit rules.toml textually, never
|
||||
by re-serialising the parsed rules**, because comments and formatting are not
|
||||
recoverable from `[]Rule`. A rule owns the comment lines directly above it, so
|
||||
deleting takes them with it; a comment separated by a blank line is a heading
|
||||
for what follows and stays. Replacing keeps them — they say why the rule is
|
||||
there, which editing its glob rarely changes — and rewrites only the rule's own
|
||||
lines, leaving its *position* alone: position still breaks ties, so a rule that
|
||||
moved could start beating an equally specific one it never used to. Every
|
||||
writer goes through `writeFileAtomic`, and both editors re-parse the result
|
||||
before replacing the file.
|
||||
|
||||
**An edit must not lose what the builder does not show.** The form has four
|
||||
fields and a `Rule` has five, so `editRule` takes `Type` from the rule on disk —
|
||||
never from the request — and the form says it is keeping it. Nothing may
|
||||
round-trip a rule through those four fields alone — a pattern the user was
|
||||
never shown is not a pattern they chose to remove. A new field on `Rule` needs
|
||||
the same treatment or a field of its own. Covered by `TestEditKeepsTypePattern`.
|
||||
|
||||
**The rule builder's preview is what the rule is judged against, which is not
|
||||
always "what is untagged".** For a new rule those are the same thing. For an
|
||||
edit they are not: the rule's own transactions are tagged, so an untagged-only
|
||||
preview would be empty for a rule that works. `previewGroups` adds them back,
|
||||
and `rulePreview` keeps the ones the new glob stops catching on screen marked
|
||||
`−` instead of dropping them silently, because giving one up is the decision
|
||||
being made. Covered by `TestEditPreviewShowsWhatIsLetGo`.
|
||||
|
||||
**The web app is a frontend, not a second implementation.** Everything that
|
||||
decides something stays in Go: amounts are formatted server-side (`money` is
|
||||
text plus a sign, never a number the browser could add up), the rule builder's
|
||||
preview is matched by `glob.Match` on the server rather than re-implemented in
|
||||
JS, and the transfer preview runs `transfers.Analyze`. Keep it that way — a
|
||||
JS glob or a JS sum is a second copy of a rule that would drift from the
|
||||
first.
|
||||
`static/app.js` only lays out what the API returns.
|
||||
|
||||
The server outlives edits to rules.toml in a way a one-shot command does not,
|
||||
so `/api/retag` and `/api/import` re-read it first (import also re-reads the
|
||||
account folders), and `/api/overview` reports `stale` when the file on disk no
|
||||
longer equals what the engines hold. Every edit and delete sends the rule or
|
||||
transfer it showed alongside the position, and is refused with 409 unless
|
||||
rules.toml still holds exactly that there — a position from a stale page can
|
||||
otherwise name a different rule. Covered by `TestStalePositionIsRefused`.
|
||||
|
||||
There is no auth by design (`--addr` defaults to loopback). Non-GET requests
|
||||
must be `application/json`, which is what keeps a cross-site form from posting
|
||||
to it; do not relax that without putting something else in its place.
|
||||
That is why `/api/upload` takes files base64 inside JSON rather than as
|
||||
multipart: multipart is precisely what a cross-site form can send.
|
||||
|
||||
**An upload only puts a file where `money import` looks.** It writes into an
|
||||
existing account folder and stops there: importing stays the user's call, made
|
||||
with the Import button, so the statements stay the source of truth and nothing
|
||||
reaches the index any other way. It never overwrites a statement (identical
|
||||
contents are a no-op, different ones a 409), refuses any name import would not
|
||||
read back — not a plain file name, a dotfile, `account.toml`, outside the
|
||||
account's `include` — and checks a whole batch before writing any of it. Covered
|
||||
by `TestUploadSavesWithoutImporting`, `TestUploadNeverReplacesAStatement` and
|
||||
`TestUploadRefusesBadNames`.
|
||||
|
||||
**The statements list is read from the folders, not the index.** `/api/files`
|
||||
walks `importer.StatementFiles` — the same list import reads — and only then
|
||||
looks each file up in `store.SourceFiles`, comparing `importer.Checksum`, so a
|
||||
file is listed exactly when import would read it and a file the index remembers
|
||||
but the disk lost shows as `missing` instead of vanishing.
|
||||
`/api/files/{account}/{name}` serves a file only by finding it in that list,
|
||||
never by joining the name onto a path. A statement is served from the app's
|
||||
origin, where a script could drive the API, so nothing is ever rendered as a
|
||||
page: text is `text/plain` under `CSP: sandbox`, anything not text or PDF is a
|
||||
sandboxed download, and PDFs — whose viewers refuse a sandbox — open in the
|
||||
browser's own isolated viewer. Covered by `TestServeFileServesOnlyStatements`.
|
||||
|
||||
**Deleting a statement marks its neighbours to be re-read.** `/api/files/delete`
|
||||
finds the file the same way `serveFile` does and deletes it — permanently, by
|
||||
the user's choice; the page says so before it asks. `store.ForgetSourceFile`
|
||||
then drops its transactions and their transfer rows, and `Link` re-pairs, which
|
||||
reads no statement. It does not import, any more than an upload does. A
|
||||
transaction two overlapping statements share is stored once, under the file
|
||||
imported first, so it goes too; `ForgetSourceFile` therefore blanks the
|
||||
checksums of the account's other statements, which show as `changed` until the
|
||||
next Import re-reads them and restores what they hold. Without that the checksum
|
||||
skip would leave those rows gone for good. Covered by
|
||||
`TestRemoveFileKeepsWhatAnotherStatementHolds`.
|
||||
|
||||
## Adding a bank parser
|
||||
|
||||
@@ -95,11 +288,44 @@ Keep PDF text extraction separate from parsing: the bank parsers expose a pure
|
||||
a runtime dependency of `nlb` and `traderepublic`; no Go library reconstructs
|
||||
column layout as well.
|
||||
|
||||
It can be carried inside the binary instead: `scripts/build-bundled.sh` builds
|
||||
a static musl `pdftotext` in a container and embeds it under `-tags bundled`,
|
||||
and `pdftotextCommand` writes it to the user cache dir (named by content hash,
|
||||
verified before reuse) and runs that. It is still the real executable run as a
|
||||
subprocess, deliberately — not poppler linked in through cgo, which would cost
|
||||
the pure-Go build, and not a different extraction API whose spacing might not
|
||||
match what the parsers were tuned on. The tag is opt-in so `go build`/`go test`
|
||||
never need the 5 MB file, which is gitignored rather than committed. Bumping
|
||||
`POPPLER_VERSION` means updating its pinned SHA-256 too, and checking real
|
||||
statements still parse — that output is what the parsers depend on.
|
||||
|
||||
Do not hardcode absolute column positions from a sample PDF. `pdftotext`
|
||||
compresses runs of spaces, so columns shift with font and page size — derive
|
||||
positions from the header line (`traderepublic`) or from the line being parsed
|
||||
(`nlb`).
|
||||
|
||||
**The other side's account number belongs in the description, not in a field
|
||||
of its own.** There used to be a `Counterparty` field; only `nlb` could fill it
|
||||
honestly, `revolut` and `traderepublic` invented it with an IBAN-shaped regex
|
||||
over the description, and the two disagreed on spacing, so one rule pattern
|
||||
could not serve both. Now `nlb` appends its IBAN column to the end of the
|
||||
description — at the end, and not in the position it held on the page, so an
|
||||
IBAN wrapped across continuation lines stays contiguous for a glob to match.
|
||||
A new parser must do the same rather than reintroduce a structured field.
|
||||
|
||||
**A parser may name its statements, and only uploads use it.** `parser.Namer` is
|
||||
optional, like `parser.Warner`: `nlb` names an izpisek `izpisek_YYYY-MM-DD`
|
||||
after its *Datum izpiska*, ported from the `rename_izpiski.py` it replaced, and
|
||||
`traderepublic` names a statement `traderepublic_YYYY_MM_DD_YYYY_MM_DD` after
|
||||
the period in its header. A name is derived from what the statement says, never
|
||||
from the upload's own name, and comes back without an extension; the upload's is
|
||||
kept, lowercased. `/api/upload` stages each file as a dotfile — invisible to
|
||||
import — so the parser can read it, then numbers a chosen name that is taken by
|
||||
different contents (`_2`, as the script did) where a name the user gave is
|
||||
refused with 409. Nothing renames a file already in a folder: `source_files`
|
||||
records statements by path, so a rename would orphan its rows as `missing` until
|
||||
the index is rebuilt. Covered by `TestUploadNamesStatementsTheParserCanName`.
|
||||
|
||||
## Verifying
|
||||
|
||||
```
|
||||
@@ -115,20 +341,23 @@ money --root /tmp/.../demo import # must report 0 new
|
||||
money --root /tmp/.../demo ls --wide
|
||||
```
|
||||
|
||||
### Driving the TUI in tests
|
||||
If you changed the schema, delete that root's `index.db` first. Nothing
|
||||
migrates it, so a demo root left over from an earlier build either carries dead
|
||||
columns or fails with `no such column`, depending on which way the schema
|
||||
moved.
|
||||
|
||||
Prefer feeding `tea.Msg` values to `Model.Update` directly — that is how every
|
||||
existing TUI test works, and it covers the keymap without a terminal.
|
||||
### Testing the web app
|
||||
|
||||
If a real terminal is genuinely needed, note that piping into `script` does not
|
||||
deliver keystrokes. Use a pty and answer the two capability queries Bubble Tea
|
||||
sends on startup, or the program blocks before its first render:
|
||||
Drive the API through `Server.Handler()` with `httptest` — `newTestServer` in
|
||||
`internal/web/server_test.go` builds a data root and an index tagged and paired
|
||||
as an import would leave them, with "today" fixed so the report's default
|
||||
window is predictable. That is how every web test works, and since the page
|
||||
only lays out what the API returns, it is where behaviour gets tested.
|
||||
|
||||
- `ESC]11;?` (background colour) → reply `ESC]11;rgb:1e1e/1e1e/1e1e ESC\`
|
||||
- `ESC[6n` (cursor position) → reply `ESC[1;1R`
|
||||
|
||||
Also delete the index first, or the import you are trying to observe will be
|
||||
skipped by checksum and finish instantly.
|
||||
For a look at the real page, run `money --root <demo> serve --addr
|
||||
127.0.0.1:<port>` against a throwaway root. Delete the index first if you want
|
||||
to watch an import do work, or it will be skipped by checksum and finish
|
||||
instantly.
|
||||
|
||||
## Conventions
|
||||
|
||||
|
||||
@@ -3,18 +3,17 @@
|
||||
A personal finance tracker built around bank statements rather than manual entry.
|
||||
|
||||
You keep a **data directory with one folder per account**, drop statement exports
|
||||
into those folders, and run `money import`. Transactions are extracted, tagged by
|
||||
glob rules you write, and movements between your own accounts are marked as
|
||||
transfers so they never count as spending.
|
||||
into those folders, and run `money import`. Transactions are extracted and
|
||||
tagged by glob rules you write.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
~/money/ # the data root (see "Where the data root lives")
|
||||
rules.toml # tag + transfer rules, in order
|
||||
.money/index.db # SQLite index (rebuildable; safe to delete*)
|
||||
rules.toml # tag rules and transfer definitions, in order
|
||||
index.db # SQLite index (rebuildable; safe to delete*)
|
||||
checking/
|
||||
account.toml # currency + how to parse this bank's exports
|
||||
account.toml # currency + which parser reads this bank's exports
|
||||
2026-01.csv
|
||||
2026-02.csv
|
||||
savings/
|
||||
@@ -22,14 +21,16 @@ transfers so they never count as spending.
|
||||
2026-01.csv
|
||||
```
|
||||
|
||||
\* Deleting the index loses manual tags and manual transfer marks, which live
|
||||
only there. Everything else is re-derived from the statements.
|
||||
\* Nothing lives only in the index: it is derived entirely from the statements
|
||||
plus rules.toml, so deleting it and re-importing gets you exactly what you had.
|
||||
It is a cache and nothing more, which is why it is never migrated — if a new
|
||||
build changes its shape, delete it and run `money import` again.
|
||||
|
||||
## Where the data root lives
|
||||
|
||||
Checked in order, first one wins:
|
||||
|
||||
1. `--root DIR` on the command line
|
||||
1. `--root DIR` on the command line, before or after the command
|
||||
2. `$MONEY_ROOT`
|
||||
3. the `root` key in `$XDG_CONFIG_HOME/money/config.toml`, which is
|
||||
`~/.config/money/config.toml` unless `XDG_CONFIG_HOME` says otherwise
|
||||
@@ -55,17 +56,45 @@ Give it the package path, not `cmd/money/main.go`. Naming the file puts the go
|
||||
tool in file mode: it names the binary after the source file (`main`) and
|
||||
compiles only the files listed, which breaks the moment the package has two.
|
||||
|
||||
That build runs `pdftotext` from `PATH` for the PDF parsers, so the machine
|
||||
needs poppler-utils. To deploy one file with nothing to install alongside it,
|
||||
build with `pdftotext` inside:
|
||||
|
||||
```
|
||||
scripts/build-bundled.sh # → dist/money-linux-amd64
|
||||
scripts/build-bundled.sh arm64 # → dist/money-linux-arm64
|
||||
```
|
||||
|
||||
It needs podman or docker: `pdftotext` is compiled in a container from a
|
||||
checksum-pinned poppler release (`scripts/pdftotext.Containerfile`), as a fully
|
||||
static musl executable with only what text extraction needs, then embedded with
|
||||
`go build -tags bundled`. The result runs on any Linux of that architecture.
|
||||
The first PDF import writes the embedded copy to `~/.cache/money/` (or
|
||||
`$XDG_CACHE_HOME/money/`) and runs it from there; `money config` says which
|
||||
`pdftotext` is in use. Only Linux on amd64 and arm64 is offered — elsewhere,
|
||||
install poppler-utils.
|
||||
|
||||
Building arm64 from an x86 machine runs the container under emulation, so the
|
||||
host needs qemu-user-static and the first build is slow; the executable is then
|
||||
cached under `internal/parser/bundled/` and later builds only rebuild money.
|
||||
|
||||
Poppler is GPL, so a bundled binary is GPL-licensed as a whole. That matters
|
||||
only if you hand the binary to someone else.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
money # open the TUI (default)
|
||||
money # the web app on 127.0.0.1:8080 (same as `money serve`)
|
||||
money serve --addr :8080 # listen on every interface (there is no login)
|
||||
money import # extract new transactions from every statement
|
||||
money import --force # re-parse statements even if unchanged
|
||||
money retag # re-apply rules.toml to everything already imported
|
||||
money ls --untagged # what still needs a tag
|
||||
money ls --wide # also show type, counterparty and reported balance
|
||||
money retag # re-apply rules.toml: retag, and re-pair transfers
|
||||
money ls --untagged # what neither a rule nor a transfer has claimed
|
||||
money ls --untagged --uniq # the same, one row per distinct description
|
||||
money ls --wide # also show type and reported balance
|
||||
money ls --account checking --month 2026-01
|
||||
money report --month 2026-01 # spending by tag
|
||||
money report --sort count # ordered by number of transactions, not amount
|
||||
money accounts # balances
|
||||
money parsers # available statement parsers
|
||||
money config # which data root is in use, and why
|
||||
@@ -73,111 +102,410 @@ money config # which data root is in use, and why
|
||||
|
||||
An account folder appears in the app only once its statements have been
|
||||
imported — creating an `account.toml` is not enough on its own. Run
|
||||
`money import` (or press `i` in the TUI) after adding one.
|
||||
`money import` (or press Import in the web app) after adding one.
|
||||
|
||||
### TUI keys
|
||||
`--uniq` turns `ls` into a list of patterns still to write rather than a list of
|
||||
rows to read: one line per distinct description, since fifty visits to the same
|
||||
shop are one glob, not fifty lines. Descriptions print exactly as a glob sees
|
||||
them — upper-cased with whitespace collapsed — so two statements that worded the
|
||||
same payee differently collapse into the one row they deserve, and a pattern
|
||||
written from the list matches what the list showed you.
|
||||
|
||||
| Key | Action |
|
||||
```
|
||||
$ money ls --untagged --uniq
|
||||
DESCRIPTION
|
||||
INTEREST PAID
|
||||
KAUFLAND 4412 SOFIA
|
||||
LIDL SOFIA 4412
|
||||
|
||||
3 distinct descriptions
|
||||
```
|
||||
|
||||
The account is not part of what makes a row distinct. A rule matches on the
|
||||
description and only optionally narrows to an account, so one payee seen on two
|
||||
accounts is still one pattern to write — and it is one row in the rule builder,
|
||||
which groups the same way. Pass `--account` to scope the listing instead.
|
||||
|
||||
It composes with the other filters (`--account`, `--month`, `--search`), and
|
||||
`--limit` caps the rows printed, saying how many it held back. It cannot be
|
||||
combined with `--wide`, whose columns all belong to a single transaction.
|
||||
|
||||
## The web app
|
||||
|
||||
`money serve`, which is also what `money` runs with no command, puts eight
|
||||
screens in a browser: accounts, transactions, report, rule builder, rules,
|
||||
transfer builder, transfers and tags. It runs the same code as the CLI: amounts
|
||||
are formatted, globs matched and transfers paired on the server, and the page
|
||||
only shows the answers. It is one binary with the page built in; nothing else to
|
||||
deploy.
|
||||
|
||||
```
|
||||
money # http://127.0.0.1:8080
|
||||
money serve --root /srv/money --addr 0.0.0.0:8080
|
||||
```
|
||||
|
||||
**There is no authentication.** Anyone who can reach the port can read every
|
||||
transaction, rewrite `rules.toml` and start an import, which is why it listens
|
||||
on loopback unless `--addr` says otherwise. To use it from elsewhere, put it
|
||||
behind a reverse proxy that does the logging in, or reach it over SSH / a VPN.
|
||||
Requests that change anything must be sent as JSON, so another website open in
|
||||
the same browser cannot post a form to it.
|
||||
|
||||
The server outlives hand edits to `rules.toml`, so it treats the file the way a
|
||||
fresh command would:
|
||||
|
||||
- **Retag and Import re-read `rules.toml`** (and Import the account folders)
|
||||
before running, exactly as `money retag` / `money import` would. Until then a
|
||||
banner says the file on disk no longer matches what the index was derived
|
||||
from.
|
||||
- **Edits and deletes check the file first.** They go by rule position, and the
|
||||
page also sends the rule it showed you; if `rules.toml` no longer holds that
|
||||
rule there (another tab, a hand edit) the change is refused and you are asked
|
||||
to reload, rather than editing whichever rule moved into its place.
|
||||
|
||||
**Re-read all** is `import --force`: every statement is parsed again, including
|
||||
the ones an Import skips because they have not changed. Use it after updating
|
||||
money, when a parser has learned to read something it used to get wrong; rows
|
||||
already in the index are recognised and not duplicated.
|
||||
|
||||
### Statements
|
||||
|
||||
The Accounts screen also lists every statement file in every account
|
||||
folder, with what the index made of it:
|
||||
|
||||
| Status | Meaning |
|
||||
| --- | --- |
|
||||
| `1` `2` `3` `4` `5` / `tab` | accounts · transactions · report · rule builder · rules |
|
||||
| `enter` | open the selected account (accounts view) |
|
||||
| `t` | set the tag on the selected transaction |
|
||||
| `x` | toggle transfer on the selected transaction |
|
||||
| `c` | clear manual overrides, falling back to the rules |
|
||||
| `/` | filter by description |
|
||||
| `u` | show only untagged transactions |
|
||||
| `a` | clear the account filter |
|
||||
| `i` | import · `r` re-apply rules · `q` quit |
|
||||
| `✓` imported | read, and unchanged since |
|
||||
| `⚠` changed | the file differs from what was imported; Import re-reads it |
|
||||
| `•` not imported | new since the last import — or an import tried and failed, in which case the error is in the import report |
|
||||
| `✗` gone from disk | imported once, then removed; its transactions stay in the index only until it is rebuilt |
|
||||
|
||||
### Rule builder (`4`)
|
||||
**Transactions** reads like `3 rows · 0 new`: how many transactions the file
|
||||
holds, then how many of them it was the first to bring in. Statements that
|
||||
overlap share rows, and a shared row is stored once, under whichever file was
|
||||
imported first — so a later statement covering the same days shows fewer new
|
||||
rows than it holds, down to `0 new` when an earlier one already had them all.
|
||||
That is deduplication working, not a file that failed to parse.
|
||||
|
||||
Click a file name to open it. PDFs and CSVs open in the browser; anything else
|
||||
downloads. Only files import would read are listed or served — not
|
||||
`account.toml`, dotfiles, or anything outside an account's `include` patterns.
|
||||
|
||||
**Delete** removes a statement from disk for good — there is no copy kept
|
||||
and no undo, so keep the bank's original if you might want it back. Its
|
||||
transactions leave the index, and transfers that used them lose their pairing:
|
||||
the other leg shows as unpaired, which is the truth once one side is gone.
|
||||
Deleting does not import. A row that an overlapping statement also contains goes
|
||||
too, for now: the account's other statements are marked *changed*, and the next
|
||||
**Import** re-reads them and puts it back. For a file already gone from disk the
|
||||
button says **Forget** and only clears the index.
|
||||
|
||||
### Adding statements from the browser
|
||||
|
||||
The Accounts screen has an **Add statements** panel: pick the account,
|
||||
drop files on it (or click to choose them) and press **Upload**. The files are
|
||||
saved into that account's folder, exactly where you would have copied them by
|
||||
hand — the folder stays the source of truth, so deleting `index.db` and
|
||||
re-importing still gets everything back. Uploading does not import: the files
|
||||
wait on the statements list as *not imported* until you press **Import**, so you
|
||||
can put a batch together and look it over first.
|
||||
|
||||
An upload never replaces a statement. A file whose name is already in the folder
|
||||
is skipped if its contents are identical and refused if they differ; rename it
|
||||
or remove the old one first. Names import would not read back — dotfiles,
|
||||
`account.toml`, anything outside an account's `include` patterns — are refused
|
||||
too, and a batch with one bad file writes none of them. The folder itself must
|
||||
already exist with an `account.toml`: an upload adds statements to an account,
|
||||
it does not create one.
|
||||
|
||||
Some banks name their downloads unhelpfully, so a parser may name the statement
|
||||
instead — lowercase throughout, extension included — and the status line says
|
||||
what each file became. An NLB izpisek is saved as `izpisek_YYYY-MM-DD.pdf` after
|
||||
the *Datum izpiska* in its header, and a Trade Republic statement, which
|
||||
downloads as `document-N.pdf`, as `traderepublic_YYYY_MM_DD_YYYY_MM_DD.pdf`
|
||||
after the period on its first page (`DATE 01 May 2025 - 31 Jul 2026`). Either
|
||||
way the folder sorts by date. Downloading the same statement twice then lands on
|
||||
the same name and is recognised as already there; a different statement with the
|
||||
same name (a reissue) is numbered `_2`, `_3` rather than refused. A statement
|
||||
with no date it can find keeps its own name. Only uploads are named — files
|
||||
already in a folder are never renamed, since the index records them by path.
|
||||
|
||||
### Navigating
|
||||
|
||||
Every screen is a link in the bar along the top, and every action is a button or
|
||||
a click; there are no keyboard shortcuts. Clicking an account on the Accounts
|
||||
screen filters the transaction list to it.
|
||||
|
||||
There is no control that tags a transaction. Tags come from `rules.toml` and
|
||||
nowhere else, so tagging what you are looking at means writing a rule for it in
|
||||
the rule builder — which is why it shows you what a glob catches before you
|
||||
save.
|
||||
|
||||
### Report
|
||||
|
||||
The report opens on **last month**, not on everything you have ever imported:
|
||||
an all-time total is the one number a spending report is least often asked for,
|
||||
and the month that has just ended is the last one your statements can be
|
||||
complete for. If you have not downloaded that month yet, the screen says the
|
||||
period is empty rather than quietly showing you a different one.
|
||||
|
||||
The time axis beside the totals runs from the widest window down to the oldest
|
||||
month in the index — `all time`, `this year`, `last 12 months`, `last 3
|
||||
months`, `this month`, `last month`, then one entry per month. Click one to
|
||||
report on it.
|
||||
|
||||
The three rolling windows run to the end of *this* month rather than to the
|
||||
last complete one — you ask for "last 3 months" to see what is happening now,
|
||||
and leaving out the days since the 1st would answer a different question. Each
|
||||
month below `last month` is a month the index actually holds; months that a
|
||||
named window above already covers are not repeated.
|
||||
|
||||
Click a column heading to change how the rows are arranged; the marked heading
|
||||
says which one they are arranged by: `Out ▾` largest spend first (the default
|
||||
question a spending report answers), then `In ▾`, `Net ▾` lowest first so the
|
||||
biggest losses lead, `N ▾` most transactions first, and `Tag ▴` A→Z. `money
|
||||
report` takes the same choice as `--sort out|in|net|count|tag`.
|
||||
|
||||
Currency is always the outer grouping and no sort changes that — there are no
|
||||
exchange rates here, so two currencies interleaved by amount would invite a
|
||||
comparison between numbers that cannot be compared. Rows that tie fall back to
|
||||
the tag, so they keep a fixed position rather than shuffling between reloads.
|
||||
The sort rearranges rows; it never changes which rows there are.
|
||||
|
||||
The period narrows the report and only the report. The account, the search and
|
||||
the untagged toggle are shared with the transaction list, so opening on last
|
||||
month does not hide the rest of the index from the transaction list.
|
||||
|
||||
`money report` takes `--month` instead; there is no command-line equivalent of
|
||||
the wider windows.
|
||||
|
||||
### Rule builder
|
||||
|
||||
Writing rules by hand means guessing what a glob will catch. This screen shows
|
||||
the answer as you type: the form is on the left, and on the right is every
|
||||
still-untagged description in the data, sorted alphabetically and grouped, with
|
||||
a `▸` against each one the glob currently matches and a running
|
||||
"*n* of *m* descriptions match" count.
|
||||
the answer as you type: the form is on the left — glob, account, tag, note —
|
||||
and on the right the still-untagged descriptions the glob currently matches,
|
||||
marked `▸`, grouped by description, under a running "*n* of *m* descriptions
|
||||
match" count.
|
||||
|
||||
```
|
||||
glob Untagged description N
|
||||
╭────────────────────────────╮ ACME PAYROLL JAN 1
|
||||
│ *LIDL* │ BOLT RIDE 1
|
||||
╰────────────────────────────╯ ▸ LIDL SOFIA 4412 2
|
||||
vs. the description ▸ LIDL VARNA 9911 1
|
||||
ZARA MLADOST 1
|
||||
▸ tag
|
||||
╭────────────────────────────╮
|
||||
│ groceries │
|
||||
╰────────────────────────────╯
|
||||
tab completes · ctrl+n: 1 more
|
||||
The list narrows as you type, so what is on screen is what the rule would
|
||||
claim — nothing else is left there to read past. The count keeps the context
|
||||
the rows no longer can: *m* is everything still in view, so `2 of 7` says the
|
||||
glob picked two descriptions out of seven, and `0 of 7` says the glob is wrong.
|
||||
With the glob still empty there is nothing to filter by, and the list is every
|
||||
untagged description in the data — which is the other question this screen
|
||||
answers, and where you go looking for the next thing to write a rule for.
|
||||
Clicking a description fills the glob with `*THAT DESCRIPTION*`, as a starting
|
||||
point to narrow down.
|
||||
|
||||
note
|
||||
╭────────────────────────────╮
|
||||
│ optional │
|
||||
╰────────────────────────────╯
|
||||
why this rule exists
|
||||
`enter` (or **Save rule**) appends the rule to `rules.toml` and retags
|
||||
immediately, so the rows it caught disappear from the list. The account stays
|
||||
filled in, since the next rule is usually for the same one.
|
||||
|
||||
2 of 7 descriptions match
|
||||
```
|
||||
Clicking the column headings switches the list between by name and by count,
|
||||
most seen first; the `↓` says which column the order is read from. The two
|
||||
answer different questions: by name finds the payee you are looking at, by count
|
||||
finds the rule worth writing next, since one pattern claiming forty rows is
|
||||
worth more than the first of forty claiming one. Ties keep the alphabetical
|
||||
order, so the list does not reshuffle under you, and the choice lasts until you
|
||||
change it.
|
||||
|
||||
Only `gro` was typed in the tag field; `ceries` is the ghosted completion.
|
||||
|
||||
`tab` / `↑↓` move between the glob, account, tag and note fields, `pgup` /
|
||||
`pgdn` scroll the list, and `enter` appends the rule to `rules.toml` and retags
|
||||
immediately, so the rows it caught disappear from the list. `esc` goes back.
|
||||
On a short window the form gives up its spacing and then its hints, so all four
|
||||
fields stay on screen.
|
||||
|
||||
The account and tag fields complete as you type: the rest of the match is
|
||||
ghosted in grey after the cursor, and `tab` (or `→` at the end of the line)
|
||||
takes it. When several candidates share the prefix, the hint under the box says
|
||||
how many, and `ctrl+n` / `ctrl+p` cycle through them. Nothing is committed until
|
||||
you accept it, so a new tag is still just typed out in full. Accounts come from
|
||||
The account and tag fields offer completions as you type. Accounts come from
|
||||
the folders on disk and the index; tags from every tag in use plus any named in
|
||||
`rules.toml`, so a tag is completable from the moment a rule mentions it —
|
||||
which is what stops `groceries` acquiring a `grocery` twin.
|
||||
which is what stops `groceries` acquiring a `grocery` twin. A new tag is just
|
||||
typed out in full.
|
||||
|
||||
Leaving the account blank applies the rule everywhere; filling it in also
|
||||
narrows the preview to that account. Rules are appended, so anything already in
|
||||
`rules.toml` keeps precedence — the preview accounts for that automatically,
|
||||
since it only ever lists transactions no existing rule has already tagged.
|
||||
narrows the preview to that account. The preview lists only transactions no
|
||||
existing rule has already tagged, so it answers "what is still waiting for a
|
||||
rule" rather than "what would this rule win". Those differ when the new rule is
|
||||
narrower than an existing one: a `*LIDL SOFIA*` written while `*LIDL*` is
|
||||
already tagging those rows previews as nothing and still takes them on save,
|
||||
because the more specific rule wins. Legs of a matched transfer are left out
|
||||
too: they are already accounted for by the definition that paired them, and the
|
||||
report leaves them out anyway.
|
||||
|
||||
### Rules (`5`)
|
||||
#### Editing a rule
|
||||
|
||||
Every rule in file order, with the number of transactions it actually claims.
|
||||
Rules that claim none are marked `✗`.
|
||||
**Edit** on the rules screen opens the rule in this same form with the
|
||||
fields filled in, and saving rewrites that rule where it sits instead of
|
||||
appending a new one. It keeps its position, since position still breaks ties
|
||||
between equally specific rules; `rules.toml` is edited textually, so the
|
||||
comments and formatting around it survive, exactly as when deleting.
|
||||
|
||||
An edit is judged against different rows to a new rule. What the rule already
|
||||
claims is tagged, so none of it would be in an untagged preview and a rule that
|
||||
works perfectly would preview as nothing; those rows are put back, and any the
|
||||
new glob stops catching stays on screen marked `−`, under a "*n* no longer
|
||||
claimed" line, rather than quietly disappearing the way an ordinary non-match
|
||||
does. Narrowing `*LIDL*` to `*LIDL SOFIA*` therefore shows you the Varna branch
|
||||
you are about to hand back to whatever rule catches it next.
|
||||
|
||||
The form has no `type` field, so a rule that sets one carries it through
|
||||
unchanged rather than losing it; it is shown under the glob as `+ type:… ·
|
||||
kept`. Saving lands back on the rules screen with the new counts; **Cancel**, or
|
||||
leaving the screen leaves the rule as it was.
|
||||
|
||||
### Rules
|
||||
|
||||
Every rule in file order — the order they are written in, not the order they
|
||||
are tried in — with the number of transactions it actually claims. Rules that
|
||||
claim none are marked `✗`.
|
||||
|
||||
```
|
||||
money · rules · 5 rules · 2 match nothing
|
||||
Rules · 5 rules · 2 match nothing
|
||||
|
||||
# Pattern Account Tag T Txns Note
|
||||
1 *LIDL* (all) groceries 3 the weekly shop
|
||||
2 ✗ *LIDL SOFIA* (all) shadowed 0
|
||||
# Pattern Account Tag Txns Note
|
||||
1 *LIDL SOFIA* (all) groceries 3 the weekly shop
|
||||
2 ✗ *LIDL* (all) shadowed 0
|
||||
3 ✗ *OLD BANK NAME* (all) dead 0 closed in 2025
|
||||
4 *ZARA* (all) clothes 1
|
||||
5 *КАУФЛАНД* checking groceries 1 4412 is the branch
|
||||
```
|
||||
|
||||
The count is how many transactions the rule *wins*, not how many its glob could
|
||||
match. Rule 2 above matches `LIDL SOFIA 4412` perfectly well, but rule 1 claims
|
||||
it first, so rule 2 is dead weight — which is the point of the screen. A rule
|
||||
can therefore reach zero either by matching nothing or by being shadowed, and
|
||||
both are worth deleting.
|
||||
match. Rule 2 above matches `LIDL SOFIA 4412` perfectly well, but rule 1 spells
|
||||
out more of it and so claims it first; with nothing else here that says `LIDL`,
|
||||
rule 2 is dead weight — which is the point of the screen. A rule can therefore
|
||||
reach zero either by matching nothing or by being shadowed, and both are worth
|
||||
deleting. Note that shadowing has nothing to do with the numbering: rule 2 would
|
||||
be just as dead written above rule 1, because precedence is decided by how
|
||||
specific a rule is and not by where it sits.
|
||||
|
||||
`d` deletes the selected rule, `p` deletes every rule marked `✗` at once, and
|
||||
both ask for a `y` first. Deleting edits `rules.toml` textually, so your
|
||||
comments, ordering and formatting survive; a comment sitting directly above a
|
||||
deleted rule goes with it, while one separated by a blank line is treated as a
|
||||
section heading and left alone.
|
||||
**Edit** opens a rule in the builder, **Delete** deletes it, and **Prune
|
||||
unused** deletes every rule marked `✗` at once; both deletions ask first.
|
||||
**Refresh counts** recounts against what is currently in the index. Editing and
|
||||
deleting both work on `rules.toml` textually, so your comments, ordering and
|
||||
formatting survive. A comment sitting directly above a deleted rule goes with
|
||||
it, while one separated by a blank line is treated as a section heading and
|
||||
left alone; an edited rule keeps its comments, since they say why it is there
|
||||
and changing its glob rarely changes that.
|
||||
|
||||
### Transfer builder
|
||||
|
||||
Same idea as the rule builder, for money moved between your own accounts. The
|
||||
form names both sides — from account, from desc, to account, to desc — plus an
|
||||
optional tolerance and note, and the preview beside it shows what the pair
|
||||
would be: `▸` for a movement it matches end to end, `⚠` for a leg it catches on
|
||||
one side and cannot pair with anything on the other.
|
||||
|
||||
```
|
||||
Date Amount Movement Description
|
||||
⚠ 2026-04-01 500.00 checking → ? TRANSFER TO SAVINGS
|
||||
▸ 2026-03-01 500.00 checking → savings TRANSFER TO SAVINGS
|
||||
|
||||
1 pairs · 1 unpaired
|
||||
```
|
||||
|
||||
The account fields complete exactly as the rule builder's do. `enter` (or
|
||||
**Save transfer**) appends the definition to `rules.toml` and re-pairs
|
||||
immediately. Saving keeps both account names in place and clears the two globs,
|
||||
since the next transfer you write is usually the same route in the other
|
||||
direction. The tolerance is cleared with them: carried over silently it would
|
||||
loosen a route that never asked for one.
|
||||
|
||||
`Tolerance %` is the one field worth previewing before you save. Type a
|
||||
percentage and the pairs it buys appear immediately, showing both amounts —
|
||||
`500.00 → 495.00` — with what they cost summarised beside the counts:
|
||||
|
||||
```
|
||||
1 pairs · 0 unpaired · 5.00 in fees
|
||||
```
|
||||
|
||||
A half-written definition previews too: fill in one side and its legs show up as
|
||||
unpaired, which is the quickest way to see that a glob is wrong before you have
|
||||
written the other half.
|
||||
|
||||
### Transfers
|
||||
|
||||
Every definition in file order, with what it currently pairs.
|
||||
|
||||
```
|
||||
Transfers · 4 definitions · 1 leg(s) unpaired
|
||||
|
||||
# From To Pairs Unpaired Tol Note
|
||||
1 ⚠ checking *TO SAVINGS* savings *FROM CHECKING* 11 1 monthly saving
|
||||
2 checking *TO BROKER* traderepublic *FROM NLB* 4 0
|
||||
3 checking *WIRE TO SAVINGS* savings *WIRE FROM CHECK* 2 0 1.5% the bank keeps a fee
|
||||
4 ✗ checking *TO OLD BANK* savings *FROM OLD BANK* 0 0 closed in 2025
|
||||
```
|
||||
|
||||
The two counts mean different things, which is the point of the screen. `⚠` is a
|
||||
definition catching legs it cannot complete: a wrong glob on the other side, a
|
||||
statement not imported yet, or a movement that genuinely went missing. `✗` is a
|
||||
definition matching nothing at all, which is just dead weight.
|
||||
|
||||
Click a `⚠` row to see the legs behind it, newest first. Each one says which
|
||||
side is missing and what that side would have had to look like:
|
||||
|
||||
```
|
||||
Date Amount Movement Missing Would pair with
|
||||
2026-04-01 500.00 checking → ? arriving leg +500.00 in savings matching *FROM CHECKING*,
|
||||
dated 2026-03-27 to 2026-04-06
|
||||
```
|
||||
|
||||
That is the search to run on the other statement. If it is not there, the
|
||||
statement probably is not imported yet — when nothing at all from that account
|
||||
is, the line says so. If something close is there, the difference tells you
|
||||
which condition failed: a date outside the window, an amount off by a fee (see
|
||||
[When the bank takes a fee](#when-the-bank-takes-a-fee)), or a description the
|
||||
glob does not match. Across currencies the line says *any amount*, since only
|
||||
the dates are checked there.
|
||||
|
||||
`Tol` is the definition's `tolerance_pct`, and it qualifies the counts beside
|
||||
it: row 3's two pairs were matched on slack rather than on the amount agreeing.
|
||||
It is blank for every definition that requires the exact amount — which is the
|
||||
default, so an empty column means nothing here is pairing on a mismatch.
|
||||
|
||||
**Delete** deletes a definition and **Prune unmatched** deletes every `✗` one at
|
||||
once, both after asking. Definitions marked `⚠` are never pruned — they are
|
||||
doing something, just not finishing it, and deleting one would hide the problem
|
||||
rather than fix it. **Refresh pairing** re-reads what is currently in the index.
|
||||
|
||||
### Tags
|
||||
|
||||
Every distinct tag, A→Z: the ones your transactions carry and any `rules.toml`
|
||||
names, with how many transactions carry each and the rules that write it.
|
||||
|
||||
```
|
||||
Tags · 2 tags
|
||||
|
||||
Tag Txns Rules
|
||||
clothes 0 #4 *ZARA* (0)
|
||||
groceries 2 #1 *LIDL* (0) #2 *KAUFLAND* (1) #3 *LIDL SOFIA* (1)
|
||||
```
|
||||
|
||||
Rules are numbered by their position in `rules.toml`, as on the rules screen,
|
||||
and the count after each is how many transactions it wins — so a tag's rules
|
||||
add up to its transactions, and a rule at `(0)` is matching nothing or being
|
||||
beaten by a more specific one. It is the place to spot near-duplicate tags
|
||||
(`grocery` beside `groceries`) and tags that several rules feed. A tag no rule
|
||||
names any more — `rules.toml` was edited since the last retag — says so; the
|
||||
next Retag clears it. The transfer label is not a tag and never appears here.
|
||||
|
||||
## rules.toml
|
||||
|
||||
Rules are evaluated in file order and the **first match wins**, so put transfer
|
||||
rules above general tag rules. Patterns are globs (`*` and `?`) matched
|
||||
case-insensitively, with whitespace collapsed.
|
||||
**The most specific rule wins**, so `*NIKOLA*` claims what it names even with a
|
||||
broad `*NIK*` sitting above it, and a catch-all can be written anywhere without
|
||||
swallowing the file. Patterns are globs (`*` and `?`) matched case-insensitively,
|
||||
with whitespace collapsed.
|
||||
|
||||
A rule matches on `match` (the description), `counterparty` (the other side's
|
||||
account number) and `type` (the bank's own classification). Setting several is
|
||||
an "and": all must match.
|
||||
Specificity is measured on what a rule spells out, in this order:
|
||||
|
||||
1. how many literal characters its patterns pin down — `*LIDL SOFIA*` (10)
|
||||
beats `*LIDL*` (4), and a bare `*` (0) is tried last of all. A rule setting
|
||||
both `match` and `type` has to satisfy both, so both count.
|
||||
2. how few `*` it uses, the only wildcard that swallows a run of any length:
|
||||
`LIDL` fixes both ends where `*LIDL*` does not, so it goes first. (`?` is
|
||||
not counted either way: it fixes a length, not a character.)
|
||||
3. whether it names an `account`, which is a narrowing the unscoped rule with
|
||||
the same patterns does not have.
|
||||
|
||||
Rules that tie on all three fall back to file order, and the earlier one wins —
|
||||
which is what keeps appending a rule from disturbing one already written.
|
||||
|
||||
A rule matches on `match` (the description) and `type` (the bank's own
|
||||
classification). Setting both is an "and": both must match.
|
||||
|
||||
`note` is free text for you, never for the matcher: why the rule is there, or
|
||||
what the unrecognisable payee behind the glob actually is. It shows in the last
|
||||
@@ -195,24 +523,6 @@ tag = "salary"
|
||||
match = "*ACME PAYROLL*"
|
||||
note = "paid on the 4th; the December one lands early"
|
||||
|
||||
# Money moved between your own accounts. Both legs need a rule.
|
||||
[[rule]]
|
||||
match = "*TO SAVINGS*"
|
||||
transfer = true
|
||||
tag = "transfer"
|
||||
|
||||
[[rule]]
|
||||
match = "*FROM CHECKING*"
|
||||
transfer = true
|
||||
tag = "transfer"
|
||||
|
||||
# Transfers are often only identifiable by the counterparty IBAN, whatever
|
||||
# the description happens to say.
|
||||
[[rule]]
|
||||
counterparty = "SI56*"
|
||||
transfer = true
|
||||
tag = "transfer"
|
||||
|
||||
# A rule can be limited to one account, and can require several patterns.
|
||||
[[rule]]
|
||||
account = "checking"
|
||||
@@ -225,9 +535,142 @@ match = "*LIDL*"
|
||||
tag = "groceries"
|
||||
```
|
||||
|
||||
Editing rules never touches tags you set by hand: rule verdicts and manual
|
||||
overrides are stored separately, and the manual one always wins. `money retag`
|
||||
is therefore safe to run whenever you change the file.
|
||||
Every tag comes from this file, so a tag is never something you have to keep
|
||||
safe: `money retag` recomputes all of them from the current rules and is safe
|
||||
to run whenever you change it. A transaction no rule matches simply stays
|
||||
untagged, and shows up under `money ls --untagged`, **Untagged only**, and
|
||||
`(untagged)` in the report — unless it is one leg of a matched transfer, which
|
||||
is already spoken for and is left out of all three.
|
||||
|
||||
### Transfers
|
||||
|
||||
Money moved between your own accounts is one movement the statements show
|
||||
twice: an outflow leaving one account and an inflow arriving in another. A
|
||||
`[[transfer]]` block names both legs, in the same file and with the same globs
|
||||
as everything else.
|
||||
|
||||
```toml
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*TO SAVINGS*"
|
||||
to_account = "savings"
|
||||
to_desc = "*FROM CHECKING*"
|
||||
note = "monthly saving, usually lands the next working day"
|
||||
|
||||
# Such movements are often only identifiable by the other side's account
|
||||
# number, whatever the rest of the description happens to say. Every bank
|
||||
# parser keeps that number in the description, so an ordinary glob finds it.
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*SI56*"
|
||||
to_account = "traderepublic"
|
||||
to_desc = "*FROM NLB*"
|
||||
```
|
||||
|
||||
All four `_account` / `_desc` keys are required; `note` and `tolerance_pct` are
|
||||
optional, and `note`, as on a rule, never takes part in matching. `from_desc` is
|
||||
matched against outflows on `from_account` and `to_desc` against inflows on
|
||||
`to_account`, so a definition written backwards pairs nothing.
|
||||
|
||||
Two legs are the same movement when they match one definition and are dated no
|
||||
more than **5 days** apart. Within one currency they must also be the exact
|
||||
opposite amount, because a movement that arrives short is not obviously the same
|
||||
movement leaving.
|
||||
|
||||
#### When the bank takes a fee
|
||||
|
||||
Some routes charge, and then the two statements really do disagree: 500.00
|
||||
leaves and 495.00 arrives. `tolerance_pct` says how far short (or over) the
|
||||
arriving leg may be, as a percentage of the leg that left:
|
||||
|
||||
```toml
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*WIRE TO SAVINGS*"
|
||||
to_account = "savings"
|
||||
to_desc = "*WIRE FROM CHECKING*"
|
||||
tolerance_pct = 1.5
|
||||
note = "the bank keeps a wire fee"
|
||||
```
|
||||
|
||||
It belongs to that one definition and loosens nothing else — every other route
|
||||
still needs the exact amount. It defaults to 0, which is what every definition
|
||||
written without the key means, and it must stay under 100: at 100% any amount
|
||||
would pair with any other and the dates would be deciding alone. Keep it as
|
||||
tight as the fee actually requires. Every percent of slack is also a percent
|
||||
more chance of pairing two unrelated movements that happen to fall in the same
|
||||
five days, and where two candidates are the same number of days away the nearer
|
||||
amount wins.
|
||||
|
||||
**The difference is not forgiven, it is reported.** A pair leaves the report
|
||||
entirely, so a fee hidden inside one would be spending that appears nowhere at
|
||||
all. `money report` names it on the excluded line and the web app's report gives it
|
||||
its own row (see below). Across currencies there is no fee to compute — the two
|
||||
numbers are in different units — so a tolerance there means nothing and is
|
||||
ignored.
|
||||
|
||||
**Across currencies the amount is not checked at all.** An exchange between two
|
||||
of your own accounts — the same money in EUR and in BGN — has two unrelated
|
||||
numbers, and the tool holds no exchange rates, so the dates carry the pairing on
|
||||
their own and the nearest one wins. That is a weaker rule than the
|
||||
same-currency one, deliberately: use the same route twice inside one window and
|
||||
it will pair the wrong two legs. The transfer builder shows both amounts for
|
||||
such a pair (`500.00 → 977.90`), which is the only place the rate the bank
|
||||
actually used appears.
|
||||
|
||||
Definitions are evaluated in file order and each transaction belongs to at most
|
||||
one transfer, so — exactly as with rules — the first definition to claim a leg
|
||||
keeps it.
|
||||
|
||||
**A matched pair leaves the report entirely.** Both legs go together, so a
|
||||
report over everything is unchanged in total by money you shuffled between your
|
||||
own accounts, and `money report` says what it held out, per currency and per
|
||||
direction — an exchange puts its two legs in different currencies, so each side
|
||||
reports only the leg it saw:
|
||||
|
||||
```
|
||||
transfers excluded: 4 legs in EUR, 2000.00 out, 1500.00 in
|
||||
transfers excluded: 1 legs in BGN, 0.00 out, 977.90 in
|
||||
```
|
||||
|
||||
Where a `tolerance_pct` definition let a fee through, the line says so too, and
|
||||
that is the number your spending is short by:
|
||||
|
||||
```
|
||||
transfers excluded: 2 legs in EUR, 500.00 out, 495.00 in, 5.00 in fees
|
||||
```
|
||||
|
||||
The report screen shows the same thing under its `TOTAL`, the fee on its
|
||||
own row (for whichever period it is on):
|
||||
|
||||
```
|
||||
TAG CUR OUT IN NET N
|
||||
groceries EUR 20.00 0.00 -20.00 1
|
||||
TOTAL EUR 20.00 0.00 -20.00
|
||||
(transfers) EUR 500.00 495.00 -5.00 2
|
||||
⤷ fees EUR 5.00 -5.00 1
|
||||
```
|
||||
|
||||
The fee is counted only for pairs whose legs are *both* in the report you are
|
||||
looking at, for the same reason the line counts legs and not transfers: filter
|
||||
to one account or one month and you usually have one side of a movement, and
|
||||
half a pair cannot say what the other half received.
|
||||
|
||||
A paired leg shows as `(transfer)` in the tag column of the transaction list
|
||||
and of `money ls`, so it reads as accounted for rather than as a blank waiting
|
||||
for a rule. That label is display only — nothing writes it to the index, and it
|
||||
is never offered as a tag to complete against. A rule tag wins where there is
|
||||
one, since that is your own word for it.
|
||||
|
||||
An *unpaired* leg is not a transfer and still counts, tagged like anything
|
||||
else. That is deliberate: money that left an account and cannot be shown to
|
||||
have arrived is exactly what you want to see, not something to quietly drop.
|
||||
`money import` and `money retag` report unpaired legs on stderr, and the
|
||||
transfers screen shows which definition they belong to.
|
||||
|
||||
The pairing is derived state, like the tags: it lives in the index, is
|
||||
recomputed wholesale by `money retag` and after every import, and disappears
|
||||
with the definition that made it.
|
||||
|
||||
## account.toml
|
||||
|
||||
@@ -236,44 +679,21 @@ Every account folder needs one. `currency` and `parser` are required.
|
||||
```toml
|
||||
name = "Main Checking"
|
||||
currency = "EUR"
|
||||
parser = "csv"
|
||||
parser = "nlb"
|
||||
# minor_digits = 2 # decimal places for this currency
|
||||
# include = ["*.csv"] # only treat matching files as statements
|
||||
```
|
||||
|
||||
### `parser = "csv"`
|
||||
### Parsers
|
||||
|
||||
Column positions are 0-based.
|
||||
|
||||
```toml
|
||||
[csv]
|
||||
delimiter = "," # default ","
|
||||
skip_rows = 1 # header rows to drop
|
||||
date = { col = 0, layout = "02.01.2006" } # Go reference layout
|
||||
description = { col = 3 }
|
||||
amount = { col = 4, decimal = ",", thousands = "." }
|
||||
# invert = true # if outflows are written as positive
|
||||
```
|
||||
|
||||
For statements with separate debit and credit columns, replace `amount`:
|
||||
|
||||
```toml
|
||||
debit = { col = 4 } # both written as positive numbers
|
||||
credit = { col = 5 }
|
||||
```
|
||||
|
||||
Amount parsing is deliberately tolerant: `1.234,56`, `-45.20`, `45,20-`,
|
||||
`(45.20)` and `45.20 EUR` all work.
|
||||
|
||||
### Bank-specific parsers
|
||||
|
||||
Three are built in, ported from the original Python extractors. They need no
|
||||
`[csv]` block — the layout is baked in.
|
||||
Three are built in, ported from the original Python extractors. A statement
|
||||
layout is described in Go rather than in a table of column indexes, so there is
|
||||
nothing else to configure — `parser` names one of these and that is all.
|
||||
|
||||
| `parser` | Statement | Notes |
|
||||
| --- | --- | --- |
|
||||
| `nlb` | NLB izpisek PDF | Wrapped descriptions and the counterparty IBAN are folded in from continuation lines. |
|
||||
| `traderepublic` | Trade Republic PDF | Handles both the single-line and the stacked layout by measuring column positions. |
|
||||
| `nlb` | NLB izpisek PDF | Wrapped descriptions are folded in from continuation lines, and the IBAN column is appended to the description. Uploads are saved as `izpisek_YYYY-MM-DD.pdf` after the statement date. |
|
||||
| `traderepublic` | Trade Republic PDF | Handles both the single-line and the stacked layout by measuring column positions. Uploads are saved as `traderepublic_YYYY_MM_DD_YYYY_MM_DD.pdf` after the statement period. |
|
||||
| `revolut` | `account-statement*.csv` | Skips non-COMPLETED rows, folds the fee into the amount. |
|
||||
|
||||
```toml
|
||||
@@ -284,7 +704,11 @@ parser = "traderepublic"
|
||||
|
||||
The two PDF parsers shell out to `pdftotext -layout` (poppler-utils), exactly
|
||||
as the Python versions did; its layout reconstruction is what makes the
|
||||
column-based parsing work.
|
||||
column-based parsing work. A build made with `scripts/build-bundled.sh`
|
||||
carries its own copy instead (see Install).
|
||||
|
||||
Amount parsing is shared and deliberately tolerant: `1.234,56`, `-45.20`,
|
||||
`45,20-`, `(45.20)` and `45.20 EUR` all work, whichever parser reads them.
|
||||
|
||||
**Revolut and multiple currencies.** One export can hold several currencies,
|
||||
but an account here has exactly one. Rows in other currencies are skipped and
|
||||
@@ -292,27 +716,10 @@ reported at import. To keep them, give that currency its own account folder
|
||||
with its own `currency` and `minor_digits` (JPY wants `minor_digits = 0`) and
|
||||
put a copy or symlink of the export in it.
|
||||
|
||||
### `parser = "cmd"`
|
||||
## Adding a parser
|
||||
|
||||
Runs an external extractor and reads normalised CSV (`date,description,amount`)
|
||||
from its stdout. This is how PDF statements and any existing Python extractor
|
||||
are handled without porting them first.
|
||||
|
||||
```toml
|
||||
parser = "cmd"
|
||||
[cmd]
|
||||
argv = ["python3", "../extract_bankx.py", "{{file}}"]
|
||||
skip_rows = 1 # if the script prints a header
|
||||
# layout = "2006-01-02" # date format the script emits (default)
|
||||
```
|
||||
|
||||
`{{file}}` is replaced with the statement's path, and the command runs with the
|
||||
account folder as its working directory, so relative script paths work.
|
||||
|
||||
## Adding a native parser
|
||||
|
||||
When a Python extractor is ported to Go, drop it in `internal/parser` and
|
||||
register it — no other package changes:
|
||||
A new bank means a new parser. Drop it in `internal/parser` and register it —
|
||||
no other package changes:
|
||||
|
||||
```go
|
||||
func init() {
|
||||
|
||||
+251
-52
@@ -2,15 +2,23 @@
|
||||
//
|
||||
// The data root holds one folder per account. Drop statements into those
|
||||
// folders, run `money import`, and tag what comes out with glob rules in
|
||||
// rules.toml. Running it with no subcommand opens the TUI.
|
||||
// rules.toml. Running it with no subcommand serves the web app.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"sort"
|
||||
"strings"
|
||||
"syscall"
|
||||
"text/tabwriter"
|
||||
"time"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
"git.petrovv.com/nikola/money/internal/importer"
|
||||
@@ -19,17 +27,20 @@ import (
|
||||
"git.petrovv.com/nikola/money/internal/report"
|
||||
"git.petrovv.com/nikola/money/internal/rules"
|
||||
"git.petrovv.com/nikola/money/internal/store"
|
||||
"git.petrovv.com/nikola/money/internal/tui"
|
||||
"git.petrovv.com/nikola/money/internal/transfers"
|
||||
"git.petrovv.com/nikola/money/internal/web"
|
||||
)
|
||||
|
||||
const usage = `money - statement-driven personal finance tracker
|
||||
|
||||
usage: money [--root DIR] <command> [flags]
|
||||
usage: money [--root DIR] <command> [flags] (--root may also follow the command)
|
||||
|
||||
commands:
|
||||
tui browse and tag transactions, build and prune rules (default)
|
||||
serve browse transactions, build the rules that tag them and the
|
||||
transfers that pair them across accounts, in a browser
|
||||
(--addr, default 127.0.0.1:8080; the default command)
|
||||
import extract transactions from every statement into the index
|
||||
retag re-apply rules.toml to everything already imported
|
||||
retag re-apply rules.toml: retag everything and re-pair transfers
|
||||
ls list transactions
|
||||
report spending by tag
|
||||
accounts list accounts with balances
|
||||
@@ -71,11 +82,22 @@ func run(args []string) error {
|
||||
}
|
||||
}
|
||||
|
||||
cmd := "tui"
|
||||
cmd := "serve"
|
||||
if len(args) > 0 {
|
||||
cmd, args = args[0], args[1:]
|
||||
}
|
||||
|
||||
// `money serve --root DIR` is too natural to reject, so --root is also
|
||||
// accepted after the subcommand; each subcommand's own flag set never
|
||||
// sees it.
|
||||
after, args, err := takeRoot(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if after != "" {
|
||||
root = after
|
||||
}
|
||||
|
||||
abs, source, err := config.ResolveRoot(root)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -97,8 +119,8 @@ func run(args []string) error {
|
||||
case "parsers":
|
||||
fmt.Println(strings.Join(parser.Names(), "\n"))
|
||||
return nil
|
||||
case "tui":
|
||||
return cmdTUI(abs, args)
|
||||
case "serve":
|
||||
return cmdServe(abs, args)
|
||||
case "help", "-h", "--help":
|
||||
fmt.Print(usage)
|
||||
return nil
|
||||
@@ -123,6 +145,7 @@ func cmdConfig(root string, source config.RootSource) error {
|
||||
fmt.Fprintf(w, "data root\t%s\n", root)
|
||||
fmt.Fprintf(w, "chosen by\t%s\n", source)
|
||||
fmt.Fprintf(w, "config file\t%s (%s)\n", path, exists)
|
||||
fmt.Fprintf(w, "pdftotext\t%s\n", parser.PdftotextSource())
|
||||
if _, err := os.Stat(root); err != nil {
|
||||
fmt.Fprintf(w, "status\tdoes not exist yet\n")
|
||||
} else {
|
||||
@@ -137,24 +160,33 @@ func cmdConfig(root string, source config.RootSource) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// opened is everything a command needs: the index, the account folders on
|
||||
// disk, and the two engines rules.toml describes.
|
||||
type opened struct {
|
||||
db *store.DB
|
||||
accounts []*config.Account
|
||||
engine *rules.Engine
|
||||
links *transfers.Engine
|
||||
}
|
||||
|
||||
// open loads the config and index that every command needs.
|
||||
func open(root string) (*store.DB, []*config.Account, *rules.Engine, error) {
|
||||
func open(root string) (*opened, error) {
|
||||
if _, err := os.Stat(root); err != nil {
|
||||
return nil, nil, nil, fmt.Errorf("data root %s is not readable: %w", root, err)
|
||||
return nil, fmt.Errorf("data root %s is not readable: %w", root, err)
|
||||
}
|
||||
accounts, err := config.LoadAccounts(root)
|
||||
if err != nil {
|
||||
return nil, nil, nil, err
|
||||
return nil, err
|
||||
}
|
||||
r, err := config.LoadRules(root)
|
||||
if err != nil {
|
||||
return nil, nil, nil, err
|
||||
return nil, err
|
||||
}
|
||||
db, err := store.Open(config.IndexPath(root))
|
||||
if err != nil {
|
||||
return nil, nil, nil, err
|
||||
return nil, err
|
||||
}
|
||||
return db, accounts, rules.New(r), nil
|
||||
return &opened{db: db, accounts: accounts, engine: rules.New(r), links: transfers.New(r)}, nil
|
||||
}
|
||||
|
||||
func cmdImport(root string, args []string) error {
|
||||
@@ -164,17 +196,17 @@ func cmdImport(root string, args []string) error {
|
||||
return err
|
||||
}
|
||||
|
||||
db, accounts, engine, err := open(root)
|
||||
o, err := open(root)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
defer o.db.Close()
|
||||
|
||||
if len(accounts) == 0 {
|
||||
if len(o.accounts) == 0 {
|
||||
return fmt.Errorf("no accounts found in %s (an account is a folder containing %s)", root, config.AccountFile)
|
||||
}
|
||||
|
||||
res, err := importer.Run(root, db, accounts, engine, importer.Options{Force: *force})
|
||||
res, err := importer.Run(root, o.db, o.accounts, o.engine, o.links, importer.Options{Force: *force})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -199,6 +231,7 @@ func cmdImport(root string, args []string) error {
|
||||
|
||||
parsed, added, skipped := res.Total()
|
||||
fmt.Printf("\n%d new, %d duplicate, %d parsed; %d rows retagged\n", added, skipped, parsed, res.Retagged)
|
||||
reportPairing(res.Paired, res.Unpaired)
|
||||
|
||||
failures := res.Errs()
|
||||
for _, f := range failures {
|
||||
@@ -210,98 +243,187 @@ func cmdImport(root string, args []string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// cmdRetag re-derives everything rules.toml decides: the tags and the transfer
|
||||
// pairing. They are one command because they are one file, and leaving half of
|
||||
// the derived state stale would be worse than not offering it at all.
|
||||
func cmdRetag(root string, _ []string) error {
|
||||
db, _, engine, err := open(root)
|
||||
o, err := open(root)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
defer o.db.Close()
|
||||
|
||||
n, err := engine.Retag(db)
|
||||
n, err := o.engine.Retag(o.db)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("%d transactions retagged\n", n)
|
||||
|
||||
paired, unpaired, err := o.links.Link(o.db)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reportPairing(paired, unpaired)
|
||||
return nil
|
||||
}
|
||||
|
||||
// reportPairing prints the state of the transfer pairing. Unmatched legs are
|
||||
// sent to stderr: a leg that never found its other side is money that left an
|
||||
// account and cannot be shown to have arrived, which is a warning, not a
|
||||
// statistic.
|
||||
func reportPairing(paired, unpaired int) {
|
||||
if paired == 0 && unpaired == 0 {
|
||||
return
|
||||
}
|
||||
fmt.Printf("%d transfers matched\n", paired)
|
||||
if unpaired > 0 {
|
||||
fmt.Fprintf(os.Stderr, "warning: %d transfer leg(s) have no counterpart\n", unpaired)
|
||||
}
|
||||
}
|
||||
|
||||
func cmdLs(root string, args []string) error {
|
||||
fs := flag.NewFlagSet("ls", flag.ContinueOnError)
|
||||
account := fs.String("account", "", "only this account slug")
|
||||
month := fs.String("month", "", "only this month (YYYY-MM)")
|
||||
search := fs.String("search", "", "only descriptions containing this text")
|
||||
untagged := fs.Bool("untagged", false, "only transactions with no effective tag")
|
||||
untagged := fs.Bool("untagged", false, "only transactions no rule tagged and no transfer claimed")
|
||||
limit := fs.Int("limit", 0, "maximum rows (0 = no limit)")
|
||||
wide := fs.Bool("wide", false, "also show type, counterparty and reported balance")
|
||||
wide := fs.Bool("wide", false, "also show type and reported balance")
|
||||
uniq := fs.Bool("uniq", false, "one row per distinct description")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if *uniq && *wide {
|
||||
// --wide's columns all belong to a single transaction, and nothing
|
||||
// sensible can be printed for a whole group of them.
|
||||
return fmt.Errorf("--uniq and --wide cannot be combined")
|
||||
}
|
||||
|
||||
db, _, _, err := open(root)
|
||||
o, err := open(root)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
defer o.db.Close()
|
||||
|
||||
txns, err := db.Transactions(store.Filter{
|
||||
f := store.Filter{
|
||||
AccountSlug: *account,
|
||||
Month: *month,
|
||||
Search: *search,
|
||||
Untagged: *untagged,
|
||||
Limit: *limit,
|
||||
})
|
||||
}
|
||||
// Deduplicating first would make a limit mean "the distinct descriptions
|
||||
// among the newest N rows", which is not what it says. Under --uniq it caps
|
||||
// what is printed instead.
|
||||
if *uniq {
|
||||
f.Limit = 0
|
||||
}
|
||||
txns, err := o.db.Transactions(f)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if *uniq {
|
||||
return printUniq(os.Stdout, txns, *limit)
|
||||
}
|
||||
|
||||
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
|
||||
if *wide {
|
||||
fmt.Fprintln(w, "DATE\tACCOUNT\tAMOUNT\tCUR\tBALANCE\tTAG\tT\tTYPE\tCOUNTERPARTY\tDESCRIPTION")
|
||||
fmt.Fprintln(w, "DATE\tACCOUNT\tAMOUNT\tCUR\tBALANCE\tTAG\tTYPE\tDESCRIPTION")
|
||||
} else {
|
||||
fmt.Fprintln(w, "DATE\tACCOUNT\tAMOUNT\tCUR\tTAG\tT\tDESCRIPTION")
|
||||
fmt.Fprintln(w, "DATE\tACCOUNT\tAMOUNT\tCUR\tTAG\tDESCRIPTION")
|
||||
}
|
||||
for _, t := range txns {
|
||||
transfer := ""
|
||||
if t.IsTransfer() {
|
||||
transfer = "T"
|
||||
}
|
||||
if *wide {
|
||||
balance := ""
|
||||
if t.BalanceMinor != nil {
|
||||
balance = model.FormatMinor(*t.BalanceMinor, t.MinorDigits)
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||
t.Date, t.AccountSlug, t.FormatAmount(), t.Currency, balance,
|
||||
t.Tag(), transfer, t.Type, t.Counterparty, t.Description)
|
||||
t.DisplayTag(), t.Type, t.Description)
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||
t.Date, t.AccountSlug, t.FormatAmount(), t.Currency, t.Tag(), transfer, t.Description)
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||
t.Date, t.AccountSlug, t.FormatAmount(), t.Currency, t.DisplayTag(), t.Description)
|
||||
}
|
||||
w.Flush()
|
||||
fmt.Printf("\n%d transactions\n", len(txns))
|
||||
return nil
|
||||
}
|
||||
|
||||
// printUniq lists each description once, which is the shape of the question
|
||||
// "what still needs a rule?" — fifty visits to one shop are one pattern to
|
||||
// write, not fifty rows to read.
|
||||
//
|
||||
// The account is deliberately not part of the key. A rule matches on the
|
||||
// description and only optionally narrows to an account, so the same payee
|
||||
// seen on two accounts is still one pattern; splitting the row per account
|
||||
// would overstate the work left and disagree with the rule builder's list,
|
||||
// which groups the same way. Use --account to scope the listing instead.
|
||||
//
|
||||
// Descriptions are printed as rules.Engine matches them: normalised, since that
|
||||
// is the string a glob is actually tested against, so a pattern written from
|
||||
// this list behaves the way the list reads. It also means two statements that
|
||||
// differ only in spacing or case collapse to the one row they deserve.
|
||||
func printUniq(out io.Writer, txns []model.Transaction, limit int) error {
|
||||
seen := map[string]bool{}
|
||||
var rows []string
|
||||
for _, t := range txns {
|
||||
desc := model.NormalizeDescription(t.Description)
|
||||
if seen[desc] {
|
||||
continue
|
||||
}
|
||||
seen[desc] = true
|
||||
rows = append(rows, desc)
|
||||
}
|
||||
// Alphabetical, so the same payee under slightly different wordings lands
|
||||
// on adjacent lines, where one glob covering both is easy to see.
|
||||
sort.Strings(rows)
|
||||
|
||||
total := len(rows)
|
||||
if limit > 0 && len(rows) > limit {
|
||||
rows = rows[:limit]
|
||||
}
|
||||
|
||||
w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
|
||||
fmt.Fprintln(w, "DESCRIPTION")
|
||||
for _, desc := range rows {
|
||||
fmt.Fprintln(w, desc)
|
||||
}
|
||||
w.Flush()
|
||||
|
||||
if len(rows) < total {
|
||||
fmt.Fprintf(out, "\n%d of %d distinct descriptions\n", len(rows), total)
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(out, "\n%d distinct descriptions\n", total)
|
||||
return nil
|
||||
}
|
||||
|
||||
func cmdReport(root string, args []string) error {
|
||||
fs := flag.NewFlagSet("report", flag.ContinueOnError)
|
||||
month := fs.String("month", "", "only this month (YYYY-MM)")
|
||||
account := fs.String("account", "", "only this account slug")
|
||||
sortBy := fs.String("sort", report.OrderOut.String(), "order rows by: out, in, net, count, tag")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
db, _, _, err := open(root)
|
||||
order, err := report.ParseOrder(*sortBy)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
txns, err := db.Transactions(store.Filter{Month: *month, AccountSlug: *account})
|
||||
o, err := open(root)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rows := report.ByTag(txns)
|
||||
defer o.db.Close()
|
||||
|
||||
txns, err := o.db.Transactions(store.Filter{Month: *month, AccountSlug: *account})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rows := report.ByTag(txns, order)
|
||||
|
||||
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
|
||||
fmt.Fprintln(w, "TAG\tCUR\tOUT\tIN\tNET\tN")
|
||||
@@ -316,29 +438,44 @@ func cmdReport(root string, args []string) error {
|
||||
model.FormatMinor(c.Net(), c.Digits))
|
||||
}
|
||||
w.Flush()
|
||||
fmt.Println("\ntransfers between own accounts are excluded")
|
||||
|
||||
// Say what was held out, or the report silently disagrees with the account
|
||||
// balances by exactly the amount moved between accounts. Both directions
|
||||
// are named: they came off the OUT and IN columns respectively, and an
|
||||
// exchange puts only one of them in this currency.
|
||||
for _, x := range report.Excluded(txns) {
|
||||
line := fmt.Sprintf("\ntransfers excluded: %d legs in %s, %s out, %s in",
|
||||
x.Legs, x.Currency,
|
||||
model.FormatMinor(x.Out, x.Digits), model.FormatMinor(x.In, x.Digits))
|
||||
// A fee went out with the pair that carried it, so unless it is named
|
||||
// here it is money that left the report without ever being spent.
|
||||
if x.Fee != 0 {
|
||||
line += fmt.Sprintf(", %s in fees", model.FormatMinor(x.Fee, x.Digits))
|
||||
}
|
||||
fmt.Println(line)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cmdAccounts(root string, _ []string) error {
|
||||
db, _, _, err := open(root)
|
||||
o, err := open(root)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
defer o.db.Close()
|
||||
|
||||
accounts, err := db.Accounts()
|
||||
accounts, err := o.db.Accounts()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
|
||||
fmt.Fprintln(w, "SLUG\tNAME\tBALANCE\tCUR\tTXNS")
|
||||
for _, a := range accounts {
|
||||
bal, err := db.Balance(a.ID)
|
||||
bal, err := o.db.Balance(a.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
n, err := db.Count(a.ID)
|
||||
n, err := o.db.Count(a.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -348,11 +485,73 @@ func cmdAccounts(root string, _ []string) error {
|
||||
return w.Flush()
|
||||
}
|
||||
|
||||
func cmdTUI(root string, _ []string) error {
|
||||
db, accounts, engine, err := open(root)
|
||||
// takeRoot removes --root/-root (as `--root DIR` or `--root=DIR`) from a
|
||||
// subcommand's arguments, returning the last value given. It stops at `--`,
|
||||
// after which everything is positional.
|
||||
func takeRoot(args []string) (root string, rest []string, err error) {
|
||||
rest = make([]string, 0, len(args))
|
||||
for i := 0; i < len(args); i++ {
|
||||
a := args[i]
|
||||
if a == "--" {
|
||||
return root, append(rest, args[i:]...), nil
|
||||
}
|
||||
if !strings.HasPrefix(a, "-") {
|
||||
rest = append(rest, a)
|
||||
continue
|
||||
}
|
||||
name, inline, hasInline := strings.Cut(strings.TrimLeft(a, "-"), "=")
|
||||
if name != "root" {
|
||||
rest = append(rest, a)
|
||||
continue
|
||||
}
|
||||
if hasInline {
|
||||
root = inline
|
||||
continue
|
||||
}
|
||||
if i+1 >= len(args) {
|
||||
return "", nil, fmt.Errorf("--root needs a directory")
|
||||
}
|
||||
root = args[i+1]
|
||||
i++
|
||||
}
|
||||
return root, rest, nil
|
||||
}
|
||||
|
||||
// cmdServe serves the web app until interrupted. There is no
|
||||
// authentication, so it listens on loopback unless told otherwise: anyone who
|
||||
// can reach the port can rewrite rules.toml and start an import.
|
||||
func cmdServe(root string, args []string) error {
|
||||
fs := flag.NewFlagSet("serve", flag.ContinueOnError)
|
||||
addr := fs.String("addr", "127.0.0.1:8080", "address to listen on")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
o, err := open(root)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer db.Close()
|
||||
return tui.Run(root, db, accounts, engine)
|
||||
defer o.db.Close()
|
||||
|
||||
srv := &http.Server{
|
||||
Addr: *addr,
|
||||
Handler: web.New(root, o.db, o.accounts, o.engine, o.links).Handler(),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
go func() {
|
||||
<-ctx.Done()
|
||||
// Let an import in flight finish writing the index before it closes.
|
||||
shutdown, cancel := context.WithTimeout(context.Background(), time.Minute)
|
||||
defer cancel()
|
||||
srv.Shutdown(shutdown)
|
||||
}()
|
||||
|
||||
fmt.Printf("serving %s on http://%s\n", root, *addr)
|
||||
if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/model"
|
||||
)
|
||||
|
||||
func txn(account, desc string) model.Transaction {
|
||||
return model.Transaction{AccountSlug: account, Description: desc}
|
||||
}
|
||||
|
||||
func uniqOutput(t *testing.T, limit int, txns ...model.Transaction) string {
|
||||
t.Helper()
|
||||
var b strings.Builder
|
||||
if err := printUniq(&b, txns, limit); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// The point of the listing is one row per pattern still to write, so fifty
|
||||
// visits to one shop must not be fifty rows.
|
||||
func TestUniqCollapsesRepeats(t *testing.T) {
|
||||
out := uniqOutput(t, 0,
|
||||
txn("checking", "LIDL SOFIA 4412"),
|
||||
txn("checking", "LIDL SOFIA 4412"),
|
||||
txn("checking", "ZARA"),
|
||||
)
|
||||
if n := strings.Count(out, "LIDL SOFIA 4412"); n != 1 {
|
||||
t.Errorf("the shop appears %d times, want once:\n%s", n, out)
|
||||
}
|
||||
if !strings.Contains(out, "2 distinct descriptions") {
|
||||
t.Errorf("count is wrong:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// Descriptions are printed as a glob sees them, so two statements differing
|
||||
// only in spacing or case are the one pattern they really are -- and what is
|
||||
// printed is exactly what a rule written from it will match.
|
||||
func TestUniqNormalisesLikeAGlob(t *testing.T) {
|
||||
out := uniqOutput(t, 0,
|
||||
txn("checking", "Lidl Sofia 4412"),
|
||||
txn("checking", "LIDL SOFIA 4412"),
|
||||
)
|
||||
if !strings.Contains(out, "LIDL SOFIA 4412") {
|
||||
t.Errorf("want the normalised description:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, "1 distinct descriptions") {
|
||||
t.Errorf("spacing and case should not make a second row:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// The account is not part of the key. A rule matches on the description and
|
||||
// only optionally narrows to an account, so one payee seen on two accounts is
|
||||
// still one pattern to write -- and the rule builder's list groups the same
|
||||
// way, which is the list this one has to agree with.
|
||||
func TestUniqIgnoresTheAccount(t *testing.T) {
|
||||
out := uniqOutput(t, 0,
|
||||
txn("checking", "TRANSFER"),
|
||||
txn("savings", "TRANSFER"),
|
||||
)
|
||||
if !strings.Contains(out, "1 distinct descriptions") {
|
||||
t.Errorf("want one row for one description:\n%s", out)
|
||||
}
|
||||
if strings.Contains(out, "checking") || strings.Contains(out, "savings") {
|
||||
t.Errorf("the account has no column here:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// The limit caps the rows printed, and says so, rather than silently cutting
|
||||
// the list down to a number that looks complete.
|
||||
func TestUniqLimitReportsWhatItHeldBack(t *testing.T) {
|
||||
out := uniqOutput(t, 1,
|
||||
txn("checking", "ZARA"),
|
||||
txn("checking", "LIDL"),
|
||||
)
|
||||
if !strings.Contains(out, "1 of 2 distinct descriptions") {
|
||||
t.Errorf("want the total alongside the limit:\n%s", out)
|
||||
}
|
||||
// Alphabetical within an account, so near-identical wordings land together.
|
||||
if !strings.Contains(out, "LIDL") || strings.Contains(out, "ZARA") {
|
||||
t.Errorf("want the first row alphabetically:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRootAfterTheSubcommand(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
args []string
|
||||
root string
|
||||
rest []string
|
||||
}{
|
||||
{[]string{"--root", "/r", "--addr", ":9"}, "/r", []string{"--addr", ":9"}},
|
||||
{[]string{"--addr", ":9", "-root=/r"}, "/r", []string{"--addr", ":9"}},
|
||||
{[]string{"--untagged"}, "", []string{"--untagged"}},
|
||||
{[]string{"--", "--root", "/r"}, "", []string{"--", "--root", "/r"}},
|
||||
} {
|
||||
root, rest, err := takeRoot(tc.args)
|
||||
if err != nil {
|
||||
t.Fatalf("%q: %v", tc.args, err)
|
||||
}
|
||||
if root != tc.root || strings.Join(rest, " ") != strings.Join(tc.rest, " ") {
|
||||
t.Errorf("%q: got %q %q, want %q %q", tc.args, root, rest, tc.root, tc.rest)
|
||||
}
|
||||
}
|
||||
if _, _, err := takeRoot([]string{"--root"}); err == nil {
|
||||
t.Error("--root without a directory was accepted")
|
||||
}
|
||||
}
|
||||
@@ -4,38 +4,16 @@ go 1.25.6
|
||||
|
||||
require (
|
||||
github.com/BurntSushi/toml v1.6.0
|
||||
github.com/charmbracelet/bubbles v1.0.0
|
||||
github.com/charmbracelet/bubbletea v1.3.10
|
||||
github.com/charmbracelet/lipgloss v1.1.0
|
||||
modernc.org/sqlite v1.56.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/atotto/clipboard v0.1.4 // indirect
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||
github.com/charmbracelet/colorprofile v0.4.1 // indirect
|
||||
github.com/charmbracelet/x/ansi v0.11.6 // indirect
|
||||
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
|
||||
github.com/charmbracelet/x/term v0.2.2 // indirect
|
||||
github.com/clipperhouse/displaywidth v0.9.0 // indirect
|
||||
github.com/clipperhouse/stringish v0.1.1 // indirect
|
||||
github.com/clipperhouse/uax29/v2 v2.5.0 // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
|
||||
github.com/mattn/go-isatty v0.0.24 // indirect
|
||||
github.com/mattn/go-localereader v0.0.1 // indirect
|
||||
github.com/mattn/go-runewidth v0.0.19 // indirect
|
||||
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
|
||||
github.com/muesli/cancelreader v0.2.2 // indirect
|
||||
github.com/muesli/termenv v0.16.0 // indirect
|
||||
github.com/ncruces/go-strftime v1.0.0 // indirect
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||
github.com/rivo/uniseg v0.4.7 // indirect
|
||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.3.8 // indirect
|
||||
modernc.org/libc v1.74.4 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
modernc.org/memory v1.11.0 // indirect
|
||||
|
||||
@@ -1,76 +1,25 @@
|
||||
github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk=
|
||||
github.com/BurntSushi/toml v1.6.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
|
||||
github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
|
||||
github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
|
||||
github.com/aymanbagabas/go-udiff v0.3.1 h1:LV+qyBQ2pqe0u42ZsUEtPiCaUoqgA9gYRDs3vj1nolY=
|
||||
github.com/aymanbagabas/go-udiff v0.3.1/go.mod h1:G0fsKmG+P6ylD0r6N/KgQD/nWzgfnl8ZBcNLgcbrw8E=
|
||||
github.com/charmbracelet/bubbles v1.0.0 h1:12J8/ak/uCZEMQ6KU7pcfwceyjLlWsDLAxB5fXonfvc=
|
||||
github.com/charmbracelet/bubbles v1.0.0/go.mod h1:9d/Zd5GdnauMI5ivUIVisuEm3ave1XwXtD1ckyV6r3E=
|
||||
github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
|
||||
github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
|
||||
github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=
|
||||
github.com/charmbracelet/colorprofile v0.4.1/go.mod h1:U1d9Dljmdf9DLegaJ0nGZNJvoXAhayhmidOdcBwAvKk=
|
||||
github.com/charmbracelet/lipgloss v1.1.0 h1:vYXsiLHVkK7fp74RkV7b2kq9+zDLoEU4MZoFqR/noCY=
|
||||
github.com/charmbracelet/lipgloss v1.1.0/go.mod h1:/6Q8FR2o+kj8rz4Dq0zQc3vYf7X+B0binUUBwA0aL30=
|
||||
github.com/charmbracelet/x/ansi v0.11.6 h1:GhV21SiDz/45W9AnV2R61xZMRri5NlLnl6CVF7ihZW8=
|
||||
github.com/charmbracelet/x/ansi v0.11.6/go.mod h1:2JNYLgQUsyqaiLovhU2Rv/pb8r6ydXKS3NIttu3VGZQ=
|
||||
github.com/charmbracelet/x/cellbuf v0.0.15 h1:ur3pZy0o6z/R7EylET877CBxaiE1Sp1GMxoFPAIztPI=
|
||||
github.com/charmbracelet/x/cellbuf v0.0.15/go.mod h1:J1YVbR7MUuEGIFPCaaZ96KDl5NoS0DAWkskup+mOY+Q=
|
||||
github.com/charmbracelet/x/exp/golden v0.0.0-20241011142426-46044092ad91 h1:payRxjMjKgx2PaCWLZ4p3ro9y97+TVLZNaRZgJwSVDQ=
|
||||
github.com/charmbracelet/x/exp/golden v0.0.0-20241011142426-46044092ad91/go.mod h1:wDlXFlCrmJ8J+swcL/MnGUuYnqgQdW9rhSD61oNMb6U=
|
||||
github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk=
|
||||
github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI=
|
||||
github.com/clipperhouse/displaywidth v0.9.0 h1:Qb4KOhYwRiN3viMv1v/3cTBlz3AcAZX3+y9OLhMtAtA=
|
||||
github.com/clipperhouse/displaywidth v0.9.0/go.mod h1:aCAAqTlh4GIVkhQnJpbL0T/WfcrJXHcj8C0yjYcjOZA=
|
||||
github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
|
||||
github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
|
||||
github.com/clipperhouse/uax29/v2 v2.5.0 h1:x7T0T4eTHDONxFJsL94uKNKPHrclyFI0lm7+w94cO8U=
|
||||
github.com/clipperhouse/uax29/v2 v2.5.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6baUTXGLOoWe4PQhGxaX0KpnayAqC48p4=
|
||||
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
|
||||
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3 h1:LMLX+LgTNWpfvCBdFebv6EsYotImrt/Ppc5cXIriCSo=
|
||||
github.com/google/pprof v0.0.0-20260802141513-ef3492d7dac3/go.mod h1:jl5iWTm0/hd5PjEYEOuwAJ57L/CibdZfrqZ5XA5GrCk=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
|
||||
github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
|
||||
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
|
||||
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
|
||||
github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4=
|
||||
github.com/mattn/go-localereader v0.0.1/go.mod h1:8fBrzywKY7BI3czFoHkuzRoWE9C+EiG4R1k4Cjx5p88=
|
||||
github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
|
||||
github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
|
||||
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 h1:ZK8zHtRHOkbHy6Mmr5D264iyp3TiX5OmNcI5cIARiQI=
|
||||
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6/go.mod h1:CJlz5H+gyd6CUWT45Oy4q24RdLyn7Md9Vj2/ldJBSIo=
|
||||
github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA=
|
||||
github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
|
||||
github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
|
||||
github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
|
||||
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
|
||||
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
|
||||
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
|
||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
|
||||
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
|
||||
golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
|
||||
golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
|
||||
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
|
||||
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
||||
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
|
||||
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/text v0.3.8 h1:nAL+RVCQ9uMn3vJZbV+MRnydTJFPf8qqY42YiA6MrqY=
|
||||
golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
|
||||
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||
modernc.org/cc/v4 v4.29.1 h1:MKgdCV3WykTSPqpVrnxdEDS0HEd2FHpKZDzxzU5LyeI=
|
||||
|
||||
+295
-112
@@ -18,23 +18,18 @@ const (
|
||||
RulesFile = "rules.toml"
|
||||
// AccountFile is the per-account config inside each account folder.
|
||||
AccountFile = "account.toml"
|
||||
// StateDir holds the rebuildable SQLite index.
|
||||
StateDir = ".money"
|
||||
// IndexFile is the SQLite index inside StateDir.
|
||||
// IndexFile is the rebuildable SQLite index, at the root of the data
|
||||
// directory alongside rules.toml.
|
||||
IndexFile = "index.db"
|
||||
)
|
||||
|
||||
// Rule is one entry in rules.toml. Rules are evaluated in file order and the
|
||||
// first one whose Match (and optional Account) matches wins.
|
||||
// Rule is one entry in rules.toml. The most specific rule that matches wins,
|
||||
// with file order breaking ties between equally specific ones; rules.Engine
|
||||
// owns that ordering.
|
||||
type Rule struct {
|
||||
Match string `toml:"match"`
|
||||
Tag string `toml:"tag"`
|
||||
Transfer bool `toml:"transfer"`
|
||||
Account string `toml:"account"` // optional: restrict to one account slug
|
||||
// Counterparty matches the other side's account number, which for
|
||||
// movements between the user's own accounts is often the only reliable
|
||||
// signal. Optional; when set, it must match as well as Match.
|
||||
Counterparty string `toml:"counterparty"`
|
||||
// Type matches the bank's own classification, e.g. Revolut's CARD_PAYMENT.
|
||||
// Optional; when set, it must match as well as Match.
|
||||
Type string `toml:"type"`
|
||||
@@ -45,9 +40,40 @@ type Rule struct {
|
||||
Note string `toml:"note"`
|
||||
}
|
||||
|
||||
// Rules is the parsed rules.toml.
|
||||
// Transfer is one entry in rules.toml describing money moved between two
|
||||
// accounts the user owns. Both sides are named: a transfer is only ever a pair,
|
||||
// which is what lets the report drop it without leaving half a movement behind.
|
||||
//
|
||||
// All four patterns are required. A one-sided definition would be a rule.
|
||||
type Transfer struct {
|
||||
FromAccount string `toml:"from_account"`
|
||||
FromDesc string `toml:"from_desc"` // glob vs. the leaving leg's description
|
||||
ToAccount string `toml:"to_account"`
|
||||
ToDesc string `toml:"to_desc"` // glob vs. the arriving leg's description
|
||||
// TolerancePct is how far short (or over) the arriving leg may be and still
|
||||
// count as the same movement, as a percentage of the leg that left. It exists
|
||||
// for routes where the bank takes a fee on the way, so the two statements
|
||||
// genuinely disagree about the amount.
|
||||
//
|
||||
// Zero — the default, and what every definition written before this key
|
||||
// existed means — requires the exact opposite amount. Keep it that way unless
|
||||
// a route actually charges: the wider the tolerance, the more likely two
|
||||
// unrelated movements in one window pair with each other.
|
||||
//
|
||||
// It applies within one currency only. Across currencies the amount is not
|
||||
// checked at all, so there is nothing for a tolerance to loosen.
|
||||
TolerancePct float64 `toml:"tolerance_pct"`
|
||||
// Note is free text for the reader, exactly as on a Rule: it never takes
|
||||
// part in matching.
|
||||
Note string `toml:"note"`
|
||||
}
|
||||
|
||||
// Rules is the parsed rules.toml: the tagging rules and the transfer
|
||||
// definitions, which share the file because they are both hand-maintained
|
||||
// statements about the same transactions.
|
||||
type Rules struct {
|
||||
Rule []Rule `toml:"rule"`
|
||||
Transfer []Transfer `toml:"transfer"`
|
||||
}
|
||||
|
||||
// LoadRules reads rules.toml from the data root. A missing file is not an
|
||||
@@ -62,30 +88,143 @@ func LoadRules(root string) (*Rules, error) {
|
||||
return nil, fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
for i, rule := range r.Rule {
|
||||
if rule.Match == "" && rule.Counterparty == "" && rule.Type == "" {
|
||||
return nil, fmt.Errorf("%s: rule %d has no match, counterparty or type pattern", path, i+1)
|
||||
if rule.Match == "" && rule.Type == "" {
|
||||
return nil, fmt.Errorf("%s: rule %d has no match or type pattern", path, i+1)
|
||||
}
|
||||
if rule.Tag == "" && !rule.Transfer {
|
||||
return nil, fmt.Errorf("%s: rule %d (%q) sets neither tag nor transfer", path, i+1, rule.Match)
|
||||
if rule.Tag == "" {
|
||||
return nil, fmt.Errorf("%s: rule %d (%q) sets no tag", path, i+1, rule.Match)
|
||||
}
|
||||
}
|
||||
for i, t := range r.Transfer {
|
||||
if err := checkTransfer(t); err != nil {
|
||||
return nil, fmt.Errorf("%s: transfer %d: %w", path, i+1, err)
|
||||
}
|
||||
}
|
||||
return &r, nil
|
||||
}
|
||||
|
||||
// checkTransfer rejects a half-written definition. Both sides are needed to
|
||||
// pair anything at all, so a missing one is refused on load rather than
|
||||
// silently matching nothing.
|
||||
func checkTransfer(t Transfer) error {
|
||||
switch {
|
||||
case t.FromAccount == "":
|
||||
return fmt.Errorf("no from_account")
|
||||
case t.FromDesc == "":
|
||||
return fmt.Errorf("no from_desc pattern")
|
||||
case t.ToAccount == "":
|
||||
return fmt.Errorf("no to_account")
|
||||
case t.ToDesc == "":
|
||||
return fmt.Errorf("no to_desc pattern")
|
||||
}
|
||||
// A negative tolerance is a typo, and one at 100% or beyond would let any
|
||||
// amount pair with any other — at which point the dates decide alone, which
|
||||
// is the cross-currency rule and not something to arrive at by accident.
|
||||
if t.TolerancePct < 0 || t.TolerancePct >= 100 {
|
||||
return fmt.Errorf("tolerance_pct %g is not between 0 and 100", t.TolerancePct)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// AppendRule adds a rule to the end of rules.toml, creating the file if it is
|
||||
// not there yet. Appending rather than inserting means an existing rule always
|
||||
// keeps precedence, since the first match wins.
|
||||
// not there yet. Position no longer decides precedence — the most specific rule
|
||||
// wins, see rules.Engine — but it still breaks ties, so appending rather than
|
||||
// inserting keeps a saved rule from displacing an equally specific one the user
|
||||
// wrote by hand.
|
||||
//
|
||||
// The file is rewritten through a temporary file so a failure part-way cannot
|
||||
// leave the user with a truncated config.
|
||||
func AppendRule(root string, r Rule) error {
|
||||
if r.Match == "" && r.Counterparty == "" && r.Type == "" {
|
||||
return fmt.Errorf("a rule needs a match, counterparty or type pattern")
|
||||
if err := checkRule(r); err != nil {
|
||||
return err
|
||||
}
|
||||
if r.Tag == "" && !r.Transfer {
|
||||
return fmt.Errorf("a rule needs a tag or transfer = true")
|
||||
return appendBlock(root, formatRule(r))
|
||||
}
|
||||
|
||||
// checkRule rejects a rule that would match nothing or decide nothing.
|
||||
func checkRule(r Rule) error {
|
||||
if r.Match == "" && r.Type == "" {
|
||||
return fmt.Errorf("a rule needs a match or type pattern")
|
||||
}
|
||||
if r.Tag == "" {
|
||||
return fmt.Errorf("a rule needs a tag")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ReplaceRule rewrites the rule at the given position (0-based, as loaded by
|
||||
// LoadRules) with r, leaving it where it sits. Position no longer decides
|
||||
// precedence, but it still breaks ties, so an edited rule that moved could
|
||||
// start losing to — or start beating — an equally specific one it never used to.
|
||||
//
|
||||
// Like DeleteRules this edits the file textually rather than re-serialising the
|
||||
// parsed rules, because comments and formatting are not recoverable from
|
||||
// []Rule. Only the rule's own lines are replaced: the comments directly above
|
||||
// it say why it is there, which editing its glob rarely changes, so they stay.
|
||||
func ReplaceRule(root string, pos int, r Rule) error {
|
||||
if err := checkRule(r); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
path := filepath.Join(root, RulesFile)
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read %s: %w", path, err)
|
||||
}
|
||||
lines := strings.Split(string(raw), "\n")
|
||||
|
||||
starts := blockStarts(lines)
|
||||
rules, transfers := blocksOf(starts, "rule"), blocksOf(starts, "transfer")
|
||||
if pos < 0 || pos >= len(rules) {
|
||||
return fmt.Errorf("rule %d is out of range; %s holds %d rules", pos+1, path, len(rules))
|
||||
}
|
||||
|
||||
from, to := blockExtent(lines, starts, rules[pos])
|
||||
kept := make([]string, 0, len(lines))
|
||||
kept = append(kept, lines[:from]...)
|
||||
kept = append(kept, strings.Split(strings.TrimRight(formatRule(r), "\n"), "\n")...)
|
||||
kept = append(kept, lines[to:]...)
|
||||
out := strings.Join(kept, "\n")
|
||||
|
||||
// Never write something that will not load again, and never let editing one
|
||||
// rule disturb a neighbour — of either kind, since both live in this file.
|
||||
var check Rules
|
||||
if _, err := toml.Decode(out, &check); err != nil {
|
||||
return fmt.Errorf("editing %s would produce invalid TOML: %w", path, err)
|
||||
}
|
||||
if len(check.Rule) != len(rules) || len(check.Transfer) != len(transfers) {
|
||||
return fmt.Errorf("editing %s would leave %d rules and %d transfers, expected %d and %d",
|
||||
path, len(check.Rule), len(check.Transfer), len(rules), len(transfers))
|
||||
}
|
||||
if check.Rule[pos] != r {
|
||||
return fmt.Errorf("editing %s would leave rule %d as %+v, expected %+v",
|
||||
path, pos+1, check.Rule[pos], r)
|
||||
}
|
||||
|
||||
return writeFileAtomic(root, path, out)
|
||||
}
|
||||
|
||||
// AppendTransfer adds a transfer definition to the end of rules.toml. Order
|
||||
// matters for transfers as it does for rules: an earlier definition claims a
|
||||
// transaction first, so appending cannot steal a leg from one already written.
|
||||
func AppendTransfer(root string, t Transfer) error {
|
||||
if err := checkTransfer(t); err != nil {
|
||||
// A missing side reads better as what the form still wants; anything
|
||||
// else already says what is wrong with what was typed.
|
||||
if rest, ok := strings.CutPrefix(err.Error(), "no "); ok {
|
||||
return fmt.Errorf("a transfer needs %s", rest)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return appendBlock(root, formatTransfer(t))
|
||||
}
|
||||
|
||||
// appendBlock adds a rendered TOML table to the end of rules.toml, creating the
|
||||
// file if it is not there yet.
|
||||
//
|
||||
// The file is rewritten through a temporary file so a failure part-way cannot
|
||||
// leave the user with a truncated config.
|
||||
func appendBlock(root, block string) error {
|
||||
path := filepath.Join(root, RulesFile)
|
||||
existing, err := os.ReadFile(path)
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
@@ -98,20 +237,91 @@ func AppendRule(root string, r Rule) error {
|
||||
b.WriteString("\n")
|
||||
}
|
||||
b.WriteString("\n")
|
||||
b.WriteString(formatRule(r))
|
||||
b.WriteString(block)
|
||||
|
||||
return writeFileAtomic(root, path, b.String())
|
||||
}
|
||||
|
||||
// DeleteRules removes the rules at the given positions (0-based, as loaded by
|
||||
// LoadRules) from rules.toml.
|
||||
func DeleteRules(root string, positions []int) (int, error) {
|
||||
return deleteBlocks(root, "rule", positions)
|
||||
}
|
||||
|
||||
// DeleteTransfers removes the transfer definitions at the given positions
|
||||
// (0-based, as loaded by LoadRules) from rules.toml.
|
||||
func DeleteTransfers(root string, positions []int) (int, error) {
|
||||
return deleteBlocks(root, "transfer", positions)
|
||||
}
|
||||
|
||||
// blockStart is where one array-of-tables entry begins in rules.toml.
|
||||
type blockStart struct {
|
||||
table string // "rule" or "transfer"
|
||||
line int
|
||||
}
|
||||
|
||||
// blockStarts finds every [[table]] header in the file. Every kind is
|
||||
// collected, not just the one being deleted: a block ends where the *next*
|
||||
// block of any kind begins, so deleting a rule that happens to sit above a
|
||||
// transfer must not swallow it.
|
||||
func blockStarts(lines []string) []blockStart {
|
||||
var out []blockStart
|
||||
for i, line := range lines {
|
||||
s := strings.TrimSpace(line)
|
||||
if strings.HasPrefix(s, "[[") && strings.HasSuffix(s, "]]") {
|
||||
out = append(out, blockStart{table: strings.TrimSpace(s[2 : len(s)-2]), line: i})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// blocksOf lists where the entries of one table sit among all the blocks, so a
|
||||
// position in rules.toml as LoadRules numbers it can be turned into a position
|
||||
// in the file.
|
||||
func blocksOf(starts []blockStart, table string) []int {
|
||||
var out []int
|
||||
for k, s := range starts {
|
||||
if s.table == table {
|
||||
out = append(out, k)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// commentPrefix walks back from a block's header over the comment lines it
|
||||
// owns. An entry owns the run of comments directly above it, with no blank line
|
||||
// in between; anything further up is a heading for what follows.
|
||||
func commentPrefix(lines []string, header int) int {
|
||||
i := header
|
||||
for i > 0 && strings.HasPrefix(strings.TrimSpace(lines[i-1]), "#") {
|
||||
i--
|
||||
}
|
||||
return i
|
||||
}
|
||||
|
||||
// blockExtent returns the k-th block's own lines as a half-open range: from its
|
||||
// header to the last line it owns, leaving out the comments introducing the
|
||||
// next block and the blank lines between the two, which belong to neither.
|
||||
func blockExtent(lines []string, starts []blockStart, k int) (from, to int) {
|
||||
from = starts[k].line
|
||||
to = len(lines)
|
||||
if k+1 < len(starts) {
|
||||
to = commentPrefix(lines, starts[k+1].line)
|
||||
}
|
||||
for to > from+1 && strings.TrimSpace(lines[to-1]) == "" {
|
||||
to--
|
||||
}
|
||||
return from, to
|
||||
}
|
||||
|
||||
// deleteBlocks removes entries of one table from rules.toml.
|
||||
//
|
||||
// The file is edited textually rather than re-serialised from the parsed
|
||||
// rules, so comments, ordering and formatting the user put there by hand
|
||||
// survive. A comment block sitting directly above a deleted rule goes with it,
|
||||
// since it documents that rule; a comment separated by a blank line is treated
|
||||
// values, so comments, ordering and formatting the user put there by hand
|
||||
// survive. A comment block sitting directly above a deleted entry goes with it,
|
||||
// since it documents that entry; a comment separated by a blank line is treated
|
||||
// as a section heading and left alone.
|
||||
func DeleteRules(root string, positions []int) (int, error) {
|
||||
func deleteBlocks(root, table string, positions []int) (int, error) {
|
||||
if len(positions) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
@@ -127,48 +337,26 @@ func DeleteRules(root string, positions []int) (int, error) {
|
||||
}
|
||||
lines := strings.Split(string(raw), "\n")
|
||||
|
||||
// Where each [[rule]] block begins.
|
||||
var starts []int
|
||||
for i, line := range lines {
|
||||
if strings.TrimSpace(line) == "[[rule]]" {
|
||||
starts = append(starts, i)
|
||||
}
|
||||
}
|
||||
starts := blockStarts(lines)
|
||||
// mine[p] is where the p-th entry of this table sits among all the blocks.
|
||||
mine := blocksOf(starts, table)
|
||||
for _, p := range positions {
|
||||
if p < 0 || p >= len(starts) {
|
||||
return 0, fmt.Errorf("rule %d is out of range; %s holds %d rules", p+1, path, len(starts))
|
||||
if p < 0 || p >= len(mine) {
|
||||
return 0, fmt.Errorf("%s %d is out of range; %s holds %d %ss",
|
||||
table, p+1, path, len(mine), table)
|
||||
}
|
||||
}
|
||||
|
||||
// A rule owns the run of comment lines directly above it, with no blank
|
||||
// line in between. Anything further up is a heading for what follows.
|
||||
prefix := func(k int) int {
|
||||
i := starts[k]
|
||||
for i > 0 && strings.HasPrefix(strings.TrimSpace(lines[i-1]), "#") {
|
||||
i--
|
||||
}
|
||||
return i
|
||||
}
|
||||
|
||||
drop := map[int]bool{}
|
||||
for k := range starts {
|
||||
if !doomed[k] {
|
||||
continue
|
||||
}
|
||||
// The block runs up to the next rule's comment prefix, so a comment
|
||||
// introducing the following rule is not swept up with this one.
|
||||
end := len(lines)
|
||||
if k+1 < len(starts) {
|
||||
end = prefix(k + 1)
|
||||
}
|
||||
for i := prefix(k); i < end; i++ {
|
||||
for p := range doomed {
|
||||
k := mine[p]
|
||||
// A rule's own comments document it and go with it; blockExtent leaves
|
||||
// out the blank lines below it, which are the gap between blocks and
|
||||
// part of neither, so the neighbours are not glued together.
|
||||
_, end := blockExtent(lines, starts, k)
|
||||
for i := commentPrefix(lines, starts[k].line); i < end; i++ {
|
||||
drop[i] = true
|
||||
}
|
||||
// Blank lines are the gap between rules, not part of either; leaving
|
||||
// them avoids gluing the neighbours together.
|
||||
for i := end - 1; i >= starts[k] && strings.TrimSpace(lines[i]) == ""; i-- {
|
||||
delete(drop, i)
|
||||
}
|
||||
}
|
||||
|
||||
kept := make([]string, 0, len(lines))
|
||||
@@ -179,14 +367,27 @@ func DeleteRules(root string, positions []int) (int, error) {
|
||||
}
|
||||
out := collapseBlankRuns(kept)
|
||||
|
||||
// Never write something that will not load again.
|
||||
// Never write something that will not load again, and never let deleting
|
||||
// one kind of block take a different kind with it.
|
||||
var check Rules
|
||||
if _, err := toml.Decode(out, &check); err != nil {
|
||||
return 0, fmt.Errorf("deleting from %s would produce invalid TOML: %w", path, err)
|
||||
}
|
||||
if want := len(starts) - len(doomed); len(check.Rule) != want {
|
||||
return 0, fmt.Errorf("deleting from %s would leave %d rules, expected %d",
|
||||
path, len(check.Rule), want)
|
||||
counts := map[string]int{"rule": len(check.Rule), "transfer": len(check.Transfer)}
|
||||
for _, kind := range []string{"rule", "transfer"} {
|
||||
want := 0
|
||||
for _, s := range starts {
|
||||
if s.table == kind {
|
||||
want++
|
||||
}
|
||||
}
|
||||
if kind == table {
|
||||
want -= len(doomed)
|
||||
}
|
||||
if counts[kind] != want {
|
||||
return 0, fmt.Errorf("deleting from %s would leave %d %ss, expected %d",
|
||||
path, counts[kind], kind, want)
|
||||
}
|
||||
}
|
||||
|
||||
if err := writeFileAtomic(root, path, out); err != nil {
|
||||
@@ -244,64 +445,48 @@ func writeFileAtomic(dir, path, content string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeKey renders one TOML key, skipping it when empty. The column is wide
|
||||
// enough for the longest key either block uses, so the values line up.
|
||||
func writeKey(b *strings.Builder, key, value string) {
|
||||
if value != "" {
|
||||
fmt.Fprintf(b, "%-12s = %s\n", key, strconv.Quote(value))
|
||||
}
|
||||
}
|
||||
|
||||
// formatTransfer renders a transfer as a TOML table, the two sides in the
|
||||
// order money travels.
|
||||
func formatTransfer(t Transfer) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("[[transfer]]\n")
|
||||
writeKey(&b, "from_account", t.FromAccount)
|
||||
writeKey(&b, "from_desc", t.FromDesc)
|
||||
writeKey(&b, "to_account", t.ToAccount)
|
||||
writeKey(&b, "to_desc", t.ToDesc)
|
||||
// Only when set: a zero written out would suggest the key is doing something
|
||||
// when it is exactly the default every other definition already has.
|
||||
if t.TolerancePct != 0 {
|
||||
fmt.Fprintf(&b, "%-12s = %s\n", "tolerance_pct",
|
||||
strconv.FormatFloat(t.TolerancePct, 'f', -1, 64))
|
||||
}
|
||||
writeKey(&b, "note", t.Note)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// formatRule renders a rule as a TOML table, omitting empty fields.
|
||||
func formatRule(r Rule) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("[[rule]]\n")
|
||||
write := func(key, value string) {
|
||||
if value != "" {
|
||||
fmt.Fprintf(&b, "%-12s = %s\n", key, strconv.Quote(value))
|
||||
}
|
||||
}
|
||||
write := func(key, value string) { writeKey(&b, key, value) }
|
||||
write("match", r.Match)
|
||||
write("counterparty", r.Counterparty)
|
||||
write("type", r.Type)
|
||||
write("account", r.Account)
|
||||
write("tag", r.Tag)
|
||||
if r.Transfer {
|
||||
b.WriteString("transfer = true\n")
|
||||
}
|
||||
// Last, so the patterns and the tag stay lined up above it however long
|
||||
// the note runs.
|
||||
write("note", r.Note)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Column locates one field in a CSV row.
|
||||
type Column struct {
|
||||
Col int `toml:"col"`
|
||||
Layout string `toml:"layout"` // date only, Go reference layout
|
||||
Decimal string `toml:"decimal"` // amount only, default "."
|
||||
Thousands string `toml:"thousands"` // amount only, default ""
|
||||
}
|
||||
|
||||
// CSVConfig describes how to read a delimited statement.
|
||||
type CSVConfig struct {
|
||||
Delimiter string `toml:"delimiter"`
|
||||
SkipRows int `toml:"skip_rows"`
|
||||
Encoding string `toml:"encoding"` // "" or "utf-8"; other encodings unsupported for now
|
||||
Date Column `toml:"date"`
|
||||
Description Column `toml:"description"`
|
||||
Amount *Column `toml:"amount"` // single signed column...
|
||||
Debit *Column `toml:"debit"` // ...or a debit/credit pair
|
||||
Credit *Column `toml:"credit"`
|
||||
// Invert flips the sign of the parsed amount, for statements that report
|
||||
// outflows as positive numbers.
|
||||
Invert bool `toml:"invert"`
|
||||
}
|
||||
|
||||
// CmdConfig runs an external extractor (e.g. one of the existing Python
|
||||
// scripts) and reads normalised CSV from its stdout.
|
||||
type CmdConfig struct {
|
||||
// Argv is the command to run. The literal token "{{file}}" is replaced
|
||||
// with the absolute path of the statement being imported.
|
||||
Argv []string `toml:"argv"`
|
||||
// Layout is the date layout the script emits; defaults to 2006-01-02.
|
||||
Layout string `toml:"layout"`
|
||||
// SkipRows skips leading rows of the script's output (e.g. a header).
|
||||
SkipRows int `toml:"skip_rows"`
|
||||
}
|
||||
|
||||
// Account is a parsed account.toml.
|
||||
type Account struct {
|
||||
Slug string // folder name, filled in by LoadAccounts
|
||||
@@ -313,8 +498,6 @@ type Account struct {
|
||||
// Include restricts which files in the folder are treated as statements.
|
||||
// Defaults to every regular file except account.toml and dotfiles.
|
||||
Include []string `toml:"include"`
|
||||
CSV *CSVConfig `toml:"csv"`
|
||||
Cmd *CmdConfig `toml:"cmd"`
|
||||
}
|
||||
|
||||
// Digits returns the configured minor-unit scale, defaulting to 2.
|
||||
@@ -376,7 +559,7 @@ func loadAccount(dir, slug, cfgPath string) (*Account, error) {
|
||||
|
||||
// IndexPath returns the location of the SQLite index for a data root.
|
||||
func IndexPath(root string) string {
|
||||
return filepath.Join(root, StateDir, IndexFile)
|
||||
return filepath.Join(root, IndexFile)
|
||||
}
|
||||
|
||||
// UserConfig is the small file in the user's config directory that says where
|
||||
|
||||
@@ -181,7 +181,7 @@ func TestAppendRuleRejectsIncomplete(t *testing.T) {
|
||||
// Globs and tags containing quotes must survive the round trip.
|
||||
func TestAppendRuleQuotesValues(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := AppendRule(root, Rule{Match: `*"ODD" PAYEE*`, Tag: "misc", Transfer: true}); err != nil {
|
||||
if err := AppendRule(root, Rule{Match: `*"ODD" PAYEE*`, Tag: "misc"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loaded, err := LoadRules(root)
|
||||
@@ -191,9 +191,6 @@ func TestAppendRuleQuotesValues(t *testing.T) {
|
||||
if loaded.Rule[0].Match != `*"ODD" PAYEE*` {
|
||||
t.Errorf("match = %q, want the quotes preserved", loaded.Rule[0].Match)
|
||||
}
|
||||
if !loaded.Rule[0].Transfer {
|
||||
t.Error("transfer flag was not written")
|
||||
}
|
||||
}
|
||||
|
||||
// A note is documentation carried with the rule. It is a key rather than a
|
||||
@@ -268,7 +265,6 @@ tag = "salary"
|
||||
# Moving money to myself.
|
||||
[[rule]]
|
||||
match = "*TO SAVINGS*"
|
||||
transfer = true
|
||||
tag = "transfer"
|
||||
`
|
||||
|
||||
@@ -459,3 +455,358 @@ func TestLoadUserConfigMalformed(t *testing.T) {
|
||||
t.Error("expected an error for a malformed config file")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAppendTransfer(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
existing := "[[rule]]\nmatch = \"*LIDL*\"\ntag = \"groceries\"\n"
|
||||
if err := os.WriteFile(filepath.Join(root, RulesFile), []byte(existing), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
tr := Transfer{
|
||||
FromAccount: "nlb", FromDesc: "*TO REVOLUT*",
|
||||
ToAccount: "revolut", ToDesc: "*FROM NLB*",
|
||||
Note: `the monthly "top-up"`,
|
||||
}
|
||||
if err := AppendTransfer(root, tr); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
loaded, err := LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatalf("the file no longer parses after appending: %v", err)
|
||||
}
|
||||
if len(loaded.Rule) != 1 {
|
||||
t.Errorf("got %d rules, want the existing one untouched", len(loaded.Rule))
|
||||
}
|
||||
if len(loaded.Transfer) != 1 || loaded.Transfer[0] != tr {
|
||||
t.Errorf("transfers = %+v, want %+v", loaded.Transfer, tr)
|
||||
}
|
||||
}
|
||||
|
||||
// Both sides are required: a one-sided definition can never pair anything, so
|
||||
// it is refused rather than written and silently ignored.
|
||||
func TestTransferNeedsBothSides(t *testing.T) {
|
||||
full := Transfer{FromAccount: "a", FromDesc: "*OUT*", ToAccount: "b", ToDesc: "*IN*"}
|
||||
cases := map[string]func(*Transfer){
|
||||
"no from_account": func(t *Transfer) { t.FromAccount = "" },
|
||||
"no from_desc": func(t *Transfer) { t.FromDesc = "" },
|
||||
"no to_account": func(t *Transfer) { t.ToAccount = "" },
|
||||
"no to_desc": func(t *Transfer) { t.ToDesc = "" },
|
||||
}
|
||||
for name, break_ := range cases {
|
||||
root := t.TempDir()
|
||||
tr := full
|
||||
break_(&tr)
|
||||
if err := AppendTransfer(root, tr); err == nil {
|
||||
t.Errorf("%s: expected an error", name)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, RulesFile)); !os.IsNotExist(err) {
|
||||
t.Errorf("%s: a rejected transfer must not create the file", name)
|
||||
}
|
||||
}
|
||||
|
||||
// The same check applies to a file written by hand.
|
||||
root := t.TempDir()
|
||||
body := "[[transfer]]\nfrom_account = \"nlb\"\nfrom_desc = \"*OUT*\"\nto_account = \"revolut\"\n"
|
||||
if err := os.WriteFile(filepath.Join(root, RulesFile), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := LoadRules(root); err == nil {
|
||||
t.Error("expected a half-written transfer to be refused on load")
|
||||
}
|
||||
}
|
||||
|
||||
// Rules and transfers share a file, so deleting one kind must not take a
|
||||
// neighbouring block of the other kind with it.
|
||||
func TestDeleteLeavesTheOtherKindAlone(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
path := filepath.Join(root, RulesFile)
|
||||
original := `[[rule]]
|
||||
match = "*LIDL*"
|
||||
tag = "groceries"
|
||||
|
||||
# Moving money to the broker.
|
||||
[[transfer]]
|
||||
from_account = "nlb"
|
||||
from_desc = "*TO TRADEREPUBLIC*"
|
||||
to_account = "traderepublic"
|
||||
to_desc = "*FROM NLB*"
|
||||
|
||||
[[rule]]
|
||||
match = "*ZARA*"
|
||||
tag = "clothes"
|
||||
`
|
||||
if err := os.WriteFile(path, []byte(original), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Deleting the first rule must stop at the transfer that follows it.
|
||||
if _, err := DeleteRules(root, []int{0}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loaded, err := LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(loaded.Rule) != 1 || loaded.Rule[0].Tag != "clothes" {
|
||||
t.Errorf("rules = %+v, want only the clothes rule left", loaded.Rule)
|
||||
}
|
||||
if len(loaded.Transfer) != 1 {
|
||||
t.Fatalf("transfers = %+v, want the transfer untouched", loaded.Transfer)
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(raw), "# Moving money to the broker.") {
|
||||
t.Errorf("rules.toml = %q, want the transfer's own comment kept", raw)
|
||||
}
|
||||
|
||||
// And deleting the transfer leaves the remaining rule alone.
|
||||
if _, err := DeleteTransfers(root, []int{0}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loaded, err = LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(loaded.Transfer) != 0 || len(loaded.Rule) != 1 {
|
||||
t.Errorf("after deleting the transfer: %d rules, %d transfers; want 1 and 0",
|
||||
len(loaded.Rule), len(loaded.Transfer))
|
||||
}
|
||||
if raw, err = os.ReadFile(path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(string(raw), "Moving money to the broker") {
|
||||
t.Errorf("rules.toml = %q, want the deleted transfer's comment gone with it", raw)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteTransfersOutOfRange(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := AppendTransfer(root, Transfer{
|
||||
FromAccount: "a", FromDesc: "*OUT*", ToAccount: "b", ToDesc: "*IN*",
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := DeleteTransfers(root, []int{3}); err == nil {
|
||||
t.Error("expected an out-of-range position to be refused")
|
||||
}
|
||||
}
|
||||
|
||||
// A tolerance survives the round trip through the file, and a definition
|
||||
// without one keeps meaning what it always meant: exact amounts.
|
||||
func TestAppendTransferWithTolerance(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
tr := Transfer{
|
||||
FromAccount: "nlb", FromDesc: "*TO REVOLUT*",
|
||||
ToAccount: "revolut", ToDesc: "*FROM NLB*",
|
||||
TolerancePct: 1.5,
|
||||
Note: "NLB takes a wire fee",
|
||||
}
|
||||
if err := AppendTransfer(root, tr); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
body, err := os.ReadFile(filepath.Join(root, RulesFile))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(body), "tolerance_pct = 1.5") {
|
||||
t.Errorf("file = %q, want the tolerance written unquoted", body)
|
||||
}
|
||||
|
||||
loaded, err := LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(loaded.Transfer) != 1 || loaded.Transfer[0] != tr {
|
||||
t.Errorf("transfers = %+v, want %+v", loaded.Transfer, tr)
|
||||
}
|
||||
}
|
||||
|
||||
// The default is exact, so an unset tolerance must not be written out as a key
|
||||
// suggesting the definition decided something.
|
||||
func TestAppendTransferOmitsAZeroTolerance(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
tr := Transfer{
|
||||
FromAccount: "nlb", FromDesc: "*TO REVOLUT*",
|
||||
ToAccount: "revolut", ToDesc: "*FROM NLB*",
|
||||
}
|
||||
if err := AppendTransfer(root, tr); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body, err := os.ReadFile(filepath.Join(root, RulesFile))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(string(body), "tolerance_pct") {
|
||||
t.Errorf("file = %q, want no tolerance key", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A negative tolerance is a typo, and one at 100% or beyond would let any
|
||||
// amount pair with any other, leaving the dates to decide alone.
|
||||
func TestTransferToleranceIsBounded(t *testing.T) {
|
||||
full := Transfer{FromAccount: "a", FromDesc: "*OUT*", ToAccount: "b", ToDesc: "*IN*"}
|
||||
for _, pct := range []float64{-1, 100, 250} {
|
||||
root := t.TempDir()
|
||||
tr := full
|
||||
tr.TolerancePct = pct
|
||||
if err := AppendTransfer(root, tr); err == nil {
|
||||
t.Errorf("tolerance_pct %g: expected an error", pct)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(root, RulesFile)); !os.IsNotExist(err) {
|
||||
t.Errorf("tolerance_pct %g: a rejected transfer must not create the file", pct)
|
||||
}
|
||||
}
|
||||
|
||||
// The same check applies to a file written by hand.
|
||||
root := t.TempDir()
|
||||
body := "[[transfer]]\nfrom_account = \"a\"\nfrom_desc = \"*OUT*\"\n" +
|
||||
"to_account = \"b\"\nto_desc = \"*IN*\"\ntolerance_pct = 150\n"
|
||||
if err := os.WriteFile(filepath.Join(root, RulesFile), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := LoadRules(root); err == nil {
|
||||
t.Error("expected an out-of-range tolerance to be refused on load")
|
||||
}
|
||||
}
|
||||
|
||||
// Editing rewrites the rule where it sits: the position is what still breaks
|
||||
// ties between equally specific rules, so a rule that moved could start beating
|
||||
// one it never used to.
|
||||
func TestReplaceRuleKeepsPositionAndComments(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
path := filepath.Join(root, RulesFile)
|
||||
if err := os.WriteFile(path, []byte(rulesWithComments), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
edited := Rule{Match: "*LIDL SOFIA*", Tag: "groceries", Account: "checking", Note: "just the branch"}
|
||||
if err := ReplaceRule(root, 0, edited); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
loaded, err := LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatalf("the file no longer parses after editing: %v", err)
|
||||
}
|
||||
if len(loaded.Rule) != 3 {
|
||||
t.Fatalf("got %d rules, want the same 3: %+v", len(loaded.Rule), loaded.Rule)
|
||||
}
|
||||
if loaded.Rule[0] != edited {
|
||||
t.Errorf("rule 1 = %+v, want %+v", loaded.Rule[0], edited)
|
||||
}
|
||||
if loaded.Rule[1].Match != "*PAYROLL*" || loaded.Rule[2].Match != "*TO SAVINGS*" {
|
||||
t.Errorf("the rest = %+v, want them where they were", loaded.Rule[1:])
|
||||
}
|
||||
|
||||
// The comment above a rule says why it is there, which editing its glob
|
||||
// does not change.
|
||||
text, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{"# Rules for my accounts.", "# Weekly shop.", "# Moving money to myself."} {
|
||||
if !strings.Contains(string(text), want) {
|
||||
t.Errorf("comment %q was lost:\n%s", want, text)
|
||||
}
|
||||
}
|
||||
if strings.Contains(string(text), `"*LIDL*"`) {
|
||||
t.Errorf("the old pattern is still there:\n%s", text)
|
||||
}
|
||||
}
|
||||
|
||||
// A rule and the transfer below it share the file, so a replacement has to end
|
||||
// where the next block of any kind begins.
|
||||
func TestReplaceRuleLeavesTheOtherKindAlone(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
path := filepath.Join(root, RulesFile)
|
||||
original := `[[rule]]
|
||||
match = "*LIDL*"
|
||||
tag = "groceries"
|
||||
|
||||
# Moving money to the broker.
|
||||
[[transfer]]
|
||||
from_account = "nlb"
|
||||
from_desc = "*TO TRADEREPUBLIC*"
|
||||
to_account = "traderepublic"
|
||||
to_desc = "*FROM NLB*"
|
||||
|
||||
[[rule]]
|
||||
match = "*ZARA*"
|
||||
tag = "clothes"
|
||||
`
|
||||
if err := os.WriteFile(path, []byte(original), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := ReplaceRule(root, 0, Rule{Match: "*KAUFLAND*", Tag: "groceries"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loaded, err := LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(loaded.Transfer) != 1 || loaded.Transfer[0].FromDesc != "*TO TRADEREPUBLIC*" {
|
||||
t.Errorf("transfers = %+v, want the definition untouched", loaded.Transfer)
|
||||
}
|
||||
if len(loaded.Rule) != 2 || loaded.Rule[0].Match != "*KAUFLAND*" || loaded.Rule[1].Match != "*ZARA*" {
|
||||
t.Errorf("rules = %+v, want only the first one rewritten", loaded.Rule)
|
||||
}
|
||||
raw, _ := os.ReadFile(path)
|
||||
if !strings.Contains(string(raw), "# Moving money to the broker.") {
|
||||
t.Errorf("the transfer's comment was lost:\n%s", raw)
|
||||
}
|
||||
}
|
||||
|
||||
// Editing a rule that sets a type keeps it: the caller decides what the rule
|
||||
// becomes, and a pattern left out of the replacement is a pattern removed.
|
||||
func TestReplaceRuleWritesEveryPattern(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
body := "[[rule]]\nmatch = \"*SPOTIFY*\"\ntype = \"CARD_PAYMENT\"\ntag = \"music\"\n"
|
||||
if err := os.WriteFile(filepath.Join(root, RulesFile), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
kept := Rule{Match: "*SPOTIFY*", Type: "CARD_PAYMENT", Tag: "subscriptions"}
|
||||
if err := ReplaceRule(root, 0, kept); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loaded, err := LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if loaded.Rule[0] != kept {
|
||||
t.Errorf("rule = %+v, want %+v", loaded.Rule[0], kept)
|
||||
}
|
||||
|
||||
// A type-only rule is a rule; a rule with neither pattern is not.
|
||||
if err := ReplaceRule(root, 0, Rule{Type: "CARD_PAYMENT", Tag: "cards"}); err != nil {
|
||||
t.Errorf("a type-only rule was refused: %v", err)
|
||||
}
|
||||
if err := ReplaceRule(root, 0, Rule{Tag: "nothing"}); err == nil {
|
||||
t.Error("expected a rule with no pattern to be rejected")
|
||||
}
|
||||
if err := ReplaceRule(root, 0, Rule{Match: "*X*"}); err == nil {
|
||||
t.Error("expected a rule with no tag to be rejected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReplaceRuleOutOfRange(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := AppendRule(root, Rule{Match: "*LIDL*", Tag: "groceries"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, pos := range []int{-1, 1, 7} {
|
||||
if err := ReplaceRule(root, pos, Rule{Match: "*X*", Tag: "x"}); err == nil {
|
||||
t.Errorf("position %d: expected an error", pos)
|
||||
}
|
||||
}
|
||||
loaded, _ := LoadRules(root)
|
||||
if len(loaded.Rule) != 1 || loaded.Rule[0].Match != "*LIDL*" {
|
||||
t.Errorf("rules = %+v, want the file untouched", loaded.Rule)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"git.petrovv.com/nikola/money/internal/parser"
|
||||
"git.petrovv.com/nikola/money/internal/rules"
|
||||
"git.petrovv.com/nikola/money/internal/store"
|
||||
"git.petrovv.com/nikola/money/internal/transfers"
|
||||
)
|
||||
|
||||
// FileResult is what happened to one statement file.
|
||||
@@ -38,6 +39,11 @@ type FileResult struct {
|
||||
type Result struct {
|
||||
Files []FileResult
|
||||
Retagged int
|
||||
// Paired and Unpaired are the state of the transfer pairing after the
|
||||
// import: how many movements were matched, and how many legs a definition
|
||||
// caught without finding the other side.
|
||||
Paired int
|
||||
Unpaired int
|
||||
}
|
||||
|
||||
// Total counts new rows across every file.
|
||||
@@ -69,8 +75,12 @@ type Options struct {
|
||||
Force bool
|
||||
}
|
||||
|
||||
// Run imports every account under root into db and then applies the rules.
|
||||
func Run(root string, db *store.DB, accounts []*config.Account, engine *rules.Engine, opts Options) (Result, error) {
|
||||
// Run imports every account under root into db and then applies the rules and
|
||||
// the transfer definitions. Both are derived from rules.toml and both are
|
||||
// recomputed over the whole index, since a statement imported now can complete
|
||||
// a transfer whose other leg arrived months ago.
|
||||
func Run(root string, db *store.DB, accounts []*config.Account, engine *rules.Engine,
|
||||
links *transfers.Engine, opts Options) (Result, error) {
|
||||
var res Result
|
||||
for _, acc := range accounts {
|
||||
accountID, err := db.UpsertAccount(model.Account{
|
||||
@@ -91,7 +101,7 @@ func Run(root string, db *store.DB, accounts []*config.Account, engine *rules.En
|
||||
continue
|
||||
}
|
||||
|
||||
files, err := statementFiles(acc)
|
||||
files, err := StatementFiles(acc)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
@@ -106,6 +116,12 @@ func Run(root string, db *store.DB, accounts []*config.Account, engine *rules.En
|
||||
return res, err
|
||||
}
|
||||
res.Retagged = n
|
||||
|
||||
paired, unpaired, err := links.Link(db)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
res.Paired, res.Unpaired = paired, unpaired
|
||||
return res, nil
|
||||
}
|
||||
|
||||
@@ -118,7 +134,7 @@ func importFile(root string, db *store.DB, acc *config.Account, accountID int64,
|
||||
}
|
||||
fr := FileResult{Account: acc.Slug, Path: rel}
|
||||
|
||||
sum, err := checksum(path)
|
||||
sum, err := Checksum(path)
|
||||
if err != nil {
|
||||
fr.Err = err
|
||||
return fr
|
||||
@@ -146,7 +162,7 @@ func importFile(root string, db *store.DB, acc *config.Account, accountID int64,
|
||||
}
|
||||
fr.Warnings = append(fr.Warnings, checkBalances(txns, acc.Digits())...)
|
||||
|
||||
sourceID, err := db.SourceFile(accountID, rel, sum, time.Now().UTC().Format(time.RFC3339))
|
||||
sourceID, err := db.SourceFile(accountID, rel, sum, time.Now().UTC().Format(time.RFC3339), len(txns))
|
||||
if err != nil {
|
||||
fr.Err = err
|
||||
return fr
|
||||
@@ -169,11 +185,10 @@ func importFile(root string, db *store.DB, acc *config.Account, accountID int64,
|
||||
Date: t.Date,
|
||||
Description: t.Description,
|
||||
AmountMinor: t.AmountMinor,
|
||||
Counterparty: t.Counterparty,
|
||||
Type: t.Type,
|
||||
BalanceMinor: t.BalanceMinor,
|
||||
}
|
||||
txn.RuleTag, txn.RuleTransfer = engine.ApplyTxn(acc.Slug, txn)
|
||||
txn.RuleTag = engine.ApplyTxn(acc.Slug, txn)
|
||||
added, err := db.InsertTransaction(txn)
|
||||
if err != nil {
|
||||
fr.Err = err
|
||||
@@ -237,10 +252,11 @@ func fingerprint(key string, ordinal int) string {
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// statementFiles lists the files in an account folder that should be parsed:
|
||||
// StatementFiles lists the files in an account folder that should be parsed:
|
||||
// every regular file except account.toml and dotfiles, narrowed by the
|
||||
// account's optional include globs.
|
||||
func statementFiles(acc *config.Account) ([]string, error) {
|
||||
// account's optional include globs. It is also what the web app lists and
|
||||
// serves, so a file is shown exactly when import would read it.
|
||||
func StatementFiles(acc *config.Account) ([]string, error) {
|
||||
entries, err := os.ReadDir(acc.Dir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read account dir %s: %w", acc.Dir, err)
|
||||
@@ -269,7 +285,9 @@ func matchAny(patterns []string, name string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func checksum(path string) (string, error) {
|
||||
// Checksum is the sha256 a statement is recorded under; an import skips a file
|
||||
// whose checksum has not changed.
|
||||
func Checksum(path string) (string, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
|
||||
@@ -1,28 +1,66 @@
|
||||
package importer
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
"git.petrovv.com/nikola/money/internal/parser"
|
||||
"git.petrovv.com/nikola/money/internal/rules"
|
||||
"git.petrovv.com/nikola/money/internal/store"
|
||||
"git.petrovv.com/nikola/money/internal/transfers"
|
||||
)
|
||||
|
||||
const accountTOML = `
|
||||
name = "Checking"
|
||||
currency = "EUR"
|
||||
parser = "csv"
|
||||
[csv]
|
||||
skip_rows = 1
|
||||
date = { col = 0, layout = "2006-01-02" }
|
||||
description = { col = 1 }
|
||||
amount = { col = 2 }
|
||||
parser = "test"
|
||||
`
|
||||
|
||||
// These tests are about the directory walk, the checksum skip, dedupe and
|
||||
// per-file error reporting -- not about any bank's layout. Driving them with a
|
||||
// real bank parser would drag that bank's quirks (Revolut's fee folding and
|
||||
// COMPLETED filter, the PDF parsers' dependency on pdftotext) into every
|
||||
// fixture, so they register the smallest parser that will do instead.
|
||||
func init() {
|
||||
parser.Register("test", func(acc *config.Account) (parser.Parser, error) {
|
||||
return testParser{digits: acc.Digits()}, nil
|
||||
})
|
||||
}
|
||||
|
||||
// testParser reads "date,description,amount" with one header row.
|
||||
type testParser struct{ digits int }
|
||||
|
||||
func (p testParser) Parse(path string, acc *config.Account) ([]parser.RawTxn, error) {
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var txns []parser.RawTxn
|
||||
for i, line := range strings.Split(strings.TrimSpace(string(body)), "\n") {
|
||||
if i == 0 || strings.TrimSpace(line) == "" {
|
||||
continue // header
|
||||
}
|
||||
fields := strings.Split(line, ",")
|
||||
if len(fields) != 3 {
|
||||
return nil, fmt.Errorf("row %d: got %d fields, want 3", i+1, len(fields))
|
||||
}
|
||||
if _, err := time.Parse("2006-01-02", fields[0]); err != nil {
|
||||
return nil, fmt.Errorf("row %d: %w", i+1, err)
|
||||
}
|
||||
amount, err := parser.ParseAmount(fields[2], ".", "", p.digits)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("row %d: %w", i+1, err)
|
||||
}
|
||||
txns = append(txns, parser.RawTxn{Date: fields[0], Description: fields[1], AmountMinor: amount})
|
||||
}
|
||||
return txns, nil
|
||||
}
|
||||
|
||||
// newRoot builds a data root with one account and the given statement files.
|
||||
func newRoot(t *testing.T, statements map[string]string) (string, *store.DB, []*config.Account, *rules.Engine) {
|
||||
t.Helper()
|
||||
@@ -59,7 +97,7 @@ func write(t *testing.T, path, body string) {
|
||||
|
||||
func mustRun(t *testing.T, root string, db *store.DB, accounts []*config.Account, e *rules.Engine, opts Options) Result {
|
||||
t.Helper()
|
||||
res, err := Run(root, db, accounts, e, opts)
|
||||
res, err := Run(root, db, accounts, e, transfers.New(&config.Rules{}), opts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -153,7 +191,7 @@ not-a-date,BROKEN,-1.00
|
||||
`,
|
||||
})
|
||||
|
||||
res, err := Run(root, db, accounts, engine, Options{})
|
||||
res, err := Run(root, db, accounts, engine, transfers.New(&config.Rules{}), Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("Run returned a fatal error, want a per-file report: %v", err)
|
||||
}
|
||||
|
||||
+31
-20
@@ -33,37 +33,48 @@ type Transaction struct {
|
||||
SourceFileID int64
|
||||
SourcePath string
|
||||
|
||||
// Counterparty is the other side's account number (an IBAN, where the
|
||||
// statement gives one). Transfers between the user's own accounts are
|
||||
// often only distinguishable by it.
|
||||
Counterparty string
|
||||
// Type is the bank's own classification, e.g. Revolut's CARD_PAYMENT.
|
||||
Type string
|
||||
// BalanceMinor is the running balance the statement reported after this
|
||||
// transaction, when it reports one. Used to verify the balance chain.
|
||||
BalanceMinor *int64
|
||||
|
||||
// RuleTag is the category, decided entirely by rules.toml. It is derived
|
||||
// state: `money retag` rewrites it wholesale.
|
||||
RuleTag string
|
||||
ManualTag string
|
||||
RuleTransfer bool
|
||||
ManualTransfer *bool
|
||||
|
||||
// TransferID names the matched movement this transaction is one leg of, or
|
||||
// nil when it is not part of one. Derived from the [[transfer]] blocks in
|
||||
// rules.toml and rewritten wholesale alongside the tags.
|
||||
TransferID *int64
|
||||
}
|
||||
|
||||
// Tag is the effective category: a manual tag always beats a rule tag.
|
||||
func (t Transaction) Tag() string {
|
||||
if t.ManualTag != "" {
|
||||
return t.ManualTag
|
||||
}
|
||||
// IsTransferLeg reports whether this transaction was paired with its opposite
|
||||
// number on another account, and so is money moved rather than money spent.
|
||||
func (t Transaction) IsTransferLeg() bool { return t.TransferID != nil }
|
||||
|
||||
// TransferTag is shown in the tag column for a matched transfer leg that no
|
||||
// rule has tagged, so a paired leg reads as accounted for rather than as a
|
||||
// blank waiting to be filled in. It is bracketed like report.Untagged because
|
||||
// it is the same kind of thing: a label the tool supplies, not one the user
|
||||
// wrote.
|
||||
const TransferTag = "(transfer)"
|
||||
|
||||
// DisplayTag is the tag to show for a transaction.
|
||||
//
|
||||
// It is display only. Nothing writes it back: rule_tag stays exactly what
|
||||
// rules.toml made it, so a transfer can never smuggle a tag into the index and
|
||||
// `money retag` remains safe to run at any time. A rule tag wins when there is
|
||||
// one, since that is the user's own word for the transaction and the transfers
|
||||
// screen is where the pairing is explained anyway.
|
||||
func (t Transaction) DisplayTag() string {
|
||||
if t.RuleTag != "" {
|
||||
return t.RuleTag
|
||||
}
|
||||
|
||||
// IsTransfer reports whether this line moves money between the user's own
|
||||
// accounts and so must not count as spending. A manual override wins.
|
||||
func (t Transaction) IsTransfer() bool {
|
||||
if t.ManualTransfer != nil {
|
||||
return *t.ManualTransfer
|
||||
}
|
||||
return t.RuleTransfer
|
||||
if t.IsTransferLeg() {
|
||||
return TransferTag
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// FormatAmount renders the amount using the account's minor-unit scale.
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// bundledPdftotext is a static pdftotext built into this binary, or empty
|
||||
// when it was built without one. Only `go build -tags bundled` fills it (see
|
||||
// scripts/build-bundled.sh), so an ordinary build stays pure Go and needs no
|
||||
// 5 MB executable checked out next to it.
|
||||
var bundledPdftotext []byte
|
||||
|
||||
var (
|
||||
bundledOnce sync.Once
|
||||
bundledPath string
|
||||
bundledErr error
|
||||
)
|
||||
|
||||
// pdftotextCommand names the pdftotext to run: the bundled copy when there is
|
||||
// one, otherwise whatever is on PATH. The bundled copy wins because it is the
|
||||
// version the binary was built and checked with, which is the point of
|
||||
// carrying it rather than trusting the host's.
|
||||
func pdftotextCommand() (string, error) {
|
||||
if len(bundledPdftotext) == 0 {
|
||||
return "pdftotext", nil
|
||||
}
|
||||
bundledOnce.Do(func() { bundledPath, bundledErr = installBundled(bundledPdftotext) })
|
||||
return bundledPath, bundledErr
|
||||
}
|
||||
|
||||
// PdftotextSource says which pdftotext the PDF parsers will run, for
|
||||
// `money config`: a deployment that lacks one should find out before an
|
||||
// import fails on it.
|
||||
func PdftotextSource() string {
|
||||
if len(bundledPdftotext) == 0 {
|
||||
return "pdftotext from PATH (not bundled into this build)"
|
||||
}
|
||||
path, err := pdftotextCommand()
|
||||
if err != nil {
|
||||
return fmt.Sprintf("bundled, but it cannot be installed: %v", err)
|
||||
}
|
||||
return "bundled, run from " + path
|
||||
}
|
||||
|
||||
// installBundled writes the executable to the user's cache directory, where it
|
||||
// can be exec'd, and returns its path. The name carries the content hash, so a
|
||||
// newer build never runs an older build's copy, and a file already there is
|
||||
// only reused once its contents check out — a truncated write from a killed
|
||||
// process must not become the pdftotext every later run trusts.
|
||||
func installBundled(bin []byte) (string, error) {
|
||||
sum := sha256.Sum256(bin)
|
||||
name := "pdftotext-" + hex.EncodeToString(sum[:8])
|
||||
|
||||
// The cache directory rather than /tmp: /tmp is often mounted noexec on
|
||||
// servers, and the cache survives reboots, so this happens once per build.
|
||||
dir, err := os.UserCacheDir()
|
||||
if err != nil {
|
||||
dir = os.TempDir()
|
||||
}
|
||||
dir = filepath.Join(dir, "money")
|
||||
path := filepath.Join(dir, name)
|
||||
|
||||
if same, err := hasContents(path, sum); err != nil {
|
||||
return "", err
|
||||
} else if same {
|
||||
return path, nil
|
||||
}
|
||||
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return "", fmt.Errorf("install bundled pdftotext: %w", err)
|
||||
}
|
||||
// Written beside the target and renamed into place, so a concurrent run
|
||||
// never execs a half-written file.
|
||||
tmp, err := os.CreateTemp(dir, name+".*")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("install bundled pdftotext: %w", err)
|
||||
}
|
||||
defer os.Remove(tmp.Name()) // a no-op once renamed
|
||||
if _, err := tmp.Write(bin); err != nil {
|
||||
tmp.Close()
|
||||
return "", fmt.Errorf("install bundled pdftotext to %s: %w", dir, err)
|
||||
}
|
||||
if err := tmp.Chmod(0o755); err != nil {
|
||||
tmp.Close()
|
||||
return "", fmt.Errorf("install bundled pdftotext to %s: %w", dir, err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return "", fmt.Errorf("install bundled pdftotext to %s: %w", dir, err)
|
||||
}
|
||||
if err := os.Rename(tmp.Name(), path); err != nil {
|
||||
return "", fmt.Errorf("install bundled pdftotext to %s: %w", dir, err)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
// hasContents reports whether the file at path exists and hashes to sum.
|
||||
func hasContents(path string, sum [sha256.Size]byte) (bool, error) {
|
||||
f, err := os.Open(path)
|
||||
if os.IsNotExist(err) {
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("check bundled pdftotext: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
h := sha256.New()
|
||||
if _, err := io.Copy(h, f); err != nil {
|
||||
return false, fmt.Errorf("check bundled pdftotext: %w", err)
|
||||
}
|
||||
return bytes.Equal(h.Sum(nil), sum[:]), nil
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
//go:build bundled
|
||||
|
||||
package parser
|
||||
|
||||
import _ "embed"
|
||||
|
||||
// Built by scripts/build-bundled.sh amd64, which also runs the go build that
|
||||
// needs it. The file is not checked in, so this only compiles after that.
|
||||
//
|
||||
//go:embed bundled/pdftotext-linux-amd64
|
||||
var embeddedPdftotext []byte
|
||||
|
||||
func init() { bundledPdftotext = embeddedPdftotext }
|
||||
@@ -0,0 +1,13 @@
|
||||
//go:build bundled
|
||||
|
||||
package parser
|
||||
|
||||
import _ "embed"
|
||||
|
||||
// Built by scripts/build-bundled.sh arm64, which also runs the go build that
|
||||
// needs it. The file is not checked in, so this only compiles after that.
|
||||
//
|
||||
//go:embed bundled/pdftotext-linux-arm64
|
||||
var embeddedPdftotext []byte
|
||||
|
||||
func init() { bundledPdftotext = embeddedPdftotext }
|
||||
@@ -0,0 +1,79 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakePdftotext stands in for the real executable: what matters here is how it
|
||||
// is put on disk, not what it does once there.
|
||||
var fakePdftotext = []byte("#!/bin/sh\necho bundled\n")
|
||||
|
||||
// The bundled copy has to land somewhere it can be exec'd, and run.
|
||||
func TestInstallBundledRuns(t *testing.T) {
|
||||
t.Setenv("XDG_CACHE_HOME", t.TempDir())
|
||||
path, err := installBundled(fakePdftotext)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
out, err := exec.Command(path).Output()
|
||||
if err != nil {
|
||||
t.Fatalf("run %s: %v", path, err)
|
||||
}
|
||||
if string(out) != "bundled\n" {
|
||||
t.Errorf("output %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A copy already in place is reused, but only once its contents check out: a
|
||||
// write cut short by a killed process must not become what every later run
|
||||
// trusts.
|
||||
func TestInstallBundledReplacesADamagedCopy(t *testing.T) {
|
||||
t.Setenv("XDG_CACHE_HOME", t.TempDir())
|
||||
path, err := installBundled(fakePdftotext)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, fakePdftotext[:5], 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again, err := installBundled(fakePdftotext)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if again != path {
|
||||
t.Errorf("reinstalled to %s, want %s", again, path)
|
||||
}
|
||||
if got, _ := os.ReadFile(path); string(got) != string(fakePdftotext) {
|
||||
t.Errorf("damaged copy kept: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Each build's copy has its own name, so a newer binary never runs an older
|
||||
// one's pdftotext left in the same cache.
|
||||
func TestInstallBundledNamesByContent(t *testing.T) {
|
||||
t.Setenv("XDG_CACHE_HOME", t.TempDir())
|
||||
a, err := installBundled(fakePdftotext)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, err := installBundled(append([]byte(nil), "#!/bin/sh\necho newer\n"...))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a == b || filepath.Dir(a) != filepath.Dir(b) {
|
||||
t.Errorf("paths %s and %s, want two names in one directory", a, b)
|
||||
}
|
||||
}
|
||||
|
||||
// Without -tags bundled, nothing is embedded and PATH decides, as before.
|
||||
func TestUnbundledUsesPath(t *testing.T) {
|
||||
if len(bundledPdftotext) != 0 {
|
||||
t.Skip("built with -tags bundled")
|
||||
}
|
||||
if cmd, err := pdftotextCommand(); err != nil || cmd != "pdftotext" {
|
||||
t.Errorf("command = %q, %v; want pdftotext from PATH", cmd, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
//go:build bundled && !(linux && (amd64 || arm64))
|
||||
|
||||
package parser
|
||||
|
||||
// pdftotext is only built for Linux, as a static musl executable; there is
|
||||
// nothing to embed for this platform. Build without -tags bundled and install
|
||||
// poppler-utils instead.
|
||||
var _ = bundled_pdftotext_is_only_available_for_linux_amd64_and_linux_arm64
|
||||
@@ -1,124 +0,0 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/csv"
|
||||
"fmt"
|
||||
"io"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
)
|
||||
|
||||
func init() {
|
||||
Register("cmd", newCmdParser)
|
||||
}
|
||||
|
||||
// FileToken is replaced with the statement's absolute path in a cmd parser's argv.
|
||||
const FileToken = "{{file}}"
|
||||
|
||||
// cmdRunTimeout bounds an extractor run so a hung script cannot wedge an import.
|
||||
const cmdRunTimeout = 2 * time.Minute
|
||||
|
||||
// cmdParser runs an external extractor and reads normalised CSV from its
|
||||
// stdout: date,description,amount with any further columns ignored. This is
|
||||
// the bridge that lets the existing Python extractors be used unchanged.
|
||||
type cmdParser struct {
|
||||
cfg config.CmdConfig
|
||||
digits int
|
||||
}
|
||||
|
||||
func newCmdParser(acc *config.Account) (Parser, error) {
|
||||
if acc.Cmd == nil || len(acc.Cmd.Argv) == 0 {
|
||||
return nil, fmt.Errorf("account %s: parser \"cmd\" requires [cmd] with a non-empty argv", acc.Slug)
|
||||
}
|
||||
cfg := *acc.Cmd
|
||||
if cfg.Layout == "" {
|
||||
cfg.Layout = "2006-01-02"
|
||||
}
|
||||
if !hasFileToken(cfg.Argv) {
|
||||
return nil, fmt.Errorf("account %s: [cmd] argv must contain %s so the script knows which file to read",
|
||||
acc.Slug, FileToken)
|
||||
}
|
||||
return &cmdParser{cfg: cfg, digits: acc.Digits()}, nil
|
||||
}
|
||||
|
||||
func hasFileToken(argv []string) bool {
|
||||
for _, a := range argv {
|
||||
if strings.Contains(a, FileToken) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (p *cmdParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
||||
argv := make([]string, len(p.cfg.Argv))
|
||||
for i, a := range p.cfg.Argv {
|
||||
argv[i] = strings.ReplaceAll(a, FileToken, path)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), cmdRunTimeout)
|
||||
defer cancel()
|
||||
|
||||
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
|
||||
cmd.Dir = acc.Dir
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout = &stdout
|
||||
cmd.Stderr = &stderr
|
||||
|
||||
if err := cmd.Run(); err != nil {
|
||||
msg := strings.TrimSpace(stderr.String())
|
||||
if ctx.Err() == context.DeadlineExceeded {
|
||||
return nil, fmt.Errorf("extractor %v timed out after %s", argv, cmdRunTimeout)
|
||||
}
|
||||
if msg != "" {
|
||||
return nil, fmt.Errorf("extractor %v failed: %w: %s", argv, err, msg)
|
||||
}
|
||||
return nil, fmt.Errorf("extractor %v failed: %w", argv, err)
|
||||
}
|
||||
|
||||
return p.parseOutput(&stdout, argv)
|
||||
}
|
||||
|
||||
func (p *cmdParser) parseOutput(out io.Reader, argv []string) ([]RawTxn, error) {
|
||||
r := csv.NewReader(out)
|
||||
r.FieldsPerRecord = -1
|
||||
r.LazyQuotes = true
|
||||
|
||||
var txns []RawTxn
|
||||
for row := 0; ; row++ {
|
||||
rec, err := r.Read()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("extractor %v: output row %d: %w", argv, row+1, err)
|
||||
}
|
||||
if row < p.cfg.SkipRows || isBlank(rec) {
|
||||
continue
|
||||
}
|
||||
if len(rec) < 3 {
|
||||
return nil, fmt.Errorf("extractor %v: output row %d has %d columns, want date,description,amount",
|
||||
argv, row+1, len(rec))
|
||||
}
|
||||
d, err := time.Parse(p.cfg.Layout, strings.TrimSpace(rec[0]))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("extractor %v: output row %d: date %q does not match layout %q",
|
||||
argv, row+1, rec[0], p.cfg.Layout)
|
||||
}
|
||||
amount, err := ParseAmount(rec[2], ".", "", p.digits)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("extractor %v: output row %d: %w", argv, row+1, err)
|
||||
}
|
||||
txns = append(txns, RawTxn{
|
||||
Date: d.Format("2006-01-02"),
|
||||
Description: strings.TrimSpace(rec[1]),
|
||||
AmountMinor: amount,
|
||||
})
|
||||
}
|
||||
return txns, nil
|
||||
}
|
||||
@@ -1,175 +0,0 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"encoding/csv"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
)
|
||||
|
||||
func init() {
|
||||
Register("csv", newCSVParser)
|
||||
}
|
||||
|
||||
// csvParser reads a delimited statement using column positions from
|
||||
// account.toml. Amounts come either from one signed column, or from a
|
||||
// debit/credit pair.
|
||||
type csvParser struct {
|
||||
cfg config.CSVConfig
|
||||
digits int
|
||||
}
|
||||
|
||||
func newCSVParser(acc *config.Account) (Parser, error) {
|
||||
if acc.CSV == nil {
|
||||
return nil, fmt.Errorf("account %s: parser \"csv\" requires a [csv] section", acc.Slug)
|
||||
}
|
||||
cfg := *acc.CSV
|
||||
if cfg.Amount == nil && cfg.Debit == nil && cfg.Credit == nil {
|
||||
return nil, fmt.Errorf("account %s: [csv] needs either amount or debit/credit columns", acc.Slug)
|
||||
}
|
||||
if cfg.Amount != nil && (cfg.Debit != nil || cfg.Credit != nil) {
|
||||
return nil, fmt.Errorf("account %s: [csv] sets both amount and debit/credit; pick one", acc.Slug)
|
||||
}
|
||||
if cfg.Date.Layout == "" {
|
||||
return nil, fmt.Errorf("account %s: [csv] date needs a layout, e.g. layout = \"02.01.2006\"", acc.Slug)
|
||||
}
|
||||
return &csvParser{cfg: cfg, digits: acc.Digits()}, nil
|
||||
}
|
||||
|
||||
func (p *csvParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
r := csv.NewReader(f)
|
||||
r.FieldsPerRecord = -1 // statements are ragged more often than not
|
||||
r.LazyQuotes = true
|
||||
if d := p.cfg.Delimiter; d != "" {
|
||||
runes := []rune(d)
|
||||
if len(runes) != 1 {
|
||||
return nil, fmt.Errorf("delimiter %q must be a single character", d)
|
||||
}
|
||||
r.Comma = runes[0]
|
||||
}
|
||||
|
||||
var out []RawTxn
|
||||
for row := 0; ; row++ {
|
||||
rec, err := r.Read()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: row %d: %w", path, row+1, err)
|
||||
}
|
||||
if row < p.cfg.SkipRows {
|
||||
continue
|
||||
}
|
||||
if isBlank(rec) {
|
||||
continue
|
||||
}
|
||||
txn, err := p.row(rec)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: row %d: %w", path, row+1, err)
|
||||
}
|
||||
out = append(out, txn)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (p *csvParser) row(rec []string) (RawTxn, error) {
|
||||
var t RawTxn
|
||||
|
||||
raw, err := field(rec, p.cfg.Date.Col)
|
||||
if err != nil {
|
||||
return t, fmt.Errorf("date column: %w", err)
|
||||
}
|
||||
d, err := time.Parse(p.cfg.Date.Layout, strings.TrimSpace(raw))
|
||||
if err != nil {
|
||||
return t, fmt.Errorf("date %q does not match layout %q", raw, p.cfg.Date.Layout)
|
||||
}
|
||||
t.Date = d.Format("2006-01-02")
|
||||
|
||||
desc, err := field(rec, p.cfg.Description.Col)
|
||||
if err != nil {
|
||||
return t, fmt.Errorf("description column: %w", err)
|
||||
}
|
||||
t.Description = strings.TrimSpace(desc)
|
||||
|
||||
switch {
|
||||
case p.cfg.Amount != nil:
|
||||
raw, err := field(rec, p.cfg.Amount.Col)
|
||||
if err != nil {
|
||||
return t, fmt.Errorf("amount column: %w", err)
|
||||
}
|
||||
v, err := ParseAmount(raw, p.cfg.Amount.Decimal, p.cfg.Amount.Thousands, p.digits)
|
||||
if err != nil {
|
||||
return t, err
|
||||
}
|
||||
t.AmountMinor = v
|
||||
default:
|
||||
// Debit/credit pair: exactly one of the two carries a value, and both
|
||||
// are written as positive numbers.
|
||||
debit, err := p.optional(rec, p.cfg.Debit)
|
||||
if err != nil {
|
||||
return t, fmt.Errorf("debit column: %w", err)
|
||||
}
|
||||
credit, err := p.optional(rec, p.cfg.Credit)
|
||||
if err != nil {
|
||||
return t, fmt.Errorf("credit column: %w", err)
|
||||
}
|
||||
if debit != 0 && credit != 0 {
|
||||
return t, fmt.Errorf("both debit (%d) and credit (%d) are set", debit, credit)
|
||||
}
|
||||
t.AmountMinor = credit - abs(debit)
|
||||
}
|
||||
|
||||
if p.cfg.Invert {
|
||||
t.AmountMinor = -t.AmountMinor
|
||||
}
|
||||
return t, nil
|
||||
}
|
||||
|
||||
// optional parses a column that may legitimately be blank, as debit/credit
|
||||
// columns always are for half the rows.
|
||||
func (p *csvParser) optional(rec []string, c *config.Column) (int64, error) {
|
||||
if c == nil {
|
||||
return 0, nil
|
||||
}
|
||||
raw, err := field(rec, c.Col)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if strings.TrimSpace(raw) == "" {
|
||||
return 0, nil
|
||||
}
|
||||
return ParseAmount(raw, c.Decimal, c.Thousands, p.digits)
|
||||
}
|
||||
|
||||
func field(rec []string, i int) (string, error) {
|
||||
if i < 0 || i >= len(rec) {
|
||||
return "", fmt.Errorf("index %d out of range, row has %d columns", i, len(rec))
|
||||
}
|
||||
return rec[i], nil
|
||||
}
|
||||
|
||||
func isBlank(rec []string) bool {
|
||||
for _, f := range rec {
|
||||
if strings.TrimSpace(f) != "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func abs(v int64) int64 {
|
||||
if v < 0 {
|
||||
return -v
|
||||
}
|
||||
return v
|
||||
}
|
||||
@@ -1,112 +0,0 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
)
|
||||
|
||||
func writeFile(t *testing.T, name, content string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), name)
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestCSVSignedAmountColumn(t *testing.T) {
|
||||
path := writeFile(t, "st.csv", `Date,Ref,Description,Amount
|
||||
02.01.2026,X1,LIDL SOFIA 1234,"-45,20"
|
||||
03.01.2026,X2,ACME PAYROLL,"1.500,00"
|
||||
|
||||
04.01.2026,X3,COFFEE,"-3,50"
|
||||
`)
|
||||
acc := &config.Account{Slug: "checking", Currency: "EUR", Parser: "csv", CSV: &config.CSVConfig{
|
||||
SkipRows: 1,
|
||||
Date: config.Column{Col: 0, Layout: "02.01.2006"},
|
||||
Description: config.Column{Col: 2},
|
||||
Amount: &config.Column{Col: 3, Decimal: ",", Thousands: "."},
|
||||
}}
|
||||
p, err := For(acc)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := p.Parse(path, acc)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []RawTxn{
|
||||
{Date: "2026-01-02", Description: "LIDL SOFIA 1234", AmountMinor: -4520},
|
||||
{Date: "2026-01-03", Description: "ACME PAYROLL", AmountMinor: 150000},
|
||||
{Date: "2026-01-04", Description: "COFFEE", AmountMinor: -350},
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("got %d txns, want %d: %+v", len(got), len(want), got)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Errorf("txn %d = %+v, want %+v", i, got[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCSVDebitCreditPair(t *testing.T) {
|
||||
path := writeFile(t, "st.csv", `2026-02-01;RENT;800.00;
|
||||
2026-02-05;SALARY;;2500.00
|
||||
`)
|
||||
acc := &config.Account{Slug: "checking", Currency: "EUR", Parser: "csv", CSV: &config.CSVConfig{
|
||||
Delimiter: ";",
|
||||
Date: config.Column{Col: 0, Layout: "2006-01-02"},
|
||||
Description: config.Column{Col: 1},
|
||||
Debit: &config.Column{Col: 2},
|
||||
Credit: &config.Column{Col: 3},
|
||||
}}
|
||||
p, err := For(acc)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := p.Parse(path, acc)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("got %d txns, want 2: %+v", len(got), got)
|
||||
}
|
||||
if got[0].AmountMinor != -80000 {
|
||||
t.Errorf("debit row = %d, want -80000", got[0].AmountMinor)
|
||||
}
|
||||
if got[1].AmountMinor != 250000 {
|
||||
t.Errorf("credit row = %d, want 250000", got[1].AmountMinor)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCSVConfigErrors(t *testing.T) {
|
||||
cases := map[string]*config.Account{
|
||||
"no csv section": {Slug: "a", Parser: "csv"},
|
||||
"no amount columns": {Slug: "a", Parser: "csv", CSV: &config.CSVConfig{
|
||||
Date: config.Column{Layout: "2006-01-02"},
|
||||
}},
|
||||
"amount and debit together": {Slug: "a", Parser: "csv", CSV: &config.CSVConfig{
|
||||
Date: config.Column{Layout: "2006-01-02"},
|
||||
Amount: &config.Column{Col: 1},
|
||||
Debit: &config.Column{Col: 2},
|
||||
}},
|
||||
"no date layout": {Slug: "a", Parser: "csv", CSV: &config.CSVConfig{
|
||||
Amount: &config.Column{Col: 1},
|
||||
}},
|
||||
}
|
||||
for name, acc := range cases {
|
||||
if _, err := For(acc); err == nil {
|
||||
t.Errorf("%s: expected an error", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownParser(t *testing.T) {
|
||||
if _, err := For(&config.Account{Slug: "a", Parser: "nope"}); err == nil {
|
||||
t.Error("expected an error for an unregistered parser")
|
||||
}
|
||||
}
|
||||
+39
-10
@@ -18,8 +18,8 @@ func init() {
|
||||
// nlbParser reads an NLB izpisek PDF.
|
||||
//
|
||||
// A transaction starts on a line beginning with a dd.mm.yy date and ends with
|
||||
// the signed amount and the running balance. Long descriptions and the
|
||||
// counterparty account wrap onto indented continuation lines below.
|
||||
// the signed amount and the running balance. Long descriptions and the other
|
||||
// side's account number wrap onto indented continuation lines below.
|
||||
type nlbParser struct {
|
||||
digits int
|
||||
}
|
||||
@@ -37,10 +37,16 @@ var (
|
||||
`(?P<balance>[-\x{2212}]?[\d.,]+[-\x{2212}]?)\s*$`)
|
||||
// Columns inside a line are separated by four or more spaces.
|
||||
nlbFieldSplit = regexp.MustCompile(`\s{4,}`)
|
||||
// A Slovenian IBAN, which is the counterparty account when present.
|
||||
// A Slovenian IBAN, which NLB prints in a column of its own. It is split
|
||||
// out only so that its wrapped fragments can be rejoined; the result goes
|
||||
// onto the end of the description, where every other parser keeps it.
|
||||
nlbAccount = regexp.MustCompile(`^SI\d{2}(?:\s?\d{4}){3}\s?\d{3}$`)
|
||||
)
|
||||
|
||||
// nlbStatementDate is the issue date in an izpisek's header. It names the
|
||||
// statement: NLB's downloads are not named for what they hold.
|
||||
var nlbStatementDate = regexp.MustCompile(`Datum izpiska\s+(\d{2})\.(\d{2})\.(\d{4})`)
|
||||
|
||||
// nlbMinContinuationIndent is the shallowest indent a wrapped description line
|
||||
// may have. The real threshold is the description column of the transaction
|
||||
// the line belongs to, measured per line rather than hardcoded: pdftotext
|
||||
@@ -59,12 +65,33 @@ func (p *nlbParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
||||
return parseNLBText(text, p.digits)
|
||||
}
|
||||
|
||||
// StatementName names an izpisek after its statement date, izpisek_YYYY-MM-DD,
|
||||
// so the folder sorts by date and two downloads of one statement collide.
|
||||
func (p *nlbParser) StatementName(path string) (string, error) {
|
||||
text, err := pdfToText(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return nlbStatementName(text), nil
|
||||
}
|
||||
|
||||
func nlbStatementName(text string) string {
|
||||
m := nlbStatementDate.FindStringSubmatch(text)
|
||||
if m == nil {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf("izpisek_%s-%s-%s", m[3], m[2], m[1])
|
||||
}
|
||||
|
||||
// parseNLBText holds the whole parser, separated from PDF extraction so it can
|
||||
// be tested against captured pdftotext output.
|
||||
func parseNLBText(text string, digits int) ([]RawTxn, error) {
|
||||
var (
|
||||
txns []RawTxn
|
||||
accts []string // counterparty per transaction, built up alongside
|
||||
// The account column per transaction, built up alongside because it
|
||||
// wraps in fragments of its own and has to be rejoined before it can
|
||||
// be appended to the description.
|
||||
trailing []string
|
||||
descAt int // description column of the last transaction line
|
||||
)
|
||||
|
||||
@@ -84,7 +111,7 @@ func parseNLBText(text string, digits int) ([]RawTxn, error) {
|
||||
return nil, fmt.Errorf("line %d: %w", i+1, err)
|
||||
}
|
||||
txns = append(txns, txn)
|
||||
accts = append(accts, account)
|
||||
trailing = append(trailing, account)
|
||||
descAt = col
|
||||
continue
|
||||
}
|
||||
@@ -104,20 +131,22 @@ func parseNLBText(text string, digits int) ([]RawTxn, error) {
|
||||
last := len(txns) - 1
|
||||
txns[last].Description += " " + parts[0]
|
||||
if len(parts) > 1 {
|
||||
accts[last] = strings.TrimSpace(accts[last] + " " + parts[1])
|
||||
trailing[last] = strings.TrimSpace(trailing[last] + " " + parts[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The account column goes last rather than in the position it occupied on
|
||||
// the page, so that an IBAN wrapped over several lines stays contiguous
|
||||
// and a glob can match it.
|
||||
for i := range txns {
|
||||
txns[i].Counterparty = accts[i]
|
||||
txns[i].Description = strings.Join(strings.Fields(txns[i].Description), " ")
|
||||
txns[i].Description = strings.Join(strings.Fields(txns[i].Description+" "+trailing[i]), " ")
|
||||
}
|
||||
return txns, nil
|
||||
}
|
||||
|
||||
// parseNLBLine parses one transaction line, returning the transaction, the
|
||||
// counterparty account found among its columns, and the column at which the
|
||||
// account number found among its columns, and the column at which the
|
||||
// description starts, which is where its wrapped lines will sit.
|
||||
func parseNLBLine(line string, digits int) (RawTxn, string, int, error) {
|
||||
trimmed := strings.TrimRight(line, " \t\r")
|
||||
@@ -154,7 +183,7 @@ func parseNLBLine(line string, digits int) (RawTxn, string, int, error) {
|
||||
return RawTxn{}, "", 0, fmt.Errorf("balance %q: %w", balance, err)
|
||||
}
|
||||
|
||||
// The middle holds the description and, sometimes, the counterparty IBAN.
|
||||
// The middle holds the description and, sometimes, the IBAN.
|
||||
var (
|
||||
account string
|
||||
desc []string
|
||||
|
||||
+20
-10
@@ -4,7 +4,7 @@ import "testing"
|
||||
|
||||
// A page as pdftotext -layout renders it: a header, transaction lines ending
|
||||
// in amount and balance, and indented continuation lines carrying wrapped
|
||||
// descriptions and the counterparty IBAN.
|
||||
// descriptions and the other side's IBAN.
|
||||
// The description column sits at 15, which is what the script's
|
||||
// CONTINUATION_INDENT tells us about the real layout.
|
||||
const nlbPage = ` NLB d.d.
|
||||
@@ -35,14 +35,12 @@ func TestParseNLBText(t *testing.T) {
|
||||
if first.Date != "2026-01-02" {
|
||||
t.Errorf("date = %q, want 2026-01-02", first.Date)
|
||||
}
|
||||
// The continuation line is appended to the description.
|
||||
if first.Description != "PLACILO S KARTICO LIDL SOFIA 4412" {
|
||||
// The continuation line is appended to the description, and the IBAN in
|
||||
// its second column goes last so that it stays contiguous however many
|
||||
// lines it wrapped over.
|
||||
if first.Description != "PLACILO S KARTICO LIDL SOFIA 4412 SI56 1234 5678 9012 345" {
|
||||
t.Errorf("description = %q", first.Description)
|
||||
}
|
||||
// ...and the IBAN in its second column becomes the counterparty.
|
||||
if first.Counterparty != "SI56 1234 5678 9012 345" {
|
||||
t.Errorf("counterparty = %q", first.Counterparty)
|
||||
}
|
||||
if first.AmountMinor != -4520 {
|
||||
t.Errorf("amount = %d, want -4520", first.AmountMinor)
|
||||
}
|
||||
@@ -57,9 +55,9 @@ func TestParseNLBText(t *testing.T) {
|
||||
t.Errorf("description = %q", txns[1].Description)
|
||||
}
|
||||
|
||||
// A transaction with no continuation line keeps an empty counterparty.
|
||||
if txns[2].Counterparty != "" {
|
||||
t.Errorf("counterparty = %q, want empty", txns[2].Counterparty)
|
||||
// A transaction with no continuation line gets nothing appended.
|
||||
if txns[2].Description != "PRENOS NA VARCEVALNI" {
|
||||
t.Errorf("description = %q, want no trailing account number", txns[2].Description)
|
||||
}
|
||||
|
||||
// A trailing minus marks a negative balance.
|
||||
@@ -114,3 +112,15 @@ func TestParseNLBRejectsMalformedLine(t *testing.T) {
|
||||
t.Error("expected an error for a transaction line with no amount")
|
||||
}
|
||||
}
|
||||
|
||||
// An izpisek is named after its statement date, as rename_izpiski.py did.
|
||||
func TestNLBStatementName(t *testing.T) {
|
||||
header := " IZPISEK 001/2026\n" +
|
||||
" Datum izpiska 31.01.2026 Stran 1\n"
|
||||
if got := nlbStatementName(header + nlbPage); got != "izpisek_2026-01-31" {
|
||||
t.Errorf("name = %q, want izpisek_2026-01-31", got)
|
||||
}
|
||||
if got := nlbStatementName(nlbPage); got != "" {
|
||||
t.Errorf("name without a statement date = %q, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
// Package parser turns a statement file into raw transactions.
|
||||
//
|
||||
// Every bank needs its own extraction logic, so parsers are looked up by name
|
||||
// from a registry. Two are built in: "csv" for delimited exports with
|
||||
// configurable columns, and "cmd" for shelling out to an external extractor
|
||||
// (which is how the existing Python scripts are used until they are ported).
|
||||
// Ported extractors register themselves here and become usable by putting
|
||||
// their name in an account.toml.
|
||||
// from a registry that each one joins from an init. A parser becomes usable by
|
||||
// putting its name in an account.toml; there is no generic column-mapped
|
||||
// parser, because a statement layout is better described in Go, where it can
|
||||
// be tested, than in a table of column indexes.
|
||||
package parser
|
||||
|
||||
import (
|
||||
@@ -23,9 +22,6 @@ type RawTxn struct {
|
||||
Description string
|
||||
AmountMinor int64 // signed; negative is an outflow
|
||||
|
||||
// Counterparty is the other side's account number, where the statement
|
||||
// gives one. Optional.
|
||||
Counterparty string
|
||||
// Type is the bank's own classification of the transaction. Optional.
|
||||
Type string
|
||||
// BalanceMinor is the running balance after this transaction, when the
|
||||
@@ -47,6 +43,17 @@ type Warner interface {
|
||||
Warnings() []string
|
||||
}
|
||||
|
||||
// Namer is an optional interface for parsers whose bank names its downloads
|
||||
// unhelpfully. StatementName reads the statement at path and returns the name,
|
||||
// without extension, it should be stored under — the upload's extension is
|
||||
// kept, lowercased — or "" when the statement does not say, in which case it
|
||||
// keeps the name it came with. Only an upload uses
|
||||
// it; files already in a folder are never renamed, since the index records
|
||||
// them by path.
|
||||
type Namer interface {
|
||||
StatementName(path string) (string, error)
|
||||
}
|
||||
|
||||
// Factory builds a parser from an account's config, validating it up front so
|
||||
// a bad account.toml fails before any file is read.
|
||||
type Factory func(acc *config.Account) (Parser, error)
|
||||
|
||||
@@ -17,12 +17,18 @@ const pdfToTextTimeout = 2 * time.Minute
|
||||
//
|
||||
// This shells out to poppler's pdftotext rather than decoding the PDF in Go:
|
||||
// the layout reconstruction it does is the whole reason the column-based
|
||||
// parsers work, and no Go library matches it.
|
||||
// parsers work, and no Go library matches it. A build made with -tags bundled
|
||||
// carries its own copy, so a server needs nothing installed.
|
||||
func pdfToText(path string) (string, error) {
|
||||
bin, err := pdftotextCommand()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), pdfToTextTimeout)
|
||||
defer cancel()
|
||||
|
||||
cmd := exec.CommandContext(ctx, "pdftotext", "-layout", path, "-")
|
||||
cmd := exec.CommandContext(ctx, bin, "-layout", path, "-")
|
||||
var stdout, stderr bytes.Buffer
|
||||
cmd.Stdout = &stdout
|
||||
cmd.Stderr = &stderr
|
||||
@@ -34,6 +40,12 @@ func pdfToText(path string) (string, error) {
|
||||
if errors := strings.TrimSpace(stderr.String()); errors != "" {
|
||||
return "", fmt.Errorf("pdftotext %s: %w: %s", path, err, errors)
|
||||
}
|
||||
if bin != "pdftotext" {
|
||||
// The copy was installed, so failing to start it is about where it
|
||||
// was put: a cache directory on a noexec mount is the usual cause.
|
||||
return "", fmt.Errorf("bundled pdftotext at %s did not run (is that directory mounted noexec? "+
|
||||
"point XDG_CACHE_HOME somewhere else): %w", bin, err)
|
||||
}
|
||||
if _, lookErr := exec.LookPath("pdftotext"); lookErr != nil {
|
||||
return "", fmt.Errorf("pdftotext is not installed (it ships with poppler-utils): %w", lookErr)
|
||||
}
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -36,9 +35,6 @@ type revolutParser struct {
|
||||
warnings []string
|
||||
}
|
||||
|
||||
// revolutIBAN finds a counterparty account inside a description.
|
||||
var revolutIBAN = regexp.MustCompile(`\b[A-Z]{2}\d{2}[A-Z0-9]{11,30}\b`)
|
||||
|
||||
// Columns the parser needs; a missing one is a hard error rather than a
|
||||
// silently empty field.
|
||||
var revolutColumns = []string{
|
||||
@@ -164,18 +160,26 @@ func (p *revolutParser) row(get func(string) string, digits int) (RawTxn, error)
|
||||
desc += fmt.Sprintf(" (fee %s)", model.FormatMinor(fee, digits))
|
||||
}
|
||||
|
||||
counterparty := revolutIBAN.FindString(desc)
|
||||
|
||||
return RawTxn{
|
||||
Date: d.Format("2006-01-02"),
|
||||
Description: desc,
|
||||
AmountMinor: amount - fee,
|
||||
Counterparty: counterparty,
|
||||
Type: strings.TrimSpace(get("Type")),
|
||||
BalanceMinor: &balance,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// isBlank reports whether a CSV record holds nothing but whitespace, which is
|
||||
// what a trailing newline in the export reads as.
|
||||
func isBlank(rec []string) bool {
|
||||
for _, f := range rec {
|
||||
if strings.TrimSpace(f) != "" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// headerIndex maps a Revolut column name to its position.
|
||||
type headerIndex map[string]int
|
||||
|
||||
|
||||
@@ -1,12 +1,24 @@
|
||||
package parser
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
)
|
||||
|
||||
// writeFile drops a statement in a temporary directory and returns its path.
|
||||
func writeFile(t *testing.T, name, content string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), name)
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
// A Revolut export: out of date order, mixed currencies, a pending row, a row
|
||||
// with a fee, and a transfer carrying an IBAN.
|
||||
const revolutCSV = `Type,Product,Started Date,Completed Date,Description,Amount,Fee,Currency,State,Balance
|
||||
@@ -60,9 +72,6 @@ func TestRevolutParse(t *testing.T) {
|
||||
if !strings.Contains(transfer.Description, "(fee 0.35)") {
|
||||
t.Errorf("description = %q, want a fee note", transfer.Description)
|
||||
}
|
||||
if transfer.Counterparty != "SI56123456789012345" {
|
||||
t.Errorf("counterparty = %q, want the IBAN from the description", transfer.Counterparty)
|
||||
}
|
||||
if transfer.BalanceMinor == nil || *transfer.BalanceMinor != 73441 {
|
||||
t.Errorf("balance = %v, want 73441", transfer.BalanceMinor)
|
||||
}
|
||||
|
||||
@@ -32,7 +32,6 @@ var (
|
||||
trAmount = regexp.MustCompile(`[-\x{2212}]?€\s?[-\x{2212}]?[\d,]+\.\d{2}`)
|
||||
trToken = regexp.MustCompile(`\S+`)
|
||||
trFullDate = regexp.MustCompile(`^\d{2} [A-Z][a-z]{2} \d{4}$`)
|
||||
trIBAN = regexp.MustCompile(`\b[A-Z]{2}\d{2}[A-Z0-9]{11,30}\b`)
|
||||
)
|
||||
|
||||
var (
|
||||
@@ -47,6 +46,12 @@ const trAmountSlack = 5
|
||||
// trDateLayout is the date Trade Republic prints, e.g. "05 Jan 2026".
|
||||
const trDateLayout = "02 Jan 2006"
|
||||
|
||||
// trPeriod is the statement period in the address block at the top of the
|
||||
// first page, e.g. "DATE 01 May 2025 - 31 Jul 2026". It names the
|
||||
// statement: Trade Republic's downloads are all called document-N.pdf. The
|
||||
// transaction table's own DATE heading never matches, since no dates follow it.
|
||||
var trPeriod = regexp.MustCompile(`\bDATE\s+(\d{2} [A-Z][a-z]{2} \d{4})\s*[-\x{2013}\x{2212}]\s*(\d{2} [A-Z][a-z]{2} \d{4})`)
|
||||
|
||||
func (p *tradeRepublicParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
||||
text, err := pdfToText(path)
|
||||
if err != nil {
|
||||
@@ -55,6 +60,33 @@ func (p *tradeRepublicParser) Parse(path string, acc *config.Account) ([]RawTxn,
|
||||
return parseTradeRepublicText(text, p.digits)
|
||||
}
|
||||
|
||||
// StatementName names a statement after its period,
|
||||
// traderepublic_YYYY_MM_DD_YYYY_MM_DD, so the folder sorts by when it starts
|
||||
// and two downloads of one statement collide.
|
||||
func (p *tradeRepublicParser) StatementName(path string) (string, error) {
|
||||
text, err := pdfToText(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return tradeRepublicStatementName(text), nil
|
||||
}
|
||||
|
||||
func tradeRepublicStatementName(text string) string {
|
||||
m := trPeriod.FindStringSubmatch(text)
|
||||
if m == nil {
|
||||
return ""
|
||||
}
|
||||
from, err := time.Parse(trDateLayout, m[1])
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
to, err := time.Parse(trDateLayout, m[2])
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return "traderepublic_" + from.Format("2006_01_02") + "_" + to.Format("2006_01_02")
|
||||
}
|
||||
|
||||
// parseTradeRepublicText holds the whole parser, separated from PDF extraction
|
||||
// so it can be tested against captured pdftotext output.
|
||||
func parseTradeRepublicText(text string, digits int) ([]RawTxn, error) {
|
||||
@@ -197,7 +229,6 @@ func parseTradeRepublicBlock(lines []string, cols map[string]int, digits int) (R
|
||||
Date: d.Format("2006-01-02"),
|
||||
Description: desc,
|
||||
AmountMinor: amount,
|
||||
Counterparty: trIBAN.FindString(desc),
|
||||
Type: strings.Join(words["TYPE"], " "),
|
||||
BalanceMinor: amounts["BALANCE"],
|
||||
}, true, nil
|
||||
|
||||
@@ -120,9 +120,6 @@ func TestParseTradeRepublicText(t *testing.T) {
|
||||
if transfer.Description != "Standing order to sav-ings SI56123456789012345" {
|
||||
t.Errorf("description = %q, want the wrapped fragment joined without a space", transfer.Description)
|
||||
}
|
||||
if transfer.Counterparty != "SI56123456789012345" {
|
||||
t.Errorf("counterparty = %q", transfer.Counterparty)
|
||||
}
|
||||
if transfer.AmountMinor != -50000 {
|
||||
t.Errorf("amount = %d, want -50000", transfer.AmountMinor)
|
||||
}
|
||||
@@ -165,3 +162,19 @@ func TestParseTradeRepublicIgnoresAmountsInDescription(t *testing.T) {
|
||||
t.Errorf("description = %q, want the inline figure kept", txns[0].Description)
|
||||
}
|
||||
}
|
||||
|
||||
// A statement is named after the period in its header; the transaction
|
||||
// table's DATE heading, with no dates after it, is not mistaken for it.
|
||||
func TestTradeRepublicStatementName(t *testing.T) {
|
||||
header := "NIKOLA PETROV DATE 01 May 2025 - 31 Jul 2026\n" +
|
||||
"Ulica 21 IBAN DE00000000000000000000\n\n" +
|
||||
"DATE TYPE DESCRIPTION MONEY IN MONEY OUT BALANCE\n"
|
||||
if got := tradeRepublicStatementName(header); got != "traderepublic_2025_05_01_2026_07_31" {
|
||||
t.Errorf("name = %q, want traderepublic_2025_05_01_2026_07_31", got)
|
||||
}
|
||||
table := "DATE TYPE DESCRIPTION MONEY IN MONEY OUT BALANCE\n" +
|
||||
"01 Jul 2026 Interest Interest payment €1.17 €671.58\n"
|
||||
if got := tradeRepublicStatementName(table); got != "" {
|
||||
t.Errorf("name without a period = %q, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Period is the window a report covers: an inclusive range of ISO dates, with
|
||||
// either side empty for unbounded. Dates are stored as YYYY-MM-DD, so comparing
|
||||
// them as strings compares them as dates and nothing has to be parsed to filter
|
||||
// on one.
|
||||
type Period struct {
|
||||
// Label is what to call the window: "last month", "all time", "2026-06".
|
||||
Label string
|
||||
// Span names the dates Label stands for, because "last month" says nothing
|
||||
// on its own about which month it was. Empty when the label already is the
|
||||
// span, as it is for a single month and for all time.
|
||||
Span string
|
||||
From string
|
||||
To string
|
||||
}
|
||||
|
||||
// String names the window, and what it covers when the label does not.
|
||||
func (p Period) String() string {
|
||||
if p.Span == "" {
|
||||
return p.Label
|
||||
}
|
||||
return p.Label + " · " + p.Span
|
||||
}
|
||||
|
||||
// LastMonth is the label of the window a report opens on. The month that has
|
||||
// just ended is the last one a statement can be complete for: this month is
|
||||
// still filling up, so its total is not comparable with any other.
|
||||
const LastMonth = "last month"
|
||||
|
||||
// Periods is the axis a report steps along, widest window first: the named
|
||||
// ranges, then every month the index holds that is older than last month, most
|
||||
// recent of those first. months is expected newest first, as store.Months
|
||||
// returns it; anything in it that a named range already covers is left out of
|
||||
// the tail rather than offered twice.
|
||||
//
|
||||
// The named ranges are relative to today, not to the newest statement. "Last
|
||||
// month" means the month that has just ended, so if no statement covers it yet
|
||||
// the report for it is empty — which is the honest answer, and one the user can
|
||||
// act on, where silently reporting an older month they did not ask for is not.
|
||||
func Periods(today time.Time, months []string) []Period {
|
||||
thisMonth := time.Date(today.Year(), today.Month(), 1, 0, 0, 0, 0, time.UTC)
|
||||
lastMonth := thisMonth.AddDate(0, -1, 0)
|
||||
|
||||
// A range ends with this month rather than with the last complete one:
|
||||
// "last 3 months" is asked in order to see what is happening now, and
|
||||
// leaving out the days since the 1st would answer a different question.
|
||||
rolling := func(back int, label string) Period {
|
||||
from := thisMonth.AddDate(0, -back, 0)
|
||||
return Period{
|
||||
Label: label,
|
||||
Span: from.Format("2006-01") + " … " + thisMonth.Format("2006-01"),
|
||||
From: isoDate(from),
|
||||
To: isoDate(endOfMonth(thisMonth)),
|
||||
}
|
||||
}
|
||||
month := func(start time.Time, label string) Period {
|
||||
p := Period{Label: label, From: isoDate(start), To: isoDate(endOfMonth(start))}
|
||||
if label != start.Format("2006-01") {
|
||||
p.Span = start.Format("2006-01")
|
||||
}
|
||||
return p
|
||||
}
|
||||
year := time.Date(today.Year(), time.January, 1, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
out := []Period{
|
||||
{Label: "all time"},
|
||||
{
|
||||
Label: "this year",
|
||||
Span: fmt.Sprintf("%d", today.Year()),
|
||||
From: isoDate(year),
|
||||
To: isoDate(year.AddDate(1, 0, -1)),
|
||||
},
|
||||
rolling(11, "last 12 months"),
|
||||
rolling(2, "last 3 months"),
|
||||
month(thisMonth, "this month"),
|
||||
month(lastMonth, LastMonth),
|
||||
}
|
||||
|
||||
cutoff := lastMonth.Format("2006-01")
|
||||
for _, ym := range months {
|
||||
// Months at or after last month are already on the axis above, and one
|
||||
// dated in the future is a parsing bug rather than a window to offer.
|
||||
if ym >= cutoff {
|
||||
continue
|
||||
}
|
||||
start, err := time.Parse("2006-01", ym)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, month(start, ym))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// DefaultIndex is where on the axis a report opens.
|
||||
func DefaultIndex(ps []Period) int {
|
||||
for i, p := range ps {
|
||||
if p.Label == LastMonth {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func isoDate(t time.Time) string { return t.Format("2006-01-02") }
|
||||
|
||||
// endOfMonth takes the first of a month to its last day, whatever its length.
|
||||
func endOfMonth(first time.Time) time.Time { return first.AddDate(0, 1, -1) }
|
||||
@@ -0,0 +1,102 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func on(s string) time.Time {
|
||||
t, err := time.Parse("2006-01-02", s)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
// The axis opens on the month that has just ended, whatever the index holds.
|
||||
func TestPeriodsDefaultsToLastMonth(t *testing.T) {
|
||||
ps := Periods(on("2026-08-23"), []string{"2026-08", "2026-07", "2026-06"})
|
||||
p := ps[DefaultIndex(ps)]
|
||||
if p.Label != LastMonth {
|
||||
t.Fatalf("default = %q, want %q", p.Label, LastMonth)
|
||||
}
|
||||
if p.From != "2026-07-01" || p.To != "2026-07-31" {
|
||||
t.Errorf("last month = %s…%s, want the whole of July", p.From, p.To)
|
||||
}
|
||||
if p.Span != "2026-07" {
|
||||
t.Errorf("span = %q, want the month it stands for", p.Span)
|
||||
}
|
||||
}
|
||||
|
||||
// An index that stops short of last month still opens there: an empty report
|
||||
// for the window asked for beats a full one for a window nobody chose.
|
||||
func TestPeriodsDefaultsToLastMonthWithNothingInIt(t *testing.T) {
|
||||
ps := Periods(on("2026-08-23"), []string{"2026-03", "2026-02"})
|
||||
if p := ps[DefaultIndex(ps)]; p.Label != LastMonth {
|
||||
t.Fatalf("default = %q, want %q", p.Label, LastMonth)
|
||||
}
|
||||
}
|
||||
|
||||
// The named windows come first, widest to narrowest, and the months follow in
|
||||
// reverse order, skipping the ones the named windows already name.
|
||||
func TestPeriodsOrder(t *testing.T) {
|
||||
ps := Periods(on("2026-08-23"), []string{"2026-09", "2026-08", "2026-07", "2026-06", "2026-01"})
|
||||
var got []string
|
||||
for _, p := range ps {
|
||||
got = append(got, p.Label)
|
||||
}
|
||||
want := []string{
|
||||
"all time", "this year", "last 12 months", "last 3 months",
|
||||
"this month", LastMonth, "2026-06", "2026-01",
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("periods = %v, want %v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("periods = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A month is bounded by its own length, not by a fixed 30 or 31 days.
|
||||
func TestPeriodsMonthEnds(t *testing.T) {
|
||||
ps := Periods(on("2026-03-15"), []string{"2026-02"})
|
||||
p := ps[DefaultIndex(ps)]
|
||||
if p.From != "2026-02-01" || p.To != "2026-02-28" {
|
||||
t.Errorf("February 2026 = %s…%s, want the 1st to the 28th", p.From, p.To)
|
||||
}
|
||||
}
|
||||
|
||||
// A rolling window runs to the end of this month rather than to the last
|
||||
// complete one: it is asked in order to see what is happening now.
|
||||
func TestPeriodsRollingWindows(t *testing.T) {
|
||||
ps := Periods(on("2026-08-23"), nil)
|
||||
for _, tc := range []struct{ label, from, to string }{
|
||||
{"last 3 months", "2026-06-01", "2026-08-31"},
|
||||
{"last 12 months", "2025-09-01", "2026-08-31"},
|
||||
{"this year", "2026-01-01", "2026-12-31"},
|
||||
} {
|
||||
var found bool
|
||||
for _, p := range ps {
|
||||
if p.Label != tc.label {
|
||||
continue
|
||||
}
|
||||
found = true
|
||||
if p.From != tc.from || p.To != tc.to {
|
||||
t.Errorf("%s = %s…%s, want %s…%s", tc.label, p.From, p.To, tc.from, tc.to)
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Errorf("no %q window", tc.label)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// All time is unbounded on both sides, so nothing has to special-case it.
|
||||
func TestPeriodsAllTimeIsUnbounded(t *testing.T) {
|
||||
ps := Periods(on("2026-08-23"), []string{"2026-01"})
|
||||
if ps[0].Label != "all time" || ps[0].From != "" || ps[0].To != "" {
|
||||
t.Errorf("first window = %+v, want an unbounded all time", ps[0])
|
||||
}
|
||||
}
|
||||
+214
-21
@@ -1,11 +1,10 @@
|
||||
// Package report aggregates transactions for the summary views.
|
||||
//
|
||||
// Transfers between the user's own accounts are excluded everywhere: moving
|
||||
// money from checking to savings is not income and not spending.
|
||||
package report
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/model"
|
||||
)
|
||||
@@ -27,16 +26,125 @@ func (t TagTotal) Net() int64 { return t.In - t.Out }
|
||||
// collected under "(untagged)" so they stay visible instead of vanishing.
|
||||
const Untagged = "(untagged)"
|
||||
|
||||
// ByTag returns totals sorted by currency, then by largest outflow first.
|
||||
func ByTag(txns []model.Transaction) []TagTotal {
|
||||
// Order is how the rows of a breakdown are arranged within a currency.
|
||||
//
|
||||
// Currency is always the outer key and no order can change that: the tool holds
|
||||
// no exchange rates, so two currencies interleaved by amount would invite a
|
||||
// comparison between numbers that cannot be compared.
|
||||
type Order int
|
||||
|
||||
const (
|
||||
// OrderOut is the default, and the question a spending report is usually
|
||||
// opened to answer: where did the money go.
|
||||
OrderOut Order = iota // largest outflow first
|
||||
OrderIn // largest inflow first
|
||||
OrderNet // lowest net first, so the biggest losses lead
|
||||
OrderCount // most transactions first
|
||||
OrderTag // tag name, A→Z
|
||||
|
||||
numOrders // how many there are, for cycling; not an order itself
|
||||
)
|
||||
|
||||
// Next is the order after o, wrapping round. There are too many for a toggle,
|
||||
// so a view offering the choice cycles through them.
|
||||
func (o Order) Next() Order { return (o + 1) % numOrders }
|
||||
|
||||
// String is the short name the `--sort` flag takes, and ParseOrder accepts.
|
||||
func (o Order) String() string {
|
||||
switch o {
|
||||
case OrderIn:
|
||||
return "in"
|
||||
case OrderNet:
|
||||
return "net"
|
||||
case OrderCount:
|
||||
return "count"
|
||||
case OrderTag:
|
||||
return "tag"
|
||||
}
|
||||
return "out"
|
||||
}
|
||||
|
||||
// Label says what the order actually does, for a view with room to say it.
|
||||
func (o Order) Label() string {
|
||||
switch o {
|
||||
case OrderIn:
|
||||
return "largest in first"
|
||||
case OrderNet:
|
||||
return "lowest net first"
|
||||
case OrderCount:
|
||||
return "most transactions first"
|
||||
case OrderTag:
|
||||
return "tag A→Z"
|
||||
}
|
||||
return "largest out first"
|
||||
}
|
||||
|
||||
// Column is the heading the order sorts on, so a view can mark the column the
|
||||
// rows are arranged by without keeping its own copy of this mapping.
|
||||
func (o Order) Column() string {
|
||||
switch o {
|
||||
case OrderIn:
|
||||
return "In"
|
||||
case OrderNet:
|
||||
return "Net"
|
||||
case OrderCount:
|
||||
return "N"
|
||||
case OrderTag:
|
||||
return "Tag"
|
||||
}
|
||||
return "Out"
|
||||
}
|
||||
|
||||
// Ascending reports which way the order runs. Only the tag reads naturally
|
||||
// upwards; every amount and count is asked for biggest-first.
|
||||
func (o Order) Ascending() bool { return o == OrderTag }
|
||||
|
||||
// Orders is every order, in the cycle Next follows.
|
||||
func Orders() []Order {
|
||||
out := make([]Order, 0, numOrders)
|
||||
for o := Order(0); o < numOrders; o++ {
|
||||
out = append(out, o)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ParseOrder turns a short name back into an order, naming the alternatives
|
||||
// when it cannot, since a misspelt sort would otherwise silently report in an
|
||||
// order the caller did not ask for.
|
||||
func ParseOrder(s string) (Order, error) {
|
||||
for _, o := range Orders() {
|
||||
if o.String() == s {
|
||||
return o, nil
|
||||
}
|
||||
}
|
||||
var names []string
|
||||
for _, o := range Orders() {
|
||||
names = append(names, o.String())
|
||||
}
|
||||
return OrderOut, fmt.Errorf("unknown sort %q: want one of %s", s, strings.Join(names, ", "))
|
||||
}
|
||||
|
||||
// ByTag returns totals grouped by currency and arranged within each by order.
|
||||
//
|
||||
// Matched transfer legs are left out entirely. Both legs of a transfer are
|
||||
// dropped together, so a report over everything is unchanged in total by money
|
||||
// the user moved between their own accounts; a report scoped to one account or
|
||||
// one month may see only one leg, and dropping it is still right, since the
|
||||
// money was not spent. What was left out is available from Excluded.
|
||||
//
|
||||
// A pair whose definition sets a tolerance_pct is dropped the same way, fee and
|
||||
// all — so with such a definition in play the report *is* short by the fee, and
|
||||
// callers are expected to show Excluded alongside it rather than leave that
|
||||
// difference unexplained.
|
||||
func ByTag(txns []model.Transaction, order Order) []TagTotal {
|
||||
type key struct{ currency, tag string }
|
||||
acc := map[key]*TagTotal{}
|
||||
|
||||
for _, t := range txns {
|
||||
if t.IsTransfer() {
|
||||
if t.IsTransferLeg() {
|
||||
continue
|
||||
}
|
||||
tag := t.Tag()
|
||||
tag := t.RuleTag
|
||||
if tag == "" {
|
||||
tag = Untagged
|
||||
}
|
||||
@@ -59,13 +167,33 @@ func ByTag(txns []model.Transaction) []TagTotal {
|
||||
out = append(out, *row)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].Currency != out[j].Currency {
|
||||
return out[i].Currency < out[j].Currency
|
||||
a, b := out[i], out[j]
|
||||
if a.Currency != b.Currency {
|
||||
return a.Currency < b.Currency
|
||||
}
|
||||
if out[i].Out != out[j].Out {
|
||||
return out[i].Out > out[j].Out
|
||||
switch order {
|
||||
case OrderIn:
|
||||
if a.In != b.In {
|
||||
return a.In > b.In
|
||||
}
|
||||
return out[i].Tag < out[j].Tag
|
||||
case OrderNet:
|
||||
if a.Net() != b.Net() {
|
||||
return a.Net() < b.Net()
|
||||
}
|
||||
case OrderCount:
|
||||
if a.Count != b.Count {
|
||||
return a.Count > b.Count
|
||||
}
|
||||
case OrderTag:
|
||||
// The tag is the whole order here, and also the tie-break below.
|
||||
default:
|
||||
if a.Out != b.Out {
|
||||
return a.Out > b.Out
|
||||
}
|
||||
}
|
||||
// Every order falls back to the tag, so rows that tie on the chosen
|
||||
// column keep a fixed position instead of shuffling between reloads.
|
||||
return a.Tag < b.Tag
|
||||
})
|
||||
return out
|
||||
}
|
||||
@@ -102,18 +230,83 @@ func Totals(rows []TagTotal) []CurrencyTotal {
|
||||
return out
|
||||
}
|
||||
|
||||
// Months lists the distinct YYYY-MM present in txns, most recent first.
|
||||
func Months(txns []model.Transaction) []string {
|
||||
seen := map[string]bool{}
|
||||
// TransferTotal is what ByTag left out for one currency.
|
||||
//
|
||||
// It counts legs rather than pairs on purpose: a report scoped to one account
|
||||
// or month usually holds only one side of a movement, and claiming a whole
|
||||
// transfer was excluded when only half of it was in view would be a lie. The
|
||||
// same goes for the two directions, which is why they are separate totals: an
|
||||
// exchange between currencies has its legs in different columns entirely, so
|
||||
// one currency can see only what left and the other only what arrived.
|
||||
type TransferTotal struct {
|
||||
Currency string
|
||||
Out int64 // sum of the leaving legs in view, positive
|
||||
In int64 // sum of the arriving legs in view
|
||||
Legs int
|
||||
Digits int
|
||||
// Fee is what those movements cost: the amount that left less the amount
|
||||
// that arrived, for pairs whose legs are *both* in view. A definition with a
|
||||
// tolerance_pct pairs legs that disagree, and since the pair leaves the
|
||||
// report entirely, this is the only place that difference is still money
|
||||
// rather than nothing. Zero when no definition allows a mismatch.
|
||||
Fee int64
|
||||
// Pairs counts the movements Fee was computed from — complete pairs in this
|
||||
// currency, not legs. A report scoped to one account or month usually holds
|
||||
// one side of a movement, and half a pair cannot say what the other half
|
||||
// received, so it contributes to Legs and Out/In but not here.
|
||||
Pairs int
|
||||
}
|
||||
|
||||
// Excluded summarises the transfer legs ByTag dropped, so a report can account
|
||||
// for the difference between its total and the account balances.
|
||||
func Excluded(txns []model.Transaction) []TransferTotal {
|
||||
// Both legs of a pair carry the same TransferID, so the ones whose partner
|
||||
// is also in view can be found without going back to the index.
|
||||
byID := map[int64][]model.Transaction{}
|
||||
for _, t := range txns {
|
||||
if len(t.Date) >= 7 {
|
||||
seen[t.Date[:7]] = true
|
||||
if t.IsTransferLeg() {
|
||||
byID[*t.TransferID] = append(byID[*t.TransferID], t)
|
||||
}
|
||||
}
|
||||
out := make([]string, 0, len(seen))
|
||||
for m := range seen {
|
||||
out = append(out, m)
|
||||
|
||||
acc := map[string]*TransferTotal{}
|
||||
row := func(t model.Transaction) *TransferTotal {
|
||||
r, ok := acc[t.Currency]
|
||||
if !ok {
|
||||
r = &TransferTotal{Currency: t.Currency, Digits: t.MinorDigits}
|
||||
acc[t.Currency] = r
|
||||
}
|
||||
sort.Sort(sort.Reverse(sort.StringSlice(out)))
|
||||
return r
|
||||
}
|
||||
|
||||
for _, t := range txns {
|
||||
if !t.IsTransferLeg() {
|
||||
continue
|
||||
}
|
||||
r := row(t)
|
||||
r.Legs++
|
||||
if t.AmountMinor < 0 {
|
||||
r.Out += -t.AmountMinor
|
||||
} else {
|
||||
r.In += t.AmountMinor
|
||||
}
|
||||
}
|
||||
|
||||
for _, legs := range byID {
|
||||
// An exchange has its legs in two currencies, and there is no rate here
|
||||
// to subtract one from the other, so it has no fee to report.
|
||||
if len(legs) != 2 || legs[0].Currency != legs[1].Currency {
|
||||
continue
|
||||
}
|
||||
r := row(legs[0])
|
||||
r.Pairs++
|
||||
r.Fee -= legs[0].AmountMinor + legs[1].AmountMinor
|
||||
}
|
||||
|
||||
out := make([]TransferTotal, 0, len(acc))
|
||||
for _, r := range acc {
|
||||
out = append(out, *r)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Currency < out[j].Currency })
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
package report
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/model"
|
||||
)
|
||||
|
||||
func leg(id int64, amount int64, transferID *int64) model.Transaction {
|
||||
return model.Transaction{
|
||||
ID: id,
|
||||
AccountSlug: "nlb",
|
||||
Currency: "EUR",
|
||||
MinorDigits: 2,
|
||||
Date: "2026-03-06",
|
||||
AmountMinor: amount,
|
||||
RuleTag: "moving",
|
||||
TransferID: transferID,
|
||||
}
|
||||
}
|
||||
|
||||
// A matched transfer is money moved, not money spent, so neither leg reaches
|
||||
// the report at all.
|
||||
func TestByTagLeavesTransferLegsOut(t *testing.T) {
|
||||
id := int64(1)
|
||||
rows := ByTag([]model.Transaction{
|
||||
leg(1, -50000, &id),
|
||||
leg(2, 50000, &id),
|
||||
{Currency: "EUR", MinorDigits: 2, AmountMinor: -2000, RuleTag: "groceries"},
|
||||
}, OrderOut)
|
||||
|
||||
if len(rows) != 1 || rows[0].Tag != "groceries" {
|
||||
t.Fatalf("rows = %+v, want only the groceries row", rows)
|
||||
}
|
||||
if rows[0].Out != 2000 {
|
||||
t.Errorf("out = %d, want only the shopping", rows[0].Out)
|
||||
}
|
||||
}
|
||||
|
||||
// An unpaired leg is not a transfer, so it still counts. Money that left an
|
||||
// account and cannot be shown to have arrived must not quietly vanish from the
|
||||
// report.
|
||||
func TestByTagKeepsUnpairedLegs(t *testing.T) {
|
||||
rows := ByTag([]model.Transaction{leg(1, -50000, nil)}, OrderOut)
|
||||
if len(rows) != 1 || rows[0].Out != 50000 {
|
||||
t.Fatalf("rows = %+v, want the unpaired leg counted", rows)
|
||||
}
|
||||
}
|
||||
|
||||
// A report scoped to one account or month usually holds one side of a
|
||||
// movement, so the summary counts legs rather than claiming whole transfers.
|
||||
func TestExcludedCountsLegsInView(t *testing.T) {
|
||||
id := int64(1)
|
||||
got := Excluded([]model.Transaction{
|
||||
leg(1, -50000, &id),
|
||||
{Currency: "EUR", MinorDigits: 2, AmountMinor: -2000, RuleTag: "groceries"},
|
||||
})
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("excluded = %+v, want one currency", got)
|
||||
}
|
||||
if got[0].Legs != 1 || got[0].Out != 50000 || got[0].In != 0 || got[0].Currency != "EUR" {
|
||||
t.Errorf("excluded = %+v, want 1 leg and 50000 out in EUR", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
// An exchange has its legs in two currencies, so each side reports only what it
|
||||
// saw. Summing the two would be meaningless: there are no rates here.
|
||||
func TestExcludedSplitsAnExchangeByCurrency(t *testing.T) {
|
||||
id := int64(1)
|
||||
arriving := leg(2, 97790, &id)
|
||||
arriving.Currency = "BGN"
|
||||
|
||||
got := Excluded([]model.Transaction{leg(1, -50000, &id), arriving})
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("excluded = %+v, want a row per currency", got)
|
||||
}
|
||||
bgn, eur := got[0], got[1]
|
||||
if bgn.Currency != "BGN" || bgn.In != 97790 || bgn.Out != 0 {
|
||||
t.Errorf("BGN row = %+v, want only the arriving leg", bgn)
|
||||
}
|
||||
if eur.Currency != "EUR" || eur.Out != 50000 || eur.In != 0 {
|
||||
t.Errorf("EUR row = %+v, want only the leaving leg", eur)
|
||||
}
|
||||
}
|
||||
|
||||
// A definition with a tolerance pairs legs that disagree, and the pair leaves
|
||||
// the report with the difference inside it. That difference is real money, so
|
||||
// the summary has to name it or it is simply lost.
|
||||
func TestExcludedReportsTheFee(t *testing.T) {
|
||||
id := int64(1)
|
||||
got := Excluded([]model.Transaction{leg(1, -50000, &id), leg(2, 49500, &id)})
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("excluded = %+v, want one currency", got)
|
||||
}
|
||||
if got[0].Fee != 500 || got[0].Pairs != 1 || got[0].Legs != 2 {
|
||||
t.Errorf("excluded = %+v, want a 5.00 fee over 1 pair and 2 legs", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Half a pair cannot say what the other half received, so a report scoped to
|
||||
// one account or month reports the leg it saw and claims no fee at all.
|
||||
func TestExcludedClaimsNoFeeFromHalfAPair(t *testing.T) {
|
||||
id := int64(1)
|
||||
got := Excluded([]model.Transaction{leg(1, -50000, &id)})
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("excluded = %+v, want one currency", got)
|
||||
}
|
||||
if got[0].Fee != 0 || got[0].Pairs != 0 || got[0].Legs != 1 {
|
||||
t.Errorf("excluded = %+v, want no fee and no complete pair", got[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Without a tolerance the two legs are exact opposites, so there is nothing to
|
||||
// report and the line stays as it was.
|
||||
func TestExcludedReportsNoFeeForAnExactPair(t *testing.T) {
|
||||
id := int64(1)
|
||||
got := Excluded([]model.Transaction{leg(1, -50000, &id), leg(2, 50000, &id)})
|
||||
if len(got) != 1 || got[0].Fee != 0 || got[0].Pairs != 1 {
|
||||
t.Errorf("excluded = %+v, want one pair and no fee", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The legs of an exchange are in different units, so subtracting one from the
|
||||
// other would produce a number meaning nothing.
|
||||
func TestExcludedClaimsNoFeeAcrossCurrencies(t *testing.T) {
|
||||
id := int64(1)
|
||||
arriving := leg(2, 97790, &id)
|
||||
arriving.Currency = "BGN"
|
||||
|
||||
for _, row := range Excluded([]model.Transaction{leg(1, -50000, &id), arriving}) {
|
||||
if row.Fee != 0 || row.Pairs != 0 {
|
||||
t.Errorf("%s row = %+v, want no fee across currencies", row.Currency, row)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// row is a transaction under one tag, for the ordering tests.
|
||||
func row(tag string, amount int64) model.Transaction {
|
||||
return model.Transaction{Currency: "EUR", MinorDigits: 2, AmountMinor: amount, RuleTag: tag}
|
||||
}
|
||||
|
||||
// tags is the order the breakdown came out in.
|
||||
func tags(rows []TagTotal) []string {
|
||||
var out []string
|
||||
for _, r := range rows {
|
||||
out = append(out, r.Tag)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestByTagOrders(t *testing.T) {
|
||||
txns := []model.Transaction{
|
||||
row("rent", -100000),
|
||||
row("groceries", -2000),
|
||||
row("groceries", -3000),
|
||||
row("groceries", -1000),
|
||||
row("salary", 250000),
|
||||
row("refund", 500),
|
||||
row("refund", -400),
|
||||
}
|
||||
|
||||
for _, tc := range []struct {
|
||||
order Order
|
||||
want []string
|
||||
}{
|
||||
{OrderOut, []string{"rent", "groceries", "refund", "salary"}},
|
||||
{OrderIn, []string{"salary", "refund", "groceries", "rent"}},
|
||||
{OrderNet, []string{"rent", "groceries", "refund", "salary"}},
|
||||
{OrderCount, []string{"groceries", "refund", "rent", "salary"}},
|
||||
{OrderTag, []string{"groceries", "refund", "rent", "salary"}},
|
||||
} {
|
||||
got := tags(ByTag(txns, tc.order))
|
||||
if len(got) != len(tc.want) {
|
||||
t.Errorf("%s: rows = %v, want %v", tc.order, got, tc.want)
|
||||
continue
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != tc.want[i] {
|
||||
t.Errorf("%s: rows = %v, want %v", tc.order, got, tc.want)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Every order falls back to the tag, so rows that tie do not shuffle about
|
||||
// between one reload and the next.
|
||||
func TestByTagTiesBreakOnTheTag(t *testing.T) {
|
||||
txns := []model.Transaction{row("zebra", -1000), row("apple", -1000)}
|
||||
for _, o := range Orders() {
|
||||
if got := tags(ByTag(txns, o)); got[0] != "apple" {
|
||||
t.Errorf("%s: rows = %v, want the tie broken on the tag", o, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// No order may interleave currencies: with no exchange rates here, two amounts
|
||||
// in different columns are not comparable and must not be sorted against each
|
||||
// other.
|
||||
func TestByTagNeverInterleavesCurrencies(t *testing.T) {
|
||||
txns := []model.Transaction{
|
||||
{Currency: "EUR", MinorDigits: 2, AmountMinor: -100, RuleTag: "small"},
|
||||
{Currency: "BGN", MinorDigits: 2, AmountMinor: -900000, RuleTag: "huge"},
|
||||
{Currency: "EUR", MinorDigits: 2, AmountMinor: -200, RuleTag: "medium"},
|
||||
}
|
||||
for _, o := range Orders() {
|
||||
rows := ByTag(txns, o)
|
||||
if rows[0].Currency != "BGN" || rows[1].Currency != "EUR" || rows[2].Currency != "EUR" {
|
||||
t.Errorf("%s: currencies = %+v, want them grouped", o, rows)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The cycle visits every order and comes back to where it started.
|
||||
func TestOrderNextCycles(t *testing.T) {
|
||||
seen := map[Order]bool{}
|
||||
o := OrderOut
|
||||
for range len(Orders()) {
|
||||
if seen[o] {
|
||||
t.Fatalf("order %s came round twice in one cycle", o)
|
||||
}
|
||||
seen[o] = true
|
||||
o = o.Next()
|
||||
}
|
||||
if o != OrderOut {
|
||||
t.Errorf("the cycle ended on %s, want it back at %s", o, OrderOut)
|
||||
}
|
||||
}
|
||||
|
||||
// A misspelt --sort is refused rather than silently reporting in the default
|
||||
// order, and the error names what it would have accepted.
|
||||
func TestParseOrder(t *testing.T) {
|
||||
for _, o := range Orders() {
|
||||
got, err := ParseOrder(o.String())
|
||||
if err != nil || got != o {
|
||||
t.Errorf("ParseOrder(%q) = %s, %v, want %s", o.String(), got, err, o)
|
||||
}
|
||||
}
|
||||
_, err := ParseOrder("descending")
|
||||
if err == nil {
|
||||
t.Fatal("ParseOrder accepted a name that is not an order")
|
||||
}
|
||||
for _, want := range []string{"descending", "out", "tag"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("error %q does not mention %q", err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Each order marks a different column, and only headings the table actually
|
||||
// has, or the marker would land nowhere.
|
||||
func TestOrderColumnsAreDistinct(t *testing.T) {
|
||||
headings := map[string]bool{"Tag": true, "Cur": true, "Out": true, "In": true, "Net": true, "N": true}
|
||||
seen := map[string]bool{}
|
||||
for _, o := range Orders() {
|
||||
c := o.Column()
|
||||
if !headings[c] {
|
||||
t.Errorf("%s sorts on %q, which is not a report column", o, c)
|
||||
}
|
||||
if seen[c] {
|
||||
t.Errorf("%s marks %q, which another order already marks", o, c)
|
||||
}
|
||||
seen[c] = true
|
||||
}
|
||||
}
|
||||
+94
-32
@@ -1,29 +1,94 @@
|
||||
// Package rules applies the ordered glob rules from rules.toml to
|
||||
// transactions, deciding their automatic tag and whether they are a transfer
|
||||
// between the user's own accounts.
|
||||
//
|
||||
// Only the rule_* columns are ever written. Manual edits made in the TUI live
|
||||
// in separate columns and survive any number of re-runs.
|
||||
// Package rules applies the glob rules from rules.toml to transactions,
|
||||
// deciding their tag. The most specific rule that fits wins, so a general rule
|
||||
// and the narrower one carving an exception out of it can be written in either
|
||||
// order. It is the only thing that decides a tag, so rule_tag is derived state
|
||||
// and Retag can rewrite it wholesale at any time.
|
||||
package rules
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
"git.petrovv.com/nikola/money/internal/glob"
|
||||
"git.petrovv.com/nikola/money/internal/model"
|
||||
"git.petrovv.com/nikola/money/internal/store"
|
||||
)
|
||||
|
||||
// Engine evaluates rules in file order; the first match wins.
|
||||
// Engine evaluates rules most specific first; the first match wins.
|
||||
type Engine struct {
|
||||
// rules is file order, which is what Rules and Usage report and what the
|
||||
// rules screen numbers and deletes by. Evaluation does not use it.
|
||||
rules []config.Rule
|
||||
// order indexes into rules, most specific first. Specificity decides
|
||||
// precedence rather than position, so "*NIKOLA*" claims what it names
|
||||
// wherever it sits relative to the "*NIK*" that would otherwise swallow it.
|
||||
order []int
|
||||
}
|
||||
|
||||
// New builds an engine from the parsed rules file.
|
||||
// New builds an engine from the parsed rules file, working out once which rule
|
||||
// is tried before which.
|
||||
func New(r *config.Rules) *Engine {
|
||||
return &Engine{rules: r.Rule}
|
||||
e := &Engine{rules: r.Rule, order: make([]int, len(r.Rule))}
|
||||
for i := range e.order {
|
||||
e.order[i] = i
|
||||
}
|
||||
// Stable, so rules of equal specificity keep file order between them and
|
||||
// the earlier one still wins.
|
||||
sort.SliceStable(e.order, func(a, b int) bool {
|
||||
return moreSpecific(e.rules[e.order[a]], e.rules[e.order[b]])
|
||||
})
|
||||
return e
|
||||
}
|
||||
|
||||
// Rules returns the ordered rules, as loaded from rules.toml.
|
||||
// moreSpecific reports whether a is tried before b.
|
||||
//
|
||||
// The literal characters a rule spells out are the evidence: they are what it
|
||||
// commits to, and what it hands to a '*' is what it gives up. So "*NIKOLA*"
|
||||
// beats "*NIK*", and a bare "*" sits last of all — the catch-all can be written
|
||||
// anywhere in the file and still catch only what nothing else wanted.
|
||||
//
|
||||
// Two rules spelling out the same amount are separated by what else they pin
|
||||
// down: fewer '*' first, since "NIKOLA" also fixes both ends where "*NIKOLA*"
|
||||
// does not, and then an account-scoped rule over one that applies everywhere.
|
||||
// Rules that tie on all three are left to file order by the stable sort.
|
||||
func moreSpecific(a, b config.Rule) bool {
|
||||
if x, y := literals(a), literals(b); x != y {
|
||||
return x > y
|
||||
}
|
||||
if x, y := stars(a), stars(b); x != y {
|
||||
return x < y
|
||||
}
|
||||
return a.Account != "" && b.Account == ""
|
||||
}
|
||||
|
||||
// literals counts the characters a rule pins down exactly, across every pattern
|
||||
// it sets: a rule with both match and type has to satisfy both, so both count.
|
||||
// '?' is not one of them — it fixes a length, not a character.
|
||||
func literals(r config.Rule) int {
|
||||
n := 0
|
||||
for _, c := range r.Match + r.Type {
|
||||
if c != '*' && c != '?' {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// stars counts the unbounded wildcards, the only ones that let a pattern match
|
||||
// a run of any length.
|
||||
func stars(r config.Rule) int {
|
||||
n := 0
|
||||
for _, c := range r.Match + r.Type {
|
||||
if c == '*' {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// Rules returns the rules in file order, as loaded from rules.toml. That is the
|
||||
// order the rules screen shows and deletes by; it is no longer the order they
|
||||
// are tried in.
|
||||
func (e *Engine) Rules() []config.Rule { return e.rules }
|
||||
|
||||
// Match returns the first rule matching a transaction on the given account, or
|
||||
@@ -35,20 +100,21 @@ func (e *Engine) Match(accountSlug string, t model.Transaction) *config.Rule {
|
||||
return nil
|
||||
}
|
||||
|
||||
// MatchIndex returns the position of the first rule matching a transaction, or
|
||||
// MatchIndex returns the file position of the rule claiming a transaction, or
|
||||
// -1 if none does. Every pattern a rule sets must match: a rule with both
|
||||
// match and counterparty is an "and", not an "or".
|
||||
// match and type is an "and", not an "or".
|
||||
//
|
||||
// The position matters as well as the rule: because the first match wins, a
|
||||
// rule that is fully shadowed by an earlier one never applies to anything, and
|
||||
// only the index reveals that.
|
||||
// Rules are tried most specific first, so the answer is the narrowest rule that
|
||||
// fits and not merely the topmost. The returned index is still the position in
|
||||
// rules.toml, because that is what the caller can point the user at — and it is
|
||||
// what reveals a rule that is fully shadowed by a more specific one and so
|
||||
// never applies to anything.
|
||||
func (e *Engine) MatchIndex(accountSlug string, t model.Transaction) int {
|
||||
var (
|
||||
description = model.NormalizeDescription(t.Description)
|
||||
counterparty = model.NormalizeDescription(t.Counterparty)
|
||||
kind = model.NormalizeDescription(t.Type)
|
||||
)
|
||||
for i := range e.rules {
|
||||
for _, i := range e.order {
|
||||
r := &e.rules[i]
|
||||
if r.Account != "" && r.Account != accountSlug {
|
||||
continue
|
||||
@@ -56,9 +122,6 @@ func (e *Engine) MatchIndex(accountSlug string, t model.Transaction) int {
|
||||
if r.Match != "" && !glob.Match(r.Match, description) {
|
||||
continue
|
||||
}
|
||||
if r.Counterparty != "" && !glob.Match(r.Counterparty, counterparty) {
|
||||
continue
|
||||
}
|
||||
if r.Type != "" && !glob.Match(r.Type, kind) {
|
||||
continue
|
||||
}
|
||||
@@ -67,9 +130,9 @@ func (e *Engine) MatchIndex(accountSlug string, t model.Transaction) int {
|
||||
return -1
|
||||
}
|
||||
|
||||
// Usage counts how many transactions each rule actually claims. A rule with a
|
||||
// count of zero is dead: either nothing matches it, or an earlier rule takes
|
||||
// everything it would have caught.
|
||||
// Usage counts how many transactions each rule actually claims, indexed by file
|
||||
// position. A rule with a count of zero is dead: either nothing matches it, or
|
||||
// a more specific rule takes everything it would have caught.
|
||||
func (e *Engine) Usage(txns []model.Transaction) []int {
|
||||
counts := make([]int, len(e.rules))
|
||||
for _, t := range txns {
|
||||
@@ -80,17 +143,16 @@ func (e *Engine) Usage(txns []model.Transaction) []int {
|
||||
return counts
|
||||
}
|
||||
|
||||
// ApplyTxn returns the tag and transfer flag for a transaction. An unmatched
|
||||
// transaction gets an empty tag and is not a transfer.
|
||||
func (e *Engine) ApplyTxn(accountSlug string, t model.Transaction) (tag string, transfer bool) {
|
||||
// ApplyTxn returns the tag for a transaction, empty if no rule matches.
|
||||
func (e *Engine) ApplyTxn(accountSlug string, t model.Transaction) string {
|
||||
if r := e.Match(accountSlug, t); r != nil {
|
||||
return r.Tag, r.Transfer
|
||||
return r.Tag
|
||||
}
|
||||
return "", false
|
||||
return ""
|
||||
}
|
||||
|
||||
// Apply is the description-only shorthand, for callers that have nothing else.
|
||||
func (e *Engine) Apply(accountSlug, description string) (tag string, transfer bool) {
|
||||
func (e *Engine) Apply(accountSlug, description string) string {
|
||||
return e.ApplyTxn(accountSlug, model.Transaction{Description: description})
|
||||
}
|
||||
|
||||
@@ -103,11 +165,11 @@ func (e *Engine) Retag(db *store.DB) (int, error) {
|
||||
}
|
||||
var changed []store.RuleAssignment
|
||||
for _, t := range txns {
|
||||
tag, transfer := e.ApplyTxn(t.AccountSlug, t)
|
||||
if tag == t.RuleTag && transfer == t.RuleTransfer {
|
||||
tag := e.ApplyTxn(t.AccountSlug, t)
|
||||
if tag == t.RuleTag {
|
||||
continue
|
||||
}
|
||||
changed = append(changed, store.RuleAssignment{ID: t.ID, Tag: tag, Transfer: transfer})
|
||||
changed = append(changed, store.RuleAssignment{ID: t.ID, Tag: tag})
|
||||
}
|
||||
if len(changed) == 0 {
|
||||
return 0, nil
|
||||
|
||||
+99
-143
@@ -28,7 +28,7 @@ func seed(t *testing.T, db *store.DB, descriptions ...string) int64 {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sourceID, err := db.SourceFile(accountID, "checking/st.csv", "sha", "2026-01-01T00:00:00Z")
|
||||
sourceID, err := db.SourceFile(accountID, "checking/st.csv", "sha", "2026-01-01T00:00:00Z", 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -66,92 +66,114 @@ func byDescription(t *testing.T, db *store.DB, desc string) model.Transaction {
|
||||
return model.Transaction{}
|
||||
}
|
||||
|
||||
// A manual tag must outlive any number of rule changes and re-runs. This is
|
||||
// the property that makes retag safe to run at will.
|
||||
func TestManualTagSurvivesRetag(t *testing.T) {
|
||||
// A tag is derived state: rules.toml is the only thing that decides it, so
|
||||
// retagging rewrites every tag from the current rules however often it runs,
|
||||
// including back to nothing when no rule matches any more.
|
||||
func TestRetagRewritesEveryTag(t *testing.T) {
|
||||
db := testDB(t)
|
||||
seed(t, db, "LIDL SOFIA", "SOME RANDOM SHOP")
|
||||
|
||||
engine := New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*LIDL*", Tag: "groceries"},
|
||||
}})
|
||||
if _, err := engine.Retag(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// The user hand-tags the row the rules did not match, and overrides the
|
||||
// one they did.
|
||||
shop := byDescription(t, db, "SOME RANDOM SHOP")
|
||||
if err := db.SetManualTag(shop.ID, "hobby"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lidl := byDescription(t, db, "LIDL SOFIA")
|
||||
if err := db.SetManualTag(lidl.ID, "party supplies"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Rules change completely, and are re-applied twice for good measure.
|
||||
engine = New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*LIDL*", Tag: "food"},
|
||||
{Match: "*SHOP*", Tag: "shopping"},
|
||||
}})
|
||||
for range 2 {
|
||||
n, err := engine.Retag(db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n != 2 {
|
||||
t.Errorf("retagged %d rows, want 2", n)
|
||||
}
|
||||
if got := byDescription(t, db, "LIDL SOFIA").RuleTag; got != "groceries" {
|
||||
t.Errorf("tag = %q, want groceries", got)
|
||||
}
|
||||
|
||||
// Re-running with the same rules changes nothing and says so.
|
||||
if n, err := engine.Retag(db); err != nil || n != 0 {
|
||||
t.Errorf("second retag: %d rows, %v; want 0 and no error", n, err)
|
||||
}
|
||||
|
||||
// New rules win, and a row no rule matches loses its tag entirely.
|
||||
engine = New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*LIDL*", Tag: "food"},
|
||||
}})
|
||||
if _, err := engine.Retag(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := byDescription(t, db, "LIDL SOFIA").RuleTag; got != "food" {
|
||||
t.Errorf("tag = %q, want food", got)
|
||||
}
|
||||
|
||||
shop = byDescription(t, db, "SOME RANDOM SHOP")
|
||||
if got := shop.Tag(); got != "hobby" {
|
||||
t.Errorf("manual tag = %q, want %q (manual tags must win)", got, "hobby")
|
||||
}
|
||||
if shop.RuleTag != "shopping" {
|
||||
t.Errorf("rule_tag = %q, want %q (rules must still be recorded underneath)", shop.RuleTag, "shopping")
|
||||
}
|
||||
|
||||
lidl = byDescription(t, db, "LIDL SOFIA")
|
||||
if got := lidl.Tag(); got != "party supplies" {
|
||||
t.Errorf("manual override = %q, want %q", got, "party supplies")
|
||||
}
|
||||
|
||||
// Clearing the override falls back to whatever the rules now say.
|
||||
if err := db.ClearOverrides(lidl.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := byDescription(t, db, "LIDL SOFIA").Tag(); got != "food" {
|
||||
t.Errorf("after clearing override, tag = %q, want %q", got, "food")
|
||||
if got := byDescription(t, db, "SOME RANDOM SHOP").RuleTag; got != "" {
|
||||
t.Errorf("tag = %q, want empty once no rule matches", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirstMatchWins(t *testing.T) {
|
||||
// The narrower rule claims what it names wherever it sits in the file: the
|
||||
// catch-all above it must not swallow the exception written below.
|
||||
func TestMoreSpecificRuleWins(t *testing.T) {
|
||||
both := [][]config.Rule{
|
||||
{{Match: "NIK*", Tag: "misc"}, {Match: "NIKOLA*", Tag: "family"}},
|
||||
{{Match: "NIKOLA*", Tag: "family"}, {Match: "NIK*", Tag: "misc"}},
|
||||
}
|
||||
for _, rs := range both {
|
||||
engine := New(&config.Rules{Rule: rs})
|
||||
if tag := engine.Apply("checking", "NIKOLA PETROV"); tag != "family" {
|
||||
t.Errorf("%q first: tag = %q, want family", rs[0].Match, tag)
|
||||
}
|
||||
if tag := engine.Apply("checking", "NIKI TODOROV"); tag != "misc" {
|
||||
t.Errorf("%q first: tag = %q, want misc", rs[0].Match, tag)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Specificity is measured on what the pattern spells out, so the wildcards
|
||||
// around a literal do not buy it precedence, and a bare "*" is always last.
|
||||
func TestSpecificityOrder(t *testing.T) {
|
||||
engine := New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*LIDL EXPRESS*", Tag: "snacks"},
|
||||
{Match: "*", Tag: "other"},
|
||||
{Match: "*LIDL*", Tag: "groceries"},
|
||||
{Match: "*LIDL EXPRESS*", Tag: "snacks"},
|
||||
}})
|
||||
if tag, _ := engine.Apply("checking", "CARD LIDL EXPRESS 12"); tag != "snacks" {
|
||||
t.Errorf("tag = %q, want snacks (earlier rule must win)", tag)
|
||||
for _, c := range []struct{ desc, want string }{
|
||||
{"CARD LIDL EXPRESS 12", "snacks"},
|
||||
{"CARD LIDL 12", "groceries"},
|
||||
{"SOMETHING ELSE", "other"},
|
||||
} {
|
||||
if tag := engine.Apply("checking", c.desc); tag != c.want {
|
||||
t.Errorf("%q: tag = %q, want %q", c.desc, tag, c.want)
|
||||
}
|
||||
if tag, _ := engine.Apply("checking", "CARD LIDL 12"); tag != "groceries" {
|
||||
t.Errorf("tag = %q, want groceries", tag)
|
||||
}
|
||||
if tag, _ := engine.Apply("checking", "SOMETHING ELSE"); tag != "" {
|
||||
}
|
||||
|
||||
// Two rules that are equally specific are still decided by the file: the
|
||||
// earlier one wins, which is what makes appending a rule safe.
|
||||
func TestEqualSpecificityKeepsFileOrder(t *testing.T) {
|
||||
engine := New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*ACME*", Tag: "first"},
|
||||
{Match: "*ACME*", Tag: "second"},
|
||||
}})
|
||||
if tag := engine.Apply("checking", "ACME LTD"); tag != "first" {
|
||||
t.Errorf("tag = %q, want first", tag)
|
||||
}
|
||||
if tag := engine.Apply("checking", "SOMETHING ELSE"); tag != "" {
|
||||
t.Errorf("tag = %q, want empty for an unmatched description", tag)
|
||||
}
|
||||
}
|
||||
|
||||
// A rule can be dead two ways: nothing matches it, or an earlier rule already
|
||||
// claimed everything it would have caught. Usage must report both as zero.
|
||||
func TestUsageCountsFirstMatchOnly(t *testing.T) {
|
||||
// A rule can be dead two ways: nothing matches it, or a rule that beats it
|
||||
// already claimed everything it would have caught. Usage must report both as
|
||||
// zero, indexed by file position however the rules are ordered for matching.
|
||||
func TestUsageCountsWinnersOnly(t *testing.T) {
|
||||
engine := New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*LIDL*", Tag: "groceries"}, // claims both LIDL rows
|
||||
{Match: "*LIDL SOFIA*", Tag: "shadowed"}, // fully shadowed by the above
|
||||
{Match: "*LIDL SOFIA*", Tag: "groceries"}, // the narrower of the two
|
||||
{Match: "*LIDL*", Tag: "shadowed"}, // every LIDL row here is Sofia
|
||||
{Match: "*NOTHING MATCHES ME*", Tag: "no"}, // matches nothing at all
|
||||
{Match: "*PAYROLL*", Tag: "salary"}, // claims one row
|
||||
}})
|
||||
|
||||
txns := []model.Transaction{
|
||||
{AccountSlug: "checking", Description: "LIDL SOFIA 4412"},
|
||||
{AccountSlug: "checking", Description: "LIDL VARNA 9911"},
|
||||
{AccountSlug: "checking", Description: "LIDL SOFIA 9911"},
|
||||
{AccountSlug: "checking", Description: "ACME PAYROLL"},
|
||||
{AccountSlug: "checking", Description: "UNMATCHED SHOP"},
|
||||
}
|
||||
@@ -168,64 +190,57 @@ func TestUsageCountsFirstMatchOnly(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The index is the rule's position in rules.toml, not its position in the order
|
||||
// it was tried in — that is the number the rules screen shows and deletes by.
|
||||
func TestMatchIndex(t *testing.T) {
|
||||
engine := New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*LIDL EXPRESS*", Tag: "snacks"},
|
||||
{Match: "*LIDL*", Tag: "groceries"},
|
||||
{Match: "*LIDL EXPRESS*", Tag: "snacks"},
|
||||
}})
|
||||
|
||||
if got := engine.MatchIndex("checking", model.Transaction{Description: "LIDL EXPRESS 1"}); got != 0 {
|
||||
t.Errorf("index = %d, want 0", got)
|
||||
}
|
||||
if got := engine.MatchIndex("checking", model.Transaction{Description: "LIDL 1"}); got != 1 {
|
||||
if got := engine.MatchIndex("checking", model.Transaction{Description: "LIDL EXPRESS 1"}); got != 1 {
|
||||
t.Errorf("index = %d, want 1", got)
|
||||
}
|
||||
if got := engine.MatchIndex("checking", model.Transaction{Description: "LIDL 1"}); got != 0 {
|
||||
t.Errorf("index = %d, want 0", got)
|
||||
}
|
||||
if got := engine.MatchIndex("checking", model.Transaction{Description: "OTHER"}); got != -1 {
|
||||
t.Errorf("index = %d, want -1 for no match", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Naming an account is itself a narrowing, so the scoped rule beats the
|
||||
// identical unscoped one even when the unscoped one is written first.
|
||||
func TestAccountScopedRule(t *testing.T) {
|
||||
engine := New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*TRANSFER*", Tag: "transfer", Transfer: true, Account: "savings"},
|
||||
{Match: "*TRANSFER*", Tag: "misc"},
|
||||
{Match: "*TRANSFER*", Tag: "transfer", Account: "savings"},
|
||||
}})
|
||||
|
||||
tag, transfer := engine.Apply("savings", "TRANSFER FROM CHECKING")
|
||||
if tag != "transfer" || !transfer {
|
||||
t.Errorf("savings: tag=%q transfer=%v, want transfer/true", tag, transfer)
|
||||
if tag := engine.Apply("savings", "TRANSFER FROM CHECKING"); tag != "transfer" {
|
||||
t.Errorf("savings: tag = %q, want transfer", tag)
|
||||
}
|
||||
tag, transfer = engine.Apply("checking", "TRANSFER FROM CHECKING")
|
||||
if tag != "misc" || transfer {
|
||||
t.Errorf("checking: tag=%q transfer=%v, want misc/false", tag, transfer)
|
||||
if tag := engine.Apply("checking", "TRANSFER FROM CHECKING"); tag != "misc" {
|
||||
t.Errorf("checking: tag = %q, want misc", tag)
|
||||
}
|
||||
}
|
||||
|
||||
// Movements between the user's own accounts are often only identifiable by
|
||||
// the counterparty IBAN, so rules can match on it.
|
||||
func TestCounterpartyAndTypeRules(t *testing.T) {
|
||||
// Besides the description, a rule can match the bank's own classification of
|
||||
// the transaction, on its own or alongside a description glob.
|
||||
func TestTypeRules(t *testing.T) {
|
||||
engine := New(&config.Rules{Rule: []config.Rule{
|
||||
{Counterparty: "SI56123456789012345", Tag: "transfer", Transfer: true},
|
||||
{Type: "CARD_PAYMENT", Match: "*LIDL*", Tag: "groceries"},
|
||||
{Type: "ATM", Tag: "cash"},
|
||||
}})
|
||||
|
||||
// Counterparty alone is enough, whatever the description says.
|
||||
tag, transfer := engine.ApplyTxn("checking", model.Transaction{
|
||||
Description: "Standing order", Counterparty: "SI56123456789012345",
|
||||
})
|
||||
if tag != "transfer" || !transfer {
|
||||
t.Errorf("tag=%q transfer=%v, want transfer/true", tag, transfer)
|
||||
}
|
||||
|
||||
// A rule setting several patterns requires all of them to match.
|
||||
tag, _ = engine.ApplyTxn("checking", model.Transaction{
|
||||
tag := engine.ApplyTxn("checking", model.Transaction{
|
||||
Description: "LIDL SOFIA", Type: "CARD_PAYMENT",
|
||||
})
|
||||
if tag != "groceries" {
|
||||
t.Errorf("tag = %q, want groceries", tag)
|
||||
}
|
||||
tag, _ = engine.ApplyTxn("checking", model.Transaction{
|
||||
tag = engine.ApplyTxn("checking", model.Transaction{
|
||||
Description: "LIDL SOFIA", Type: "TRANSFER",
|
||||
})
|
||||
if tag != "" {
|
||||
@@ -233,68 +248,9 @@ func TestCounterpartyAndTypeRules(t *testing.T) {
|
||||
}
|
||||
|
||||
// Type alone works too.
|
||||
if tag, _ := engine.ApplyTxn("checking", model.Transaction{
|
||||
if tag := engine.ApplyTxn("checking", model.Transaction{
|
||||
Description: "WITHDRAWAL", Type: "ATM",
|
||||
}); tag != "cash" {
|
||||
t.Errorf("tag = %q, want cash", tag)
|
||||
}
|
||||
}
|
||||
|
||||
// Transfers can also be overridden by hand, independently of the tag.
|
||||
func TestManualTransferOverride(t *testing.T) {
|
||||
db := testDB(t)
|
||||
seed(t, db, "TRANSFER TO SAVINGS")
|
||||
|
||||
engine := New(&config.Rules{Rule: []config.Rule{
|
||||
{Match: "*TO SAVINGS*", Tag: "transfer", Transfer: true},
|
||||
}})
|
||||
if _, err := engine.Retag(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
txn := byDescription(t, db, "TRANSFER TO SAVINGS")
|
||||
if !txn.IsTransfer() {
|
||||
t.Fatal("expected the rule to mark this as a transfer")
|
||||
}
|
||||
|
||||
no := false
|
||||
if err := db.SetManualTransfer(txn.ID, &no); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := engine.Retag(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if byDescription(t, db, "TRANSFER TO SAVINGS").IsTransfer() {
|
||||
t.Error("manual transfer override was lost on retag")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUntaggedFilter(t *testing.T) {
|
||||
db := testDB(t)
|
||||
seed(t, db, "LIDL SOFIA", "SOME RANDOM SHOP")
|
||||
|
||||
engine := New(&config.Rules{Rule: []config.Rule{{Match: "*LIDL*", Tag: "groceries"}}})
|
||||
if _, err := engine.Retag(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
untagged, err := db.Transactions(store.Filter{Untagged: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(untagged) != 1 || untagged[0].Description != "SOME RANDOM SHOP" {
|
||||
t.Fatalf("untagged = %+v, want only SOME RANDOM SHOP", untagged)
|
||||
}
|
||||
|
||||
// A manual tag must remove the row from the untagged view.
|
||||
if err := db.SetManualTag(untagged[0].ID, "hobby"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
untagged, err = db.Transactions(store.Filter{Untagged: true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(untagged) != 0 {
|
||||
t.Errorf("untagged = %+v, want none", untagged)
|
||||
}
|
||||
}
|
||||
|
||||
+215
-129
@@ -1,6 +1,6 @@
|
||||
// Package store is the SQLite index over the statements. It is entirely
|
||||
// rebuildable: delete .money/index.db and re-import to get it back, except for
|
||||
// manual tags and manual transfer overrides, which live only here.
|
||||
// rebuildable: delete index.db and re-import to get it back. Nothing lives
|
||||
// only here.
|
||||
package store
|
||||
|
||||
import (
|
||||
@@ -37,6 +37,9 @@ CREATE TABLE IF NOT EXISTS source_files (
|
||||
path TEXT NOT NULL,
|
||||
sha256 TEXT NOT NULL,
|
||||
imported_at TEXT NOT NULL,
|
||||
-- How many transactions the statement holds, as parsed. Not how many it
|
||||
-- brought in: rows an overlapping statement already had are deduplicated.
|
||||
rows INTEGER NOT NULL DEFAULT 0,
|
||||
UNIQUE(account_id, path)
|
||||
);
|
||||
|
||||
@@ -48,77 +51,35 @@ CREATE TABLE IF NOT EXISTS transactions (
|
||||
date TEXT NOT NULL,
|
||||
description TEXT NOT NULL,
|
||||
amount_minor INTEGER NOT NULL,
|
||||
counterparty TEXT NOT NULL DEFAULT '',
|
||||
type TEXT NOT NULL DEFAULT '',
|
||||
balance_minor INTEGER,
|
||||
rule_tag TEXT,
|
||||
manual_tag TEXT,
|
||||
rule_transfer INTEGER NOT NULL DEFAULT 0,
|
||||
manual_transfer INTEGER,
|
||||
UNIQUE(account_id, fingerprint)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_txn_date ON transactions(date);
|
||||
CREATE INDEX IF NOT EXISTS idx_txn_account ON transactions(account_id);
|
||||
|
||||
-- One row per matched movement between the user's own accounts, derived from
|
||||
-- the [[transfer]] blocks in rules.toml. A leg belongs to at most one transfer,
|
||||
-- which UNIQUE enforces rather than trusting the pairing to be well behaved.
|
||||
CREATE TABLE IF NOT EXISTS transfers (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
def_index INTEGER NOT NULL,
|
||||
out_txn_id INTEGER NOT NULL UNIQUE REFERENCES transactions(id),
|
||||
in_txn_id INTEGER NOT NULL UNIQUE REFERENCES transactions(id)
|
||||
);
|
||||
`
|
||||
|
||||
// migrations bring an index created by an older build up to date. SQLite
|
||||
// errors on a duplicate column, which is how we detect "already applied".
|
||||
var migrations = []string{
|
||||
`ALTER TABLE transactions ADD COLUMN counterparty TEXT NOT NULL DEFAULT ''`,
|
||||
`ALTER TABLE transactions ADD COLUMN type TEXT NOT NULL DEFAULT ''`,
|
||||
`ALTER TABLE transactions ADD COLUMN balance_minor INTEGER`,
|
||||
}
|
||||
|
||||
// migrate applies any column that this index is missing.
|
||||
func migrate(db *sql.DB) error {
|
||||
have, err := columns(db, "transactions")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, stmt := range migrations {
|
||||
name := addedColumn(stmt)
|
||||
if have[name] {
|
||||
continue
|
||||
}
|
||||
if _, err := db.Exec(stmt); err != nil {
|
||||
return fmt.Errorf("migrate (%s): %w", stmt, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func columns(db *sql.DB, table string) (map[string]bool, error) {
|
||||
rows, err := db.Query(`SELECT name FROM pragma_table_info(?)`, table)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("inspect %s: %w", table, err)
|
||||
}
|
||||
defer rows.Close()
|
||||
out := map[string]bool{}
|
||||
for rows.Next() {
|
||||
var name string
|
||||
if err := rows.Scan(&name); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[name] = true
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// addedColumn pulls the column name out of an ADD COLUMN statement.
|
||||
func addedColumn(stmt string) string {
|
||||
_, rest, ok := strings.Cut(stmt, "ADD COLUMN ")
|
||||
if !ok {
|
||||
return ""
|
||||
}
|
||||
name, _, _ := strings.Cut(rest, " ")
|
||||
return name
|
||||
}
|
||||
|
||||
// Open opens (creating if needed) the index at path.
|
||||
//
|
||||
// There is no schema migration: the index caches what the statements and
|
||||
// rules.toml already say, so a build that changes the schema is answered by
|
||||
// deleting index.db and importing again, not by patching the old one in place.
|
||||
// Until it is deleted, queries against it fail on the columns it lacks.
|
||||
func Open(path string) (*DB, error) {
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return nil, fmt.Errorf("create state dir: %w", err)
|
||||
return nil, fmt.Errorf("create index dir: %w", err)
|
||||
}
|
||||
sqlDB, err := sql.Open("sqlite", path)
|
||||
if err != nil {
|
||||
@@ -128,10 +89,6 @@ func Open(path string) (*DB, error) {
|
||||
sqlDB.Close()
|
||||
return nil, fmt.Errorf("apply schema: %w", err)
|
||||
}
|
||||
if err := migrate(sqlDB); err != nil {
|
||||
sqlDB.Close()
|
||||
return nil, err
|
||||
}
|
||||
return &DB{sql: sqlDB}, nil
|
||||
}
|
||||
|
||||
@@ -176,15 +133,17 @@ func (d *DB) Accounts() ([]model.Account, error) {
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SourceFile records that a statement file was imported, returning its id.
|
||||
func (d *DB) SourceFile(accountID int64, path, sha, importedAt string) (int64, error) {
|
||||
// SourceFile records that a statement file was imported, holding rows
|
||||
// transactions, and returns its id.
|
||||
func (d *DB) SourceFile(accountID int64, path, sha, importedAt string, rows int) (int64, error) {
|
||||
_, err := d.sql.Exec(`
|
||||
INSERT INTO source_files (account_id, path, sha256, imported_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
INSERT INTO source_files (account_id, path, sha256, imported_at, rows)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(account_id, path) DO UPDATE SET
|
||||
sha256 = excluded.sha256,
|
||||
imported_at = excluded.imported_at`,
|
||||
accountID, path, sha, importedAt)
|
||||
imported_at = excluded.imported_at,
|
||||
rows = excluded.rows`,
|
||||
accountID, path, sha, importedAt, rows)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("record source file %s: %w", path, err)
|
||||
}
|
||||
@@ -214,8 +173,8 @@ func (d *DB) SourceFileSHA(accountID int64, path string) (string, bool, error) {
|
||||
// InsertTransaction adds a transaction unless its fingerprint already exists
|
||||
// for that account. It reports whether a new row was created.
|
||||
//
|
||||
// Existing rows are deliberately left untouched: re-importing an overlapping
|
||||
// statement must never clobber tags the user set by hand.
|
||||
// Existing rows are deliberately left untouched, so re-importing a statement
|
||||
// that overlaps one already imported adds nothing rather than duplicating it.
|
||||
func (d *DB) InsertTransaction(t model.Transaction) (bool, error) {
|
||||
var balance any
|
||||
if t.BalanceMinor != nil {
|
||||
@@ -224,11 +183,11 @@ func (d *DB) InsertTransaction(t model.Transaction) (bool, error) {
|
||||
res, err := d.sql.Exec(`
|
||||
INSERT INTO transactions
|
||||
(account_id, source_file_id, fingerprint, date, description, amount_minor,
|
||||
counterparty, type, balance_minor, rule_tag, rule_transfer)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, NULLIF(?, ''), ?)
|
||||
type, balance_minor, rule_tag)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, NULLIF(?, ''))
|
||||
ON CONFLICT(account_id, fingerprint) DO NOTHING`,
|
||||
t.AccountID, t.SourceFileID, t.Fingerprint, t.Date, t.Description,
|
||||
t.AmountMinor, t.Counterparty, t.Type, balance, t.RuleTag, boolToInt(t.RuleTransfer))
|
||||
t.AmountMinor, t.Type, balance, t.RuleTag)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("insert transaction: %w", err)
|
||||
}
|
||||
@@ -242,8 +201,18 @@ func (d *DB) InsertTransaction(t model.Transaction) (bool, error) {
|
||||
// Filter narrows a transaction query.
|
||||
type Filter struct {
|
||||
AccountSlug string
|
||||
Untagged bool // only rows with no effective tag
|
||||
// Untagged selects the rows still waiting for a verdict: no tag, and not a
|
||||
// leg of a matched transfer. A paired leg has been accounted for by the
|
||||
// transfer that claimed it, so listing it as untagged would ask the user to
|
||||
// write a rule for something that is already spoken for and that the report
|
||||
// leaves out anyway.
|
||||
Untagged bool
|
||||
Month string // YYYY-MM
|
||||
// From and To bound the dates inclusively, as YYYY-MM-DD, either side empty
|
||||
// for unbounded. Dates are stored ISO-8601, so a string comparison is a date
|
||||
// comparison; nothing has to be parsed to filter on a range.
|
||||
From string
|
||||
To string
|
||||
Search string // case-insensitive substring of the description
|
||||
Limit int
|
||||
}
|
||||
@@ -253,12 +222,12 @@ func (d *DB) Transactions(f Filter) ([]model.Transaction, error) {
|
||||
q := `
|
||||
SELECT t.id, t.account_id, a.slug, a.currency, a.minor_digits,
|
||||
t.fingerprint, t.date, t.description, t.amount_minor,
|
||||
COALESCE(s.path, ''), t.counterparty, t.type, t.balance_minor,
|
||||
COALESCE(t.rule_tag, ''), COALESCE(t.manual_tag, ''),
|
||||
t.rule_transfer, t.manual_transfer
|
||||
COALESCE(s.path, ''), t.type, t.balance_minor,
|
||||
COALESCE(t.rule_tag, ''), x.id
|
||||
FROM transactions t
|
||||
JOIN accounts a ON a.id = t.account_id
|
||||
LEFT JOIN source_files s ON s.id = t.source_file_id
|
||||
LEFT JOIN transfers x ON x.out_txn_id = t.id OR x.in_txn_id = t.id
|
||||
WHERE 1 = 1`
|
||||
var args []any
|
||||
if f.AccountSlug != "" {
|
||||
@@ -266,12 +235,20 @@ func (d *DB) Transactions(f Filter) ([]model.Transaction, error) {
|
||||
args = append(args, f.AccountSlug)
|
||||
}
|
||||
if f.Untagged {
|
||||
q += ` AND COALESCE(NULLIF(t.manual_tag, ''), NULLIF(t.rule_tag, '')) IS NULL`
|
||||
q += ` AND NULLIF(t.rule_tag, '') IS NULL AND x.id IS NULL`
|
||||
}
|
||||
if f.Month != "" {
|
||||
q += ` AND substr(t.date, 1, 7) = ?`
|
||||
args = append(args, f.Month)
|
||||
}
|
||||
if f.From != "" {
|
||||
q += ` AND t.date >= ?`
|
||||
args = append(args, f.From)
|
||||
}
|
||||
if f.To != "" {
|
||||
q += ` AND t.date <= ?`
|
||||
args = append(args, f.To)
|
||||
}
|
||||
q += ` ORDER BY t.date DESC, t.id DESC`
|
||||
// Search and Limit are applied in Go: SQLite's upper()/LIKE fold ASCII
|
||||
// only, which would silently fail on Cyrillic statement descriptions.
|
||||
@@ -287,24 +264,21 @@ func (d *DB) Transactions(f Filter) ([]model.Transaction, error) {
|
||||
for rows.Next() {
|
||||
var (
|
||||
t model.Transaction
|
||||
ruleTransfer int
|
||||
manualTransfer sql.NullInt64
|
||||
balance sql.NullInt64
|
||||
transfer sql.NullInt64
|
||||
)
|
||||
if err := rows.Scan(&t.ID, &t.AccountID, &t.AccountSlug, &t.Currency, &t.MinorDigits,
|
||||
&t.Fingerprint, &t.Date, &t.Description, &t.AmountMinor, &t.SourcePath,
|
||||
&t.Counterparty, &t.Type, &balance,
|
||||
&t.RuleTag, &t.ManualTag, &ruleTransfer, &manualTransfer); err != nil {
|
||||
&t.Type, &balance, &t.RuleTag, &transfer); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
t.RuleTransfer = ruleTransfer != 0
|
||||
if balance.Valid {
|
||||
v := balance.Int64
|
||||
t.BalanceMinor = &v
|
||||
}
|
||||
if manualTransfer.Valid {
|
||||
v := manualTransfer.Int64 != 0
|
||||
t.ManualTransfer = &v
|
||||
if transfer.Valid {
|
||||
v := transfer.Int64
|
||||
t.TransferID = &v
|
||||
}
|
||||
if needle != "" && !strings.Contains(model.NormalizeDescription(t.Description), needle) {
|
||||
continue
|
||||
@@ -317,39 +291,14 @@ func (d *DB) Transactions(f Filter) ([]model.Transaction, error) {
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SetManualTag sets (or, with an empty tag, clears) the user's tag override.
|
||||
func (d *DB) SetManualTag(id int64, tag string) error {
|
||||
_, err := d.sql.Exec(`UPDATE transactions SET manual_tag = NULLIF(?, '') WHERE id = ?`, tag, id)
|
||||
return err
|
||||
}
|
||||
|
||||
// SetManualTransfer sets the user's transfer override; nil restores the
|
||||
// rule-derived value.
|
||||
func (d *DB) SetManualTransfer(id int64, v *bool) error {
|
||||
var arg any
|
||||
if v != nil {
|
||||
arg = boolToInt(*v)
|
||||
}
|
||||
_, err := d.sql.Exec(`UPDATE transactions SET manual_transfer = ? WHERE id = ?`, arg, id)
|
||||
return err
|
||||
}
|
||||
|
||||
// ClearOverrides drops both manual overrides for a transaction.
|
||||
func (d *DB) ClearOverrides(id int64) error {
|
||||
_, err := d.sql.Exec(
|
||||
`UPDATE transactions SET manual_tag = NULL, manual_transfer = NULL WHERE id = ?`, id)
|
||||
return err
|
||||
}
|
||||
|
||||
// RuleAssignment is one row's recomputed rule verdict.
|
||||
type RuleAssignment struct {
|
||||
ID int64
|
||||
Tag string
|
||||
Transfer bool
|
||||
}
|
||||
|
||||
// ApplyRuleResults rewrites rule_tag and rule_transfer for every listed row in
|
||||
// a single transaction. Manual columns are never touched.
|
||||
// ApplyRuleResults rewrites rule_tag for every listed row in a single
|
||||
// transaction. The manual column is never touched.
|
||||
func (d *DB) ApplyRuleResults(rs []RuleAssignment) error {
|
||||
tx, err := d.sql.Begin()
|
||||
if err != nil {
|
||||
@@ -358,22 +307,57 @@ func (d *DB) ApplyRuleResults(rs []RuleAssignment) error {
|
||||
defer tx.Rollback()
|
||||
|
||||
stmt, err := tx.Prepare(
|
||||
`UPDATE transactions SET rule_tag = NULLIF(?, ''), rule_transfer = ? WHERE id = ?`)
|
||||
`UPDATE transactions SET rule_tag = NULLIF(?, '') WHERE id = ?`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer stmt.Close()
|
||||
|
||||
for _, r := range rs {
|
||||
if _, err := stmt.Exec(r.Tag, boolToInt(r.Transfer), r.ID); err != nil {
|
||||
if _, err := stmt.Exec(r.Tag, r.ID); err != nil {
|
||||
return fmt.Errorf("apply rules to txn %d: %w", r.ID, err)
|
||||
}
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// Balance sums every transaction in an account, transfers included: a transfer
|
||||
// really does move the account's money.
|
||||
// TransferLink is one matched pair, as decided by the transfer definitions.
|
||||
type TransferLink struct {
|
||||
DefIndex int
|
||||
OutID int64
|
||||
InID int64
|
||||
}
|
||||
|
||||
// ReplaceTransfers rewrites the whole pairing in one transaction. Like the
|
||||
// tags, it is derived from a file the user edits, so it is replaced wholesale
|
||||
// rather than patched: a definition removed from rules.toml must take its pairs
|
||||
// with it.
|
||||
func (d *DB) ReplaceTransfers(links []TransferLink) error {
|
||||
tx, err := d.sql.Begin()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
if _, err := tx.Exec(`DELETE FROM transfers`); err != nil {
|
||||
return fmt.Errorf("clear transfers: %w", err)
|
||||
}
|
||||
stmt, err := tx.Prepare(
|
||||
`INSERT INTO transfers (def_index, out_txn_id, in_txn_id) VALUES (?, ?, ?)`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer stmt.Close()
|
||||
|
||||
for _, l := range links {
|
||||
if _, err := stmt.Exec(l.DefIndex, l.OutID, l.InID); err != nil {
|
||||
return fmt.Errorf("link transfer %d→%d: %w", l.OutID, l.InID, err)
|
||||
}
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
// Balance sums every transaction in an account.
|
||||
func (d *DB) Balance(accountID int64) (int64, error) {
|
||||
var v sql.NullInt64
|
||||
err := d.sql.QueryRow(
|
||||
@@ -392,11 +376,120 @@ func (d *DB) Count(accountID int64) (int, error) {
|
||||
return n, err
|
||||
}
|
||||
|
||||
// Tags lists every effective tag in use, for completion in the TUI.
|
||||
// Months lists the distinct YYYY-MM the index holds, most recent first. It is
|
||||
// asked of the whole index rather than of a filtered view on purpose: it is the
|
||||
// time axis the report screen steps along, and an axis that grew and shrank as
|
||||
// the account or search filter changed would move under the cursor.
|
||||
func (d *DB) Months() ([]string, error) {
|
||||
rows, err := d.sql.Query(
|
||||
`SELECT DISTINCT substr(date, 1, 7) FROM transactions ORDER BY 1 DESC`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list months: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []string
|
||||
for rows.Next() {
|
||||
var s string
|
||||
if err := rows.Scan(&s); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SourceFileInfo is what the index recorded about one imported statement.
|
||||
type SourceFileInfo struct {
|
||||
AccountSlug string
|
||||
Path string // relative to the data root, as the importer records it
|
||||
SHA256 string
|
||||
ImportedAt string
|
||||
// Rows is how many transactions the statement holds, as parsed.
|
||||
Rows int
|
||||
// Added counts the transactions this file introduced. A row seen again in
|
||||
// an overlapping statement is deduplicated and stays with the file that
|
||||
// brought it first, so this is not how many rows the file holds.
|
||||
Added int
|
||||
}
|
||||
|
||||
// SourceFiles lists every statement the index has imported.
|
||||
func (d *DB) SourceFiles() ([]SourceFileInfo, error) {
|
||||
rows, err := d.sql.Query(`
|
||||
SELECT a.slug, s.path, s.sha256, s.imported_at, s.rows, COUNT(t.id)
|
||||
FROM source_files s
|
||||
JOIN accounts a ON a.id = s.account_id
|
||||
LEFT JOIN transactions t ON t.source_file_id = s.id
|
||||
GROUP BY s.id
|
||||
ORDER BY a.slug, s.path`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list source files: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []SourceFileInfo
|
||||
for rows.Next() {
|
||||
var f SourceFileInfo
|
||||
if err := rows.Scan(&f.AccountSlug, &f.Path, &f.SHA256, &f.ImportedAt, &f.Rows, &f.Added); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, f)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// ForgetSourceFile removes a statement from the index: its transactions, any
|
||||
// transfer pairing that used them, and the record of it, returning how many
|
||||
// transactions went. Pairing is derived state that the next Link rewrites, so
|
||||
// dropping a pair here loses nothing.
|
||||
//
|
||||
// A transaction two overlapping statements share is stored once, under the
|
||||
// file that brought it first, so removing that file can take rows another
|
||||
// statement still holds. The account's other statements therefore forget
|
||||
// their checksums, and the next import re-reads them and restores those rows.
|
||||
func (d *DB) ForgetSourceFile(accountSlug, path string) (int, error) {
|
||||
tx, err := d.sql.Begin()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
var id, accountID int64
|
||||
err = tx.QueryRow(`
|
||||
SELECT s.id, s.account_id FROM source_files s JOIN accounts a ON a.id = s.account_id
|
||||
WHERE a.slug = ? AND s.path = ?`, accountSlug, path).Scan(&id, &accountID)
|
||||
if err == sql.ErrNoRows {
|
||||
return 0, nil
|
||||
}
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("find source file %s: %w", path, err)
|
||||
}
|
||||
if _, err := tx.Exec(`
|
||||
DELETE FROM transfers WHERE
|
||||
out_txn_id IN (SELECT id FROM transactions WHERE source_file_id = ?) OR
|
||||
in_txn_id IN (SELECT id FROM transactions WHERE source_file_id = ?)`, id, id); err != nil {
|
||||
return 0, fmt.Errorf("forget transfers from %s: %w", path, err)
|
||||
}
|
||||
res, err := tx.Exec(`DELETE FROM transactions WHERE source_file_id = ?`, id)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("forget transactions from %s: %w", path, err)
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if _, err := tx.Exec(`DELETE FROM source_files WHERE id = ?`, id); err != nil {
|
||||
return 0, fmt.Errorf("forget %s: %w", path, err)
|
||||
}
|
||||
if _, err := tx.Exec(`UPDATE source_files SET sha256 = '' WHERE account_id = ?`, accountID); err != nil {
|
||||
return 0, fmt.Errorf("mark %s's statements for re-reading: %w", accountSlug, err)
|
||||
}
|
||||
return int(n), tx.Commit()
|
||||
}
|
||||
|
||||
// Tags lists every tag in use, for completion in the rule builder.
|
||||
func (d *DB) Tags() ([]string, error) {
|
||||
rows, err := d.sql.Query(`
|
||||
SELECT DISTINCT COALESCE(NULLIF(manual_tag, ''), NULLIF(rule_tag, '')) AS tag
|
||||
FROM transactions WHERE tag IS NOT NULL ORDER BY tag`)
|
||||
SELECT DISTINCT rule_tag FROM transactions
|
||||
WHERE rule_tag IS NOT NULL AND rule_tag != '' ORDER BY rule_tag`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -411,10 +504,3 @@ func (d *DB) Tags() ([]string, error) {
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
func boolToInt(b bool) int {
|
||||
if b {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
@@ -0,0 +1,296 @@
|
||||
// Package transfers pairs the two legs of money moved between the user's own
|
||||
// accounts, as described by the [[transfer]] blocks in rules.toml.
|
||||
//
|
||||
// A transfer is always a pair. One leg on its own is not a transfer, it is an
|
||||
// unmatched leg: money that left an account and cannot be shown to have
|
||||
// arrived. That distinction is the whole point of pairing here rather than
|
||||
// flagging single transactions, because only a complete pair can be dropped
|
||||
// from the report without unbalancing it.
|
||||
package transfers
|
||||
|
||||
import (
|
||||
"math"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
"git.petrovv.com/nikola/money/internal/glob"
|
||||
"git.petrovv.com/nikola/money/internal/model"
|
||||
"git.petrovv.com/nikola/money/internal/store"
|
||||
)
|
||||
|
||||
// WindowDays is how far apart the two legs may be dated. A transfer between
|
||||
// two banks is one movement seen twice, but the statements rarely agree on the
|
||||
// day: the money leaves on Friday and lands on Monday.
|
||||
const WindowDays = 5
|
||||
|
||||
// Engine pairs legs using the definitions in file order. As with rules, the
|
||||
// first definition to claim a transaction keeps it, so an earlier definition
|
||||
// can never have a leg stolen by a later one.
|
||||
type Engine struct {
|
||||
defs []config.Transfer
|
||||
}
|
||||
|
||||
// New builds an engine from the parsed rules file.
|
||||
func New(r *config.Rules) *Engine { return &Engine{defs: r.Transfer} }
|
||||
|
||||
// Transfers returns the ordered definitions, as loaded from rules.toml.
|
||||
func (e *Engine) Transfers() []config.Transfer { return e.defs }
|
||||
|
||||
// Pair is one matched movement: the leg that left and the leg that arrived.
|
||||
type Pair struct {
|
||||
Def int // index of the definition that claimed it
|
||||
Out model.Transaction
|
||||
In model.Transaction
|
||||
}
|
||||
|
||||
// Fee is what the movement lost on the way: the amount that left, less the
|
||||
// amount that arrived. It is non-zero only for a definition carrying a
|
||||
// tolerance_pct, and it is money genuinely spent — a pair leaves the report
|
||||
// entirely, so this is the one number that has to be reported separately or it
|
||||
// vanishes with the legs. Negative would mean more arrived than left.
|
||||
//
|
||||
// Across currencies it is always zero: the two amounts are in different units,
|
||||
// so subtracting them would produce a number that means nothing.
|
||||
func (p Pair) Fee() int64 {
|
||||
if p.In.Currency != p.Out.Currency {
|
||||
return 0
|
||||
}
|
||||
return -(p.Out.AmountMinor + p.In.AmountMinor)
|
||||
}
|
||||
|
||||
// Leg is a transaction a definition caught on one side but could not pair.
|
||||
type Leg struct {
|
||||
Def int
|
||||
Txn model.Transaction
|
||||
// Out reports which side it was caught on: true for the leaving leg
|
||||
// (from_account, from_desc), false for the arriving one.
|
||||
Out bool
|
||||
}
|
||||
|
||||
// Result is what a definition set makes of a set of transactions.
|
||||
type Result struct {
|
||||
Pairs []Pair
|
||||
Unmatched []Leg
|
||||
// Paired and Orphaned are per-definition counts, positionally matching the
|
||||
// definitions. A definition with no pairs and no orphans matches nothing at
|
||||
// all; one with orphans is catching transactions but not completing them.
|
||||
Paired []int
|
||||
Orphaned []int
|
||||
}
|
||||
|
||||
// Analyze pairs every leg it can. Transactions are claimed at most once across
|
||||
// the whole run, so the result is a partition, not a set of overlapping
|
||||
// interpretations.
|
||||
func (e *Engine) Analyze(txns []model.Transaction) Result {
|
||||
res := Result{
|
||||
Paired: make([]int, len(e.defs)),
|
||||
Orphaned: make([]int, len(e.defs)),
|
||||
}
|
||||
|
||||
// Deterministic input order: the index returns newest first, and pairing
|
||||
// walks forward in time so the earliest leg gets the earliest counterpart.
|
||||
ordered := append([]model.Transaction(nil), txns...)
|
||||
sort.Slice(ordered, func(i, j int) bool {
|
||||
if ordered[i].Date != ordered[j].Date {
|
||||
return ordered[i].Date < ordered[j].Date
|
||||
}
|
||||
return ordered[i].ID < ordered[j].ID
|
||||
})
|
||||
|
||||
claimed := map[int64]bool{}
|
||||
for d := range e.defs {
|
||||
def := &e.defs[d]
|
||||
var outs, ins []model.Transaction
|
||||
for _, t := range ordered {
|
||||
if claimed[t.ID] {
|
||||
continue
|
||||
}
|
||||
switch {
|
||||
case t.AmountMinor < 0 && matches(def.FromAccount, def.FromDesc, t):
|
||||
outs = append(outs, t)
|
||||
case t.AmountMinor > 0 && matches(def.ToAccount, def.ToDesc, t):
|
||||
ins = append(ins, t)
|
||||
}
|
||||
}
|
||||
|
||||
used := map[int64]bool{}
|
||||
for _, out := range outs {
|
||||
j := bestCounterpart(out, ins, used, def.TolerancePct)
|
||||
if j < 0 {
|
||||
continue
|
||||
}
|
||||
in := ins[j]
|
||||
used[in.ID], used[out.ID] = true, true
|
||||
claimed[in.ID], claimed[out.ID] = true, true
|
||||
res.Pairs = append(res.Pairs, Pair{Def: d, Out: out, In: in})
|
||||
res.Paired[d]++
|
||||
}
|
||||
|
||||
for _, t := range outs {
|
||||
if !used[t.ID] {
|
||||
res.Unmatched = append(res.Unmatched, Leg{Def: d, Txn: t, Out: true})
|
||||
res.Orphaned[d]++
|
||||
}
|
||||
}
|
||||
for _, t := range ins {
|
||||
if !used[t.ID] {
|
||||
res.Unmatched = append(res.Unmatched, Leg{Def: d, Txn: t})
|
||||
res.Orphaned[d]++
|
||||
}
|
||||
}
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// matches reports whether a transaction is on the named account and its
|
||||
// description fits the glob. Descriptions are normalised the same way the
|
||||
// tagging rules normalise them, so one pattern behaves the same in both places.
|
||||
func matches(account, pattern string, t model.Transaction) bool {
|
||||
return t.AccountSlug == account && glob.Match(pattern, model.NormalizeDescription(t.Description))
|
||||
}
|
||||
|
||||
// bestCounterpart finds the arriving leg for out, dated within the window. The
|
||||
// closest date wins, so two identical monthly transfers pair up in order
|
||||
// instead of crossing over.
|
||||
//
|
||||
// Within one currency the amount is the evidence: an exact opposite is near
|
||||
// proof that two legs are one movement, so it is what is required by default.
|
||||
// tolerancePct widens that, and only that, for a route where the bank takes a
|
||||
// fee on the way and the two statements therefore disagree. The difference it
|
||||
// admits is not forgiven — it is real money, it is reported as the pair's Fee,
|
||||
// and report.Excluded carries it out of the report so it cannot be lost inside
|
||||
// a transfer. Which is why the default stays zero: every percent of slack is
|
||||
// also a percent more chance of pairing two unrelated movements.
|
||||
//
|
||||
// Across currencies there is no such evidence. The tool holds no exchange
|
||||
// rates, so the two numbers are unrelated and the dates carry the pairing on
|
||||
// their own. That is weaker, and it is meant to be: it pairs an exchange
|
||||
// between your own accounts, and it will pick the wrong counterpart if the
|
||||
// same route is used twice inside one window. A tolerance means nothing there
|
||||
// and is ignored.
|
||||
func bestCounterpart(out model.Transaction, ins []model.Transaction, used map[int64]bool, tolerancePct float64) int {
|
||||
outDay, ok := day(out.Date)
|
||||
if !ok {
|
||||
return -1
|
||||
}
|
||||
allowed := allowance(out.AmountMinor, tolerancePct)
|
||||
|
||||
best, bestGap, bestOff := -1, 0, int64(0)
|
||||
for j, in := range ins {
|
||||
if used[in.ID] {
|
||||
continue
|
||||
}
|
||||
off := int64(0)
|
||||
if in.Currency == out.Currency {
|
||||
off = in.AmountMinor + out.AmountMinor
|
||||
if off < 0 {
|
||||
off = -off
|
||||
}
|
||||
if off > allowed {
|
||||
continue
|
||||
}
|
||||
}
|
||||
inDay, ok := day(in.Date)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
gap := int(inDay.Sub(outDay).Hours() / 24)
|
||||
if gap < 0 {
|
||||
gap = -gap
|
||||
}
|
||||
if gap > WindowDays {
|
||||
continue
|
||||
}
|
||||
// Strictly closer, so an equal gap keeps the candidate already found.
|
||||
// ins is in date order, which makes that the earlier one. Under a
|
||||
// tolerance an equal gap can still be decided on the amount, and the
|
||||
// nearer amount is the better evidence; with no tolerance every
|
||||
// candidate is exact and this never fires.
|
||||
if best < 0 || gap < bestGap || (gap == bestGap && off < bestOff) {
|
||||
best, bestGap, bestOff = j, gap, off
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
// Wanted describes the counterpart an unpaired leg is missing, in the terms
|
||||
// bestCounterpart judges it by, so a person can go and look for it.
|
||||
type Wanted struct {
|
||||
Account string // where the other leg would be
|
||||
Desc string // the glob it would have to match
|
||||
From string // the window it would have to fall in, inclusive
|
||||
To string
|
||||
// Amount is the other leg's signed amount, exact unless TolerancePct
|
||||
// widens it. AnyAmount is set across currencies, where the amount is not
|
||||
// checked at all and the dates carry the pairing alone.
|
||||
Amount int64
|
||||
TolerancePct float64
|
||||
AnyAmount bool
|
||||
}
|
||||
|
||||
// Want says what would have paired l. currencyOf names an account's currency,
|
||||
// or "" for one the index holds nothing for.
|
||||
func (e *Engine) Want(l Leg, currencyOf func(slug string) string) Wanted {
|
||||
def := e.defs[l.Def]
|
||||
w := Wanted{Account: def.ToAccount, Desc: def.ToDesc, Amount: -l.Txn.AmountMinor}
|
||||
if !l.Out {
|
||||
w.Account, w.Desc = def.FromAccount, def.FromDesc
|
||||
}
|
||||
if c := currencyOf(w.Account); c != "" && c != l.Txn.Currency {
|
||||
w.AnyAmount = true
|
||||
} else {
|
||||
w.TolerancePct = def.TolerancePct
|
||||
}
|
||||
if d, ok := day(l.Txn.Date); ok {
|
||||
w.From = d.AddDate(0, 0, -WindowDays).Format("2006-01-02")
|
||||
w.To = d.AddDate(0, 0, WindowDays).Format("2006-01-02")
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
// allowance is how far the arriving leg may miss the leaving one, in minor
|
||||
// units. It is a share of the amount that left, not of the difference, so the
|
||||
// same percentage means the same thing on a large transfer as on a small one.
|
||||
//
|
||||
// Rounded rather than truncated: at 1% of 10.00 a truncating allowance would be
|
||||
// 0.09 and miss the 0.10 fee the percentage was chosen to admit.
|
||||
func allowance(outMinor int64, pct float64) int64 {
|
||||
if pct <= 0 {
|
||||
return 0
|
||||
}
|
||||
if outMinor < 0 {
|
||||
outMinor = -outMinor
|
||||
}
|
||||
return int64(math.Round(float64(outMinor) * pct / 100))
|
||||
}
|
||||
|
||||
// day parses a statement date. An unparseable one cannot be windowed, so it
|
||||
// simply never pairs rather than pairing wrongly.
|
||||
func day(s string) (time.Time, bool) {
|
||||
t, err := time.Parse("2006-01-02", s)
|
||||
return t, err == nil
|
||||
}
|
||||
|
||||
// Link recomputes the pairing over every transaction in the index and writes it
|
||||
// back, replacing whatever was there. Like Retag, it is derived state rewritten
|
||||
// wholesale, so it is safe to run at any time.
|
||||
//
|
||||
// It runs over the whole index deliberately: pairing inside a filtered view
|
||||
// would let a movement count as a transfer in one report and not in another.
|
||||
func (e *Engine) Link(db *store.DB) (pairs, unmatched int, err error) {
|
||||
txns, err := db.Transactions(store.Filter{})
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
res := e.Analyze(txns)
|
||||
|
||||
links := make([]store.TransferLink, 0, len(res.Pairs))
|
||||
for _, p := range res.Pairs {
|
||||
links = append(links, store.TransferLink{DefIndex: p.Def, OutID: p.Out.ID, InID: p.In.ID})
|
||||
}
|
||||
if err := db.ReplaceTransfers(links); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
return len(res.Pairs), len(res.Unmatched), nil
|
||||
}
|
||||
@@ -0,0 +1,392 @@
|
||||
package transfers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
"git.petrovv.com/nikola/money/internal/model"
|
||||
)
|
||||
|
||||
// txn builds a transaction the way the index hands them out.
|
||||
func txn(id int64, account, date, desc string, amount int64) model.Transaction {
|
||||
return model.Transaction{
|
||||
ID: id,
|
||||
AccountSlug: account,
|
||||
Currency: "EUR",
|
||||
MinorDigits: 2,
|
||||
Date: date,
|
||||
Description: desc,
|
||||
AmountMinor: amount,
|
||||
}
|
||||
}
|
||||
|
||||
func engine(defs ...config.Transfer) *Engine {
|
||||
return New(&config.Rules{Transfer: defs})
|
||||
}
|
||||
|
||||
var topUp = config.Transfer{
|
||||
FromAccount: "nlb", FromDesc: "*TO REVOLUT*",
|
||||
ToAccount: "revolut", ToDesc: "*FROM NLB*",
|
||||
}
|
||||
|
||||
// The two legs of one movement, dated a weekend apart, are one transfer.
|
||||
func TestPairsTheTwoLegs(t *testing.T) {
|
||||
res := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-09", "Top-up from NLB", 50000),
|
||||
txn(3, "nlb", "2026-03-07", "LIDL SOFIA", -2000),
|
||||
})
|
||||
|
||||
if len(res.Pairs) != 1 {
|
||||
t.Fatalf("pairs = %+v, want exactly one", res.Pairs)
|
||||
}
|
||||
if res.Pairs[0].Out.ID != 1 || res.Pairs[0].In.ID != 2 {
|
||||
t.Errorf("paired %d→%d, want 1→2", res.Pairs[0].Out.ID, res.Pairs[0].In.ID)
|
||||
}
|
||||
if len(res.Unmatched) != 0 {
|
||||
t.Errorf("unmatched = %+v, want none; the shopping matches no side", res.Unmatched)
|
||||
}
|
||||
if res.Paired[0] != 1 || res.Orphaned[0] != 0 {
|
||||
t.Errorf("counts = %d paired, %d orphaned; want 1 and 0", res.Paired[0], res.Orphaned[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Money that left and never arrived is the case the whole feature exists to
|
||||
// surface: it is not a transfer, it is one leg on its own.
|
||||
func TestLegWithoutACounterpartIsUnmatched(t *testing.T) {
|
||||
res := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
})
|
||||
|
||||
if len(res.Pairs) != 0 {
|
||||
t.Fatalf("pairs = %+v, want none", res.Pairs)
|
||||
}
|
||||
if len(res.Unmatched) != 1 || res.Unmatched[0].Txn.ID != 1 || !res.Unmatched[0].Out {
|
||||
t.Fatalf("unmatched = %+v, want the leaving leg", res.Unmatched)
|
||||
}
|
||||
if res.Paired[0] != 0 || res.Orphaned[0] != 1 {
|
||||
t.Errorf("counts = %d paired, %d orphaned; want 0 and 1", res.Paired[0], res.Orphaned[0])
|
||||
}
|
||||
}
|
||||
|
||||
// A definition that catches nothing at all is dead, and reports as such
|
||||
// separately from one that catches legs it cannot pair.
|
||||
func TestDefinitionMatchingNothing(t *testing.T) {
|
||||
res := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "LIDL SOFIA", -2000),
|
||||
})
|
||||
if res.Paired[0] != 0 || res.Orphaned[0] != 0 {
|
||||
t.Errorf("counts = %d paired, %d orphaned; want both zero", res.Paired[0], res.Orphaned[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Within one currency the amount is the evidence, so by default it must agree
|
||||
// exactly: a movement that arrives short a fee is not the same movement
|
||||
// leaving, unless the definition says the route charges one.
|
||||
func TestAmountMustAgreeWithinACurrency(t *testing.T) {
|
||||
res := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 49500),
|
||||
})
|
||||
if len(res.Pairs) != 0 {
|
||||
t.Errorf("pairs = %+v, want none: the amounts differ", res.Pairs)
|
||||
}
|
||||
if len(res.Unmatched) != 2 {
|
||||
t.Errorf("unmatched = %+v, want both legs reported", res.Unmatched)
|
||||
}
|
||||
}
|
||||
|
||||
// Across currencies the amounts are unrelated -- there are no exchange rates
|
||||
// here -- so the dates carry the pairing on their own.
|
||||
func TestCrossCurrencyPairsOnDateAlone(t *testing.T) {
|
||||
arrived := txn(2, "revolut", "2026-03-07", "Top-up from NLB", 97790)
|
||||
arrived.Currency = "BGN"
|
||||
|
||||
res := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
arrived,
|
||||
})
|
||||
if len(res.Pairs) != 1 {
|
||||
t.Fatalf("pairs = %+v, want the exchange paired", res.Pairs)
|
||||
}
|
||||
if res.Pairs[0].In.Currency == res.Pairs[0].Out.Currency {
|
||||
t.Error("expected the pair to span two currencies")
|
||||
}
|
||||
|
||||
// The window still bounds it.
|
||||
late := arrived
|
||||
late.Date = "2026-03-20"
|
||||
res = engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
late,
|
||||
})
|
||||
if len(res.Pairs) != 0 {
|
||||
t.Errorf("pairs = %+v, want none beyond the window", res.Pairs)
|
||||
}
|
||||
}
|
||||
|
||||
// With no amount to go on, the nearest date decides, so two exchanges in flight
|
||||
// at once pair in order rather than crossing over.
|
||||
func TestCrossCurrencyPicksTheNearestDate(t *testing.T) {
|
||||
arriving := func(id int64, date string, amount int64) model.Transaction {
|
||||
in := txn(id, "revolut", date, "Top-up from NLB", amount)
|
||||
in.Currency = "BGN"
|
||||
return in
|
||||
}
|
||||
res := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-02", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(3, "nlb", "2026-03-10", "TRANSFER TO REVOLUT LTD", -20000),
|
||||
arriving(2, "2026-03-03", 97790),
|
||||
arriving(4, "2026-03-11", 39116),
|
||||
})
|
||||
if len(res.Pairs) != 2 {
|
||||
t.Fatalf("pairs = %+v, want both exchanges paired", res.Pairs)
|
||||
}
|
||||
for _, p := range res.Pairs {
|
||||
if p.In.ID != p.Out.ID+1 {
|
||||
t.Errorf("paired %d->%d, want each exchange with its own counterpart",
|
||||
p.Out.ID, p.In.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Legs further apart than the window are not the same movement.
|
||||
func TestWindowBoundsThePairing(t *testing.T) {
|
||||
inside := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-01", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-06", "Top-up from NLB", 50000),
|
||||
})
|
||||
if len(inside.Pairs) != 1 {
|
||||
t.Errorf("pairs = %+v, want one at exactly the window", inside.Pairs)
|
||||
}
|
||||
|
||||
outside := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-01", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 50000),
|
||||
})
|
||||
if len(outside.Pairs) != 0 {
|
||||
t.Errorf("pairs = %+v, want none beyond the window", outside.Pairs)
|
||||
}
|
||||
}
|
||||
|
||||
// Two identical monthly transfers must pair in order rather than crossing over,
|
||||
// or the dates in the report would be wrong even though the totals were right.
|
||||
func TestIdenticalTransfersPairInOrder(t *testing.T) {
|
||||
res := engine(topUp).Analyze([]model.Transaction{
|
||||
txn(4, "revolut", "2026-04-02", "Top-up from NLB", 50000),
|
||||
txn(1, "nlb", "2026-03-01", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(3, "nlb", "2026-04-01", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-02", "Top-up from NLB", 50000),
|
||||
})
|
||||
if len(res.Pairs) != 2 {
|
||||
t.Fatalf("pairs = %+v, want two", res.Pairs)
|
||||
}
|
||||
for _, p := range res.Pairs {
|
||||
if p.In.ID != p.Out.ID+1 {
|
||||
t.Errorf("paired %d→%d, want each transfer with its own month", p.Out.ID, p.In.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A transaction belongs to one transfer. The first definition to claim a leg
|
||||
// keeps it, exactly as the first matching rule keeps a tag.
|
||||
func TestFirstDefinitionClaimsTheLeg(t *testing.T) {
|
||||
broad := config.Transfer{
|
||||
FromAccount: "nlb", FromDesc: "*TRANSFER*",
|
||||
ToAccount: "revolut", ToDesc: "*NLB*",
|
||||
}
|
||||
res := engine(topUp, broad).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 50000),
|
||||
})
|
||||
if len(res.Pairs) != 1 || res.Pairs[0].Def != 0 {
|
||||
t.Fatalf("pairs = %+v, want one claimed by the first definition", res.Pairs)
|
||||
}
|
||||
if res.Paired[1] != 0 || res.Orphaned[1] != 0 {
|
||||
t.Errorf("the shadowed definition reports %d paired, %d orphaned; want zero",
|
||||
res.Paired[1], res.Orphaned[1])
|
||||
}
|
||||
}
|
||||
|
||||
// Direction is part of the definition: the arriving leg is an inflow and the
|
||||
// leaving leg an outflow, so a definition written backwards pairs nothing.
|
||||
func TestDirectionMatters(t *testing.T) {
|
||||
backwards := config.Transfer{
|
||||
FromAccount: "revolut", FromDesc: "*FROM NLB*",
|
||||
ToAccount: "nlb", ToDesc: "*TO REVOLUT*",
|
||||
}
|
||||
res := engine(backwards).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 50000),
|
||||
})
|
||||
if len(res.Pairs) != 0 {
|
||||
t.Errorf("pairs = %+v, want none: neither leg is on the side it was named for", res.Pairs)
|
||||
}
|
||||
if len(res.Unmatched) != 0 {
|
||||
t.Errorf("unmatched = %+v, want none either", res.Unmatched)
|
||||
}
|
||||
}
|
||||
|
||||
// Money moved inside one account (a savings pocket, say) is still a pair.
|
||||
func TestSameAccountBothSides(t *testing.T) {
|
||||
pocket := config.Transfer{
|
||||
FromAccount: "revolut", FromDesc: "*TO VAULT*",
|
||||
ToAccount: "revolut", ToDesc: "*FROM VAULT*",
|
||||
}
|
||||
res := engine(pocket).Analyze([]model.Transaction{
|
||||
txn(1, "revolut", "2026-03-06", "Move to Vault", -10000),
|
||||
txn(2, "revolut", "2026-03-06", "Move from Vault", 10000),
|
||||
})
|
||||
if len(res.Pairs) != 1 {
|
||||
t.Errorf("pairs = %+v, want one within the account", res.Pairs)
|
||||
}
|
||||
}
|
||||
|
||||
// tolerant is the same route on a bank that takes a fee on the way: 1% of the
|
||||
// leaving leg, so a 500.00 transfer may arrive as little as 495.00.
|
||||
var tolerant = config.Transfer{
|
||||
FromAccount: "nlb", FromDesc: "*TO REVOLUT*",
|
||||
ToAccount: "revolut", ToDesc: "*FROM NLB*",
|
||||
TolerancePct: 1,
|
||||
}
|
||||
|
||||
// A route that charges pairs anyway, and says what it cost.
|
||||
func TestToleranceAdmitsAFee(t *testing.T) {
|
||||
res := engine(tolerant).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 49500),
|
||||
})
|
||||
if len(res.Pairs) != 1 {
|
||||
t.Fatalf("pairs = %+v, want the fee tolerated", res.Pairs)
|
||||
}
|
||||
if fee := res.Pairs[0].Fee(); fee != 500 {
|
||||
t.Errorf("fee = %d, want 500: the pair leaves the report, so the fee has to be reported", fee)
|
||||
}
|
||||
}
|
||||
|
||||
// The tolerance is a bound, not an invitation: past it the legs are still two
|
||||
// separate things.
|
||||
func TestToleranceStopsAtItsBound(t *testing.T) {
|
||||
res := engine(tolerant).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 49499),
|
||||
})
|
||||
if len(res.Pairs) != 0 {
|
||||
t.Errorf("pairs = %+v, want none: 5.01 is more than 1%% of 500.00", res.Pairs)
|
||||
}
|
||||
}
|
||||
|
||||
// The allowance is a share of the amount that left, so the same percentage
|
||||
// means the same thing on a small transfer as on a large one -- and it is
|
||||
// rounded, or 1% of 10.00 would admit 0.09 and miss the 0.10 fee it was
|
||||
// chosen for.
|
||||
func TestToleranceScalesWithTheAmount(t *testing.T) {
|
||||
res := engine(tolerant).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -1000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 990),
|
||||
})
|
||||
if len(res.Pairs) != 1 || res.Pairs[0].Fee() != 10 {
|
||||
t.Fatalf("pairs = %+v, want a 0.10 fee on 10.00 tolerated", res.Pairs)
|
||||
}
|
||||
|
||||
res = engine(tolerant).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -1000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 950),
|
||||
})
|
||||
if len(res.Pairs) != 0 {
|
||||
t.Errorf("pairs = %+v, want none: 0.50 is 5%% of 10.00", res.Pairs)
|
||||
}
|
||||
}
|
||||
|
||||
// A tolerance belongs to the definition that declares it and to no other, so
|
||||
// one tolerant route cannot loosen a strict one written beside it.
|
||||
func TestToleranceIsPerDefinition(t *testing.T) {
|
||||
strict := config.Transfer{
|
||||
FromAccount: "nlb", FromDesc: "*TO SAVINGS*",
|
||||
ToAccount: "savings", ToDesc: "*FROM NLB*",
|
||||
}
|
||||
res := engine(tolerant, strict).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-07", "Top-up from NLB", 49500),
|
||||
txn(3, "nlb", "2026-03-06", "TRANSFER TO SAVINGS", -50000),
|
||||
txn(4, "savings", "2026-03-07", "FROM NLB", 49500),
|
||||
})
|
||||
if len(res.Pairs) != 1 || res.Pairs[0].Def != 0 {
|
||||
t.Fatalf("pairs = %+v, want only the tolerant definition to pair", res.Pairs)
|
||||
}
|
||||
if len(res.Unmatched) != 2 {
|
||||
t.Errorf("unmatched = %+v, want both legs of the strict route reported", res.Unmatched)
|
||||
}
|
||||
}
|
||||
|
||||
// Across currencies the amounts are in different units, so a fee cannot be
|
||||
// computed from them -- subtracting one from the other would be a number
|
||||
// meaning nothing.
|
||||
func TestNoFeeAcrossCurrencies(t *testing.T) {
|
||||
arrived := txn(2, "revolut", "2026-03-07", "Top-up from NLB", 97790)
|
||||
arrived.Currency = "BGN"
|
||||
|
||||
res := engine(tolerant).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
arrived,
|
||||
})
|
||||
if len(res.Pairs) != 1 {
|
||||
t.Fatalf("pairs = %+v, want the exchange paired", res.Pairs)
|
||||
}
|
||||
if fee := res.Pairs[0].Fee(); fee != 0 {
|
||||
t.Errorf("fee = %d, want 0 across currencies", fee)
|
||||
}
|
||||
}
|
||||
|
||||
// With slack in the amount two candidates can sit the same number of days
|
||||
// away, and then the nearer amount is the better evidence.
|
||||
func TestEqualGapPrefersTheNearerAmount(t *testing.T) {
|
||||
res := engine(tolerant).Analyze([]model.Transaction{
|
||||
txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT LTD", -50000),
|
||||
txn(2, "revolut", "2026-03-05", "Top-up from NLB", 49600),
|
||||
txn(3, "revolut", "2026-03-07", "Top-up from NLB", 50000),
|
||||
})
|
||||
if len(res.Pairs) != 1 {
|
||||
t.Fatalf("pairs = %+v, want one", res.Pairs)
|
||||
}
|
||||
if res.Pairs[0].In.ID != 3 {
|
||||
t.Errorf("paired with %d, want 3: same gap, exact amount", res.Pairs[0].In.ID)
|
||||
}
|
||||
}
|
||||
|
||||
// Want describes the missing side in the terms the pairing judges it by: the
|
||||
// other definition half, the window, and the amount — exact by default,
|
||||
// widened only by the definition's own tolerance, unchecked across currencies.
|
||||
func TestWantDescribesTheMissingSide(t *testing.T) {
|
||||
eur := func(string) string { return "EUR" }
|
||||
|
||||
out := Leg{Def: 0, Out: true, Txn: txn(1, "nlb", "2026-03-06", "TRANSFER TO REVOLUT", -50000)}
|
||||
got := engine(topUp).Want(out, eur)
|
||||
want := Wanted{Account: "revolut", Desc: "*FROM NLB*", From: "2026-03-01", To: "2026-03-11", Amount: 50000}
|
||||
if got != want {
|
||||
t.Errorf("arriving side = %+v, want %+v", got, want)
|
||||
}
|
||||
|
||||
in := Leg{Def: 0, Txn: txn(2, "revolut", "2026-03-09", "Top-up from NLB", 50000)}
|
||||
got = engine(topUp).Want(in, eur)
|
||||
want = Wanted{Account: "nlb", Desc: "*TO REVOLUT*", From: "2026-03-04", To: "2026-03-14", Amount: -50000}
|
||||
if got != want {
|
||||
t.Errorf("leaving side = %+v, want %+v", got, want)
|
||||
}
|
||||
|
||||
fee := topUp
|
||||
fee.TolerancePct = 1.5
|
||||
if got := engine(fee).Want(out, eur); got.TolerancePct != 1.5 || got.AnyAmount {
|
||||
t.Errorf("tolerant = %+v, want the definition's 1.5%%", got)
|
||||
}
|
||||
|
||||
usd := func(slug string) string {
|
||||
if slug == "revolut" {
|
||||
return "USD"
|
||||
}
|
||||
return "EUR"
|
||||
}
|
||||
if got := engine(fee).Want(out, usd); !got.AnyAmount || got.TolerancePct != 0 {
|
||||
t.Errorf("cross-currency = %+v, want any amount and no tolerance", got)
|
||||
}
|
||||
}
|
||||
-1429
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,516 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
"git.petrovv.com/nikola/money/internal/model"
|
||||
"git.petrovv.com/nikola/money/internal/rules"
|
||||
"git.petrovv.com/nikola/money/internal/store"
|
||||
"git.petrovv.com/nikola/money/internal/transfers"
|
||||
)
|
||||
|
||||
type fixtureTxn struct {
|
||||
account, date, desc string
|
||||
amount int64
|
||||
}
|
||||
|
||||
// newTestServer builds a server over a data root holding rules.toml and an
|
||||
// index with the given transactions, tagged and paired as an import would
|
||||
// leave them. "Today" is fixed so the report's default window is predictable.
|
||||
func newTestServer(t *testing.T, rulesToml string, txns ...fixtureTxn) (*Server, http.Handler) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(root, config.RulesFile), []byte(rulesToml), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := store.Open(config.IndexPath(root))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { db.Close() })
|
||||
|
||||
ids := map[string][2]int64{}
|
||||
for _, slug := range []string{"checking", "savings"} {
|
||||
id, err := db.UpsertAccount(model.Account{Slug: slug, Name: slug, Currency: "EUR", MinorDigits: 2})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
src, err := db.SourceFile(id, slug+"/st.csv", "sha", "2026-01-01T00:00:00Z", 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ids[slug] = [2]int64{id, src}
|
||||
}
|
||||
for i, x := range txns {
|
||||
if _, err := db.InsertTransaction(model.Transaction{
|
||||
AccountID: ids[x.account][0], SourceFileID: ids[x.account][1],
|
||||
Fingerprint: x.desc + x.date + string(rune('a'+i)),
|
||||
Date: x.date, Description: x.desc, AmountMinor: x.amount,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
loaded, err := config.LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
engine, links := rules.New(loaded), transfers.New(loaded)
|
||||
if _, err := engine.Retag(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := links.Link(db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
s := New(root, db, nil, engine, links)
|
||||
s.now = func() time.Time { return time.Date(2026, 3, 15, 0, 0, 0, 0, time.UTC) }
|
||||
return s, s.Handler()
|
||||
}
|
||||
|
||||
// call sends a request and decodes the JSON answer, failing unless the status
|
||||
// is the one expected.
|
||||
func call(t *testing.T, h http.Handler, method, path string, body any, want int, out any) {
|
||||
t.Helper()
|
||||
var r *http.Request
|
||||
if body == nil {
|
||||
r = httptest.NewRequest(method, path, nil)
|
||||
} else {
|
||||
b, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r = httptest.NewRequest(method, path, bytes.NewReader(b))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != want {
|
||||
t.Fatalf("%s %s: status %d, want %d: %s", method, path, w.Code, want, w.Body.String())
|
||||
}
|
||||
if out != nil {
|
||||
if err := json.Unmarshal(w.Body.Bytes(), out); err != nil {
|
||||
t.Fatalf("%s %s: decode %q: %v", method, path, w.Body.String(), err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func readRules(t *testing.T, s *Server) *config.Rules {
|
||||
t.Helper()
|
||||
r, err := config.LoadRules(s.root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
const lidlRule = `
|
||||
[[rule]]
|
||||
match = "*LIDL*"
|
||||
tag = "groceries"
|
||||
`
|
||||
|
||||
// The period narrows the report and nothing else: the report opens on last
|
||||
// month while the transaction list still holds the whole index.
|
||||
func TestReportPeriodLeavesTransactionsAlone(t *testing.T) {
|
||||
_, h := newTestServer(t, lidlRule,
|
||||
fixtureTxn{"checking", "2026-02-10", "LIDL SOFIA", -1000},
|
||||
fixtureTxn{"checking", "2025-11-03", "LIDL VARNA", -500},
|
||||
)
|
||||
|
||||
var rep reportJSON
|
||||
call(t, h, "GET", "/api/report", nil, http.StatusOK, &rep)
|
||||
if got := rep.Periods[rep.Period].Label; got != "last month" {
|
||||
t.Fatalf("report opens on %q, want last month", got)
|
||||
}
|
||||
if len(rep.Rows) != 1 || rep.Rows[0].Out.Text != "10.00" {
|
||||
t.Errorf("last month's report = %+v, want groceries 10.00 only", rep.Rows)
|
||||
}
|
||||
|
||||
var txns struct{ Rows []txnRow }
|
||||
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
|
||||
if len(txns.Rows) != 2 {
|
||||
t.Errorf("transactions = %d rows, want both", len(txns.Rows))
|
||||
}
|
||||
|
||||
call(t, h, "GET", "/api/report?period=all+time", nil, http.StatusOK, &rep)
|
||||
if len(rep.Rows) != 1 || rep.Rows[0].Out.Text != "15.00" {
|
||||
t.Errorf("all-time report = %+v, want groceries 15.00", rep.Rows)
|
||||
}
|
||||
}
|
||||
|
||||
// The edit form has no type field, so the type must come from the rule on
|
||||
// disk; a pattern the user was never shown is not one they chose to remove.
|
||||
func TestEditKeepsTypePattern(t *testing.T) {
|
||||
s, h := newTestServer(t, `
|
||||
[[rule]]
|
||||
match = "*LIDL*"
|
||||
type = "CARD_PAYMENT"
|
||||
tag = "groceries"
|
||||
`)
|
||||
expect := ruleJSON{Match: "*LIDL*", Type: "CARD_PAYMENT", Tag: "groceries"}
|
||||
call(t, h, "PUT", "/api/rules/0", editRuleReq{
|
||||
Rule: ruleForm{Match: "*LIDL SOFIA*", Tag: "food"}, Expect: expect,
|
||||
}, http.StatusOK, nil)
|
||||
|
||||
got := readRules(t, s).Rule[0]
|
||||
if got.Type != "CARD_PAYMENT" || got.Match != "*LIDL SOFIA*" || got.Tag != "food" {
|
||||
t.Errorf("edited rule = %+v, want the type kept", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Positions are what rules.toml is edited by, so a page that saw a different
|
||||
// rule in that position must be refused rather than edit the wrong one.
|
||||
func TestStalePositionIsRefused(t *testing.T) {
|
||||
s, h := newTestServer(t, lidlRule)
|
||||
before, _ := os.ReadFile(filepath.Join(s.root, config.RulesFile))
|
||||
|
||||
call(t, h, "POST", "/api/rules/delete", deleteRulesReq{
|
||||
Positions: []int{0}, Expect: []ruleJSON{{Match: "*ZARA*", Tag: "clothes"}},
|
||||
}, http.StatusConflict, nil)
|
||||
call(t, h, "PUT", "/api/rules/0", editRuleReq{
|
||||
Rule: ruleForm{Match: "*X*", Tag: "x"}, Expect: ruleJSON{Match: "*ZARA*", Tag: "clothes"},
|
||||
}, http.StatusConflict, nil)
|
||||
|
||||
after, _ := os.ReadFile(filepath.Join(s.root, config.RulesFile))
|
||||
if !bytes.Equal(before, after) {
|
||||
t.Errorf("rules.toml changed on a refused request:\n%s", after)
|
||||
}
|
||||
}
|
||||
|
||||
// A body that is not JSON is refused before it reaches a handler: with no
|
||||
// authentication, that is what stops a cross-site form posting to the server.
|
||||
func TestFormPostsAreRefused(t *testing.T) {
|
||||
s, h := newTestServer(t, "")
|
||||
r := httptest.NewRequest("POST", "/api/rules", strings.NewReader("match=*&tag=x"))
|
||||
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusUnsupportedMediaType {
|
||||
t.Fatalf("status %d, want 415", w.Code)
|
||||
}
|
||||
if n := len(readRules(t, s).Rule); n != 0 {
|
||||
t.Errorf("%d rules written by a form post", n)
|
||||
}
|
||||
}
|
||||
|
||||
// Saving a rule writes rules.toml and retags, so the description it caught is
|
||||
// no longer offered as untagged.
|
||||
func TestCreateRuleRetags(t *testing.T) {
|
||||
s, h := newTestServer(t, "",
|
||||
fixtureTxn{"checking", "2026-02-10", "ZARA SOFIA", -1000},
|
||||
)
|
||||
var p rulePreviewJSON
|
||||
call(t, h, "POST", "/api/rules/preview", rulePreviewReq{Glob: "*zara*"}, http.StatusOK, &p)
|
||||
if p.Matches != 1 || p.Candidates != 1 || p.Rows[0].Marker != "▸" {
|
||||
t.Fatalf("preview = %+v, want the one description matched", p)
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/rules", ruleForm{Match: "*ZARA*", Tag: "clothes"}, http.StatusOK, nil)
|
||||
if r := readRules(t, s).Rule; len(r) != 1 || r[0].Tag != "clothes" {
|
||||
t.Fatalf("rules.toml = %+v", r)
|
||||
}
|
||||
call(t, h, "POST", "/api/rules/preview", rulePreviewReq{}, http.StatusOK, &p)
|
||||
if p.Candidates != 0 {
|
||||
t.Errorf("after saving, %d descriptions still untagged", p.Candidates)
|
||||
}
|
||||
|
||||
var list struct{ Rules []ruleRow }
|
||||
call(t, h, "GET", "/api/rules", nil, http.StatusOK, &list)
|
||||
if list.Rules[0].Usage != 1 {
|
||||
t.Errorf("usage = %d, want 1", list.Rules[0].Usage)
|
||||
}
|
||||
}
|
||||
|
||||
// An edit's preview includes what the rule claims now, and keeps a description
|
||||
// the new glob would let go of on screen, marked, instead of dropping it.
|
||||
func TestEditPreviewShowsWhatIsLetGo(t *testing.T) {
|
||||
_, h := newTestServer(t, lidlRule,
|
||||
fixtureTxn{"checking", "2026-02-10", "LIDL SOFIA", -1000},
|
||||
fixtureTxn{"checking", "2026-02-11", "LIDL VARNA", -1000},
|
||||
)
|
||||
edit := 0
|
||||
var p rulePreviewJSON
|
||||
call(t, h, "POST", "/api/rules/preview", rulePreviewReq{Glob: "*LIDL SOFIA*", Edit: &edit}, http.StatusOK, &p)
|
||||
if p.Matches != 1 || p.Dropped != 1 {
|
||||
t.Fatalf("preview = %+v, want 1 match and 1 dropped", p)
|
||||
}
|
||||
markers := map[string]string{}
|
||||
for _, r := range p.Rows {
|
||||
markers[r.Description] = r.Marker
|
||||
}
|
||||
if markers["LIDL SOFIA"] != "▸" || markers["LIDL VARNA"] != "−" {
|
||||
t.Errorf("markers = %v", markers)
|
||||
}
|
||||
}
|
||||
|
||||
// A transfer is a pair or nothing: the preview shows a leg with no other side
|
||||
// as unpaired, and only a matched pair leaves the report, with its fee named.
|
||||
func TestTransferPairsLeaveTheReportWithTheirFee(t *testing.T) {
|
||||
_, h := newTestServer(t, "",
|
||||
fixtureTxn{"checking", "2026-02-01", "WIRE TO SAVINGS", -50000},
|
||||
fixtureTxn{"savings", "2026-02-02", "WIRE FROM CHECKING", 49500},
|
||||
fixtureTxn{"checking", "2026-02-20", "WIRE TO SAVINGS", -10000},
|
||||
)
|
||||
form := transferForm{
|
||||
FromAccount: "checking", FromDesc: "*WIRE TO SAVINGS*",
|
||||
ToAccount: "savings", ToDesc: "*WIRE FROM CHECKING*",
|
||||
}
|
||||
|
||||
var p transferPreviewJSON
|
||||
call(t, h, "POST", "/api/transfers/preview", form, http.StatusOK, &p)
|
||||
if p.Pairs != 0 || p.Unpaired != 3 {
|
||||
t.Fatalf("exact preview = %+v, want nothing paired on a mismatch", p)
|
||||
}
|
||||
|
||||
form.Tolerance = "1.5"
|
||||
call(t, h, "POST", "/api/transfers/preview", form, http.StatusOK, &p)
|
||||
if p.Pairs != 1 || p.Unpaired != 1 || p.Fees != "5.00" {
|
||||
t.Fatalf("tolerant preview = %+v, want 1 pair, 1 unpaired, 5.00 in fees", p)
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/transfers", form, http.StatusOK, nil)
|
||||
|
||||
var rep reportJSON
|
||||
call(t, h, "GET", "/api/report?period=all+time", nil, http.StatusOK, &rep)
|
||||
// The unpaired leg keeps counting; the pair is excluded with its fee.
|
||||
if len(rep.Rows) != 1 || rep.Rows[0].Out.Text != "100.00" {
|
||||
t.Errorf("report rows = %+v, want only the unpaired 100.00", rep.Rows)
|
||||
}
|
||||
if len(rep.Excluded) != 1 || rep.Excluded[0].Fee == nil || rep.Excluded[0].Fee.Text != "5.00" {
|
||||
t.Errorf("excluded = %+v, want the 5.00 fee reported", rep.Excluded)
|
||||
}
|
||||
|
||||
var list struct {
|
||||
Transfers []transferRow
|
||||
Unpaired int
|
||||
}
|
||||
call(t, h, "GET", "/api/transfers", nil, http.StatusOK, &list)
|
||||
if list.Unpaired != 1 || list.Transfers[0].Paired != 1 || list.Transfers[0].Tolerance != "1.5%" {
|
||||
t.Errorf("transfer list = %+v", list)
|
||||
}
|
||||
}
|
||||
|
||||
// The tolerance may be half typed while previewing, but saving refuses it
|
||||
// rather than writing a silent zero.
|
||||
func TestBadToleranceIsRefusedOnSave(t *testing.T) {
|
||||
s, h := newTestServer(t, "")
|
||||
call(t, h, "POST", "/api/transfers", transferForm{
|
||||
FromAccount: "checking", FromDesc: "*A*", ToAccount: "savings", ToDesc: "*B*", Tolerance: "1,5",
|
||||
}, http.StatusBadRequest, nil)
|
||||
if n := len(readRules(t, s).Transfer); n != 0 {
|
||||
t.Errorf("%d transfers written", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The server outlives hand edits to rules.toml. The overview says when the
|
||||
// file has moved on, and retag re-reads it, as a fresh `money retag` would.
|
||||
func TestRetagRereadsRulesFromDisk(t *testing.T) {
|
||||
s, h := newTestServer(t, "",
|
||||
fixtureTxn{"checking", "2026-02-10", "ZARA", -1000},
|
||||
)
|
||||
var o overview
|
||||
call(t, h, "GET", "/api/overview", nil, http.StatusOK, &o)
|
||||
if o.Stale {
|
||||
t.Fatal("stale before anything changed")
|
||||
}
|
||||
|
||||
path := filepath.Join(s.root, config.RulesFile)
|
||||
if err := os.WriteFile(path, []byte("[[rule]]\nmatch = \"*ZARA*\"\ntag = \"clothes\"\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
call(t, h, "GET", "/api/overview", nil, http.StatusOK, &o)
|
||||
if !o.Stale {
|
||||
t.Fatal("a hand edit to rules.toml is not reported")
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/retag", struct{}{}, http.StatusOK, nil)
|
||||
call(t, h, "GET", "/api/overview", nil, http.StatusOK, &o)
|
||||
if o.Stale {
|
||||
t.Error("still stale after retag")
|
||||
}
|
||||
var txns struct{ Rows []txnRow }
|
||||
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
|
||||
if txns.Rows[0].Tag != "clothes" {
|
||||
t.Errorf("tag = %q after retag, want clothes", txns.Rows[0].Tag)
|
||||
}
|
||||
}
|
||||
|
||||
// (transfer) is a label for a paired leg, never a tag: it is flagged as
|
||||
// supplied, and never offered for completion.
|
||||
func TestTransferLabelIsNotATag(t *testing.T) {
|
||||
_, h := newTestServer(t, `
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*TO SAVINGS*"
|
||||
to_account = "savings"
|
||||
to_desc = "*FROM CHECKING*"
|
||||
`,
|
||||
fixtureTxn{"checking", "2026-02-01", "TO SAVINGS", -1000},
|
||||
fixtureTxn{"savings", "2026-02-01", "FROM CHECKING", 1000},
|
||||
)
|
||||
var txns struct{ Rows []txnRow }
|
||||
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
|
||||
for _, r := range txns.Rows {
|
||||
if r.Tag != model.TransferTag || !r.Supplied {
|
||||
t.Errorf("row %+v, want a supplied transfer label", r)
|
||||
}
|
||||
}
|
||||
call(t, h, "GET", "/api/transactions?untagged=1", nil, http.StatusOK, &txns)
|
||||
if len(txns.Rows) != 0 {
|
||||
t.Errorf("paired legs listed as untagged: %+v", txns.Rows)
|
||||
}
|
||||
var o overview
|
||||
call(t, h, "GET", "/api/overview", nil, http.StatusOK, &o)
|
||||
if len(o.Tags) != 0 {
|
||||
t.Errorf("tags offered = %v, want none", o.Tags)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPageIsServed(t *testing.T) {
|
||||
_, h := newTestServer(t, "")
|
||||
for _, path := range []string{"/", "/app.js", "/style.css"} {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest("GET", path, nil))
|
||||
if w.Code != http.StatusOK || w.Body.Len() == 0 {
|
||||
t.Errorf("GET %s: status %d, %d bytes", path, w.Code, w.Body.Len())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A paired leg is spoken for, so it is neither listed as untagged nor offered
|
||||
// to the rule builder; an unpaired leg is untagged like anything else, which
|
||||
// is how it gets noticed.
|
||||
func TestPairedLegsAreNotUntagged(t *testing.T) {
|
||||
_, h := newTestServer(t, `
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*TO SAVINGS*"
|
||||
to_account = "savings"
|
||||
to_desc = "*FROM CHECKING*"
|
||||
`,
|
||||
fixtureTxn{"checking", "2026-02-01", "TRANSFER TO SAVINGS", -1000},
|
||||
fixtureTxn{"savings", "2026-02-01", "TRANSFER FROM CHECKING", 1000},
|
||||
fixtureTxn{"checking", "2026-02-20", "TRANSFER TO SAVINGS", -2000},
|
||||
fixtureTxn{"checking", "2026-02-05", "LIDL SOFIA", -500},
|
||||
)
|
||||
|
||||
var txns struct{ Rows []txnRow }
|
||||
call(t, h, "GET", "/api/transactions?untagged=1", nil, http.StatusOK, &txns)
|
||||
var seen []string
|
||||
for _, r := range txns.Rows {
|
||||
seen = append(seen, r.Date+" "+r.Description)
|
||||
}
|
||||
want := "2026-02-20 TRANSFER TO SAVINGS|2026-02-05 LIDL SOFIA"
|
||||
if strings.Join(seen, "|") != want {
|
||||
t.Errorf("untagged = %v, want only the unpaired leg and the shopping", seen)
|
||||
}
|
||||
|
||||
var p rulePreviewJSON
|
||||
call(t, h, "POST", "/api/rules/preview", rulePreviewReq{Glob: "*"}, http.StatusOK, &p)
|
||||
for _, r := range p.Rows {
|
||||
if r.Description == "TRANSFER FROM CHECKING" {
|
||||
t.Error("the arriving leg of a matched transfer is still offered for tagging")
|
||||
}
|
||||
if r.Description == "TRANSFER TO SAVINGS" && r.Count != 1 {
|
||||
t.Errorf("departing leg offered %d times, want only the unpaired one", r.Count)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A ⚠ on the transfers screen opens onto the legs behind it: which side each
|
||||
// is missing, and what that side would have to be.
|
||||
func TestTransferListNamesTheMissingLegs(t *testing.T) {
|
||||
_, h := newTestServer(t, `
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*TO SAVINGS*"
|
||||
to_account = "savings"
|
||||
to_desc = "*FROM CHECKING*"
|
||||
`,
|
||||
fixtureTxn{"checking", "2026-02-01", "TO SAVINGS", -1000},
|
||||
fixtureTxn{"savings", "2026-02-02", "FROM CHECKING", 1000},
|
||||
fixtureTxn{"checking", "2026-02-20", "TO SAVINGS", -2500},
|
||||
fixtureTxn{"savings", "2026-01-10", "FROM CHECKING", 700},
|
||||
)
|
||||
var list struct {
|
||||
Transfers []transferRow
|
||||
Unpaired int
|
||||
}
|
||||
call(t, h, "GET", "/api/transfers", nil, http.StatusOK, &list)
|
||||
legs := list.Transfers[0].Legs
|
||||
if list.Unpaired != 2 || len(legs) != 2 {
|
||||
t.Fatalf("unpaired = %d, legs = %+v; want the two lone legs", list.Unpaired, legs)
|
||||
}
|
||||
newest, oldest := legs[0], legs[1]
|
||||
if newest.Movement != "checking → ?" || newest.Missing != "arriving" ||
|
||||
newest.Want != "+25.00 in savings matching *FROM CHECKING*, dated 2026-02-15 to 2026-02-25" {
|
||||
t.Errorf("newest leg = %+v", newest)
|
||||
}
|
||||
if oldest.Movement != "? → savings" || oldest.Missing != "leaving" ||
|
||||
oldest.Want != "-7.00 in checking matching *TO SAVINGS*, dated 2026-01-05 to 2026-01-15" {
|
||||
t.Errorf("oldest leg = %+v", oldest)
|
||||
}
|
||||
}
|
||||
|
||||
// The tags screen lists each tag once, with the transactions carrying it and
|
||||
// every rule that writes it — a shadowed one at zero — and never the
|
||||
// transfer label, which no rule wrote.
|
||||
func TestTagListGroupsRulesByTag(t *testing.T) {
|
||||
_, h := newTestServer(t, `
|
||||
[[rule]]
|
||||
match = "*LIDL*"
|
||||
tag = "groceries"
|
||||
|
||||
[[rule]]
|
||||
match = "*KAUFLAND*"
|
||||
tag = "groceries"
|
||||
|
||||
[[rule]]
|
||||
match = "*LIDL SOFIA*"
|
||||
tag = "groceries"
|
||||
|
||||
[[rule]]
|
||||
match = "*ZARA*"
|
||||
tag = "clothes"
|
||||
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*TO SAVINGS*"
|
||||
to_account = "savings"
|
||||
to_desc = "*FROM CHECKING*"
|
||||
`,
|
||||
fixtureTxn{"checking", "2026-02-01", "LIDL SOFIA", -1000},
|
||||
fixtureTxn{"checking", "2026-02-02", "KAUFLAND", -500},
|
||||
fixtureTxn{"checking", "2026-02-03", "TO SAVINGS", -100},
|
||||
fixtureTxn{"savings", "2026-02-03", "FROM CHECKING", 100},
|
||||
)
|
||||
var res struct{ Tags []tagEntry }
|
||||
call(t, h, "GET", "/api/tags", nil, http.StatusOK, &res)
|
||||
if len(res.Tags) != 2 || res.Tags[0].Tag != "clothes" || res.Tags[1].Tag != "groceries" {
|
||||
t.Fatalf("tags = %+v, want clothes and groceries only", res.Tags)
|
||||
}
|
||||
clothes, groceries := res.Tags[0], res.Tags[1]
|
||||
if clothes.Transactions != 0 || len(clothes.Rules) != 1 {
|
||||
t.Errorf("clothes = %+v, want its rule and no transactions", clothes)
|
||||
}
|
||||
if groceries.Transactions != 2 || len(groceries.Rules) != 3 {
|
||||
t.Fatalf("groceries = %+v, want 2 transactions from 3 rules", groceries)
|
||||
}
|
||||
usage := map[string]int{}
|
||||
for _, r := range groceries.Rules {
|
||||
usage[r.Pattern] = r.Usage
|
||||
}
|
||||
if usage["*LIDL*"] != 0 || usage["*LIDL SOFIA*"] != 1 || usage["*KAUFLAND*"] != 1 {
|
||||
t.Errorf("usage = %v, want *LIDL* shadowed by *LIDL SOFIA*", usage)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,8 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
|
||||
<rect width="32" height="32" rx="7" fill="#4b46c8"/>
|
||||
<g fill="#fff">
|
||||
<rect x="7" y="8" width="18" height="4" rx="2"/>
|
||||
<rect x="7" y="14" width="11" height="4" rx="2"/>
|
||||
<rect x="7" y="20" width="15" height="4" rx="2"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 309 B |
@@ -0,0 +1,24 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>money</title>
|
||||
<link rel="icon" href="icon.svg" type="image/svg+xml">
|
||||
<link rel="stylesheet" href="style.css">
|
||||
</head>
|
||||
<body>
|
||||
<header class="top">
|
||||
<a class="brand" href="#accounts">money</a>
|
||||
<nav id="nav" aria-label="Screens"></nav>
|
||||
<div class="actions">
|
||||
<button type="button" id="retag" title="Re-apply rules.toml: retag everything and re-pair transfers">Retag</button>
|
||||
<button type="button" id="reread" title="Re-read every statement, including ones unchanged since they were imported">Re-read all</button>
|
||||
<button type="button" id="import" class="primary" title="Import new statements from every account folder">Import</button>
|
||||
</div>
|
||||
</header>
|
||||
<div id="banners"></div>
|
||||
<main id="main"></main>
|
||||
<script src="app.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,218 @@
|
||||
:root {
|
||||
--bg: #f7f7f5;
|
||||
--surface: #ffffff;
|
||||
--text: #1d1d1f;
|
||||
--muted: #6b6b70;
|
||||
--border: #e2e2de;
|
||||
--accent: #4b46c8;
|
||||
--accent-text: #ffffff;
|
||||
--accent-soft: #ecebfb;
|
||||
--negative: #b3261e;
|
||||
--positive: #1d7a3a;
|
||||
--warn: #9a5b00;
|
||||
--warn-soft: #fff4e0;
|
||||
--error-soft: #fdecea;
|
||||
--row-hover: #f2f2ef;
|
||||
--mono: ui-monospace, "SF Mono", "Cascadia Mono", Menlo, Consolas, monospace;
|
||||
color-scheme: light;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #161618;
|
||||
--surface: #1e1e21;
|
||||
--text: #e8e8ea;
|
||||
--muted: #9a9aa2;
|
||||
--border: #333338;
|
||||
--accent: #8f8cf0;
|
||||
--accent-text: #111114;
|
||||
--accent-soft: #2a2950;
|
||||
--negative: #f2867d;
|
||||
--positive: #6fcf8e;
|
||||
--warn: #f0b45a;
|
||||
--warn-soft: #3a2c14;
|
||||
--error-soft: #3d1c1a;
|
||||
--row-hover: #26262a;
|
||||
color-scheme: dark;
|
||||
}
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0;
|
||||
background: var(--bg);
|
||||
color: var(--text);
|
||||
font: 14px/1.45 system-ui, -apple-system, "Segoe UI", sans-serif;
|
||||
}
|
||||
button, input, select { font: inherit; color: inherit; }
|
||||
|
||||
.top {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 16px;
|
||||
padding: 10px 16px;
|
||||
background: var(--surface);
|
||||
border-bottom: 1px solid var(--border);
|
||||
flex-wrap: wrap;
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 2;
|
||||
}
|
||||
.brand { font-weight: 700; font-size: 16px; color: var(--text); text-decoration: none; }
|
||||
#nav { display: flex; gap: 2px; flex-wrap: wrap; flex: 1; }
|
||||
#nav a {
|
||||
padding: 5px 10px;
|
||||
border-radius: 6px;
|
||||
color: var(--muted);
|
||||
text-decoration: none;
|
||||
white-space: nowrap;
|
||||
}
|
||||
#nav a:hover { color: var(--text); background: var(--row-hover); }
|
||||
#nav a.active { color: var(--accent); background: var(--accent-soft); font-weight: 600; }
|
||||
.actions { display: flex; gap: 8px; }
|
||||
|
||||
button {
|
||||
padding: 5px 12px;
|
||||
border-radius: 6px;
|
||||
border: 1px solid var(--border);
|
||||
background: var(--surface);
|
||||
cursor: pointer;
|
||||
}
|
||||
button:hover:not(:disabled) { border-color: var(--muted); }
|
||||
button:disabled { opacity: .5; cursor: default; }
|
||||
button.primary { background: var(--accent); border-color: var(--accent); color: var(--accent-text); }
|
||||
button.danger { color: var(--negative); }
|
||||
button.link { border: none; background: none; padding: 0 4px; color: var(--accent); }
|
||||
|
||||
#banners { padding: 0 16px; }
|
||||
.banner {
|
||||
margin-top: 10px;
|
||||
padding: 8px 12px;
|
||||
border-radius: 6px;
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
.banner.info { background: var(--accent-soft); }
|
||||
.banner.warn { background: var(--warn-soft); color: var(--warn); }
|
||||
.banner.error { background: var(--error-soft); color: var(--negative); }
|
||||
.banner button { flex: none; }
|
||||
|
||||
main { padding: 16px; max-width: 1400px; }
|
||||
h2 { font-size: 16px; margin: 0 0 12px; font-weight: 600; }
|
||||
h2 .sub { color: var(--muted); font-weight: 400; }
|
||||
|
||||
.toolbar { display: flex; gap: 10px; align-items: center; flex-wrap: wrap; margin-bottom: 12px; }
|
||||
.toolbar input[type=search] { min-width: 220px; }
|
||||
input[type=text], input[type=search], select {
|
||||
padding: 5px 8px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 6px;
|
||||
background: var(--surface);
|
||||
}
|
||||
input:focus, select:focus { outline: 2px solid var(--accent); outline-offset: -1px; border-color: transparent; }
|
||||
label.check { display: inline-flex; gap: 6px; align-items: center; cursor: pointer; }
|
||||
|
||||
.panel {
|
||||
background: var(--surface);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 8px;
|
||||
overflow: auto;
|
||||
}
|
||||
table { border-collapse: collapse; width: 100%; }
|
||||
th, td { padding: 5px 10px; text-align: left; white-space: nowrap; }
|
||||
th {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
background: var(--surface);
|
||||
border-bottom: 1px solid var(--border);
|
||||
font-weight: 600;
|
||||
z-index: 1;
|
||||
}
|
||||
tbody tr:hover { background: var(--row-hover); }
|
||||
tbody tr.clickable { cursor: pointer; }
|
||||
td.num, th.num { text-align: right; font-family: var(--mono); font-variant-numeric: tabular-nums; }
|
||||
td.desc { white-space: normal; min-width: 240px; }
|
||||
td.mark { width: 1em; text-align: center; padding-right: 0; }
|
||||
.neg { color: var(--negative); }
|
||||
.muted { color: var(--muted); }
|
||||
.warn { color: var(--warn); }
|
||||
th.sortable { cursor: pointer; user-select: none; }
|
||||
th.sortable:hover { color: var(--accent); }
|
||||
tr.total td { font-weight: 600; border-top: 1px solid var(--border); }
|
||||
tr.excluded td { color: var(--muted); }
|
||||
tr.excluded.first td { border-top: 1px dashed var(--border); }
|
||||
td.actions { text-align: right; }
|
||||
.mono { font-family: var(--mono); }
|
||||
.tag-rule { display: inline-block; margin-right: 14px; white-space: nowrap; }
|
||||
tr.legs > td { padding: 0 10px 10px 36px; background: var(--bg); }
|
||||
tr.legs:hover { background: none; }
|
||||
table.sub th { background: var(--bg); font-weight: 500; color: var(--muted); }
|
||||
td.actions button { padding: 1px 8px; font-size: 12px; }
|
||||
|
||||
.empty { padding: 32px 16px; color: var(--muted); white-space: pre-line; text-align: center; }
|
||||
.more { padding: 10px; text-align: center; }
|
||||
|
||||
.report { display: grid; grid-template-columns: 220px 1fr; gap: 12px; align-items: start; }
|
||||
.periods { padding: 6px; }
|
||||
.periods button {
|
||||
display: block;
|
||||
width: 100%;
|
||||
text-align: left;
|
||||
border: none;
|
||||
background: none;
|
||||
padding: 4px 8px;
|
||||
border-radius: 5px;
|
||||
color: var(--muted);
|
||||
}
|
||||
.periods button:hover { background: var(--row-hover); color: var(--text); }
|
||||
.periods button.active { background: var(--accent-soft); color: var(--accent); font-weight: 600; }
|
||||
.periods .head { font-weight: 600; padding: 4px 8px; }
|
||||
|
||||
.builder { display: grid; grid-template-columns: 320px 1fr; gap: 16px; align-items: start; }
|
||||
.form { padding: 14px; display: flex; flex-direction: column; gap: 12px; }
|
||||
.field label { display: block; font-weight: 600; margin-bottom: 3px; }
|
||||
.field input { width: 100%; font-family: var(--mono); }
|
||||
.field .hint { color: var(--muted); font-size: 12px; margin-top: 2px; }
|
||||
.form .summary { font-weight: 600; white-space: pre-line; }
|
||||
.form .buttons { display: flex; gap: 8px; }
|
||||
.kept { font-size: 12px; color: var(--muted); font-family: var(--mono); }
|
||||
.preview { max-height: calc(100vh - 170px); }
|
||||
|
||||
h3.section { font-size: 14px; font-weight: 600; margin: 20px 0 8px; }
|
||||
h3.section .sub { color: var(--muted); font-weight: 400; }
|
||||
.files a { color: var(--accent); text-decoration: none; }
|
||||
.files a:hover { text-decoration: underline; }
|
||||
|
||||
.upload { margin-top: 16px; padding: 14px; overflow: visible; }
|
||||
.upload h3 { font-size: 14px; margin: 0 0 10px; }
|
||||
.upload .toolbar { margin-bottom: 10px; }
|
||||
.dropzone {
|
||||
border: 2px dashed var(--border);
|
||||
border-radius: 8px;
|
||||
padding: 22px 16px;
|
||||
text-align: center;
|
||||
color: var(--muted);
|
||||
cursor: pointer;
|
||||
}
|
||||
.dropzone:hover, .dropzone:focus, .dropzone.over {
|
||||
border-color: var(--accent);
|
||||
color: var(--accent);
|
||||
background: var(--accent-soft);
|
||||
outline: none;
|
||||
}
|
||||
.upload .chosen { margin-top: 8px; font-family: var(--mono); font-size: 12px; overflow-wrap: anywhere; }
|
||||
.upload .hint { margin-top: 8px; font-size: 12px; }
|
||||
|
||||
.import-files { margin-top: 6px; font-size: 13px; }
|
||||
.import-files li { font-family: var(--mono); }
|
||||
|
||||
@media (max-width: 760px) {
|
||||
.report, .builder { grid-template-columns: 1fr; }
|
||||
.periods { display: flex; flex-wrap: wrap; gap: 2px; }
|
||||
.periods button { width: auto; }
|
||||
.periods .head { width: 100%; }
|
||||
.toolbar input[type=search] { min-width: 0; flex: 1; }
|
||||
.preview { max-height: none; }
|
||||
}
|
||||
@@ -0,0 +1,442 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.petrovv.com/nikola/money/internal/config"
|
||||
"git.petrovv.com/nikola/money/internal/parser"
|
||||
"git.petrovv.com/nikola/money/internal/rules"
|
||||
"git.petrovv.com/nikola/money/internal/store"
|
||||
"git.petrovv.com/nikola/money/internal/transfers"
|
||||
)
|
||||
|
||||
// The upload tests are about where files land and what is refused, not about
|
||||
// any bank's layout, so they read "date,description,amount" with a header.
|
||||
func init() {
|
||||
parser.Register("webtest", func(acc *config.Account) (parser.Parser, error) {
|
||||
return csvParser{digits: acc.Digits()}, nil
|
||||
})
|
||||
parser.Register("webnamed", func(acc *config.Account) (parser.Parser, error) {
|
||||
return namingParser{csvParser{digits: acc.Digits()}}, nil
|
||||
})
|
||||
}
|
||||
|
||||
type csvParser struct{ digits int }
|
||||
|
||||
// namingParser is csvParser for a bank whose downloads are named unhelpfully:
|
||||
// it names a statement after its first date, as nlb does after the statement
|
||||
// date, and says nothing for a statement with no rows.
|
||||
type namingParser struct{ csvParser }
|
||||
|
||||
func (namingParser) StatementName(path string) (string, error) {
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(body)), "\n")
|
||||
if len(lines) < 2 {
|
||||
return "", nil
|
||||
}
|
||||
return "stmt_" + strings.Split(lines[1], ",")[0], nil
|
||||
}
|
||||
|
||||
func (p csvParser) Parse(path string, _ *config.Account) ([]parser.RawTxn, error) {
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var txns []parser.RawTxn
|
||||
for i, line := range strings.Split(strings.TrimSpace(string(body)), "\n")[1:] {
|
||||
f := strings.Split(line, ",")
|
||||
if len(f) != 3 {
|
||||
return nil, fmt.Errorf("row %d: want 3 fields", i+2)
|
||||
}
|
||||
amount, err := parser.ParseAmount(f[2], ".", "", p.digits)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
txns = append(txns, parser.RawTxn{Date: f[0], Description: f[1], AmountMinor: amount})
|
||||
}
|
||||
return txns, nil
|
||||
}
|
||||
|
||||
// newUploadServer builds a server over a data root with one empty account
|
||||
// folder, as `money serve` sees it before the first import.
|
||||
func newUploadServer(t *testing.T, accountTOML string) (string, http.Handler) {
|
||||
t.Helper()
|
||||
root := t.TempDir()
|
||||
dir := filepath.Join(root, "checking")
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(dir, config.AccountFile), []byte(accountTOML), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := store.Open(config.IndexPath(root))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { db.Close() })
|
||||
accounts, err := config.LoadAccounts(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loaded, err := config.LoadRules(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s := New(root, db, accounts, rules.New(loaded), transfers.New(loaded))
|
||||
s.now = func() time.Time { return time.Date(2026, 3, 15, 0, 0, 0, 0, time.UTC) }
|
||||
return dir, s.Handler()
|
||||
}
|
||||
|
||||
const checkingTOML = "currency = \"EUR\"\nparser = \"webtest\"\n"
|
||||
|
||||
const statement = "date,description,amount\n2026-02-01,LIDL SOFIA,-12.50\n2026-02-03,SALARY,1000.00\n"
|
||||
|
||||
// An upload only saves: the file waits on the statements list as new until
|
||||
// the user presses Import.
|
||||
func TestUploadSavesWithoutImporting(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
req := uploadReq{Account: "checking", Files: []uploadFile{{Name: "2026-02.csv", Data: []byte(statement)}}}
|
||||
|
||||
var res status
|
||||
call(t, h, "POST", "/api/upload", req, http.StatusOK, &res)
|
||||
if res.Status != "uploaded 1 file(s) to checking · press Import to read them" {
|
||||
t.Errorf("status = %q", res.Status)
|
||||
}
|
||||
if got, err := os.ReadFile(filepath.Join(dir, "2026-02.csv")); err != nil || string(got) != statement {
|
||||
t.Errorf("file on disk = %q, %v", got, err)
|
||||
}
|
||||
if got := descriptions(t, h); got != "" {
|
||||
t.Errorf("upload imported %s; it should only save", got)
|
||||
}
|
||||
var files struct{ Files []fileRow }
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
|
||||
if len(files.Files) != 1 || files.Files[0].Status != "new" {
|
||||
t.Errorf("files = %+v, want the upload waiting as new", files.Files)
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
if got := descriptions(t, h); got != "SALARY,LIDL SOFIA" {
|
||||
t.Errorf("after Import the index holds %s", got)
|
||||
}
|
||||
|
||||
// The same file again is not an error, and saves nothing.
|
||||
call(t, h, "POST", "/api/upload", req, http.StatusOK, &res)
|
||||
if res.Status != "uploaded 0 file(s) to checking (1 already there)" {
|
||||
t.Errorf("re-upload status = %q", res.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// A statement already in the folder is the source of truth for what it
|
||||
// imported, so an upload never replaces one with different contents.
|
||||
func TestUploadNeverReplacesAStatement(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
path := filepath.Join(dir, "2026-02.csv")
|
||||
if err := os.WriteFile(path, []byte(statement), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req := uploadReq{Account: "checking", Files: []uploadFile{{Name: "2026-02.csv", Data: []byte("date,description,amount\n")}}}
|
||||
call(t, h, "POST", "/api/upload", req, http.StatusConflict, nil)
|
||||
if got, _ := os.ReadFile(path); string(got) != statement {
|
||||
t.Errorf("statement was overwritten: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A batch is checked as a whole before anything is written, and nothing may
|
||||
// land outside the account folder or where import would not read it back.
|
||||
func TestUploadRefusesBadNames(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML+"include = [\"*.csv\"]\n")
|
||||
for _, bad := range []string{"../escape.csv", "sub/x.csv", ".hidden.csv", config.AccountFile, "notes.txt", ""} {
|
||||
req := uploadReq{Account: "checking", Files: []uploadFile{
|
||||
{Name: "good.csv", Data: []byte(statement)},
|
||||
{Name: bad, Data: []byte(statement)},
|
||||
}}
|
||||
call(t, h, "POST", "/api/upload", req, http.StatusBadRequest, nil)
|
||||
}
|
||||
entries, _ := os.ReadDir(dir)
|
||||
if len(entries) != 1 {
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
t.Errorf("folder holds %v, want only %s", names, config.AccountFile)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(filepath.Dir(dir), "escape.csv")); err == nil {
|
||||
t.Error("a file escaped the account folder")
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/upload", uploadReq{Account: "nope", Files: []uploadFile{{Name: "a.csv"}}},
|
||||
http.StatusBadRequest, nil)
|
||||
}
|
||||
|
||||
// Multipart is what a cross-site form can send, so it is refused like any
|
||||
// other non-JSON write.
|
||||
func TestUploadRefusesMultipart(t *testing.T) {
|
||||
_, h := newUploadServer(t, checkingTOML)
|
||||
r := httptest.NewRequest("POST", "/api/upload", strings.NewReader("--x\r\n"))
|
||||
r.Header.Set("Content-Type", "multipart/form-data; boundary=x")
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusUnsupportedMediaType {
|
||||
t.Errorf("status %d, want %d", w.Code, http.StatusUnsupportedMediaType)
|
||||
}
|
||||
}
|
||||
|
||||
// The statements list is the folders, each beside what the index recorded:
|
||||
// imported, changed since, not imported yet, and gone from disk.
|
||||
func TestFileListStatuses(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
write := func(name, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
write("a.csv", statement)
|
||||
write("b.csv", "date,description,amount\n2026-03-01,ZARA,-40.00\n")
|
||||
write("c.csv", "date,description,amount\n2026-04-01,KAUFLAND,-9.00\n")
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
|
||||
write("b.csv", "date,description,amount\n2026-03-01,ZARA,-41.00\n")
|
||||
if err := os.Remove(filepath.Join(dir, "c.csv")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write("d.csv", statement)
|
||||
write(".hidden.csv", statement)
|
||||
|
||||
var res struct{ Files []fileRow }
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &res)
|
||||
got := map[string]fileRow{}
|
||||
var names []string
|
||||
for _, f := range res.Files {
|
||||
got[f.Name] = f
|
||||
names = append(names, f.Name)
|
||||
}
|
||||
if strings.Join(names, " ") != "a.csv b.csv c.csv d.csv" {
|
||||
t.Fatalf("files = %v, want the four statements and nothing import would skip", names)
|
||||
}
|
||||
for name, want := range map[string]string{"a.csv": "imported", "b.csv": "changed", "c.csv": "missing", "d.csv": "new"} {
|
||||
if got[name].Status != want {
|
||||
t.Errorf("%s status = %q, want %q", name, got[name].Status, want)
|
||||
}
|
||||
}
|
||||
if got["a.csv"].Rows != 2 || got["a.csv"].Added != 2 || got["c.csv"].Rows != 1 || got["c.csv"].Added != 1 ||
|
||||
got["a.csv"].ImportedAt == "" {
|
||||
t.Errorf("a.csv = %+v, c.csv = %+v", got["a.csv"], got["c.csv"])
|
||||
}
|
||||
if got["a.csv"].Size != int64(len(statement)) {
|
||||
t.Errorf("a.csv size = %d, want %d", got["a.csv"].Size, len(statement))
|
||||
}
|
||||
}
|
||||
|
||||
// Only a file import would read is served, and never as a page.
|
||||
func TestServeFileServesOnlyStatements(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
for name, body := range map[string]string{"a.csv": statement, "page.html": "<script>alert(1)</script>", ".secret": "x"} {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
get := func(path string) *httptest.ResponseRecorder {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest("GET", path, nil))
|
||||
return w
|
||||
}
|
||||
|
||||
w := get("/api/files/checking/a.csv")
|
||||
if w.Code != http.StatusOK || w.Body.String() != statement {
|
||||
t.Fatalf("a.csv: %d %q", w.Code, w.Body.String())
|
||||
}
|
||||
if ct := w.Header().Get("Content-Type"); ct != "text/plain; charset=utf-8" {
|
||||
t.Errorf("a.csv content type = %q", ct)
|
||||
}
|
||||
|
||||
w = get("/api/files/checking/page.html")
|
||||
if w.Code != http.StatusOK || w.Header().Get("Content-Type") != "application/octet-stream" ||
|
||||
w.Header().Get("Content-Security-Policy") != "sandbox" ||
|
||||
!strings.HasPrefix(w.Header().Get("Content-Disposition"), "attachment") {
|
||||
t.Errorf("page.html served as %d %v, want a sandboxed download", w.Code, w.Header())
|
||||
}
|
||||
|
||||
for _, path := range []string{
|
||||
"/api/files/checking/" + config.AccountFile,
|
||||
"/api/files/checking/.secret",
|
||||
"/api/files/checking/..%2F" + config.RulesFile,
|
||||
"/api/files/nope/a.csv",
|
||||
} {
|
||||
if w := get(path); w.Code != http.StatusNotFound {
|
||||
t.Errorf("%s: status %d, want 404", path, w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A parser that names its statements decides the stored name, so the bank's
|
||||
// "download (3).pdf" lands as something that sorts by date. A reissue of the
|
||||
// same date is numbered rather than refused, and a statement that does not
|
||||
// say keeps the name it came with.
|
||||
func TestUploadNamesStatementsTheParserCanName(t *testing.T) {
|
||||
dir, h := newUploadServer(t, "currency = \"EUR\"\nparser = \"webnamed\"\n")
|
||||
upload := func(files ...uploadFile) string {
|
||||
t.Helper()
|
||||
var res status
|
||||
call(t, h, "POST", "/api/upload", uploadReq{Account: "checking", Files: files}, http.StatusOK, &res)
|
||||
return res.Status
|
||||
}
|
||||
reissue := "date,description,amount\n2026-02-01,LIDL SOFIA,-12.50\n2026-02-04,ZARA,-30.00\n"
|
||||
other := "date,description,amount\n2026-03-01,KAUFLAND,-9.00\n"
|
||||
|
||||
status := upload(uploadFile{Name: "download (3).csv", Data: []byte(statement)})
|
||||
if !strings.Contains(status, "download (3).csv → stmt_2026-02-01.csv") {
|
||||
t.Errorf("status = %q, want the rename named", status)
|
||||
}
|
||||
// The same statement downloaded again is recognised under its new name.
|
||||
if status := upload(uploadFile{Name: "download (4).csv", Data: []byte(statement)}); !strings.Contains(status, "1 already there") {
|
||||
t.Errorf("second download: status = %q", status)
|
||||
}
|
||||
// A different statement of the same date, and two of one date in a batch.
|
||||
upload(uploadFile{Name: "x.csv", Data: []byte(reissue)},
|
||||
uploadFile{Name: "y.csv", Data: []byte(reissue + "2026-02-05,BOLT,-4.00\n")},
|
||||
uploadFile{Name: "Z.CSV", Data: []byte(other)}, // a chosen name is lowercase
|
||||
uploadFile{Name: "empty.csv", Data: []byte("date,description,amount\n")})
|
||||
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
want := config.AccountFile + " empty.csv stmt_2026-02-01.csv stmt_2026-02-01_2.csv stmt_2026-02-01_3.csv stmt_2026-03-01.csv"
|
||||
if strings.Join(names, " ") != want {
|
||||
t.Errorf("folder = %v, want %s", names, want)
|
||||
}
|
||||
}
|
||||
|
||||
func writeFile(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func descriptions(t *testing.T, h http.Handler) string {
|
||||
t.Helper()
|
||||
var txns struct{ Rows []txnRow }
|
||||
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
|
||||
var out []string
|
||||
for _, r := range txns.Rows {
|
||||
out = append(out, r.Description)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
// Removing a statement deletes the file and takes out what it brought in. A
|
||||
// row an overlapping statement also holds was stored under the file imported
|
||||
// first; the next Import brings it back from the other one.
|
||||
func TestRemoveFileKeepsWhatAnotherStatementHolds(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
writeFile(t, filepath.Join(dir, "a.csv"), "date,description,amount\n2026-01-30,ONLY IN A,-1.00\n2026-02-01,SHARED,-2.00\n")
|
||||
writeFile(t, filepath.Join(dir, "b.csv"), "date,description,amount\n2026-02-01,SHARED,-2.00\n2026-02-03,ONLY IN B,-3.00\n")
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
|
||||
// Each holds two rows; b.csv was imported second, so its shared row
|
||||
// stayed with a.csv and it brought in only one.
|
||||
var before struct{ Files []fileRow }
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &before)
|
||||
if f := before.Files; len(f) != 2 || f[0].Rows != 2 || f[0].Added != 2 || f[1].Rows != 2 || f[1].Added != 1 {
|
||||
t.Errorf("files = %+v, want a.csv 2 rows · 2 new and b.csv 2 rows · 1 new", f)
|
||||
}
|
||||
|
||||
var res status
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusOK, &res)
|
||||
if !strings.HasPrefix(res.Status, "deleted checking/a.csv, 2 transaction(s) dropped") {
|
||||
t.Errorf("status = %q", res.Status)
|
||||
}
|
||||
// Deleting does not import, so the shared row is gone for now, and b.csv
|
||||
// says it is due to be read again.
|
||||
if got := descriptions(t, h); got != "ONLY IN B" {
|
||||
t.Errorf("index holds %s right after the delete, want only b's own row", got)
|
||||
}
|
||||
var files struct{ Files []fileRow }
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
|
||||
if len(files.Files) != 1 || files.Files[0].Status != "changed" {
|
||||
t.Errorf("files = %+v, want b.csv marked to be re-read", files.Files)
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
if got := descriptions(t, h); got != "ONLY IN B,SHARED" {
|
||||
t.Errorf("after Import the index holds %s, want the shared row restored from b", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "a.csv")); !errors.Is(err, fs.ErrNotExist) {
|
||||
t.Errorf("a.csv is still on disk: %v", err)
|
||||
}
|
||||
if entries, _ := os.ReadDir(dir); len(entries) != 2 {
|
||||
t.Errorf("folder holds %d entries, want account.toml and b.csv only", len(entries))
|
||||
}
|
||||
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
|
||||
if len(files.Files) != 1 || files.Files[0].Name != "b.csv" || files.Files[0].Status != "imported" {
|
||||
t.Errorf("files = %+v, want only b.csv, imported", files.Files)
|
||||
}
|
||||
}
|
||||
|
||||
// A removed leg takes its transfer pairing with it, and the other leg is left
|
||||
// unpaired — which is the truth once one side is gone.
|
||||
func TestRemoveFileUnpairsItsTransfers(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
writeFile(t, filepath.Join(filepath.Dir(dir), config.RulesFile), `
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*OUT*"
|
||||
to_account = "checking"
|
||||
to_desc = "*IN*"
|
||||
`)
|
||||
writeFile(t, filepath.Join(dir, "out.csv"), "date,description,amount\n2026-02-01,MOVE OUT,-5.00\n")
|
||||
writeFile(t, filepath.Join(dir, "in.csv"), "date,description,amount\n2026-02-02,MOVE IN,5.00\n")
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
|
||||
var res status
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "in.csv"}, http.StatusOK, &res)
|
||||
if !strings.Contains(res.Status, "1 transfer leg(s) unpaired") {
|
||||
t.Errorf("status = %q, want the remaining leg reported unpaired", res.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// A file already gone from disk is forgotten by the index; a name the index
|
||||
// never had is not found.
|
||||
func TestRemoveFileForgetsAMissingOne(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
writeFile(t, filepath.Join(dir, "a.csv"), statement)
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
if err := os.Remove(filepath.Join(dir, "a.csv")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var res status
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusOK, &res)
|
||||
if !strings.HasPrefix(res.Status, "forgot checking/a.csv, 2 transaction(s) dropped") {
|
||||
t.Errorf("status = %q", res.Status)
|
||||
}
|
||||
if got := descriptions(t, h); got != "" {
|
||||
t.Errorf("index still holds %s", got)
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusNotFound, nil)
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "../rules.toml"}, http.StatusBadRequest, nil)
|
||||
if _, err := os.Stat(filepath.Join(dir, config.AccountFile)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: config.AccountFile}, http.StatusNotFound, nil)
|
||||
}
|
||||
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/bin/sh
|
||||
# Builds money with a static pdftotext inside it: one file to copy to a server,
|
||||
# with nothing to install there — not even poppler-utils.
|
||||
#
|
||||
# scripts/build-bundled.sh # linux, this machine's architecture
|
||||
# scripts/build-bundled.sh arm64 # linux/arm64 (podman/docker need qemu)
|
||||
#
|
||||
# pdftotext is built in a container from a checksum-pinned poppler release
|
||||
# (scripts/pdftotext.Containerfile) and cached under internal/parser/bundled/,
|
||||
# so later runs only rebuild money. Delete that file to rebuild it.
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
arch=${1:-$(go env GOARCH)}
|
||||
case $arch in
|
||||
amd64 | arm64) ;;
|
||||
*)
|
||||
echo "build-bundled: no static pdftotext for linux/$arch; use amd64 or arm64" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
bin=internal/parser/bundled/pdftotext-linux-$arch
|
||||
if [ ! -f "$bin" ] || [ scripts/pdftotext.Containerfile -nt "$bin" ]; then
|
||||
engine=$(command -v podman || command -v docker) || {
|
||||
echo "build-bundled: building pdftotext needs podman or docker" >&2
|
||||
exit 1
|
||||
}
|
||||
out=$(mktemp -d)
|
||||
trap 'rm -rf "$out"' EXIT
|
||||
"$engine" build --platform "linux/$arch" -f scripts/pdftotext.Containerfile \
|
||||
--output "type=local,dest=$out" scripts
|
||||
mkdir -p internal/parser/bundled
|
||||
mv "$out/pdftotext" "$bin"
|
||||
fi
|
||||
|
||||
mkdir -p dist
|
||||
CGO_ENABLED=0 GOOS=linux GOARCH=$arch \
|
||||
go build -tags bundled -trimpath -o "dist/money-linux-$arch" ./cmd/money
|
||||
echo "dist/money-linux-$arch"
|
||||
@@ -0,0 +1,70 @@
|
||||
# Builds a fully static pdftotext for `go build -tags bundled`, so the money
|
||||
# binary can carry it rather than depend on poppler-utils on the host.
|
||||
#
|
||||
# Only what text extraction needs is compiled in. Fonts are never rendered, so
|
||||
# fontconfig, cairo and the image codecs (JPEG, JPEG 2000, TIFF) are left out,
|
||||
# as are colour management, HTTP and signature support. Freetype and zlib are
|
||||
# the two dependencies poppler will not build without.
|
||||
#
|
||||
# Built against musl because glibc cannot be linked statically in any way that
|
||||
# survives a different host.
|
||||
#
|
||||
# Run it through scripts/build-bundled.sh rather than by hand.
|
||||
|
||||
FROM docker.io/library/alpine:3.22 AS build
|
||||
|
||||
RUN apk add --no-cache build-base cmake samurai pkgconf \
|
||||
freetype-dev freetype-static zlib-dev zlib-static \
|
||||
libpng-dev libpng-static bzip2-static brotli-static
|
||||
|
||||
# Pinned by checksum: this binary ends up inside money, so the source it is
|
||||
# built from must be exactly the one that was reviewed.
|
||||
ARG POPPLER_VERSION=26.09.0
|
||||
ARG POPPLER_SHA256=8059eadb6805340768f138c465b57f8164c92b4a0773c37ef031ea6c0d987b2e
|
||||
|
||||
WORKDIR /src
|
||||
RUN wget -q "https://poppler.freedesktop.org/poppler-${POPPLER_VERSION}.tar.xz" \
|
||||
&& echo "${POPPLER_SHA256} poppler-${POPPLER_VERSION}.tar.xz" | sha256sum -c - \
|
||||
&& tar xf "poppler-${POPPLER_VERSION}.tar.xz" --strip-components=1 \
|
||||
&& rm "poppler-${POPPLER_VERSION}.tar.xz"
|
||||
|
||||
# The find modules would otherwise settle on the shared libraries, and freetype's static
|
||||
# archive does not carry its own dependencies (png, bzip2, brotli), so they are
|
||||
# appended from what pkg-config says a static freetype needs.
|
||||
RUN cmake -S . -B build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DBUILD_SHARED_LIBS=OFF \
|
||||
-DFREETYPE_LIBRARY_RELEASE=/usr/lib/libfreetype.a \
|
||||
-DPNG_LIBRARY_RELEASE=/usr/lib/libpng.a \
|
||||
-DZLIB_LIBRARY_RELEASE=/usr/lib/libz.a \
|
||||
-DCMAKE_EXE_LINKER_FLAGS=-static \
|
||||
-DCMAKE_CXX_STANDARD_LIBRARIES="$(pkg-config --static --libs freetype2)" \
|
||||
-DFONT_CONFIGURATION=generic \
|
||||
-DENABLE_UTILS=ON \
|
||||
-DENABLE_CPP=OFF \
|
||||
-DENABLE_GLIB=OFF \
|
||||
-DENABLE_GOBJECT_INTROSPECTION=OFF \
|
||||
-DENABLE_QT5=OFF \
|
||||
-DENABLE_QT6=OFF \
|
||||
-DENABLE_BOOST=OFF \
|
||||
-DENABLE_LIBOPENJPEG=OFF \
|
||||
-DENABLE_LIBJPEG=OFF \
|
||||
-DENABLE_LCMS=OFF \
|
||||
-DENABLE_LIBCURL=OFF \
|
||||
-DENABLE_LIBTIFF=OFF \
|
||||
-DENABLE_NSS3=OFF \
|
||||
-DENABLE_GPGME=OFF \
|
||||
-DENABLE_HARFBUZZ=OFF \
|
||||
-DBUILD_GTK_TESTS=OFF \
|
||||
-DBUILD_QT5_TESTS=OFF \
|
||||
-DBUILD_QT6_TESTS=OFF \
|
||||
-DBUILD_CPP_TESTS=OFF \
|
||||
-DBUILD_MANUAL_TESTS=OFF \
|
||||
&& cmake --build build --target pdftotext \
|
||||
&& strip build/utils/pdftotext \
|
||||
# Refuse to hand over anything that still wants a dynamic loader.
|
||||
&& ! readelf -l build/utils/pdftotext | grep -q INTERP \
|
||||
&& build/utils/pdftotext -v
|
||||
|
||||
FROM scratch
|
||||
COPY --from=build /src/build/utils/pdftotext /pdftotext
|
||||
Reference in New Issue
Block a user