Compare commits
8
Commits
7f5b4846ef
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a086f9242d | ||
|
|
c271cf2c8f | ||
|
|
3d72a54be1 | ||
|
|
ba7e0e2b42 | ||
|
|
ed52be3f1d | ||
|
|
bd406a110d | ||
|
|
a5f7541980 | ||
|
|
9ad6c05f9a |
@@ -143,14 +143,14 @@ Anything new that reports a rule must report its file position too.
|
||||
settles ties: a saved rule cannot displace an equally specific one written by
|
||||
hand, while a narrower one is meant to take precedence and does.
|
||||
|
||||
**The report's period narrows the report and nothing else.** The screen opens
|
||||
on last month, and `←`/`→` step along `report.Periods` — the named windows,
|
||||
then the months the index holds. The report and the transaction list share a
|
||||
scope (account, search, untagged), which `web.filterFrom` reads and
|
||||
`state.filter` holds in the page; the period is read by the report handler
|
||||
alone and kept in `state.report`, so opening on last month cannot hide the
|
||||
rest of the index from the list. Nothing may put the period into that shared
|
||||
scope, which is exactly what would make it leak. Covered by `TestReportPeriodLeavesTransactionsAlone`.
|
||||
**The report's period narrows the report and nothing else.** The screen opens on
|
||||
last month, and `←`/`→` step along `report.Periods` — the named windows, then
|
||||
the months the index holds. The report and the transaction list share a scope
|
||||
(account, search, untagged), which `web.filterFrom` reads and `state.filter`
|
||||
holds in the page; the period is read by the report handler alone and kept in
|
||||
`state.report`, so opening on last month cannot hide the rest of the index from
|
||||
the list. Nothing may put the period into that shared scope, which is exactly
|
||||
what would make it leak. Covered by `TestReportPeriodLeavesTransactionsAlone`.
|
||||
|
||||
The axis is built from `store.Months` over the *whole* index, not from the rows
|
||||
in view, or it would grow and shrink as the account or search filter changed and
|
||||
@@ -174,12 +174,12 @@ tag so ties keep a fixed position instead of shuffling between reloads.
|
||||
own copy of that mapping; a new order needs a column of its own, which
|
||||
`TestOrderColumnsAreDistinct` checks.
|
||||
|
||||
**The single-key shortcuts must not fire while typing.** The page's `keydown`
|
||||
handler ignores them whenever the target is an input, or typing `i` in the tag
|
||||
field would start an import. A view's own `key` hook runs first and is told
|
||||
whether the user is typing, so a command *inside* a builder has to be a chord —
|
||||
`ctrl+s` re-sorts the preview — because every printable key belongs to the
|
||||
field being typed in.
|
||||
**There are no keyboard shortcuts.** The page used to carry the TUI's keymap —
|
||||
screens on `1`–`8`, `u`, `/`, `i`, `r`, and chords inside the builders — and it
|
||||
was removed on purpose: every action is a link or a button. Do not add a global
|
||||
`keydown` handler back without being asked; a single-letter key fires while
|
||||
typing in a field unless every one of them is guarded, which is what the old one
|
||||
existed to do.
|
||||
|
||||
**A rule's usage count is how many transactions it wins, not how many its glob
|
||||
could match** — `Engine.Usage` counts by `MatchIndex`, so a rule shadowed by a
|
||||
@@ -241,13 +241,38 @@ That is why `/api/upload` takes files base64 inside JSON rather than as
|
||||
multipart: multipart is precisely what a cross-site form can send.
|
||||
|
||||
**An upload only puts a file where `money import` looks.** It writes into an
|
||||
existing account folder and then runs the same import as `/api/import`, so the
|
||||
statements stay the source of truth and nothing reaches the index any other
|
||||
way. It never overwrites a statement (identical contents are a no-op, different
|
||||
ones a 409), refuses any name import would not read back — not a plain file
|
||||
name, a dotfile, `account.toml`, outside the account's `include` — and checks a
|
||||
whole batch before writing any of it. Covered by
|
||||
`TestUploadNeverReplacesAStatement` and `TestUploadRefusesBadNames`.
|
||||
existing account folder and stops there: importing stays the user's call, made
|
||||
with the Import button, so the statements stay the source of truth and nothing
|
||||
reaches the index any other way. It never overwrites a statement (identical
|
||||
contents are a no-op, different ones a 409), refuses any name import would not
|
||||
read back — not a plain file name, a dotfile, `account.toml`, outside the
|
||||
account's `include` — and checks a whole batch before writing any of it. Covered
|
||||
by `TestUploadSavesWithoutImporting`, `TestUploadNeverReplacesAStatement` and
|
||||
`TestUploadRefusesBadNames`.
|
||||
|
||||
**The statements list is read from the folders, not the index.** `/api/files`
|
||||
walks `importer.StatementFiles` — the same list import reads — and only then
|
||||
looks each file up in `store.SourceFiles`, comparing `importer.Checksum`, so a
|
||||
file is listed exactly when import would read it and a file the index remembers
|
||||
but the disk lost shows as `missing` instead of vanishing.
|
||||
`/api/files/{account}/{name}` serves a file only by finding it in that list,
|
||||
never by joining the name onto a path. A statement is served from the app's
|
||||
origin, where a script could drive the API, so nothing is ever rendered as a
|
||||
page: text is `text/plain` under `CSP: sandbox`, anything not text or PDF is a
|
||||
sandboxed download, and PDFs — whose viewers refuse a sandbox — open in the
|
||||
browser's own isolated viewer. Covered by `TestServeFileServesOnlyStatements`.
|
||||
|
||||
**Deleting a statement marks its neighbours to be re-read.** `/api/files/delete`
|
||||
finds the file the same way `serveFile` does and deletes it — permanently, by
|
||||
the user's choice; the page says so before it asks. `store.ForgetSourceFile`
|
||||
then drops its transactions and their transfer rows, and `Link` re-pairs, which
|
||||
reads no statement. It does not import, any more than an upload does. A
|
||||
transaction two overlapping statements share is stored once, under the file
|
||||
imported first, so it goes too; `ForgetSourceFile` therefore blanks the
|
||||
checksums of the account's other statements, which show as `changed` until the
|
||||
next Import re-reads them and restores what they hold. Without that the checksum
|
||||
skip would leave those rows gone for good. Covered by
|
||||
`TestRemoveFileKeepsWhatAnotherStatementHolds`.
|
||||
|
||||
## Adding a bank parser
|
||||
|
||||
@@ -288,6 +313,19 @@ description — at the end, and not in the position it held on the page, so an
|
||||
IBAN wrapped across continuation lines stays contiguous for a glob to match.
|
||||
A new parser must do the same rather than reintroduce a structured field.
|
||||
|
||||
**A parser may name its statements, and only uploads use it.** `parser.Namer` is
|
||||
optional, like `parser.Warner`: `nlb` names an izpisek `izpisek_YYYY-MM-DD`
|
||||
after its *Datum izpiska*, ported from the `rename_izpiski.py` it replaced, and
|
||||
`traderepublic` names a statement `traderepublic_YYYY_MM_DD_YYYY_MM_DD` after
|
||||
the period in its header. A name is derived from what the statement says, never
|
||||
from the upload's own name, and comes back without an extension; the upload's is
|
||||
kept, lowercased. `/api/upload` stages each file as a dotfile — invisible to
|
||||
import — so the parser can read it, then numbers a chosen name that is taken by
|
||||
different contents (`_2`, as the script did) where a name the user gave is
|
||||
refused with 409. Nothing renames a file already in a folder: `source_files`
|
||||
records statements by path, so a rename would orphan its rows as `missing` until
|
||||
the index is rebuilt. Covered by `TestUploadNamesStatementsTheParserCanName`.
|
||||
|
||||
## Verifying
|
||||
|
||||
```
|
||||
|
||||
@@ -102,8 +102,7 @@ money config # which data root is in use, and why
|
||||
|
||||
An account folder appears in the app only once its statements have been
|
||||
imported — creating an `account.toml` is not enough on its own. Run
|
||||
`money import` (or press Import in the web app) after adding one, or add its
|
||||
first statements from the Accounts screen, which imports them as it saves them.
|
||||
`money import` (or press Import in the web app) after adding one.
|
||||
|
||||
`--uniq` turns `ls` into a list of patterns still to write rather than a list of
|
||||
rows to read: one line per distinct description, since fifty visits to the same
|
||||
@@ -124,9 +123,8 @@ LIDL SOFIA 4412
|
||||
|
||||
The account is not part of what makes a row distinct. A rule matches on the
|
||||
description and only optionally narrows to an account, so one payee seen on two
|
||||
accounts is still one pattern to write — and it is one row in the rule builder
|
||||
on `4`, which groups the same way. Pass `--account` to scope the listing
|
||||
instead.
|
||||
accounts is still one pattern to write — and it is one row in the rule builder,
|
||||
which groups the same way. Pass `--account` to scope the listing instead.
|
||||
|
||||
It composes with the other filters (`--account`, `--month`, `--search`), and
|
||||
`--limit` caps the rows printed, saying how many it held back. It cannot be
|
||||
@@ -134,11 +132,12 @@ combined with `--wide`, whose columns all belong to a single transaction.
|
||||
|
||||
## The web app
|
||||
|
||||
`money serve`, which is also what `money` runs with no command, puts seven
|
||||
`money serve`, which is also what `money` runs with no command, puts eight
|
||||
screens in a browser: accounts, transactions, report, rule builder, rules,
|
||||
transfer builder and transfers. It runs the same code as the CLI: amounts are formatted,
|
||||
globs matched and transfers paired on the server, and the page only shows the
|
||||
answers. It is one binary with the page built in; nothing else to deploy.
|
||||
transfer builder, transfers and tags. It runs the same code as the CLI: amounts
|
||||
are formatted, globs matched and transfers paired on the server, and the page
|
||||
only shows the answers. It is one binary with the page built in; nothing else to
|
||||
deploy.
|
||||
|
||||
```
|
||||
money # http://127.0.0.1:8080
|
||||
@@ -164,15 +163,52 @@ fresh command would:
|
||||
rule there (another tab, a hand edit) the change is refused and you are asked
|
||||
to reload, rather than editing whichever rule moved into its place.
|
||||
|
||||
Shift-click **Import** for `import --force`.
|
||||
**Re-read all** is `import --force`: every statement is parsed again, including
|
||||
the ones an Import skips because they have not changed. Use it after updating
|
||||
money, when a parser has learned to read something it used to get wrong; rows
|
||||
already in the index are recognised and not duplicated.
|
||||
|
||||
### Statements
|
||||
|
||||
The Accounts screen also lists every statement file in every account
|
||||
folder, with what the index made of it:
|
||||
|
||||
| Status | Meaning |
|
||||
| --- | --- |
|
||||
| `✓` imported | read, and unchanged since |
|
||||
| `⚠` changed | the file differs from what was imported; Import re-reads it |
|
||||
| `•` not imported | new since the last import — or an import tried and failed, in which case the error is in the import report |
|
||||
| `✗` gone from disk | imported once, then removed; its transactions stay in the index only until it is rebuilt |
|
||||
|
||||
**Transactions** reads like `3 rows · 0 new`: how many transactions the file
|
||||
holds, then how many of them it was the first to bring in. Statements that
|
||||
overlap share rows, and a shared row is stored once, under whichever file was
|
||||
imported first — so a later statement covering the same days shows fewer new
|
||||
rows than it holds, down to `0 new` when an earlier one already had them all.
|
||||
That is deduplication working, not a file that failed to parse.
|
||||
|
||||
Click a file name to open it. PDFs and CSVs open in the browser; anything else
|
||||
downloads. Only files import would read are listed or served — not
|
||||
`account.toml`, dotfiles, or anything outside an account's `include` patterns.
|
||||
|
||||
**Delete** removes a statement from disk for good — there is no copy kept
|
||||
and no undo, so keep the bank's original if you might want it back. Its
|
||||
transactions leave the index, and transfers that used them lose their pairing:
|
||||
the other leg shows as unpaired, which is the truth once one side is gone.
|
||||
Deleting does not import. A row that an overlapping statement also contains goes
|
||||
too, for now: the account's other statements are marked *changed*, and the next
|
||||
**Import** re-reads them and puts it back. For a file already gone from disk the
|
||||
button says **Forget** and only clears the index.
|
||||
|
||||
### Adding statements from the browser
|
||||
|
||||
The Accounts screen (`1`) has an **Add statements** panel: pick the account,
|
||||
drop files on it (or click to choose them) and press **Upload and import**. The
|
||||
files are saved into that account's folder, exactly where you would have copied
|
||||
them by hand, and then imported — the folder stays the source of truth, so
|
||||
deleting `index.db` and re-importing still gets everything back.
|
||||
The Accounts screen has an **Add statements** panel: pick the account,
|
||||
drop files on it (or click to choose them) and press **Upload**. The files are
|
||||
saved into that account's folder, exactly where you would have copied them by
|
||||
hand — the folder stays the source of truth, so deleting `index.db` and
|
||||
re-importing still gets everything back. Uploading does not import: the files
|
||||
wait on the statements list as *not imported* until you press **Import**, so you
|
||||
can put a batch together and look it over first.
|
||||
|
||||
An upload never replaces a statement. A file whose name is already in the folder
|
||||
is skipped if its contents are identical and refused if they differ; rename it
|
||||
@@ -182,28 +218,30 @@ too, and a batch with one bad file writes none of them. The folder itself must
|
||||
already exist with an `account.toml`: an upload adds statements to an account,
|
||||
it does not create one.
|
||||
|
||||
### Keys
|
||||
Some banks name their downloads unhelpfully, so a parser may name the statement
|
||||
instead — lowercase throughout, extension included — and the status line says
|
||||
what each file became. An NLB izpisek is saved as `izpisek_YYYY-MM-DD.pdf` after
|
||||
the *Datum izpiska* in its header, and a Trade Republic statement, which
|
||||
downloads as `document-N.pdf`, as `traderepublic_YYYY_MM_DD_YYYY_MM_DD.pdf`
|
||||
after the period on its first page (`DATE 01 May 2025 - 31 Jul 2026`). Either
|
||||
way the folder sorts by date. Downloading the same statement twice then lands on
|
||||
the same name and is recognised as already there; a different statement with the
|
||||
same name (a reissue) is numbered `_2`, `_3` rather than refused. A statement
|
||||
with no date it can find keeps its own name. Only uploads are named — files
|
||||
already in a folder are never renamed, since the index records them by path.
|
||||
|
||||
| Key | Action |
|
||||
| --- | --- |
|
||||
| `1` – `7` | accounts · transactions · report · rule builder · rules · transfer builder · transfers |
|
||||
| `/` | filter by description |
|
||||
| `u` | show only untagged transactions (matched transfer legs are not among them) |
|
||||
| `a` | clear the account filter |
|
||||
| `←` `→` | move the report's period (report) |
|
||||
| `s` | change how the report is sorted (report) |
|
||||
| `ctrl+s` | sort the rule builder's preview by name / by count |
|
||||
| `i` | import · `r` retag |
|
||||
### Navigating
|
||||
|
||||
Inside a form every printable key belongs to the field you are typing in, so
|
||||
the single-letter keys only work outside one; that is why the builder's sort is
|
||||
a chord. Clicking an account on `1` filters the transaction list to it.
|
||||
Every screen is a link in the bar along the top, and every action is a button or
|
||||
a click; there are no keyboard shortcuts. Clicking an account on the Accounts
|
||||
screen filters the transaction list to it.
|
||||
|
||||
There is no control that tags a transaction. Tags come from `rules.toml` and
|
||||
nowhere else, so tagging what you are looking at means writing a rule for it on
|
||||
`4` — which is why that screen shows you what a glob catches before you save.
|
||||
nowhere else, so tagging what you are looking at means writing a rule for it in
|
||||
the rule builder — which is why it shows you what a glob catches before you
|
||||
save.
|
||||
|
||||
### Report (`3`)
|
||||
### Report
|
||||
|
||||
The report opens on **last month**, not on everything you have ever imported:
|
||||
an all-time total is the one number a spending report is least often asked for,
|
||||
@@ -213,8 +251,8 @@ period is empty rather than quietly showing you a different one.
|
||||
|
||||
The time axis beside the totals runs from the widest window down to the oldest
|
||||
month in the index — `all time`, `this year`, `last 12 months`, `last 3
|
||||
months`, `this month`, `last month`, then one entry per month. Click one, or
|
||||
step along it with `←` and `→`.
|
||||
months`, `this month`, `last month`, then one entry per month. Click one to
|
||||
report on it.
|
||||
|
||||
The three rolling windows run to the end of *this* month rather than to the
|
||||
last complete one — you ask for "last 3 months" to see what is happening now,
|
||||
@@ -222,12 +260,11 @@ and leaving out the days since the 1st would answer a different question. Each
|
||||
month below `last month` is a month the index actually holds; months that a
|
||||
named window above already covers are not repeated.
|
||||
|
||||
Click a column heading, or press `s` to cycle, to change how the rows are
|
||||
arranged; the marked heading says which one they are arranged by: `Out ▾`
|
||||
largest spend first (the default question a spending report answers), then
|
||||
`In ▾`, `Net ▾` lowest first so the biggest losses lead, `N ▾` most
|
||||
transactions first, and `Tag ▴` A→Z. `money report` takes the same choice as
|
||||
`--sort out|in|net|count|tag`.
|
||||
Click a column heading to change how the rows are arranged; the marked heading
|
||||
says which one they are arranged by: `Out ▾` largest spend first (the default
|
||||
question a spending report answers), then `In ▾`, `Net ▾` lowest first so the
|
||||
biggest losses lead, `N ▾` most transactions first, and `Tag ▴` A→Z. `money
|
||||
report` takes the same choice as `--sort out|in|net|count|tag`.
|
||||
|
||||
Currency is always the outer grouping and no sort changes that — there are no
|
||||
exchange rates here, so two currencies interleaved by amount would invite a
|
||||
@@ -237,12 +274,12 @@ The sort rearranges rows; it never changes which rows there are.
|
||||
|
||||
The period narrows the report and only the report. The account, the search and
|
||||
the untagged toggle are shared with the transaction list, so opening on last
|
||||
month does not hide the rest of the index from `2`.
|
||||
month does not hide the rest of the index from the transaction list.
|
||||
|
||||
`money report` takes `--month` instead; there is no command-line equivalent of
|
||||
the wider windows.
|
||||
|
||||
### Rule builder (`4`)
|
||||
### Rule builder
|
||||
|
||||
Writing rules by hand means guessing what a glob will catch. This screen shows
|
||||
the answer as you type: the form is on the left — glob, account, tag, note —
|
||||
@@ -264,13 +301,13 @@ point to narrow down.
|
||||
immediately, so the rows it caught disappear from the list. The account stays
|
||||
filled in, since the next rule is usually for the same one.
|
||||
|
||||
Clicking the column headings, or `ctrl+s`, switches the list between by name and
|
||||
by count, most seen first; the `↓` says which column the order is read from.
|
||||
The two answer different questions: by name finds the payee you are looking at,
|
||||
by count finds the rule worth writing next, since one pattern claiming forty
|
||||
rows is worth more than the first of forty claiming one. Ties keep the
|
||||
alphabetical order, so the list does not reshuffle under you, and the choice
|
||||
lasts until you change it.
|
||||
Clicking the column headings switches the list between by name and by count,
|
||||
most seen first; the `↓` says which column the order is read from. The two
|
||||
answer different questions: by name finds the payee you are looking at, by count
|
||||
finds the rule worth writing next, since one pattern claiming forty rows is
|
||||
worth more than the first of forty claiming one. Ties keep the alphabetical
|
||||
order, so the list does not reshuffle under you, and the choice lasts until you
|
||||
change it.
|
||||
|
||||
The account and tag fields offer completions as you type. Accounts come from
|
||||
the folders on disk and the index; tags from every tag in use plus any named in
|
||||
@@ -290,7 +327,7 @@ report leaves them out anyway.
|
||||
|
||||
#### Editing a rule
|
||||
|
||||
**Edit** on the rules screen (`5`) opens the rule in this same form with the
|
||||
**Edit** on the rules screen opens the rule in this same form with the
|
||||
fields filled in, and saving rewrites that rule where it sits instead of
|
||||
appending a new one. It keeps its position, since position still breaks ties
|
||||
between equally specific rules; `rules.toml` is edited textually, so the
|
||||
@@ -306,10 +343,10 @@ you are about to hand back to whatever rule catches it next.
|
||||
|
||||
The form has no `type` field, so a rule that sets one carries it through
|
||||
unchanged rather than losing it; it is shown under the glob as `+ type:… ·
|
||||
kept`. Saving lands back on the rules screen with the new counts; **Cancel**,
|
||||
`esc`, or leaving the screen leaves the rule as it was.
|
||||
kept`. Saving lands back on the rules screen with the new counts; **Cancel**, or
|
||||
leaving the screen leaves the rule as it was.
|
||||
|
||||
### Rules (`5`)
|
||||
### Rules
|
||||
|
||||
Every rule in file order — the order they are written in, not the order they
|
||||
are tried in — with the number of transactions it actually claims. Rules that
|
||||
@@ -344,7 +381,7 @@ it, while one separated by a blank line is treated as a section heading and
|
||||
left alone; an edited rule keeps its comments, since they say why it is there
|
||||
and changing its glob rarely changes that.
|
||||
|
||||
### Transfer builder (`6`)
|
||||
### Transfer builder
|
||||
|
||||
Same idea as the rule builder, for money moved between your own accounts. The
|
||||
form names both sides — from account, from desc, to account, to desc — plus an
|
||||
@@ -379,7 +416,7 @@ A half-written definition previews too: fill in one side and its legs show up as
|
||||
unpaired, which is the quickest way to see that a glob is wrong before you have
|
||||
written the other half.
|
||||
|
||||
### Transfers (`7`)
|
||||
### Transfers
|
||||
|
||||
Every definition in file order, with what it currently pairs.
|
||||
|
||||
@@ -425,6 +462,27 @@ once, both after asking. Definitions marked `⚠` are never pruned — they are
|
||||
doing something, just not finishing it, and deleting one would hide the problem
|
||||
rather than fix it. **Refresh pairing** re-reads what is currently in the index.
|
||||
|
||||
### Tags
|
||||
|
||||
Every distinct tag, A→Z: the ones your transactions carry and any `rules.toml`
|
||||
names, with how many transactions carry each and the rules that write it.
|
||||
|
||||
```
|
||||
Tags · 2 tags
|
||||
|
||||
Tag Txns Rules
|
||||
clothes 0 #4 *ZARA* (0)
|
||||
groceries 2 #1 *LIDL* (0) #2 *KAUFLAND* (1) #3 *LIDL SOFIA* (1)
|
||||
```
|
||||
|
||||
Rules are numbered by their position in `rules.toml`, as on the rules screen,
|
||||
and the count after each is how many transactions it wins — so a tag's rules
|
||||
add up to its transactions, and a rule at `(0)` is matching nothing or being
|
||||
beaten by a more specific one. It is the place to spot near-duplicate tags
|
||||
(`grocery` beside `groceries`) and tags that several rules feed. A tag no rule
|
||||
names any more — `rules.toml` was edited since the last retag — says so; the
|
||||
next Retag clears it. The transfer label is not a tag and never appears here.
|
||||
|
||||
## rules.toml
|
||||
|
||||
**The most specific rule wins**, so `*NIKOLA*` claims what it names even with a
|
||||
@@ -480,7 +538,7 @@ tag = "groceries"
|
||||
Every tag comes from this file, so a tag is never something you have to keep
|
||||
safe: `money retag` recomputes all of them from the current rules and is safe
|
||||
to run whenever you change it. A transaction no rule matches simply stays
|
||||
untagged, and shows up under `money ls --untagged`, the `u` view, and
|
||||
untagged, and shows up under `money ls --untagged`, **Untagged only**, and
|
||||
`(untagged)` in the report — unless it is one leg of a matched transfer, which
|
||||
is already spoken for and is left out of all three.
|
||||
|
||||
@@ -582,7 +640,7 @@ that is the number your spending is short by:
|
||||
transfers excluded: 2 legs in EUR, 500.00 out, 495.00 in, 5.00 in fees
|
||||
```
|
||||
|
||||
The report screen (`3`) shows the same thing under its `TOTAL`, the fee on its
|
||||
The report screen shows the same thing under its `TOTAL`, the fee on its
|
||||
own row (for whichever period it is on):
|
||||
|
||||
```
|
||||
@@ -608,7 +666,7 @@ An *unpaired* leg is not a transfer and still counts, tagged like anything
|
||||
else. That is deliberate: money that left an account and cannot be shown to
|
||||
have arrived is exactly what you want to see, not something to quietly drop.
|
||||
`money import` and `money retag` report unpaired legs on stderr, and the
|
||||
transfers screen (`7`) shows which definition they belong to.
|
||||
transfers screen shows which definition they belong to.
|
||||
|
||||
The pairing is derived state, like the tags: it lives in the index, is
|
||||
recomputed wholesale by `money retag` and after every import, and disappears
|
||||
@@ -634,8 +692,8 @@ nothing else to configure — `parser` names one of these and that is all.
|
||||
|
||||
| `parser` | Statement | Notes |
|
||||
| --- | --- | --- |
|
||||
| `nlb` | NLB izpisek PDF | Wrapped descriptions are folded in from continuation lines, and the IBAN column is appended to the description. |
|
||||
| `traderepublic` | Trade Republic PDF | Handles both the single-line and the stacked layout by measuring column positions. |
|
||||
| `nlb` | NLB izpisek PDF | Wrapped descriptions are folded in from continuation lines, and the IBAN column is appended to the description. Uploads are saved as `izpisek_YYYY-MM-DD.pdf` after the statement date. |
|
||||
| `traderepublic` | Trade Republic PDF | Handles both the single-line and the stacked layout by measuring column positions. Uploads are saved as `traderepublic_YYYY_MM_DD_YYYY_MM_DD.pdf` after the statement period. |
|
||||
| `revolut` | `account-statement*.csv` | Skips non-COMPLETED rows, folds the fee into the amount. |
|
||||
|
||||
```toml
|
||||
|
||||
@@ -101,7 +101,7 @@ func Run(root string, db *store.DB, accounts []*config.Account, engine *rules.En
|
||||
continue
|
||||
}
|
||||
|
||||
files, err := statementFiles(acc)
|
||||
files, err := StatementFiles(acc)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
@@ -134,7 +134,7 @@ func importFile(root string, db *store.DB, acc *config.Account, accountID int64,
|
||||
}
|
||||
fr := FileResult{Account: acc.Slug, Path: rel}
|
||||
|
||||
sum, err := checksum(path)
|
||||
sum, err := Checksum(path)
|
||||
if err != nil {
|
||||
fr.Err = err
|
||||
return fr
|
||||
@@ -162,7 +162,7 @@ func importFile(root string, db *store.DB, acc *config.Account, accountID int64,
|
||||
}
|
||||
fr.Warnings = append(fr.Warnings, checkBalances(txns, acc.Digits())...)
|
||||
|
||||
sourceID, err := db.SourceFile(accountID, rel, sum, time.Now().UTC().Format(time.RFC3339))
|
||||
sourceID, err := db.SourceFile(accountID, rel, sum, time.Now().UTC().Format(time.RFC3339), len(txns))
|
||||
if err != nil {
|
||||
fr.Err = err
|
||||
return fr
|
||||
@@ -252,10 +252,11 @@ func fingerprint(key string, ordinal int) string {
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// statementFiles lists the files in an account folder that should be parsed:
|
||||
// StatementFiles lists the files in an account folder that should be parsed:
|
||||
// every regular file except account.toml and dotfiles, narrowed by the
|
||||
// account's optional include globs.
|
||||
func statementFiles(acc *config.Account) ([]string, error) {
|
||||
// account's optional include globs. It is also what the web app lists and
|
||||
// serves, so a file is shown exactly when import would read it.
|
||||
func StatementFiles(acc *config.Account) ([]string, error) {
|
||||
entries, err := os.ReadDir(acc.Dir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read account dir %s: %w", acc.Dir, err)
|
||||
@@ -284,7 +285,9 @@ func matchAny(patterns []string, name string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func checksum(path string) (string, error) {
|
||||
// Checksum is the sha256 a statement is recorded under; an import skips a file
|
||||
// whose checksum has not changed.
|
||||
func Checksum(path string) (string, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
|
||||
@@ -43,6 +43,10 @@ var (
|
||||
nlbAccount = regexp.MustCompile(`^SI\d{2}(?:\s?\d{4}){3}\s?\d{3}$`)
|
||||
)
|
||||
|
||||
// nlbStatementDate is the issue date in an izpisek's header. It names the
|
||||
// statement: NLB's downloads are not named for what they hold.
|
||||
var nlbStatementDate = regexp.MustCompile(`Datum izpiska\s+(\d{2})\.(\d{2})\.(\d{4})`)
|
||||
|
||||
// nlbMinContinuationIndent is the shallowest indent a wrapped description line
|
||||
// may have. The real threshold is the description column of the transaction
|
||||
// the line belongs to, measured per line rather than hardcoded: pdftotext
|
||||
@@ -61,6 +65,24 @@ func (p *nlbParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
||||
return parseNLBText(text, p.digits)
|
||||
}
|
||||
|
||||
// StatementName names an izpisek after its statement date, izpisek_YYYY-MM-DD,
|
||||
// so the folder sorts by date and two downloads of one statement collide.
|
||||
func (p *nlbParser) StatementName(path string) (string, error) {
|
||||
text, err := pdfToText(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return nlbStatementName(text), nil
|
||||
}
|
||||
|
||||
func nlbStatementName(text string) string {
|
||||
m := nlbStatementDate.FindStringSubmatch(text)
|
||||
if m == nil {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf("izpisek_%s-%s-%s", m[3], m[2], m[1])
|
||||
}
|
||||
|
||||
// parseNLBText holds the whole parser, separated from PDF extraction so it can
|
||||
// be tested against captured pdftotext output.
|
||||
func parseNLBText(text string, digits int) ([]RawTxn, error) {
|
||||
|
||||
@@ -112,3 +112,15 @@ func TestParseNLBRejectsMalformedLine(t *testing.T) {
|
||||
t.Error("expected an error for a transaction line with no amount")
|
||||
}
|
||||
}
|
||||
|
||||
// An izpisek is named after its statement date, as rename_izpiski.py did.
|
||||
func TestNLBStatementName(t *testing.T) {
|
||||
header := " IZPISEK 001/2026\n" +
|
||||
" Datum izpiska 31.01.2026 Stran 1\n"
|
||||
if got := nlbStatementName(header + nlbPage); got != "izpisek_2026-01-31" {
|
||||
t.Errorf("name = %q, want izpisek_2026-01-31", got)
|
||||
}
|
||||
if got := nlbStatementName(nlbPage); got != "" {
|
||||
t.Errorf("name without a statement date = %q, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,6 +43,17 @@ type Warner interface {
|
||||
Warnings() []string
|
||||
}
|
||||
|
||||
// Namer is an optional interface for parsers whose bank names its downloads
|
||||
// unhelpfully. StatementName reads the statement at path and returns the name,
|
||||
// without extension, it should be stored under — the upload's extension is
|
||||
// kept, lowercased — or "" when the statement does not say, in which case it
|
||||
// keeps the name it came with. Only an upload uses
|
||||
// it; files already in a folder are never renamed, since the index records
|
||||
// them by path.
|
||||
type Namer interface {
|
||||
StatementName(path string) (string, error)
|
||||
}
|
||||
|
||||
// Factory builds a parser from an account's config, validating it up front so
|
||||
// a bad account.toml fails before any file is read.
|
||||
type Factory func(acc *config.Account) (Parser, error)
|
||||
|
||||
@@ -46,6 +46,12 @@ const trAmountSlack = 5
|
||||
// trDateLayout is the date Trade Republic prints, e.g. "05 Jan 2026".
|
||||
const trDateLayout = "02 Jan 2006"
|
||||
|
||||
// trPeriod is the statement period in the address block at the top of the
|
||||
// first page, e.g. "DATE 01 May 2025 - 31 Jul 2026". It names the
|
||||
// statement: Trade Republic's downloads are all called document-N.pdf. The
|
||||
// transaction table's own DATE heading never matches, since no dates follow it.
|
||||
var trPeriod = regexp.MustCompile(`\bDATE\s+(\d{2} [A-Z][a-z]{2} \d{4})\s*[-\x{2013}\x{2212}]\s*(\d{2} [A-Z][a-z]{2} \d{4})`)
|
||||
|
||||
func (p *tradeRepublicParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
||||
text, err := pdfToText(path)
|
||||
if err != nil {
|
||||
@@ -54,6 +60,33 @@ func (p *tradeRepublicParser) Parse(path string, acc *config.Account) ([]RawTxn,
|
||||
return parseTradeRepublicText(text, p.digits)
|
||||
}
|
||||
|
||||
// StatementName names a statement after its period,
|
||||
// traderepublic_YYYY_MM_DD_YYYY_MM_DD, so the folder sorts by when it starts
|
||||
// and two downloads of one statement collide.
|
||||
func (p *tradeRepublicParser) StatementName(path string) (string, error) {
|
||||
text, err := pdfToText(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return tradeRepublicStatementName(text), nil
|
||||
}
|
||||
|
||||
func tradeRepublicStatementName(text string) string {
|
||||
m := trPeriod.FindStringSubmatch(text)
|
||||
if m == nil {
|
||||
return ""
|
||||
}
|
||||
from, err := time.Parse(trDateLayout, m[1])
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
to, err := time.Parse(trDateLayout, m[2])
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return "traderepublic_" + from.Format("2006_01_02") + "_" + to.Format("2006_01_02")
|
||||
}
|
||||
|
||||
// parseTradeRepublicText holds the whole parser, separated from PDF extraction
|
||||
// so it can be tested against captured pdftotext output.
|
||||
func parseTradeRepublicText(text string, digits int) ([]RawTxn, error) {
|
||||
|
||||
@@ -162,3 +162,19 @@ func TestParseTradeRepublicIgnoresAmountsInDescription(t *testing.T) {
|
||||
t.Errorf("description = %q, want the inline figure kept", txns[0].Description)
|
||||
}
|
||||
}
|
||||
|
||||
// A statement is named after the period in its header; the transaction
|
||||
// table's DATE heading, with no dates after it, is not mistaken for it.
|
||||
func TestTradeRepublicStatementName(t *testing.T) {
|
||||
header := "NIKOLA PETROV DATE 01 May 2025 - 31 Jul 2026\n" +
|
||||
"Ulica 21 IBAN DE00000000000000000000\n\n" +
|
||||
"DATE TYPE DESCRIPTION MONEY IN MONEY OUT BALANCE\n"
|
||||
if got := tradeRepublicStatementName(header); got != "traderepublic_2025_05_01_2026_07_31" {
|
||||
t.Errorf("name = %q, want traderepublic_2025_05_01_2026_07_31", got)
|
||||
}
|
||||
table := "DATE TYPE DESCRIPTION MONEY IN MONEY OUT BALANCE\n" +
|
||||
"01 Jul 2026 Interest Interest payment €1.17 €671.58\n"
|
||||
if got := tradeRepublicStatementName(table); got != "" {
|
||||
t.Errorf("name without a period = %q, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -28,7 +28,7 @@ func seed(t *testing.T, db *store.DB, descriptions ...string) int64 {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
sourceID, err := db.SourceFile(accountID, "checking/st.csv", "sha", "2026-01-01T00:00:00Z")
|
||||
sourceID, err := db.SourceFile(accountID, "checking/st.csv", "sha", "2026-01-01T00:00:00Z", 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
+98
-6
@@ -37,6 +37,9 @@ CREATE TABLE IF NOT EXISTS source_files (
|
||||
path TEXT NOT NULL,
|
||||
sha256 TEXT NOT NULL,
|
||||
imported_at TEXT NOT NULL,
|
||||
-- How many transactions the statement holds, as parsed. Not how many it
|
||||
-- brought in: rows an overlapping statement already had are deduplicated.
|
||||
rows INTEGER NOT NULL DEFAULT 0,
|
||||
UNIQUE(account_id, path)
|
||||
);
|
||||
|
||||
@@ -130,15 +133,17 @@ func (d *DB) Accounts() ([]model.Account, error) {
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SourceFile records that a statement file was imported, returning its id.
|
||||
func (d *DB) SourceFile(accountID int64, path, sha, importedAt string) (int64, error) {
|
||||
// SourceFile records that a statement file was imported, holding rows
|
||||
// transactions, and returns its id.
|
||||
func (d *DB) SourceFile(accountID int64, path, sha, importedAt string, rows int) (int64, error) {
|
||||
_, err := d.sql.Exec(`
|
||||
INSERT INTO source_files (account_id, path, sha256, imported_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
INSERT INTO source_files (account_id, path, sha256, imported_at, rows)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(account_id, path) DO UPDATE SET
|
||||
sha256 = excluded.sha256,
|
||||
imported_at = excluded.imported_at`,
|
||||
accountID, path, sha, importedAt)
|
||||
imported_at = excluded.imported_at,
|
||||
rows = excluded.rows`,
|
||||
accountID, path, sha, importedAt, rows)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("record source file %s: %w", path, err)
|
||||
}
|
||||
@@ -393,6 +398,93 @@ func (d *DB) Months() ([]string, error) {
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SourceFileInfo is what the index recorded about one imported statement.
|
||||
type SourceFileInfo struct {
|
||||
AccountSlug string
|
||||
Path string // relative to the data root, as the importer records it
|
||||
SHA256 string
|
||||
ImportedAt string
|
||||
// Rows is how many transactions the statement holds, as parsed.
|
||||
Rows int
|
||||
// Added counts the transactions this file introduced. A row seen again in
|
||||
// an overlapping statement is deduplicated and stays with the file that
|
||||
// brought it first, so this is not how many rows the file holds.
|
||||
Added int
|
||||
}
|
||||
|
||||
// SourceFiles lists every statement the index has imported.
|
||||
func (d *DB) SourceFiles() ([]SourceFileInfo, error) {
|
||||
rows, err := d.sql.Query(`
|
||||
SELECT a.slug, s.path, s.sha256, s.imported_at, s.rows, COUNT(t.id)
|
||||
FROM source_files s
|
||||
JOIN accounts a ON a.id = s.account_id
|
||||
LEFT JOIN transactions t ON t.source_file_id = s.id
|
||||
GROUP BY s.id
|
||||
ORDER BY a.slug, s.path`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list source files: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []SourceFileInfo
|
||||
for rows.Next() {
|
||||
var f SourceFileInfo
|
||||
if err := rows.Scan(&f.AccountSlug, &f.Path, &f.SHA256, &f.ImportedAt, &f.Rows, &f.Added); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, f)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// ForgetSourceFile removes a statement from the index: its transactions, any
|
||||
// transfer pairing that used them, and the record of it, returning how many
|
||||
// transactions went. Pairing is derived state that the next Link rewrites, so
|
||||
// dropping a pair here loses nothing.
|
||||
//
|
||||
// A transaction two overlapping statements share is stored once, under the
|
||||
// file that brought it first, so removing that file can take rows another
|
||||
// statement still holds. The account's other statements therefore forget
|
||||
// their checksums, and the next import re-reads them and restores those rows.
|
||||
func (d *DB) ForgetSourceFile(accountSlug, path string) (int, error) {
|
||||
tx, err := d.sql.Begin()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
|
||||
var id, accountID int64
|
||||
err = tx.QueryRow(`
|
||||
SELECT s.id, s.account_id FROM source_files s JOIN accounts a ON a.id = s.account_id
|
||||
WHERE a.slug = ? AND s.path = ?`, accountSlug, path).Scan(&id, &accountID)
|
||||
if err == sql.ErrNoRows {
|
||||
return 0, nil
|
||||
}
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("find source file %s: %w", path, err)
|
||||
}
|
||||
if _, err := tx.Exec(`
|
||||
DELETE FROM transfers WHERE
|
||||
out_txn_id IN (SELECT id FROM transactions WHERE source_file_id = ?) OR
|
||||
in_txn_id IN (SELECT id FROM transactions WHERE source_file_id = ?)`, id, id); err != nil {
|
||||
return 0, fmt.Errorf("forget transfers from %s: %w", path, err)
|
||||
}
|
||||
res, err := tx.Exec(`DELETE FROM transactions WHERE source_file_id = ?`, id)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("forget transactions from %s: %w", path, err)
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if _, err := tx.Exec(`DELETE FROM source_files WHERE id = ?`, id); err != nil {
|
||||
return 0, fmt.Errorf("forget %s: %w", path, err)
|
||||
}
|
||||
if _, err := tx.Exec(`UPDATE source_files SET sha256 = '' WHERE account_id = ?`, accountID); err != nil {
|
||||
return 0, fmt.Errorf("mark %s's statements for re-reading: %w", accountSlug, err)
|
||||
}
|
||||
return int(n), tx.Commit()
|
||||
}
|
||||
|
||||
// Tags lists every tag in use, for completion in the rule builder.
|
||||
func (d *DB) Tags() ([]string, error) {
|
||||
rows, err := d.sql.Query(`
|
||||
|
||||
+417
-64
@@ -12,6 +12,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"mime"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -26,6 +27,7 @@ import (
|
||||
"git.petrovv.com/nikola/money/internal/glob"
|
||||
"git.petrovv.com/nikola/money/internal/importer"
|
||||
"git.petrovv.com/nikola/money/internal/model"
|
||||
"git.petrovv.com/nikola/money/internal/parser"
|
||||
"git.petrovv.com/nikola/money/internal/report"
|
||||
"git.petrovv.com/nikola/money/internal/rules"
|
||||
"git.petrovv.com/nikola/money/internal/store"
|
||||
@@ -76,6 +78,7 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /api/report", s.read(s.report))
|
||||
|
||||
mux.HandleFunc("GET /api/rules", s.read(s.ruleList))
|
||||
mux.HandleFunc("GET /api/tags", s.read(s.tagList))
|
||||
mux.HandleFunc("POST /api/rules/preview", s.read(s.rulePreview))
|
||||
mux.HandleFunc("POST /api/rules", s.write(s.createRule))
|
||||
mux.HandleFunc("PUT /api/rules/{pos}", s.write(s.editRule))
|
||||
@@ -88,6 +91,9 @@ func (s *Server) Handler() http.Handler {
|
||||
|
||||
mux.HandleFunc("POST /api/import", s.write(s.runImport))
|
||||
mux.HandleFunc("POST /api/upload", s.write(s.upload))
|
||||
mux.HandleFunc("GET /api/files", s.read(s.fileList))
|
||||
mux.HandleFunc("GET /api/files/{account}/{name}", s.serveFile)
|
||||
mux.HandleFunc("POST /api/files/delete", s.write(s.removeFile))
|
||||
mux.HandleFunc("POST /api/retag", s.write(s.retag))
|
||||
return mux
|
||||
}
|
||||
@@ -551,6 +557,55 @@ func (s *Server) ruleList(*http.Request) (any, error) {
|
||||
return map[string]any{"rules": rows}, nil
|
||||
}
|
||||
|
||||
type tagRule struct {
|
||||
Pos int `json:"pos"` // file position, as the rules screen numbers it
|
||||
Pattern string `json:"pattern"`
|
||||
Account string `json:"account"`
|
||||
Usage int `json:"usage"`
|
||||
}
|
||||
|
||||
type tagEntry struct {
|
||||
Tag string `json:"tag"`
|
||||
Transactions int `json:"transactions"`
|
||||
Rules []tagRule `json:"rules"`
|
||||
}
|
||||
|
||||
// tagList is every distinct tag — the ones the index holds and the ones
|
||||
// rules.toml names — with how many transactions carry it and the rules that
|
||||
// write it. A tag is only ever what a rule wrote, so the transfer label is
|
||||
// never among them, and a tag no rule names any more is one a retag will
|
||||
// clear.
|
||||
func (s *Server) tagList(*http.Request) (any, error) {
|
||||
txns, err := s.db.Transactions(store.Filter{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
counts := map[string]int{}
|
||||
for _, t := range txns {
|
||||
if t.RuleTag != "" {
|
||||
counts[t.RuleTag]++
|
||||
}
|
||||
}
|
||||
rules := map[string][]tagRule{}
|
||||
usage := s.engine.Usage(txns)
|
||||
for i, r := range s.engine.Rules() {
|
||||
rules[r.Tag] = append(rules[r.Tag], tagRule{Pos: i, Pattern: rulePattern(r), Account: r.Account, Usage: usage[i]})
|
||||
}
|
||||
tags, err := s.knownTags()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rows := make([]tagEntry, 0, len(tags))
|
||||
for _, tag := range tags {
|
||||
rs := rules[tag]
|
||||
if rs == nil {
|
||||
rs = []tagRule{}
|
||||
}
|
||||
rows = append(rows, tagEntry{Tag: tag, Transactions: counts[tag], Rules: rs})
|
||||
}
|
||||
return map[string]any{"tags": rows}, nil
|
||||
}
|
||||
|
||||
// rulePattern renders whichever patterns a rule sets, labelled so a type rule
|
||||
// is not mistaken for a description one.
|
||||
func rulePattern(r config.Rule) string {
|
||||
@@ -1208,26 +1263,22 @@ func (s *Server) runImport(r *http.Request) (any, error) {
|
||||
if err := decode(r, &req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.importAll(req.Force)
|
||||
}
|
||||
|
||||
func (s *Server) importAll(force bool) (importJSON, error) {
|
||||
accounts, err := config.LoadAccounts(s.root)
|
||||
if err != nil {
|
||||
return importJSON{}, err
|
||||
return nil, err
|
||||
}
|
||||
if len(accounts) == 0 {
|
||||
return importJSON{}, badRequest("no accounts found in %s (an account is a folder containing %s)",
|
||||
return nil, badRequest("no accounts found in %s (an account is a folder containing %s)",
|
||||
s.root, config.AccountFile)
|
||||
}
|
||||
if err := s.reloadRules(); err != nil {
|
||||
return importJSON{}, err
|
||||
return nil, err
|
||||
}
|
||||
s.accounts = accounts
|
||||
|
||||
res, err := importer.Run(s.root, s.db, s.accounts, s.engine, s.links, importer.Options{Force: force})
|
||||
res, err := importer.Run(s.root, s.db, s.accounts, s.engine, s.links, importer.Options{Force: req.Force})
|
||||
if err != nil {
|
||||
return importJSON{}, err
|
||||
return nil, err
|
||||
}
|
||||
|
||||
_, added, skipped := res.Total()
|
||||
@@ -1266,10 +1317,11 @@ type uploadReq struct {
|
||||
Files []uploadFile `json:"files"`
|
||||
}
|
||||
|
||||
// upload saves statements into an account folder and imports them. The files
|
||||
// on disk are the source of truth and the index is derived from them, so an
|
||||
// upload is nothing more than putting a file where `money import` looks; the
|
||||
// import that follows is the one the Import button runs.
|
||||
// upload saves statements into an account folder. The files on disk are the
|
||||
// source of truth and the index is derived from them, so an upload is nothing
|
||||
// more than putting a file where `money import` looks. It does not import:
|
||||
// that stays the user's call, made with the Import button, so a batch can be
|
||||
// put together and checked on the statements list before it is read.
|
||||
//
|
||||
// Files arrive base64 inside JSON rather than as multipart: a multipart body
|
||||
// is exactly what a cross-site form can send, and the JSON-only rule is the
|
||||
@@ -1297,60 +1349,133 @@ func (s *Server) upload(r *http.Request) (any, error) {
|
||||
req.Account, s.root, config.AccountFile)
|
||||
}
|
||||
|
||||
// Everything is checked before anything is written, so a bad file in a
|
||||
// batch leaves the folder as it was rather than half uploaded.
|
||||
var write []uploadFile
|
||||
var same []string
|
||||
seen := map[string]bool{}
|
||||
for _, f := range req.Files {
|
||||
if err := checkStatementName(acc, f.Name); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if seen[f.Name] {
|
||||
return nil, badRequest("%s is in the upload twice", f.Name)
|
||||
}
|
||||
seen[f.Name] = true
|
||||
existing, err := os.ReadFile(filepath.Join(acc.Dir, f.Name))
|
||||
switch {
|
||||
case err == nil && bytes.Equal(existing, f.Data):
|
||||
same = append(same, f.Name)
|
||||
case err == nil:
|
||||
// A statement is the source of truth for what it already
|
||||
// imported; replacing it under the same name is not an upload's
|
||||
// decision to make.
|
||||
return nil, &apiError{http.StatusConflict, fmt.Sprintf(
|
||||
"%s/%s already exists with different contents; rename the file or remove the old one first",
|
||||
acc.Slug, f.Name)}
|
||||
case !errors.Is(err, fs.ErrNotExist):
|
||||
return nil, err
|
||||
default:
|
||||
write = append(write, f)
|
||||
}
|
||||
p, err := parser.For(acc)
|
||||
if err != nil {
|
||||
return nil, badRequest("%v", err)
|
||||
}
|
||||
for _, f := range write {
|
||||
if err := writeStatement(acc.Dir, f); err != nil {
|
||||
namer, _ := p.(parser.Namer)
|
||||
|
||||
// Each file is staged as a dotfile — invisible to import — so a parser
|
||||
// that names its statements can read it. Everything is then checked
|
||||
// before anything is moved into place, so a bad file in a batch leaves
|
||||
// the folder as it was rather than half uploaded.
|
||||
type staged struct {
|
||||
orig, name, tmp string
|
||||
data []byte
|
||||
named bool // the parser chose name, from the statement
|
||||
}
|
||||
var batch []*staged
|
||||
defer func() {
|
||||
for _, f := range batch {
|
||||
if f.tmp != "" {
|
||||
os.Remove(f.tmp)
|
||||
}
|
||||
}
|
||||
}()
|
||||
for _, f := range req.Files {
|
||||
if err := checkPlainName(f.Name); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tmp, err := stageStatement(acc.Dir, f.Data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
st := &staged{orig: f.Name, name: f.Name, tmp: tmp, data: f.Data}
|
||||
batch = append(batch, st)
|
||||
if namer != nil {
|
||||
stem, err := namer.StatementName(tmp)
|
||||
if err != nil {
|
||||
return nil, badRequest("%s: %v", f.Name, err)
|
||||
}
|
||||
if stem != "" {
|
||||
st.name, st.named = stem+strings.ToLower(filepath.Ext(f.Name)), true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
out, err := s.importAll(false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
var write []*staged
|
||||
var same, renamed []string
|
||||
claimed := map[string][]byte{}
|
||||
for _, f := range batch {
|
||||
base, n := f.name, 2
|
||||
resolve:
|
||||
for {
|
||||
if err := checkStatementName(acc, f.name); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
existing, inBatch := claimed[f.name]
|
||||
var err error
|
||||
if !inBatch {
|
||||
existing, err = os.ReadFile(filepath.Join(acc.Dir, f.name))
|
||||
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case !inBatch && err != nil: // free
|
||||
claimed[f.name] = f.data
|
||||
write = append(write, f)
|
||||
break resolve
|
||||
case bytes.Equal(existing, f.data):
|
||||
same = append(same, f.orig)
|
||||
break resolve
|
||||
case f.named:
|
||||
// A name the parser chose can be taken by a different
|
||||
// statement of the same date — a reissue — so it is
|
||||
// numbered, as rename_izpiski.py did, rather than refused.
|
||||
f.name = fmt.Sprintf("%s_%d%s", strings.TrimSuffix(base, filepath.Ext(base)), n, filepath.Ext(base))
|
||||
n++
|
||||
case inBatch:
|
||||
return nil, badRequest("%s is in the upload twice", f.orig)
|
||||
default:
|
||||
// A statement is the source of truth for what it already
|
||||
// imported; replacing it under the same name is not an
|
||||
// upload's decision to make.
|
||||
return nil, &apiError{http.StatusConflict, fmt.Sprintf(
|
||||
"%s/%s already exists with different contents; rename the file or remove the old one first",
|
||||
acc.Slug, f.name)}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, f := range write {
|
||||
path := filepath.Join(acc.Dir, f.name)
|
||||
if err := os.Rename(f.tmp, path); err != nil {
|
||||
return nil, fmt.Errorf("write %s: %w", path, err)
|
||||
}
|
||||
f.tmp = ""
|
||||
if f.name != f.orig {
|
||||
renamed = append(renamed, f.orig+" → "+f.name)
|
||||
}
|
||||
}
|
||||
|
||||
msg := fmt.Sprintf("uploaded %d file(s) to %s", len(write), acc.Slug)
|
||||
if len(same) > 0 {
|
||||
msg += fmt.Sprintf(" (%d already there)", len(same))
|
||||
}
|
||||
out.Status = msg + " · " + out.Status
|
||||
return out, nil
|
||||
if len(renamed) > 0 {
|
||||
msg += ", named by statement date: " + strings.Join(renamed, ", ")
|
||||
}
|
||||
if len(write) > 0 {
|
||||
msg += " · press Import to read them"
|
||||
}
|
||||
return status{msg}, nil
|
||||
}
|
||||
|
||||
// checkPlainName refuses anything that is not a plain file name, before the
|
||||
// file is so much as staged.
|
||||
func checkPlainName(name string) error {
|
||||
if name == "" || name != filepath.Base(name) || strings.ContainsAny(name, "/\\\x00") ||
|
||||
name == "." || name == ".." {
|
||||
return badRequest("%q is not a plain file name", name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkStatementName refuses any name the importer would not read back as a
|
||||
// statement of this account, and anything that is not a plain file name.
|
||||
func checkStatementName(acc *config.Account, name string) error {
|
||||
if name == "" || name != filepath.Base(name) || strings.ContainsAny(name, "/\\\x00") ||
|
||||
name == "." || name == ".." {
|
||||
return badRequest("%q is not a plain file name", name)
|
||||
if err := checkPlainName(name); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.HasPrefix(name, ".") || name == config.AccountFile {
|
||||
return badRequest("%s would be ignored by import (dotfiles and %s are not statements)",
|
||||
@@ -1363,29 +1488,257 @@ func checkStatementName(acc *config.Account, name string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// writeStatement writes through a dotfile and renames it into place, so a
|
||||
// stageStatement writes an upload to a dotfile in the account folder. Import
|
||||
// skips dotfiles, so it is invisible until renamed into place — and a
|
||||
// concurrent `money import` never reads half a statement.
|
||||
func writeStatement(dir string, f uploadFile) error {
|
||||
path := filepath.Join(dir, f.Name)
|
||||
func stageStatement(dir string, data []byte) (string, error) {
|
||||
tmp, err := os.CreateTemp(dir, ".upload-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("write %s: %w", path, err)
|
||||
return "", fmt.Errorf("write to %s: %w", dir, err)
|
||||
}
|
||||
defer os.Remove(tmp.Name()) // a no-op once renamed
|
||||
if _, err := tmp.Write(f.Data); err != nil {
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
tmp.Close()
|
||||
return fmt.Errorf("write %s: %w", path, err)
|
||||
os.Remove(tmp.Name())
|
||||
return "", fmt.Errorf("write %s: %w", tmp.Name(), err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("write %s: %w", path, err)
|
||||
os.Remove(tmp.Name())
|
||||
return "", fmt.Errorf("write %s: %w", tmp.Name(), err)
|
||||
}
|
||||
if err := os.Chmod(tmp.Name(), 0o644); err != nil {
|
||||
return err
|
||||
os.Remove(tmp.Name())
|
||||
return "", err
|
||||
}
|
||||
if err := os.Rename(tmp.Name(), path); err != nil {
|
||||
return fmt.Errorf("write %s: %w", path, err)
|
||||
return tmp.Name(), nil
|
||||
}
|
||||
|
||||
type fileRow struct {
|
||||
Account string `json:"account"`
|
||||
Name string `json:"name"`
|
||||
Size int64 `json:"size"`
|
||||
// Modified is the file's mtime, YYYY-MM-DD HH:MM in local time; blank for
|
||||
// a file that is gone from disk.
|
||||
Modified string `json:"modified"`
|
||||
// Status is "imported", "changed" (on disk differs from what was
|
||||
// imported), "new" (not imported yet, or its import failed) or "missing"
|
||||
// (imported, then removed from disk; its rows stay in the index until it
|
||||
// is rebuilt).
|
||||
Status string `json:"status"`
|
||||
ImportedAt string `json:"importedAt"`
|
||||
// Rows is how many transactions the statement holds; Added how many of
|
||||
// them it was the first to bring in, which overlap with another statement
|
||||
// can make fewer.
|
||||
Rows int `json:"rows"`
|
||||
Added int `json:"added"`
|
||||
}
|
||||
|
||||
// fileList is every statement on disk next to what the index recorded about
|
||||
// it. The folders are the source of truth, so they decide what is listed, and
|
||||
// a file the index remembers but the disk no longer holds is called out: its
|
||||
// rows would not survive a rebuild.
|
||||
func (s *Server) fileList(*http.Request) (any, error) {
|
||||
recorded, err := s.db.SourceFiles()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return nil
|
||||
byPath := map[string]store.SourceFileInfo{}
|
||||
for _, f := range recorded {
|
||||
byPath[f.Path] = f
|
||||
}
|
||||
accounts, err := config.LoadAccounts(s.root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
out := []fileRow{}
|
||||
onDisk := map[string]bool{}
|
||||
for _, acc := range accounts {
|
||||
paths, err := importer.StatementFiles(acc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, path := range paths {
|
||||
rel, err := filepath.Rel(s.root, path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
onDisk[rel] = true
|
||||
row := fileRow{Account: acc.Slug, Name: filepath.Base(path), Status: "new"}
|
||||
if info, err := os.Stat(path); err == nil {
|
||||
row.Size, row.Modified = info.Size(), info.ModTime().Format("2006-01-02 15:04")
|
||||
}
|
||||
if rec, ok := byPath[rel]; ok {
|
||||
row.ImportedAt, row.Rows, row.Added, row.Status = rec.ImportedAt, rec.Rows, rec.Added, "imported"
|
||||
if sum, err := importer.Checksum(path); err != nil {
|
||||
return nil, err
|
||||
} else if sum != rec.SHA256 {
|
||||
row.Status = "changed"
|
||||
}
|
||||
}
|
||||
out = append(out, row)
|
||||
}
|
||||
}
|
||||
for _, rec := range recorded {
|
||||
if !onDisk[rec.Path] {
|
||||
out = append(out, fileRow{
|
||||
Account: rec.AccountSlug, Name: filepath.Base(rec.Path), Status: "missing",
|
||||
ImportedAt: rec.ImportedAt, Rows: rec.Rows, Added: rec.Added,
|
||||
})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(out, func(i, j int) bool {
|
||||
if out[i].Account != out[j].Account {
|
||||
return out[i].Account < out[j].Account
|
||||
}
|
||||
return out[i].Name < out[j].Name
|
||||
})
|
||||
return map[string]any{"files": out}, nil
|
||||
}
|
||||
|
||||
// serveFile opens a statement in the browser. Only a file import would read
|
||||
// is served — it is looked up in the account's statement list, never joined
|
||||
// onto a path — so nothing else under the data root can be fetched.
|
||||
//
|
||||
// A statement is whatever the bank sent, and it is served from this origin,
|
||||
// where a script could drive the API; so nothing is ever rendered as a page.
|
||||
// Text opens inline as text/plain under a sandboxing CSP, and anything that is
|
||||
// neither text nor PDF downloads. A PDF opens inline without the sandbox: the
|
||||
// browsers' viewers refuse to run under one, and they render it in their own
|
||||
// isolated viewer rather than in this origin's DOM.
|
||||
func (s *Server) serveFile(w http.ResponseWriter, r *http.Request) {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
fail := func(code int, msg string) {
|
||||
writeJSON(w, code, map[string]string{"error": msg})
|
||||
}
|
||||
accounts, err := config.LoadAccounts(s.root)
|
||||
if err != nil {
|
||||
fail(http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
account, name := r.PathValue("account"), r.PathValue("name")
|
||||
var path string
|
||||
for _, acc := range accounts {
|
||||
if acc.Slug != account {
|
||||
continue
|
||||
}
|
||||
paths, err := importer.StatementFiles(acc)
|
||||
if err != nil {
|
||||
fail(http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
for _, p := range paths {
|
||||
if filepath.Base(p) == name {
|
||||
path = p
|
||||
}
|
||||
}
|
||||
}
|
||||
if path == "" {
|
||||
fail(http.StatusNotFound, fmt.Sprintf("no statement %s/%s", account, name))
|
||||
return
|
||||
}
|
||||
|
||||
disposition := "attachment"
|
||||
switch strings.ToLower(filepath.Ext(name)) {
|
||||
case ".pdf":
|
||||
w.Header().Set("Content-Type", "application/pdf")
|
||||
disposition = "inline"
|
||||
case ".csv", ".txt", ".tsv":
|
||||
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
w.Header().Set("Content-Security-Policy", "sandbox")
|
||||
disposition = "inline"
|
||||
default:
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
w.Header().Set("Content-Security-Policy", "sandbox")
|
||||
}
|
||||
w.Header().Set("Content-Disposition", mime.FormatMediaType(disposition, map[string]string{"filename": name}))
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
http.ServeFile(w, r, path)
|
||||
}
|
||||
|
||||
type removeFileReq struct {
|
||||
Account string `json:"account"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
// removeFile takes a statement out of the data: the file is deleted and its
|
||||
// transactions leave the index. A file already gone from disk is only
|
||||
// forgotten by the index. The deletion is permanent — the page says so before
|
||||
// asking — so nothing here keeps a copy.
|
||||
//
|
||||
// It does not import. Rows the file shared with an overlapping statement went
|
||||
// with it, and ForgetSourceFile marks the account's other statements so the
|
||||
// next Import re-reads them and restores those rows; until then they show as
|
||||
// changed. Pairing is re-derived here, though: it reads no statement, and a
|
||||
// leg whose partner just went should not stay counted as a transfer.
|
||||
func (s *Server) removeFile(r *http.Request) (any, error) {
|
||||
var req removeFileReq
|
||||
if err := decode(r, &req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := checkPlainName(req.Name); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rel := filepath.Join(req.Account, req.Name)
|
||||
|
||||
// The file is found in the account's statement list, never by joining the
|
||||
// request onto a path, exactly as serveFile does.
|
||||
var path string
|
||||
accounts, err := config.LoadAccounts(s.root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, acc := range accounts {
|
||||
if acc.Slug != req.Account {
|
||||
continue
|
||||
}
|
||||
paths, err := importer.StatementFiles(acc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range paths {
|
||||
if filepath.Base(p) == req.Name {
|
||||
path = p
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if path != "" {
|
||||
if err := os.Remove(path); err != nil {
|
||||
return nil, fmt.Errorf("remove %s: %w", path, err)
|
||||
}
|
||||
}
|
||||
dropped, err := s.db.ForgetSourceFile(req.Account, rel)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if path == "" && dropped == 0 {
|
||||
known, err := s.db.SourceFiles()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !slices.ContainsFunc(known, func(f store.SourceFileInfo) bool { return f.Path == rel }) {
|
||||
return nil, &apiError{http.StatusNotFound, fmt.Sprintf("no statement %s", rel)}
|
||||
}
|
||||
}
|
||||
|
||||
_, unpaired, err := s.links.Link(s.db)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
verb := "deleted"
|
||||
if path == "" {
|
||||
verb = "forgot"
|
||||
}
|
||||
msg := fmt.Sprintf("%s %s, %d transaction(s) dropped", verb, rel, dropped)
|
||||
if unpaired > 0 {
|
||||
msg += fmt.Sprintf(" · %d transfer leg(s) unpaired", unpaired)
|
||||
}
|
||||
if dropped > 0 {
|
||||
msg += " · press Import to restore any of them another statement also holds"
|
||||
}
|
||||
return status{msg}, nil
|
||||
}
|
||||
|
||||
// retag is `money retag`: both halves of what rules.toml decides, re-derived
|
||||
|
||||
@@ -44,7 +44,7 @@ func newTestServer(t *testing.T, rulesToml string, txns ...fixtureTxn) (*Server,
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
src, err := db.SourceFile(id, slug+"/st.csv", "sha", "2026-01-01T00:00:00Z")
|
||||
src, err := db.SourceFile(id, slug+"/st.csv", "sha", "2026-01-01T00:00:00Z", 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -461,3 +461,56 @@ to_desc = "*FROM CHECKING*"
|
||||
t.Errorf("oldest leg = %+v", oldest)
|
||||
}
|
||||
}
|
||||
|
||||
// The tags screen lists each tag once, with the transactions carrying it and
|
||||
// every rule that writes it — a shadowed one at zero — and never the
|
||||
// transfer label, which no rule wrote.
|
||||
func TestTagListGroupsRulesByTag(t *testing.T) {
|
||||
_, h := newTestServer(t, `
|
||||
[[rule]]
|
||||
match = "*LIDL*"
|
||||
tag = "groceries"
|
||||
|
||||
[[rule]]
|
||||
match = "*KAUFLAND*"
|
||||
tag = "groceries"
|
||||
|
||||
[[rule]]
|
||||
match = "*LIDL SOFIA*"
|
||||
tag = "groceries"
|
||||
|
||||
[[rule]]
|
||||
match = "*ZARA*"
|
||||
tag = "clothes"
|
||||
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*TO SAVINGS*"
|
||||
to_account = "savings"
|
||||
to_desc = "*FROM CHECKING*"
|
||||
`,
|
||||
fixtureTxn{"checking", "2026-02-01", "LIDL SOFIA", -1000},
|
||||
fixtureTxn{"checking", "2026-02-02", "KAUFLAND", -500},
|
||||
fixtureTxn{"checking", "2026-02-03", "TO SAVINGS", -100},
|
||||
fixtureTxn{"savings", "2026-02-03", "FROM CHECKING", 100},
|
||||
)
|
||||
var res struct{ Tags []tagEntry }
|
||||
call(t, h, "GET", "/api/tags", nil, http.StatusOK, &res)
|
||||
if len(res.Tags) != 2 || res.Tags[0].Tag != "clothes" || res.Tags[1].Tag != "groceries" {
|
||||
t.Fatalf("tags = %+v, want clothes and groceries only", res.Tags)
|
||||
}
|
||||
clothes, groceries := res.Tags[0], res.Tags[1]
|
||||
if clothes.Transactions != 0 || len(clothes.Rules) != 1 {
|
||||
t.Errorf("clothes = %+v, want its rule and no transactions", clothes)
|
||||
}
|
||||
if groceries.Transactions != 2 || len(groceries.Rules) != 3 {
|
||||
t.Fatalf("groceries = %+v, want 2 transactions from 3 rules", groceries)
|
||||
}
|
||||
usage := map[string]int{}
|
||||
for _, r := range groceries.Rules {
|
||||
usage[r.Pattern] = r.Usage
|
||||
}
|
||||
if usage["*LIDL*"] != 0 || usage["*LIDL SOFIA*"] != 1 || usage["*KAUFLAND*"] != 1 {
|
||||
t.Errorf("usage = %v, want *LIDL* shadowed by *LIDL SOFIA*", usage)
|
||||
}
|
||||
}
|
||||
|
||||
+145
-99
@@ -12,6 +12,7 @@ const VIEWS = [
|
||||
{ id: 'rules', title: 'Rules', mount: mountRules },
|
||||
{ id: 'transfer', title: 'Transfer builder', mount: mountTransferBuilder },
|
||||
{ id: 'transfers', title: 'Transfers', mount: mountTransfers },
|
||||
{ id: 'tags', title: 'Tags', mount: mountTags },
|
||||
];
|
||||
|
||||
const state = {
|
||||
@@ -174,7 +175,7 @@ function renderNav() {
|
||||
nav.replaceChildren(...VIEWS.map((v, i) => h('a', {
|
||||
href: '#' + v.id,
|
||||
class: v.id === state.view ? 'active' : null,
|
||||
}, h('kbd', {}, String(i + 1)), v.title)));
|
||||
}, v.title)));
|
||||
}
|
||||
|
||||
function go(id) {
|
||||
@@ -188,8 +189,8 @@ function go(id) {
|
||||
function show() {
|
||||
const id = location.hash.slice(1) || 'accounts';
|
||||
const view = VIEWS.find((v) => v.id === id) || VIEWS[0];
|
||||
// Leaving the builder mid-edit abandons the edit, as esc does: a new rule
|
||||
// must not start out as a copy of the one being edited.
|
||||
// Leaving the builder mid-edit abandons the edit, as Cancel does: a new
|
||||
// rule must not start out as a copy of the one being edited.
|
||||
if (view.id !== 'rule' && state.rule.editing) state.rule = blankRule();
|
||||
state.view = view.id;
|
||||
renderNav();
|
||||
@@ -205,35 +206,30 @@ async function refreshAll() {
|
||||
|
||||
// --- import and retag -----------------------------------------------------
|
||||
|
||||
function runImport(force) {
|
||||
return importing(() => api('POST', '/api/import', { force: !!force }));
|
||||
}
|
||||
|
||||
// importing runs a request that ends in an import — the Import button, or an
|
||||
// upload — and reports it the same way. It answers whether the request
|
||||
// succeeded; per-file failures inside the import still count as success.
|
||||
async function importing(request) {
|
||||
if (state.busy) return false;
|
||||
// runImport is `money import`, or with force `money import --force`: every
|
||||
// statement re-read, including the ones the checksum would skip.
|
||||
async function runImport(force) {
|
||||
if (state.busy) return;
|
||||
state.busy = true;
|
||||
const btn = document.getElementById('import');
|
||||
btn.disabled = true;
|
||||
btn.textContent = 'Importing…';
|
||||
const buttons = ['import', 'reread'].map((id) => document.getElementById(id));
|
||||
const btn = buttons[force ? 1 : 0];
|
||||
const label = btn.textContent;
|
||||
for (const b of buttons) b.disabled = true;
|
||||
btn.textContent = force ? 'Re-reading…' : 'Importing…';
|
||||
setStatus('importing… parsing PDF statements can take a while');
|
||||
try {
|
||||
const res = await request();
|
||||
const res = await api('POST', '/api/import', { force: !!force });
|
||||
state.status = res.status;
|
||||
state.error = res.failed ? `${res.failed} file(s) failed to import` : '';
|
||||
renderBanners();
|
||||
showImportDetails(res);
|
||||
await refreshAll();
|
||||
return true;
|
||||
} catch (e) {
|
||||
setError(e);
|
||||
return false;
|
||||
} finally {
|
||||
state.busy = false;
|
||||
btn.disabled = false;
|
||||
btn.textContent = 'Import';
|
||||
for (const b of buttons) b.disabled = false;
|
||||
btn.textContent = label;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -266,13 +262,15 @@ async function retag() {
|
||||
|
||||
function mountAccounts(main) {
|
||||
const body = h('div');
|
||||
const files = h('div');
|
||||
const upload = h('div');
|
||||
main.append(h('h2', {}, 'Accounts'), body, upload);
|
||||
main.append(h('h2', {}, 'Accounts'), body, files, upload);
|
||||
|
||||
async function refresh() {
|
||||
if (!state.overview) await loadOverview();
|
||||
const o = state.overview;
|
||||
upload.replaceChildren(uploader(o));
|
||||
api('GET', '/api/files').then((res) => files.replaceChildren(fileList(res.files))).catch(setError);
|
||||
if (!o.accounts.length) {
|
||||
// An account.toml is not enough on its own: folders reach the index
|
||||
// only through an import, so say that rather than show nothing.
|
||||
@@ -295,6 +293,82 @@ function mountAccounts(main) {
|
||||
return { refresh };
|
||||
}
|
||||
|
||||
const FILE_STATUS = {
|
||||
imported: ['✓', '', 'imported'],
|
||||
changed: ['⚠', 'warn', 'changed'],
|
||||
new: ['•', 'warn', 'not imported'],
|
||||
missing: ['✗', 'neg', 'gone from disk'],
|
||||
};
|
||||
|
||||
// fileList shows the statements in every account folder beside what the index
|
||||
// recorded about each, and opens one on click. The folders decide what is
|
||||
// listed; the index only says how far each file has got.
|
||||
function fileList(list) {
|
||||
const panel = h('div', { class: 'panel files' });
|
||||
if (!list.length) {
|
||||
panel.append(h('div', { class: 'empty' }, 'No statement files yet.'));
|
||||
return h('div', {}, h('h3', { class: 'section' }, 'Statements'), panel);
|
||||
}
|
||||
const pending = list.filter((f) => f.status !== 'imported').length;
|
||||
panel.append(h('table', {},
|
||||
h('thead', {}, h('tr', {}, h('th', {}, 'Account'), h('th', {}, 'File'), h('th', { class: 'num' }, 'Size'),
|
||||
h('th', {}, 'Modified'), h('th', {}, ''), h('th', {}, 'Status'),
|
||||
h('th', { class: 'num', title: 'Transactions the file holds, and how many of them it was the first to bring in — an overlapping statement imported earlier already holds the rest' }, 'Transactions'),
|
||||
h('th', {}, 'Imported'), h('th', {}, ''))),
|
||||
h('tbody', {}, list.map((f) => {
|
||||
const [mark, cls, text] = FILE_STATUS[f.status] || ['', '', f.status];
|
||||
return h('tr', {},
|
||||
h('td', {}, f.account),
|
||||
h('td', { class: 'desc' }, f.status === 'missing' ? h('span', { class: 'muted' }, f.name) : h('a', {
|
||||
href: `/api/files/${encodeURIComponent(f.account)}/${encodeURIComponent(f.name)}`,
|
||||
target: '_blank', rel: 'noopener', title: 'Open this statement',
|
||||
}, f.name)),
|
||||
h('td', { class: 'num' }, f.status === 'missing' ? '' : formatSize(f.size)),
|
||||
h('td', { class: 'muted' }, f.modified),
|
||||
h('td', { class: 'mark ' + cls }, mark),
|
||||
h('td', { class: cls || 'muted' }, text),
|
||||
h('td', { class: 'num' }, f.status === 'new' ? '' : `${f.rows} ${f.rows === 1 ? 'row' : 'rows'} · ${f.added} new`),
|
||||
h('td', { class: 'muted' }, localTime(f.importedAt)),
|
||||
h('td', { class: 'actions' }, h('button', {
|
||||
type: 'button', class: 'danger', onclick: () => removeFile(f),
|
||||
}, f.status === 'missing' ? 'Forget' : 'Delete')));
|
||||
}))));
|
||||
return h('div', {}, h('h3', { class: 'section' }, 'Statements ',
|
||||
h('span', { class: 'sub' }, `· ${list.length} file(s)` + (pending ? ` · ${pending} need attention` : ''))), panel);
|
||||
}
|
||||
|
||||
// localTime shows the index's UTC import time in the same local
|
||||
// YYYY-MM-DD HH:MM that the server gives the files' modified times.
|
||||
function localTime(iso) {
|
||||
if (!iso) return '';
|
||||
const d = new Date(iso);
|
||||
if (isNaN(d)) return iso;
|
||||
const p = (n) => String(n).padStart(2, '0');
|
||||
return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())} ${p(d.getHours())}:${p(d.getMinutes())}`;
|
||||
}
|
||||
|
||||
// removeFile deletes a statement for good. It does not import: rows another
|
||||
// statement also holds come back at the next Import, so the question asked
|
||||
// here is only about what this file alone brought in.
|
||||
function removeFile(f) {
|
||||
const where = `${f.account}/${f.name}`;
|
||||
const rows = f.status === 'new' ? 'It has not been imported, so no transactions change.'
|
||||
: `The ${f.added} transaction(s) it brought in leave the index; any another statement also holds come back at the next Import.`;
|
||||
const question = f.status === 'missing'
|
||||
? `Forget ${where}? The file is already gone from disk. ${rows}`
|
||||
: `Delete ${where} permanently? The file is removed from disk and cannot be restored from here. ${rows}`;
|
||||
if (!confirm(question)) return;
|
||||
api('POST', '/api/files/delete', { account: f.account, name: f.name })
|
||||
.then((res) => { setStatus(res.status); return refreshAll(); })
|
||||
.catch(setError);
|
||||
}
|
||||
|
||||
function formatSize(bytes) {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
if (bytes < 1024 * 1024) return `${Math.round(bytes / 1024)} KB`;
|
||||
return `${(bytes / 1024 / 1024).toFixed(1)} MB`;
|
||||
}
|
||||
|
||||
// uploader puts statement files into an account folder and imports them. The
|
||||
// folder is where they belong: the index is rebuilt from it, so a statement
|
||||
// that lived only in the index would be lost the next time it is deleted.
|
||||
@@ -318,7 +392,7 @@ function uploader(o) {
|
||||
onchange: (e) => choose(e.target.files),
|
||||
});
|
||||
const chosen = h('div', { class: 'chosen muted' }, 'No files chosen');
|
||||
const send = h('button', { type: 'button', class: 'primary', disabled: true, onclick: submit }, 'Upload and import');
|
||||
const send = h('button', { type: 'button', class: 'primary', disabled: true, onclick: submit }, 'Upload');
|
||||
const zone = h('div', {
|
||||
class: 'dropzone',
|
||||
tabindex: 0,
|
||||
@@ -345,10 +419,11 @@ function uploader(o) {
|
||||
account: account.value,
|
||||
files: await Promise.all(files.map(async (f) => ({ name: f.name, data: await base64(f) }))),
|
||||
};
|
||||
if (await importing(() => api('POST', '/api/upload', payload))) {
|
||||
picker.value = '';
|
||||
choose([]);
|
||||
}
|
||||
const res = await api('POST', '/api/upload', payload);
|
||||
picker.value = '';
|
||||
choose([]);
|
||||
setStatus(res.status);
|
||||
await refreshAll();
|
||||
} catch (e) {
|
||||
setError(e);
|
||||
} finally {
|
||||
@@ -360,7 +435,7 @@ function uploader(o) {
|
||||
h('div', { class: 'toolbar' }, h('label', {}, 'Account ', account), send),
|
||||
zone, picker, chosen,
|
||||
h('div', { class: 'hint muted' },
|
||||
'Files are saved into the account folder, next to the statements already there, and imported. ' +
|
||||
'Files are saved into the account folder, next to the statements already there; press Import to read them. ' +
|
||||
'A file with the same name and different contents is refused rather than replaced.'));
|
||||
return panel;
|
||||
}
|
||||
@@ -388,7 +463,7 @@ function scopeToolbar(onChange) {
|
||||
select.value = state.filter.account;
|
||||
|
||||
const search = h('input', {
|
||||
type: 'search', id: 'search', placeholder: 'Filter by description ( / )',
|
||||
type: 'search', id: 'search', placeholder: 'Filter by description',
|
||||
value: state.filter.search,
|
||||
oninput: debounce((e) => { state.filter.search = e.target.value.trim(); onChange(); }, 200),
|
||||
});
|
||||
@@ -397,7 +472,7 @@ function scopeToolbar(onChange) {
|
||||
onchange: (e) => { state.filter.untagged = e.target.checked; onChange(); },
|
||||
});
|
||||
return h('div', { class: 'toolbar' }, select, search,
|
||||
h('label', { class: 'check', title: 'No tag and not a matched transfer leg (u)' }, untagged, 'Untagged only'));
|
||||
h('label', { class: 'check', title: 'No tag and not a matched transfer leg' }, untagged, 'Untagged only'));
|
||||
}
|
||||
|
||||
// --- transactions ---------------------------------------------------------
|
||||
@@ -500,7 +575,7 @@ function mountReport(main) {
|
||||
// An empty period is not an empty index, and the default window is the
|
||||
// one most likely to be empty: last month's statement may not be in yet.
|
||||
content = h('div', { class: 'panel empty' }, period.bounded && !data.indexEmpty
|
||||
? `Nothing in ${period.name}.\n\nChoose another period (← →), or import.`
|
||||
? `Nothing in ${period.name}.\n\nChoose another period, or import.`
|
||||
: 'Nothing to report yet.\n\nImport some statements first.');
|
||||
} else {
|
||||
const order = data.orders.find((o) => o.name === data.order);
|
||||
@@ -540,21 +615,7 @@ function mountReport(main) {
|
||||
} catch (e) { setError(e); }
|
||||
}
|
||||
refresh();
|
||||
return {
|
||||
refresh,
|
||||
key(e, typing) {
|
||||
if (typing || !data || e.ctrlKey || e.metaKey || e.altKey) return false;
|
||||
if (e.key === 'ArrowLeft') { setPeriod(data.period - 1); return true; }
|
||||
if (e.key === 'ArrowRight') { setPeriod(data.period + 1); return true; }
|
||||
if (e.key === 's') {
|
||||
const i = data.orders.findIndex((o) => o.name === data.order);
|
||||
state.report.sort = data.orders[(i + 1) % data.orders.length].name;
|
||||
refresh();
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
},
|
||||
};
|
||||
return { refresh };
|
||||
}
|
||||
|
||||
// --- form helpers ---------------------------------------------------------
|
||||
@@ -641,8 +702,8 @@ function mountRuleBuilder(main) {
|
||||
}
|
||||
preview.replaceChildren(h('table', {},
|
||||
h('thead', {}, h('tr', {}, h('th', {}, ''),
|
||||
h('th', { class: 'sortable', onclick: toggle, title: 'Sort by name / by count (ctrl+s)' }, descTitle),
|
||||
h('th', { class: 'num sortable', onclick: toggle, title: 'Sort by name / by count (ctrl+s)' }, 'N' + (byCount ? ' ↓' : '')))),
|
||||
h('th', { class: 'sortable', onclick: toggle, title: 'Sort by name / by count' }, descTitle),
|
||||
h('th', { class: 'num sortable', onclick: toggle, title: 'Sort by name / by count' }, 'N' + (byCount ? ' ↓' : '')))),
|
||||
h('tbody', {}, res.rows.map((row) => h('tr', {
|
||||
class: 'clickable',
|
||||
title: 'Use this description as the glob',
|
||||
@@ -687,23 +748,7 @@ function mountRuleBuilder(main) {
|
||||
|
||||
inputs.match.focus();
|
||||
refreshNow();
|
||||
return {
|
||||
refresh: refreshNow,
|
||||
form: true,
|
||||
key(e) {
|
||||
if (e.key === 's' && e.ctrlKey) {
|
||||
state.ruleSort = state.ruleSort === 'count' ? 'name' : 'count';
|
||||
refreshNow();
|
||||
return true;
|
||||
}
|
||||
if (e.key === 'Escape' && editing) {
|
||||
state.rule = blankRule();
|
||||
go('rules');
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
},
|
||||
};
|
||||
return { refresh: refreshNow };
|
||||
}
|
||||
|
||||
// --- rules ----------------------------------------------------------------
|
||||
@@ -867,7 +912,7 @@ function mountTransferBuilder(main) {
|
||||
|
||||
inputs.fromAccount.focus();
|
||||
refreshNow();
|
||||
return { refresh: refreshNow, form: true };
|
||||
return { refresh: refreshNow };
|
||||
}
|
||||
|
||||
// --- transfers ------------------------------------------------------------
|
||||
@@ -979,43 +1024,44 @@ function mountTransfers(main) {
|
||||
return { refresh };
|
||||
}
|
||||
|
||||
// --- keyboard -------------------------------------------------------------
|
||||
// --- tags -----------------------------------------------------------------
|
||||
|
||||
// Single-key shortcuts, where they do not fight the browser. A builder is a
|
||||
// form, so outside of chords its keys belong to the field being typed in.
|
||||
document.addEventListener('keydown', (e) => {
|
||||
const t = e.target;
|
||||
const typing = t instanceof HTMLInputElement || t instanceof HTMLSelectElement || t instanceof HTMLTextAreaElement;
|
||||
if (current && current.key && current.key(e, typing)) {
|
||||
e.preventDefault();
|
||||
return;
|
||||
// mountTags lists every distinct tag with how many transactions carry it and
|
||||
// the rules that write it, numbered by file position as on the rules screen.
|
||||
function mountTags(main) {
|
||||
const title = h('h2');
|
||||
const body = h('div');
|
||||
main.append(title, body);
|
||||
|
||||
async function refresh() {
|
||||
let tags;
|
||||
try {
|
||||
tags = (await api('GET', '/api/tags')).tags;
|
||||
} catch (e) { setError(e); return; }
|
||||
title.replaceChildren('Tags ', h('span', { class: 'sub' }, `· ${tags.length} tags`));
|
||||
if (!tags.length) {
|
||||
body.replaceChildren(h('div', { class: 'panel empty' }, 'No tags yet.\n\nA tag comes from a rule: build one on the rule builder.'));
|
||||
return;
|
||||
}
|
||||
body.replaceChildren(h('div', { class: 'panel' }, h('table', {},
|
||||
h('thead', {}, h('tr', {}, h('th', {}, 'Tag'), h('th', { class: 'num' }, 'Txns'), h('th', {}, 'Rules'))),
|
||||
h('tbody', {}, tags.map((t) => h('tr', {},
|
||||
h('td', { class: t.transactions ? null : 'muted' }, t.tag),
|
||||
h('td', { class: 'num' + (t.transactions ? '' : ' muted') }, t.transactions),
|
||||
h('td', { class: 'desc' }, t.rules.length
|
||||
? t.rules.map((r) => h('span', { class: 'tag-rule' + (r.usage ? '' : ' muted') },
|
||||
h('span', { class: 'muted' }, `#${r.pos + 1} `),
|
||||
h('span', { class: 'mono' }, r.pattern),
|
||||
r.account ? h('span', { class: 'muted' }, ` on ${r.account}`) : null,
|
||||
` (${r.usage})`))
|
||||
: h('span', { class: 'warn' }, 'no rule writes it — Retag clears it'))))))));
|
||||
}
|
||||
if (typing || e.ctrlKey || e.metaKey || e.altKey) return;
|
||||
refresh();
|
||||
return { refresh };
|
||||
}
|
||||
|
||||
if (e.key >= '1' && e.key <= String(VIEWS.length)) {
|
||||
go(VIEWS[Number(e.key) - 1].id);
|
||||
} else if (e.key === '/') {
|
||||
if (state.view !== 'txns' && state.view !== 'report') go('txns');
|
||||
setTimeout(() => { const s = document.getElementById('search'); if (s) s.focus(); });
|
||||
} else if (e.key === 'u') {
|
||||
state.filter.untagged = !state.filter.untagged;
|
||||
if (state.view === 'txns' || state.view === 'report') show();
|
||||
else go('txns');
|
||||
} else if (e.key === 'a') {
|
||||
state.filter.account = '';
|
||||
if (state.view === 'txns' || state.view === 'report') show();
|
||||
} else if (e.key === 'i') {
|
||||
runImport(false);
|
||||
} else if (e.key === 'r') {
|
||||
retag();
|
||||
} else {
|
||||
return;
|
||||
}
|
||||
e.preventDefault();
|
||||
});
|
||||
|
||||
document.getElementById('import').addEventListener('click', (e) => runImport(e.shiftKey));
|
||||
document.getElementById('import').title += ' (shift-click re-parses unchanged files too)';
|
||||
document.getElementById('import').addEventListener('click', () => runImport(false));
|
||||
document.getElementById('reread').addEventListener('click', () => runImport(true));
|
||||
document.getElementById('retag').addEventListener('click', retag);
|
||||
window.addEventListener('hashchange', show);
|
||||
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
|
||||
<rect width="32" height="32" rx="7" fill="#4b46c8"/>
|
||||
<g fill="#fff">
|
||||
<rect x="7" y="8" width="18" height="4" rx="2"/>
|
||||
<rect x="7" y="14" width="11" height="4" rx="2"/>
|
||||
<rect x="7" y="20" width="15" height="4" rx="2"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 309 B |
@@ -4,6 +4,7 @@
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>money</title>
|
||||
<link rel="icon" href="icon.svg" type="image/svg+xml">
|
||||
<link rel="stylesheet" href="style.css">
|
||||
</head>
|
||||
<body>
|
||||
@@ -12,6 +13,7 @@
|
||||
<nav id="nav" aria-label="Screens"></nav>
|
||||
<div class="actions">
|
||||
<button type="button" id="retag" title="Re-apply rules.toml: retag everything and re-pair transfers">Retag</button>
|
||||
<button type="button" id="reread" title="Re-read every statement, including ones unchanged since they were imported">Re-read all</button>
|
||||
<button type="button" id="import" class="primary" title="Import new statements from every account folder">Import</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
@@ -68,7 +68,6 @@ button, input, select { font: inherit; color: inherit; }
|
||||
}
|
||||
#nav a:hover { color: var(--text); background: var(--row-hover); }
|
||||
#nav a.active { color: var(--accent); background: var(--accent-soft); font-weight: 600; }
|
||||
#nav kbd { font-family: var(--mono); font-size: 11px; opacity: .6; margin-right: 4px; }
|
||||
.actions { display: flex; gap: 8px; }
|
||||
|
||||
button {
|
||||
@@ -145,6 +144,8 @@ tr.total td { font-weight: 600; border-top: 1px solid var(--border); }
|
||||
tr.excluded td { color: var(--muted); }
|
||||
tr.excluded.first td { border-top: 1px dashed var(--border); }
|
||||
td.actions { text-align: right; }
|
||||
.mono { font-family: var(--mono); }
|
||||
.tag-rule { display: inline-block; margin-right: 14px; white-space: nowrap; }
|
||||
tr.legs > td { padding: 0 10px 10px 36px; background: var(--bg); }
|
||||
tr.legs:hover { background: none; }
|
||||
table.sub th { background: var(--bg); font-weight: 500; color: var(--muted); }
|
||||
@@ -179,6 +180,11 @@ td.actions button { padding: 1px 8px; font-size: 12px; }
|
||||
.kept { font-size: 12px; color: var(--muted); font-family: var(--mono); }
|
||||
.preview { max-height: calc(100vh - 170px); }
|
||||
|
||||
h3.section { font-size: 14px; font-weight: 600; margin: 20px 0 8px; }
|
||||
h3.section .sub { color: var(--muted); font-weight: 400; }
|
||||
.files a { color: var(--accent); text-decoration: none; }
|
||||
.files a:hover { text-decoration: underline; }
|
||||
|
||||
.upload { margin-top: 16px; padding: 14px; overflow: visible; }
|
||||
.upload h3 { font-size: 14px; margin: 0 0 10px; }
|
||||
.upload .toolbar { margin-bottom: 10px; }
|
||||
|
||||
+290
-14
@@ -1,7 +1,9 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
@@ -23,10 +25,30 @@ func init() {
|
||||
parser.Register("webtest", func(acc *config.Account) (parser.Parser, error) {
|
||||
return csvParser{digits: acc.Digits()}, nil
|
||||
})
|
||||
parser.Register("webnamed", func(acc *config.Account) (parser.Parser, error) {
|
||||
return namingParser{csvParser{digits: acc.Digits()}}, nil
|
||||
})
|
||||
}
|
||||
|
||||
type csvParser struct{ digits int }
|
||||
|
||||
// namingParser is csvParser for a bank whose downloads are named unhelpfully:
|
||||
// it names a statement after its first date, as nlb does after the statement
|
||||
// date, and says nothing for a statement with no rows.
|
||||
type namingParser struct{ csvParser }
|
||||
|
||||
func (namingParser) StatementName(path string) (string, error) {
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
lines := strings.Split(strings.TrimSpace(string(body)), "\n")
|
||||
if len(lines) < 2 {
|
||||
return "", nil
|
||||
}
|
||||
return "stmt_" + strings.Split(lines[1], ",")[0], nil
|
||||
}
|
||||
|
||||
func (p csvParser) Parse(path string, _ *config.Account) ([]parser.RawTxn, error) {
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
@@ -81,32 +103,38 @@ const checkingTOML = "currency = \"EUR\"\nparser = \"webtest\"\n"
|
||||
|
||||
const statement = "date,description,amount\n2026-02-01,LIDL SOFIA,-12.50\n2026-02-03,SALARY,1000.00\n"
|
||||
|
||||
func TestUploadSavesAndImports(t *testing.T) {
|
||||
// An upload only saves: the file waits on the statements list as new until
|
||||
// the user presses Import.
|
||||
func TestUploadSavesWithoutImporting(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
req := uploadReq{Account: "checking", Files: []uploadFile{{Name: "2026-02.csv", Data: []byte(statement)}}}
|
||||
|
||||
var res importJSON
|
||||
var res status
|
||||
call(t, h, "POST", "/api/upload", req, http.StatusOK, &res)
|
||||
if !strings.Contains(res.Status, "uploaded 1 file(s) to checking") || !strings.Contains(res.Status, "2 new") {
|
||||
if res.Status != "uploaded 1 file(s) to checking · press Import to read them" {
|
||||
t.Errorf("status = %q", res.Status)
|
||||
}
|
||||
if got, err := os.ReadFile(filepath.Join(dir, "2026-02.csv")); err != nil || string(got) != statement {
|
||||
t.Errorf("file on disk = %q, %v", got, err)
|
||||
}
|
||||
var txns struct{ Rows []txnRow }
|
||||
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
|
||||
if len(txns.Rows) != 2 {
|
||||
t.Errorf("index holds %d rows, want the 2 uploaded", len(txns.Rows))
|
||||
if got := descriptions(t, h); got != "" {
|
||||
t.Errorf("upload imported %s; it should only save", got)
|
||||
}
|
||||
var files struct{ Files []fileRow }
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
|
||||
if len(files.Files) != 1 || files.Files[0].Status != "new" {
|
||||
t.Errorf("files = %+v, want the upload waiting as new", files.Files)
|
||||
}
|
||||
|
||||
// The same file again is not an error, and adds nothing.
|
||||
call(t, h, "POST", "/api/upload", req, http.StatusOK, &res)
|
||||
if !strings.Contains(res.Status, "uploaded 0 file(s)") || !strings.Contains(res.Status, "1 already there") {
|
||||
t.Errorf("re-upload status = %q", res.Status)
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
if got := descriptions(t, h); got != "SALARY,LIDL SOFIA" {
|
||||
t.Errorf("after Import the index holds %s", got)
|
||||
}
|
||||
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
|
||||
if len(txns.Rows) != 2 {
|
||||
t.Errorf("re-upload left %d rows, want 2", len(txns.Rows))
|
||||
|
||||
// The same file again is not an error, and saves nothing.
|
||||
call(t, h, "POST", "/api/upload", req, http.StatusOK, &res)
|
||||
if res.Status != "uploaded 0 file(s) to checking (1 already there)" {
|
||||
t.Errorf("re-upload status = %q", res.Status)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -164,3 +192,251 @@ func TestUploadRefusesMultipart(t *testing.T) {
|
||||
t.Errorf("status %d, want %d", w.Code, http.StatusUnsupportedMediaType)
|
||||
}
|
||||
}
|
||||
|
||||
// The statements list is the folders, each beside what the index recorded:
|
||||
// imported, changed since, not imported yet, and gone from disk.
|
||||
func TestFileListStatuses(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
write := func(name, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
write("a.csv", statement)
|
||||
write("b.csv", "date,description,amount\n2026-03-01,ZARA,-40.00\n")
|
||||
write("c.csv", "date,description,amount\n2026-04-01,KAUFLAND,-9.00\n")
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
|
||||
write("b.csv", "date,description,amount\n2026-03-01,ZARA,-41.00\n")
|
||||
if err := os.Remove(filepath.Join(dir, "c.csv")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
write("d.csv", statement)
|
||||
write(".hidden.csv", statement)
|
||||
|
||||
var res struct{ Files []fileRow }
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &res)
|
||||
got := map[string]fileRow{}
|
||||
var names []string
|
||||
for _, f := range res.Files {
|
||||
got[f.Name] = f
|
||||
names = append(names, f.Name)
|
||||
}
|
||||
if strings.Join(names, " ") != "a.csv b.csv c.csv d.csv" {
|
||||
t.Fatalf("files = %v, want the four statements and nothing import would skip", names)
|
||||
}
|
||||
for name, want := range map[string]string{"a.csv": "imported", "b.csv": "changed", "c.csv": "missing", "d.csv": "new"} {
|
||||
if got[name].Status != want {
|
||||
t.Errorf("%s status = %q, want %q", name, got[name].Status, want)
|
||||
}
|
||||
}
|
||||
if got["a.csv"].Rows != 2 || got["a.csv"].Added != 2 || got["c.csv"].Rows != 1 || got["c.csv"].Added != 1 ||
|
||||
got["a.csv"].ImportedAt == "" {
|
||||
t.Errorf("a.csv = %+v, c.csv = %+v", got["a.csv"], got["c.csv"])
|
||||
}
|
||||
if got["a.csv"].Size != int64(len(statement)) {
|
||||
t.Errorf("a.csv size = %d, want %d", got["a.csv"].Size, len(statement))
|
||||
}
|
||||
}
|
||||
|
||||
// Only a file import would read is served, and never as a page.
|
||||
func TestServeFileServesOnlyStatements(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
for name, body := range map[string]string{"a.csv": statement, "page.html": "<script>alert(1)</script>", ".secret": "x"} {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
get := func(path string) *httptest.ResponseRecorder {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest("GET", path, nil))
|
||||
return w
|
||||
}
|
||||
|
||||
w := get("/api/files/checking/a.csv")
|
||||
if w.Code != http.StatusOK || w.Body.String() != statement {
|
||||
t.Fatalf("a.csv: %d %q", w.Code, w.Body.String())
|
||||
}
|
||||
if ct := w.Header().Get("Content-Type"); ct != "text/plain; charset=utf-8" {
|
||||
t.Errorf("a.csv content type = %q", ct)
|
||||
}
|
||||
|
||||
w = get("/api/files/checking/page.html")
|
||||
if w.Code != http.StatusOK || w.Header().Get("Content-Type") != "application/octet-stream" ||
|
||||
w.Header().Get("Content-Security-Policy") != "sandbox" ||
|
||||
!strings.HasPrefix(w.Header().Get("Content-Disposition"), "attachment") {
|
||||
t.Errorf("page.html served as %d %v, want a sandboxed download", w.Code, w.Header())
|
||||
}
|
||||
|
||||
for _, path := range []string{
|
||||
"/api/files/checking/" + config.AccountFile,
|
||||
"/api/files/checking/.secret",
|
||||
"/api/files/checking/..%2F" + config.RulesFile,
|
||||
"/api/files/nope/a.csv",
|
||||
} {
|
||||
if w := get(path); w.Code != http.StatusNotFound {
|
||||
t.Errorf("%s: status %d, want 404", path, w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A parser that names its statements decides the stored name, so the bank's
|
||||
// "download (3).pdf" lands as something that sorts by date. A reissue of the
|
||||
// same date is numbered rather than refused, and a statement that does not
|
||||
// say keeps the name it came with.
|
||||
func TestUploadNamesStatementsTheParserCanName(t *testing.T) {
|
||||
dir, h := newUploadServer(t, "currency = \"EUR\"\nparser = \"webnamed\"\n")
|
||||
upload := func(files ...uploadFile) string {
|
||||
t.Helper()
|
||||
var res status
|
||||
call(t, h, "POST", "/api/upload", uploadReq{Account: "checking", Files: files}, http.StatusOK, &res)
|
||||
return res.Status
|
||||
}
|
||||
reissue := "date,description,amount\n2026-02-01,LIDL SOFIA,-12.50\n2026-02-04,ZARA,-30.00\n"
|
||||
other := "date,description,amount\n2026-03-01,KAUFLAND,-9.00\n"
|
||||
|
||||
status := upload(uploadFile{Name: "download (3).csv", Data: []byte(statement)})
|
||||
if !strings.Contains(status, "download (3).csv → stmt_2026-02-01.csv") {
|
||||
t.Errorf("status = %q, want the rename named", status)
|
||||
}
|
||||
// The same statement downloaded again is recognised under its new name.
|
||||
if status := upload(uploadFile{Name: "download (4).csv", Data: []byte(statement)}); !strings.Contains(status, "1 already there") {
|
||||
t.Errorf("second download: status = %q", status)
|
||||
}
|
||||
// A different statement of the same date, and two of one date in a batch.
|
||||
upload(uploadFile{Name: "x.csv", Data: []byte(reissue)},
|
||||
uploadFile{Name: "y.csv", Data: []byte(reissue + "2026-02-05,BOLT,-4.00\n")},
|
||||
uploadFile{Name: "Z.CSV", Data: []byte(other)}, // a chosen name is lowercase
|
||||
uploadFile{Name: "empty.csv", Data: []byte("date,description,amount\n")})
|
||||
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
want := config.AccountFile + " empty.csv stmt_2026-02-01.csv stmt_2026-02-01_2.csv stmt_2026-02-01_3.csv stmt_2026-03-01.csv"
|
||||
if strings.Join(names, " ") != want {
|
||||
t.Errorf("folder = %v, want %s", names, want)
|
||||
}
|
||||
}
|
||||
|
||||
func writeFile(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func descriptions(t *testing.T, h http.Handler) string {
|
||||
t.Helper()
|
||||
var txns struct{ Rows []txnRow }
|
||||
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
|
||||
var out []string
|
||||
for _, r := range txns.Rows {
|
||||
out = append(out, r.Description)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
// Removing a statement deletes the file and takes out what it brought in. A
|
||||
// row an overlapping statement also holds was stored under the file imported
|
||||
// first; the next Import brings it back from the other one.
|
||||
func TestRemoveFileKeepsWhatAnotherStatementHolds(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
writeFile(t, filepath.Join(dir, "a.csv"), "date,description,amount\n2026-01-30,ONLY IN A,-1.00\n2026-02-01,SHARED,-2.00\n")
|
||||
writeFile(t, filepath.Join(dir, "b.csv"), "date,description,amount\n2026-02-01,SHARED,-2.00\n2026-02-03,ONLY IN B,-3.00\n")
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
|
||||
// Each holds two rows; b.csv was imported second, so its shared row
|
||||
// stayed with a.csv and it brought in only one.
|
||||
var before struct{ Files []fileRow }
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &before)
|
||||
if f := before.Files; len(f) != 2 || f[0].Rows != 2 || f[0].Added != 2 || f[1].Rows != 2 || f[1].Added != 1 {
|
||||
t.Errorf("files = %+v, want a.csv 2 rows · 2 new and b.csv 2 rows · 1 new", f)
|
||||
}
|
||||
|
||||
var res status
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusOK, &res)
|
||||
if !strings.HasPrefix(res.Status, "deleted checking/a.csv, 2 transaction(s) dropped") {
|
||||
t.Errorf("status = %q", res.Status)
|
||||
}
|
||||
// Deleting does not import, so the shared row is gone for now, and b.csv
|
||||
// says it is due to be read again.
|
||||
if got := descriptions(t, h); got != "ONLY IN B" {
|
||||
t.Errorf("index holds %s right after the delete, want only b's own row", got)
|
||||
}
|
||||
var files struct{ Files []fileRow }
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
|
||||
if len(files.Files) != 1 || files.Files[0].Status != "changed" {
|
||||
t.Errorf("files = %+v, want b.csv marked to be re-read", files.Files)
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
if got := descriptions(t, h); got != "ONLY IN B,SHARED" {
|
||||
t.Errorf("after Import the index holds %s, want the shared row restored from b", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "a.csv")); !errors.Is(err, fs.ErrNotExist) {
|
||||
t.Errorf("a.csv is still on disk: %v", err)
|
||||
}
|
||||
if entries, _ := os.ReadDir(dir); len(entries) != 2 {
|
||||
t.Errorf("folder holds %d entries, want account.toml and b.csv only", len(entries))
|
||||
}
|
||||
|
||||
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
|
||||
if len(files.Files) != 1 || files.Files[0].Name != "b.csv" || files.Files[0].Status != "imported" {
|
||||
t.Errorf("files = %+v, want only b.csv, imported", files.Files)
|
||||
}
|
||||
}
|
||||
|
||||
// A removed leg takes its transfer pairing with it, and the other leg is left
|
||||
// unpaired — which is the truth once one side is gone.
|
||||
func TestRemoveFileUnpairsItsTransfers(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
writeFile(t, filepath.Join(filepath.Dir(dir), config.RulesFile), `
|
||||
[[transfer]]
|
||||
from_account = "checking"
|
||||
from_desc = "*OUT*"
|
||||
to_account = "checking"
|
||||
to_desc = "*IN*"
|
||||
`)
|
||||
writeFile(t, filepath.Join(dir, "out.csv"), "date,description,amount\n2026-02-01,MOVE OUT,-5.00\n")
|
||||
writeFile(t, filepath.Join(dir, "in.csv"), "date,description,amount\n2026-02-02,MOVE IN,5.00\n")
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
|
||||
var res status
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "in.csv"}, http.StatusOK, &res)
|
||||
if !strings.Contains(res.Status, "1 transfer leg(s) unpaired") {
|
||||
t.Errorf("status = %q, want the remaining leg reported unpaired", res.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// A file already gone from disk is forgotten by the index; a name the index
|
||||
// never had is not found.
|
||||
func TestRemoveFileForgetsAMissingOne(t *testing.T) {
|
||||
dir, h := newUploadServer(t, checkingTOML)
|
||||
writeFile(t, filepath.Join(dir, "a.csv"), statement)
|
||||
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
|
||||
if err := os.Remove(filepath.Join(dir, "a.csv")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var res status
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusOK, &res)
|
||||
if !strings.HasPrefix(res.Status, "forgot checking/a.csv, 2 transaction(s) dropped") {
|
||||
t.Errorf("status = %q", res.Status)
|
||||
}
|
||||
if got := descriptions(t, h); got != "" {
|
||||
t.Errorf("index still holds %s", got)
|
||||
}
|
||||
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusNotFound, nil)
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "../rules.toml"}, http.StatusBadRequest, nil)
|
||||
if _, err := os.Stat(filepath.Join(dir, config.AccountFile)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: config.AccountFile}, http.StatusNotFound, nil)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user