Name NLB uploads, delete statements, and stop importing on upload

Four changes to statement handling in the web app, made together and
touching the same upload and statements-list code.

Name NLB uploads by statement date. parser.Namer is an optional
interface, like Warner, through which a parser names its statements;
nlb reads the "Datum izpiska" from the izpisek header and names it
izpisek_YYYY_MM_DD, lowercase, extension included -- ported from the
rename_izpiski.py it replaces. Uploads are staged as dotfiles, invisible
to import, so the parser can read them; two downloads of one statement
then meet under one name and the second is recognised as already there,
while a different statement of the same date is numbered _2 as the
script did. Only uploads are named: source_files records statements by
path, so renaming a file already in a folder would orphan its rows.

Delete a statement from the statements list. The file is removed from
disk for good -- the page says so before it asks -- and
store.ForgetSourceFile drops its transactions and their transfer rows.
A row two overlapping statements share is stored once, under the file
imported first, so it goes too; the account's other statements forget
their checksums and show as changed until the next Import re-reads them
and restores it. A file already gone from disk can be forgotten.

Upload and delete no longer import. Importing stays the user's call,
made with the Import button, so a batch can be put together and looked
over first. Delete still re-pairs transfers, which reads no statement.

Show rows and new rows per statement. The list read "0" for a file
whose rows an earlier, overlapping statement already held, which looked
like a file that failed to parse. source_files now records how many
transactions each statement holds, and the list reads "3 rows · 0 new".

This adds a column the code reads, so an index built by an earlier
version fails with "no such column: s.rows": delete index.db and import
again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-02 21:48:29 +02:00
co-authored by Claude Opus 5.5
parent 9ad6c05f9a
commit a5f7541980
12 changed files with 652 additions and 146 deletions
+43 -20
View File
@@ -143,14 +143,14 @@ Anything new that reports a rule must report its file position too.
settles ties: a saved rule cannot displace an equally specific one written by settles ties: a saved rule cannot displace an equally specific one written by
hand, while a narrower one is meant to take precedence and does. hand, while a narrower one is meant to take precedence and does.
**The report's period narrows the report and nothing else.** The screen opens **The report's period narrows the report and nothing else.** The screen opens on
on last month, and `←`/`→` step along `report.Periods` — the named windows, last month, and `←`/`→` step along `report.Periods` — the named windows, then
then the months the index holds. The report and the transaction list share a the months the index holds. The report and the transaction list share a scope
scope (account, search, untagged), which `web.filterFrom` reads and (account, search, untagged), which `web.filterFrom` reads and `state.filter`
`state.filter` holds in the page; the period is read by the report handler holds in the page; the period is read by the report handler alone and kept in
alone and kept in `state.report`, so opening on last month cannot hide the `state.report`, so opening on last month cannot hide the rest of the index from
rest of the index from the list. Nothing may put the period into that shared the list. Nothing may put the period into that shared scope, which is exactly
scope, which is exactly what would make it leak. Covered by `TestReportPeriodLeavesTransactionsAlone`. what would make it leak. Covered by `TestReportPeriodLeavesTransactionsAlone`.
The axis is built from `store.Months` over the *whole* index, not from the rows The axis is built from `store.Months` over the *whole* index, not from the rows
in view, or it would grow and shrink as the account or search filter changed and in view, or it would grow and shrink as the account or search filter changed and
@@ -241,13 +241,14 @@ That is why `/api/upload` takes files base64 inside JSON rather than as
multipart: multipart is precisely what a cross-site form can send. multipart: multipart is precisely what a cross-site form can send.
**An upload only puts a file where `money import` looks.** It writes into an **An upload only puts a file where `money import` looks.** It writes into an
existing account folder and then runs the same import as `/api/import`, so the existing account folder and stops there: importing stays the user's call, made
statements stay the source of truth and nothing reaches the index any other with the Import button, so the statements stay the source of truth and nothing
way. It never overwrites a statement (identical contents are a no-op, different reaches the index any other way. It never overwrites a statement (identical
ones a 409), refuses any name import would not read back — not a plain file contents are a no-op, different ones a 409), refuses any name import would not
name, a dotfile, `account.toml`, outside the account's `include` — and checks a read back — not a plain file name, a dotfile, `account.toml`, outside the
whole batch before writing any of it. Covered by account's `include` — and checks a whole batch before writing any of it. Covered
`TestUploadNeverReplacesAStatement` and `TestUploadRefusesBadNames`. by `TestUploadSavesWithoutImporting`, `TestUploadNeverReplacesAStatement` and
`TestUploadRefusesBadNames`.
**The statements list is read from the folders, not the index.** `/api/files` **The statements list is read from the folders, not the index.** `/api/files`
walks `importer.StatementFiles` — the same list import reads — and only then walks `importer.StatementFiles` — the same list import reads — and only then
@@ -255,11 +256,23 @@ looks each file up in `store.SourceFiles`, comparing `importer.Checksum`, so a
file is listed exactly when import would read it and a file the index remembers file is listed exactly when import would read it and a file the index remembers
but the disk lost shows as `missing` instead of vanishing. but the disk lost shows as `missing` instead of vanishing.
`/api/files/{account}/{name}` serves a file only by finding it in that list, `/api/files/{account}/{name}` serves a file only by finding it in that list,
never by joining the name onto a path. A statement is served from the app's origin, where a script could drive never by joining the name onto a path. A statement is served from the app's
the API, so nothing is ever rendered as a page: text is `text/plain` under origin, where a script could drive the API, so nothing is ever rendered as a
`CSP: sandbox`, anything not text or PDF is a sandboxed download, and PDFs — page: text is `text/plain` under `CSP: sandbox`, anything not text or PDF is a
whose viewers refuse a sandbox — open in the browser's own isolated viewer. sandboxed download, and PDFs — whose viewers refuse a sandbox — open in the
Covered by `TestServeFileServesOnlyStatements`. browser's own isolated viewer. Covered by `TestServeFileServesOnlyStatements`.
**Deleting a statement marks its neighbours to be re-read.** `/api/files/delete`
finds the file the same way `serveFile` does and deletes it — permanently, by
the user's choice; the page says so before it asks. `store.ForgetSourceFile`
then drops its transactions and their transfer rows, and `Link` re-pairs, which
reads no statement. It does not import, any more than an upload does. A
transaction two overlapping statements share is stored once, under the file
imported first, so it goes too; `ForgetSourceFile` therefore blanks the
checksums of the account's other statements, which show as `changed` until the
next Import re-reads them and restores what they hold. Without that the checksum
skip would leave those rows gone for good. Covered by
`TestRemoveFileKeepsWhatAnotherStatementHolds`.
## Adding a bank parser ## Adding a bank parser
@@ -300,6 +313,16 @@ description — at the end, and not in the position it held on the page, so an
IBAN wrapped across continuation lines stays contiguous for a glob to match. IBAN wrapped across continuation lines stays contiguous for a glob to match.
A new parser must do the same rather than reintroduce a structured field. A new parser must do the same rather than reintroduce a structured field.
**A parser may name its statements, and only uploads use it.** `parser.Namer`
is optional, like `parser.Warner`: `nlb` names an izpisek `izpisek_YYYY_MM_DD`
after its *Datum izpiska*, ported from the `rename_izpiski.py` it replaced.
`/api/upload` stages each file as a dotfile — invisible to import — so the
parser can read it, then numbers a chosen name that is taken by different
contents (`_2`, as the script did) where a name the user gave is refused with
409. Nothing renames a file already in a folder: `source_files` records
statements by path, so a rename would orphan its rows as `missing` until the
index is rebuilt. Covered by `TestUploadNamesStatementsTheParserCanName`.
## Verifying ## Verifying
``` ```
+33 -10
View File
@@ -102,8 +102,7 @@ money config # which data root is in use, and why
An account folder appears in the app only once its statements have been An account folder appears in the app only once its statements have been
imported — creating an `account.toml` is not enough on its own. Run imported — creating an `account.toml` is not enough on its own. Run
`money import` (or press Import in the web app) after adding one, or add its `money import` (or press Import in the web app) after adding one.
first statements from the Accounts screen, which imports them as it saves them.
`--uniq` turns `ls` into a list of patterns still to write rather than a list of `--uniq` turns `ls` into a list of patterns still to write rather than a list of
rows to read: one line per distinct description, since fifty visits to the same rows to read: one line per distinct description, since fifty visits to the same
@@ -178,21 +177,35 @@ folder, with what the index made of it:
| `•` not imported | new since the last import — or an import tried and failed, in which case the error is in the import report | | `•` not imported | new since the last import — or an import tried and failed, in which case the error is in the import report |
| `✗` gone from disk | imported once, then removed; its transactions stay in the index only until it is rebuilt | | `✗` gone from disk | imported once, then removed; its transactions stay in the index only until it is rebuilt |
**Added** is how many transactions the file brought in. Statements that overlap **Transactions** reads like `3 rows · 0 new`: how many transactions the file
share rows, and a shared row counts towards whichever file was imported first, holds, then how many of them it was the first to bring in. Statements that
so a later statement covering the same days can add fewer rows than it holds. overlap share rows, and a shared row is stored once, under whichever file was
imported first — so a later statement covering the same days shows fewer new
rows than it holds, down to `0 new` when an earlier one already had them all.
That is deduplication working, not a file that failed to parse.
Click a file name to open it. PDFs and CSVs open in the browser; anything else Click a file name to open it. PDFs and CSVs open in the browser; anything else
downloads. Only files import would read are listed or served — not downloads. Only files import would read are listed or served — not
`account.toml`, dotfiles, or anything outside an account's `include` patterns. `account.toml`, dotfiles, or anything outside an account's `include` patterns.
**Delete** removes a statement from disk for good — there is no copy kept
and no undo, so keep the bank's original if you might want it back. Its
transactions leave the index, and transfers that used them lose their pairing:
the other leg shows as unpaired, which is the truth once one side is gone.
Deleting does not import. A row that an overlapping statement also contains goes
too, for now: the account's other statements are marked *changed*, and the next
**Import** re-reads them and puts it back. For a file already gone from disk the
button says **Forget** and only clears the index.
### Adding statements from the browser ### Adding statements from the browser
The Accounts screen (`1`) has an **Add statements** panel: pick the account, The Accounts screen (`1`) has an **Add statements** panel: pick the account,
drop files on it (or click to choose them) and press **Upload and import**. The drop files on it (or click to choose them) and press **Upload**. The files are
files are saved into that account's folder, exactly where you would have copied saved into that account's folder, exactly where you would have copied them by
them by hand, and then imported — the folder stays the source of truth, so hand — the folder stays the source of truth, so deleting `index.db` and
deleting `index.db` and re-importing still gets everything back. re-importing still gets everything back. Uploading does not import: the files
wait on the statements list as *not imported* until you press **Import**, so you
can put a batch together and look it over first.
An upload never replaces a statement. A file whose name is already in the folder An upload never replaces a statement. A file whose name is already in the folder
is skipped if its contents are identical and refused if they differ; rename it is skipped if its contents are identical and refused if they differ; rename it
@@ -202,6 +215,16 @@ too, and a batch with one bad file writes none of them. The folder itself must
already exist with an `account.toml`: an upload adds statements to an account, already exist with an `account.toml`: an upload adds statements to an account,
it does not create one. it does not create one.
Some banks name their downloads unhelpfully, so a parser may name the statement
instead. An NLB izpisek is saved as `izpisek_YYYY_MM_DD.pdf` after the *Datum
izpiska* in its header — lowercase throughout, extension included — so the
folder sorts by date and the status line says what each file became. Downloading
the same statement twice then lands on the same name and is recognised as
already there; a different statement with the same date (a reissue) is numbered
`_2`, `_3` rather than refused. A statement with no date it can find keeps its
own name. Only uploads are named — files already in a folder are never renamed,
since the index records them by path.
### Keys ### Keys
| Key | Action | | Key | Action |
@@ -654,7 +677,7 @@ nothing else to configure — `parser` names one of these and that is all.
| `parser` | Statement | Notes | | `parser` | Statement | Notes |
| --- | --- | --- | | --- | --- | --- |
| `nlb` | NLB izpisek PDF | Wrapped descriptions are folded in from continuation lines, and the IBAN column is appended to the description. | | `nlb` | NLB izpisek PDF | Wrapped descriptions are folded in from continuation lines, and the IBAN column is appended to the description. Uploads are saved as `izpisek_YYYY_MM_DD.pdf` after the statement date. |
| `traderepublic` | Trade Republic PDF | Handles both the single-line and the stacked layout by measuring column positions. | | `traderepublic` | Trade Republic PDF | Handles both the single-line and the stacked layout by measuring column positions. |
| `revolut` | `account-statement*.csv` | Skips non-COMPLETED rows, folds the fee into the amount. | | `revolut` | `account-statement*.csv` | Skips non-COMPLETED rows, folds the fee into the amount. |
+1 -1
View File
@@ -162,7 +162,7 @@ func importFile(root string, db *store.DB, acc *config.Account, accountID int64,
} }
fr.Warnings = append(fr.Warnings, checkBalances(txns, acc.Digits())...) fr.Warnings = append(fr.Warnings, checkBalances(txns, acc.Digits())...)
sourceID, err := db.SourceFile(accountID, rel, sum, time.Now().UTC().Format(time.RFC3339)) sourceID, err := db.SourceFile(accountID, rel, sum, time.Now().UTC().Format(time.RFC3339), len(txns))
if err != nil { if err != nil {
fr.Err = err fr.Err = err
return fr return fr
+22
View File
@@ -43,6 +43,10 @@ var (
nlbAccount = regexp.MustCompile(`^SI\d{2}(?:\s?\d{4}){3}\s?\d{3}$`) nlbAccount = regexp.MustCompile(`^SI\d{2}(?:\s?\d{4}){3}\s?\d{3}$`)
) )
// nlbStatementDate is the issue date in an izpisek's header. It names the
// statement: NLB's downloads are not named for what they hold.
var nlbStatementDate = regexp.MustCompile(`Datum izpiska\s+(\d{2})\.(\d{2})\.(\d{4})`)
// nlbMinContinuationIndent is the shallowest indent a wrapped description line // nlbMinContinuationIndent is the shallowest indent a wrapped description line
// may have. The real threshold is the description column of the transaction // may have. The real threshold is the description column of the transaction
// the line belongs to, measured per line rather than hardcoded: pdftotext // the line belongs to, measured per line rather than hardcoded: pdftotext
@@ -61,6 +65,24 @@ func (p *nlbParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
return parseNLBText(text, p.digits) return parseNLBText(text, p.digits)
} }
// StatementName names an izpisek after its statement date, izpisek_YYYY_MM_DD,
// so the folder sorts by date and two downloads of one statement collide.
func (p *nlbParser) StatementName(path string) (string, error) {
text, err := pdfToText(path)
if err != nil {
return "", err
}
return nlbStatementName(text), nil
}
func nlbStatementName(text string) string {
m := nlbStatementDate.FindStringSubmatch(text)
if m == nil {
return ""
}
return fmt.Sprintf("izpisek_%s_%s_%s", m[3], m[2], m[1])
}
// parseNLBText holds the whole parser, separated from PDF extraction so it can // parseNLBText holds the whole parser, separated from PDF extraction so it can
// be tested against captured pdftotext output. // be tested against captured pdftotext output.
func parseNLBText(text string, digits int) ([]RawTxn, error) { func parseNLBText(text string, digits int) ([]RawTxn, error) {
+12
View File
@@ -112,3 +112,15 @@ func TestParseNLBRejectsMalformedLine(t *testing.T) {
t.Error("expected an error for a transaction line with no amount") t.Error("expected an error for a transaction line with no amount")
} }
} }
// An izpisek is named after its statement date, as rename_izpiski.py did.
func TestNLBStatementName(t *testing.T) {
header := " IZPISEK 001/2026\n" +
" Datum izpiska 31.01.2026 Stran 1\n"
if got := nlbStatementName(header + nlbPage); got != "izpisek_2026_01_31" {
t.Errorf("name = %q, want izpisek_2026_01_31", got)
}
if got := nlbStatementName(nlbPage); got != "" {
t.Errorf("name without a statement date = %q, want none", got)
}
}
+11
View File
@@ -43,6 +43,17 @@ type Warner interface {
Warnings() []string Warnings() []string
} }
// Namer is an optional interface for parsers whose bank names its downloads
// unhelpfully. StatementName reads the statement at path and returns the name,
// without extension, it should be stored under — the upload's extension is
// kept, lowercased — or "" when the statement does not say, in which case it
// keeps the name it came with. Only an upload uses
// it; files already in a folder are never renamed, since the index records
// them by path.
type Namer interface {
StatementName(path string) (string, error)
}
// Factory builds a parser from an account's config, validating it up front so // Factory builds a parser from an account's config, validating it up front so
// a bad account.toml fails before any file is read. // a bad account.toml fails before any file is read.
type Factory func(acc *config.Account) (Parser, error) type Factory func(acc *config.Account) (Parser, error)
+1 -1
View File
@@ -28,7 +28,7 @@ func seed(t *testing.T, db *store.DB, descriptions ...string) int64 {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
sourceID, err := db.SourceFile(accountID, "checking/st.csv", "sha", "2026-01-01T00:00:00Z") sourceID, err := db.SourceFile(accountID, "checking/st.csv", "sha", "2026-01-01T00:00:00Z", 0)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+64 -8
View File
@@ -37,6 +37,9 @@ CREATE TABLE IF NOT EXISTS source_files (
path TEXT NOT NULL, path TEXT NOT NULL,
sha256 TEXT NOT NULL, sha256 TEXT NOT NULL,
imported_at TEXT NOT NULL, imported_at TEXT NOT NULL,
-- How many transactions the statement holds, as parsed. Not how many it
-- brought in: rows an overlapping statement already had are deduplicated.
rows INTEGER NOT NULL DEFAULT 0,
UNIQUE(account_id, path) UNIQUE(account_id, path)
); );
@@ -130,15 +133,17 @@ func (d *DB) Accounts() ([]model.Account, error) {
return out, rows.Err() return out, rows.Err()
} }
// SourceFile records that a statement file was imported, returning its id. // SourceFile records that a statement file was imported, holding rows
func (d *DB) SourceFile(accountID int64, path, sha, importedAt string) (int64, error) { // transactions, and returns its id.
func (d *DB) SourceFile(accountID int64, path, sha, importedAt string, rows int) (int64, error) {
_, err := d.sql.Exec(` _, err := d.sql.Exec(`
INSERT INTO source_files (account_id, path, sha256, imported_at) INSERT INTO source_files (account_id, path, sha256, imported_at, rows)
VALUES (?, ?, ?, ?) VALUES (?, ?, ?, ?, ?)
ON CONFLICT(account_id, path) DO UPDATE SET ON CONFLICT(account_id, path) DO UPDATE SET
sha256 = excluded.sha256, sha256 = excluded.sha256,
imported_at = excluded.imported_at`, imported_at = excluded.imported_at,
accountID, path, sha, importedAt) rows = excluded.rows`,
accountID, path, sha, importedAt, rows)
if err != nil { if err != nil {
return 0, fmt.Errorf("record source file %s: %w", path, err) return 0, fmt.Errorf("record source file %s: %w", path, err)
} }
@@ -399,6 +404,8 @@ type SourceFileInfo struct {
Path string // relative to the data root, as the importer records it Path string // relative to the data root, as the importer records it
SHA256 string SHA256 string
ImportedAt string ImportedAt string
// Rows is how many transactions the statement holds, as parsed.
Rows int
// Added counts the transactions this file introduced. A row seen again in // Added counts the transactions this file introduced. A row seen again in
// an overlapping statement is deduplicated and stays with the file that // an overlapping statement is deduplicated and stays with the file that
// brought it first, so this is not how many rows the file holds. // brought it first, so this is not how many rows the file holds.
@@ -408,7 +415,7 @@ type SourceFileInfo struct {
// SourceFiles lists every statement the index has imported. // SourceFiles lists every statement the index has imported.
func (d *DB) SourceFiles() ([]SourceFileInfo, error) { func (d *DB) SourceFiles() ([]SourceFileInfo, error) {
rows, err := d.sql.Query(` rows, err := d.sql.Query(`
SELECT a.slug, s.path, s.sha256, s.imported_at, COUNT(t.id) SELECT a.slug, s.path, s.sha256, s.imported_at, s.rows, COUNT(t.id)
FROM source_files s FROM source_files s
JOIN accounts a ON a.id = s.account_id JOIN accounts a ON a.id = s.account_id
LEFT JOIN transactions t ON t.source_file_id = s.id LEFT JOIN transactions t ON t.source_file_id = s.id
@@ -421,7 +428,7 @@ func (d *DB) SourceFiles() ([]SourceFileInfo, error) {
var out []SourceFileInfo var out []SourceFileInfo
for rows.Next() { for rows.Next() {
var f SourceFileInfo var f SourceFileInfo
if err := rows.Scan(&f.AccountSlug, &f.Path, &f.SHA256, &f.ImportedAt, &f.Added); err != nil { if err := rows.Scan(&f.AccountSlug, &f.Path, &f.SHA256, &f.ImportedAt, &f.Rows, &f.Added); err != nil {
return nil, err return nil, err
} }
out = append(out, f) out = append(out, f)
@@ -429,6 +436,55 @@ func (d *DB) SourceFiles() ([]SourceFileInfo, error) {
return out, rows.Err() return out, rows.Err()
} }
// ForgetSourceFile removes a statement from the index: its transactions, any
// transfer pairing that used them, and the record of it, returning how many
// transactions went. Pairing is derived state that the next Link rewrites, so
// dropping a pair here loses nothing.
//
// A transaction two overlapping statements share is stored once, under the
// file that brought it first, so removing that file can take rows another
// statement still holds. The account's other statements therefore forget
// their checksums, and the next import re-reads them and restores those rows.
func (d *DB) ForgetSourceFile(accountSlug, path string) (int, error) {
tx, err := d.sql.Begin()
if err != nil {
return 0, err
}
defer tx.Rollback()
var id, accountID int64
err = tx.QueryRow(`
SELECT s.id, s.account_id FROM source_files s JOIN accounts a ON a.id = s.account_id
WHERE a.slug = ? AND s.path = ?`, accountSlug, path).Scan(&id, &accountID)
if err == sql.ErrNoRows {
return 0, nil
}
if err != nil {
return 0, fmt.Errorf("find source file %s: %w", path, err)
}
if _, err := tx.Exec(`
DELETE FROM transfers WHERE
out_txn_id IN (SELECT id FROM transactions WHERE source_file_id = ?) OR
in_txn_id IN (SELECT id FROM transactions WHERE source_file_id = ?)`, id, id); err != nil {
return 0, fmt.Errorf("forget transfers from %s: %w", path, err)
}
res, err := tx.Exec(`DELETE FROM transactions WHERE source_file_id = ?`, id)
if err != nil {
return 0, fmt.Errorf("forget transactions from %s: %w", path, err)
}
n, err := res.RowsAffected()
if err != nil {
return 0, err
}
if _, err := tx.Exec(`DELETE FROM source_files WHERE id = ?`, id); err != nil {
return 0, fmt.Errorf("forget %s: %w", path, err)
}
if _, err := tx.Exec(`UPDATE source_files SET sha256 = '' WHERE account_id = ?`, accountID); err != nil {
return 0, fmt.Errorf("mark %s's statements for re-reading: %w", accountSlug, err)
}
return int(n), tx.Commit()
}
// Tags lists every tag in use, for completion in the rule builder. // Tags lists every tag in use, for completion in the rule builder.
func (d *DB) Tags() ([]string, error) { func (d *DB) Tags() ([]string, error) {
rows, err := d.sql.Query(` rows, err := d.sql.Query(`
+227 -68
View File
@@ -27,6 +27,7 @@ import (
"git.petrovv.com/nikola/money/internal/glob" "git.petrovv.com/nikola/money/internal/glob"
"git.petrovv.com/nikola/money/internal/importer" "git.petrovv.com/nikola/money/internal/importer"
"git.petrovv.com/nikola/money/internal/model" "git.petrovv.com/nikola/money/internal/model"
"git.petrovv.com/nikola/money/internal/parser"
"git.petrovv.com/nikola/money/internal/report" "git.petrovv.com/nikola/money/internal/report"
"git.petrovv.com/nikola/money/internal/rules" "git.petrovv.com/nikola/money/internal/rules"
"git.petrovv.com/nikola/money/internal/store" "git.petrovv.com/nikola/money/internal/store"
@@ -91,6 +92,7 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /api/upload", s.write(s.upload)) mux.HandleFunc("POST /api/upload", s.write(s.upload))
mux.HandleFunc("GET /api/files", s.read(s.fileList)) mux.HandleFunc("GET /api/files", s.read(s.fileList))
mux.HandleFunc("GET /api/files/{account}/{name}", s.serveFile) mux.HandleFunc("GET /api/files/{account}/{name}", s.serveFile)
mux.HandleFunc("POST /api/files/delete", s.write(s.removeFile))
mux.HandleFunc("POST /api/retag", s.write(s.retag)) mux.HandleFunc("POST /api/retag", s.write(s.retag))
return mux return mux
} }
@@ -1211,26 +1213,22 @@ func (s *Server) runImport(r *http.Request) (any, error) {
if err := decode(r, &req); err != nil { if err := decode(r, &req); err != nil {
return nil, err return nil, err
} }
return s.importAll(req.Force)
}
func (s *Server) importAll(force bool) (importJSON, error) {
accounts, err := config.LoadAccounts(s.root) accounts, err := config.LoadAccounts(s.root)
if err != nil { if err != nil {
return importJSON{}, err return nil, err
} }
if len(accounts) == 0 { if len(accounts) == 0 {
return importJSON{}, badRequest("no accounts found in %s (an account is a folder containing %s)", return nil, badRequest("no accounts found in %s (an account is a folder containing %s)",
s.root, config.AccountFile) s.root, config.AccountFile)
} }
if err := s.reloadRules(); err != nil { if err := s.reloadRules(); err != nil {
return importJSON{}, err return nil, err
} }
s.accounts = accounts s.accounts = accounts
res, err := importer.Run(s.root, s.db, s.accounts, s.engine, s.links, importer.Options{Force: force}) res, err := importer.Run(s.root, s.db, s.accounts, s.engine, s.links, importer.Options{Force: req.Force})
if err != nil { if err != nil {
return importJSON{}, err return nil, err
} }
_, added, skipped := res.Total() _, added, skipped := res.Total()
@@ -1269,10 +1267,11 @@ type uploadReq struct {
Files []uploadFile `json:"files"` Files []uploadFile `json:"files"`
} }
// upload saves statements into an account folder and imports them. The files // upload saves statements into an account folder. The files on disk are the
// on disk are the source of truth and the index is derived from them, so an // source of truth and the index is derived from them, so an upload is nothing
// upload is nothing more than putting a file where `money import` looks; the // more than putting a file where `money import` looks. It does not import:
// import that follows is the one the Import button runs. // that stays the user's call, made with the Import button, so a batch can be
// put together and checked on the statements list before it is read.
// //
// Files arrive base64 inside JSON rather than as multipart: a multipart body // Files arrive base64 inside JSON rather than as multipart: a multipart body
// is exactly what a cross-site form can send, and the JSON-only rule is the // is exactly what a cross-site form can send, and the JSON-only rule is the
@@ -1300,60 +1299,133 @@ func (s *Server) upload(r *http.Request) (any, error) {
req.Account, s.root, config.AccountFile) req.Account, s.root, config.AccountFile)
} }
// Everything is checked before anything is written, so a bad file in a p, err := parser.For(acc)
// batch leaves the folder as it was rather than half uploaded. if err != nil {
var write []uploadFile return nil, badRequest("%v", err)
var same []string
seen := map[string]bool{}
for _, f := range req.Files {
if err := checkStatementName(acc, f.Name); err != nil {
return nil, err
}
if seen[f.Name] {
return nil, badRequest("%s is in the upload twice", f.Name)
}
seen[f.Name] = true
existing, err := os.ReadFile(filepath.Join(acc.Dir, f.Name))
switch {
case err == nil && bytes.Equal(existing, f.Data):
same = append(same, f.Name)
case err == nil:
// A statement is the source of truth for what it already
// imported; replacing it under the same name is not an upload's
// decision to make.
return nil, &apiError{http.StatusConflict, fmt.Sprintf(
"%s/%s already exists with different contents; rename the file or remove the old one first",
acc.Slug, f.Name)}
case !errors.Is(err, fs.ErrNotExist):
return nil, err
default:
write = append(write, f)
}
} }
for _, f := range write { namer, _ := p.(parser.Namer)
if err := writeStatement(acc.Dir, f); err != nil {
// Each file is staged as a dotfile — invisible to import — so a parser
// that names its statements can read it. Everything is then checked
// before anything is moved into place, so a bad file in a batch leaves
// the folder as it was rather than half uploaded.
type staged struct {
orig, name, tmp string
data []byte
named bool // the parser chose name, from the statement
}
var batch []*staged
defer func() {
for _, f := range batch {
if f.tmp != "" {
os.Remove(f.tmp)
}
}
}()
for _, f := range req.Files {
if err := checkPlainName(f.Name); err != nil {
return nil, err return nil, err
} }
tmp, err := stageStatement(acc.Dir, f.Data)
if err != nil {
return nil, err
}
st := &staged{orig: f.Name, name: f.Name, tmp: tmp, data: f.Data}
batch = append(batch, st)
if namer != nil {
stem, err := namer.StatementName(tmp)
if err != nil {
return nil, badRequest("%s: %v", f.Name, err)
}
if stem != "" {
st.name, st.named = stem+strings.ToLower(filepath.Ext(f.Name)), true
}
}
} }
out, err := s.importAll(false) var write []*staged
if err != nil { var same, renamed []string
return nil, err claimed := map[string][]byte{}
for _, f := range batch {
base, n := f.name, 2
resolve:
for {
if err := checkStatementName(acc, f.name); err != nil {
return nil, err
}
existing, inBatch := claimed[f.name]
var err error
if !inBatch {
existing, err = os.ReadFile(filepath.Join(acc.Dir, f.name))
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return nil, err
}
}
switch {
case !inBatch && err != nil: // free
claimed[f.name] = f.data
write = append(write, f)
break resolve
case bytes.Equal(existing, f.data):
same = append(same, f.orig)
break resolve
case f.named:
// A name the parser chose can be taken by a different
// statement of the same date — a reissue — so it is
// numbered, as rename_izpiski.py did, rather than refused.
f.name = fmt.Sprintf("%s_%d%s", strings.TrimSuffix(base, filepath.Ext(base)), n, filepath.Ext(base))
n++
case inBatch:
return nil, badRequest("%s is in the upload twice", f.orig)
default:
// A statement is the source of truth for what it already
// imported; replacing it under the same name is not an
// upload's decision to make.
return nil, &apiError{http.StatusConflict, fmt.Sprintf(
"%s/%s already exists with different contents; rename the file or remove the old one first",
acc.Slug, f.name)}
}
}
} }
for _, f := range write {
path := filepath.Join(acc.Dir, f.name)
if err := os.Rename(f.tmp, path); err != nil {
return nil, fmt.Errorf("write %s: %w", path, err)
}
f.tmp = ""
if f.name != f.orig {
renamed = append(renamed, f.orig+" → "+f.name)
}
}
msg := fmt.Sprintf("uploaded %d file(s) to %s", len(write), acc.Slug) msg := fmt.Sprintf("uploaded %d file(s) to %s", len(write), acc.Slug)
if len(same) > 0 { if len(same) > 0 {
msg += fmt.Sprintf(" (%d already there)", len(same)) msg += fmt.Sprintf(" (%d already there)", len(same))
} }
out.Status = msg + " · " + out.Status if len(renamed) > 0 {
return out, nil msg += ", named by statement date: " + strings.Join(renamed, ", ")
}
if len(write) > 0 {
msg += " · press Import to read them"
}
return status{msg}, nil
}
// checkPlainName refuses anything that is not a plain file name, before the
// file is so much as staged.
func checkPlainName(name string) error {
if name == "" || name != filepath.Base(name) || strings.ContainsAny(name, "/\\\x00") ||
name == "." || name == ".." {
return badRequest("%q is not a plain file name", name)
}
return nil
} }
// checkStatementName refuses any name the importer would not read back as a // checkStatementName refuses any name the importer would not read back as a
// statement of this account, and anything that is not a plain file name. // statement of this account, and anything that is not a plain file name.
func checkStatementName(acc *config.Account, name string) error { func checkStatementName(acc *config.Account, name string) error {
if name == "" || name != filepath.Base(name) || strings.ContainsAny(name, "/\\\x00") || if err := checkPlainName(name); err != nil {
name == "." || name == ".." { return err
return badRequest("%q is not a plain file name", name)
} }
if strings.HasPrefix(name, ".") || name == config.AccountFile { if strings.HasPrefix(name, ".") || name == config.AccountFile {
return badRequest("%s would be ignored by import (dotfiles and %s are not statements)", return badRequest("%s would be ignored by import (dotfiles and %s are not statements)",
@@ -1366,29 +1438,28 @@ func checkStatementName(acc *config.Account, name string) error {
return nil return nil
} }
// writeStatement writes through a dotfile and renames it into place, so a // stageStatement writes an upload to a dotfile in the account folder. Import
// skips dotfiles, so it is invisible until renamed into place — and a
// concurrent `money import` never reads half a statement. // concurrent `money import` never reads half a statement.
func writeStatement(dir string, f uploadFile) error { func stageStatement(dir string, data []byte) (string, error) {
path := filepath.Join(dir, f.Name)
tmp, err := os.CreateTemp(dir, ".upload-*") tmp, err := os.CreateTemp(dir, ".upload-*")
if err != nil { if err != nil {
return fmt.Errorf("write %s: %w", path, err) return "", fmt.Errorf("write to %s: %w", dir, err)
} }
defer os.Remove(tmp.Name()) // a no-op once renamed if _, err := tmp.Write(data); err != nil {
if _, err := tmp.Write(f.Data); err != nil {
tmp.Close() tmp.Close()
return fmt.Errorf("write %s: %w", path, err) os.Remove(tmp.Name())
return "", fmt.Errorf("write %s: %w", tmp.Name(), err)
} }
if err := tmp.Close(); err != nil { if err := tmp.Close(); err != nil {
return fmt.Errorf("write %s: %w", path, err) os.Remove(tmp.Name())
return "", fmt.Errorf("write %s: %w", tmp.Name(), err)
} }
if err := os.Chmod(tmp.Name(), 0o644); err != nil { if err := os.Chmod(tmp.Name(), 0o644); err != nil {
return err os.Remove(tmp.Name())
return "", err
} }
if err := os.Rename(tmp.Name(), path); err != nil { return tmp.Name(), nil
return fmt.Errorf("write %s: %w", path, err)
}
return nil
} }
type fileRow struct { type fileRow struct {
@@ -1404,7 +1475,11 @@ type fileRow struct {
// is rebuilt). // is rebuilt).
Status string `json:"status"` Status string `json:"status"`
ImportedAt string `json:"importedAt"` ImportedAt string `json:"importedAt"`
Added int `json:"added"` // Rows is how many transactions the statement holds; Added how many of
// them it was the first to bring in, which overlap with another statement
// can make fewer.
Rows int `json:"rows"`
Added int `json:"added"`
} }
// fileList is every statement on disk next to what the index recorded about // fileList is every statement on disk next to what the index recorded about
@@ -1443,7 +1518,7 @@ func (s *Server) fileList(*http.Request) (any, error) {
row.Size, row.Modified = info.Size(), info.ModTime().Format("2006-01-02 15:04") row.Size, row.Modified = info.Size(), info.ModTime().Format("2006-01-02 15:04")
} }
if rec, ok := byPath[rel]; ok { if rec, ok := byPath[rel]; ok {
row.ImportedAt, row.Added, row.Status = rec.ImportedAt, rec.Added, "imported" row.ImportedAt, row.Rows, row.Added, row.Status = rec.ImportedAt, rec.Rows, rec.Added, "imported"
if sum, err := importer.Checksum(path); err != nil { if sum, err := importer.Checksum(path); err != nil {
return nil, err return nil, err
} else if sum != rec.SHA256 { } else if sum != rec.SHA256 {
@@ -1457,7 +1532,7 @@ func (s *Server) fileList(*http.Request) (any, error) {
if !onDisk[rec.Path] { if !onDisk[rec.Path] {
out = append(out, fileRow{ out = append(out, fileRow{
Account: rec.AccountSlug, Name: filepath.Base(rec.Path), Status: "missing", Account: rec.AccountSlug, Name: filepath.Base(rec.Path), Status: "missing",
ImportedAt: rec.ImportedAt, Added: rec.Added, ImportedAt: rec.ImportedAt, Rows: rec.Rows, Added: rec.Added,
}) })
} }
} }
@@ -1532,6 +1607,90 @@ func (s *Server) serveFile(w http.ResponseWriter, r *http.Request) {
http.ServeFile(w, r, path) http.ServeFile(w, r, path)
} }
type removeFileReq struct {
Account string `json:"account"`
Name string `json:"name"`
}
// removeFile takes a statement out of the data: the file is deleted and its
// transactions leave the index. A file already gone from disk is only
// forgotten by the index. The deletion is permanent — the page says so before
// asking — so nothing here keeps a copy.
//
// It does not import. Rows the file shared with an overlapping statement went
// with it, and ForgetSourceFile marks the account's other statements so the
// next Import re-reads them and restores those rows; until then they show as
// changed. Pairing is re-derived here, though: it reads no statement, and a
// leg whose partner just went should not stay counted as a transfer.
func (s *Server) removeFile(r *http.Request) (any, error) {
var req removeFileReq
if err := decode(r, &req); err != nil {
return nil, err
}
if err := checkPlainName(req.Name); err != nil {
return nil, err
}
rel := filepath.Join(req.Account, req.Name)
// The file is found in the account's statement list, never by joining the
// request onto a path, exactly as serveFile does.
var path string
accounts, err := config.LoadAccounts(s.root)
if err != nil {
return nil, err
}
for _, acc := range accounts {
if acc.Slug != req.Account {
continue
}
paths, err := importer.StatementFiles(acc)
if err != nil {
return nil, err
}
for _, p := range paths {
if filepath.Base(p) == req.Name {
path = p
}
}
}
if path != "" {
if err := os.Remove(path); err != nil {
return nil, fmt.Errorf("remove %s: %w", path, err)
}
}
dropped, err := s.db.ForgetSourceFile(req.Account, rel)
if err != nil {
return nil, err
}
if path == "" && dropped == 0 {
known, err := s.db.SourceFiles()
if err != nil {
return nil, err
}
if !slices.ContainsFunc(known, func(f store.SourceFileInfo) bool { return f.Path == rel }) {
return nil, &apiError{http.StatusNotFound, fmt.Sprintf("no statement %s", rel)}
}
}
_, unpaired, err := s.links.Link(s.db)
if err != nil {
return nil, err
}
verb := "deleted"
if path == "" {
verb = "forgot"
}
msg := fmt.Sprintf("%s %s, %d transaction(s) dropped", verb, rel, dropped)
if unpaired > 0 {
msg += fmt.Sprintf(" · %d transfer leg(s) unpaired", unpaired)
}
if dropped > 0 {
msg += " · press Import to restore any of them another statement also holds"
}
return status{msg}, nil
}
// retag is `money retag`: both halves of what rules.toml decides, re-derived // retag is `money retag`: both halves of what rules.toml decides, re-derived
// from the file as it is on disk now. // from the file as it is on disk now.
func (s *Server) retag(*http.Request) (any, error) { func (s *Server) retag(*http.Request) (any, error) {
+1 -1
View File
@@ -44,7 +44,7 @@ func newTestServer(t *testing.T, rulesToml string, txns ...fixtureTxn) (*Server,
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
src, err := db.SourceFile(id, slug+"/st.csv", "sha", "2026-01-01T00:00:00Z") src, err := db.SourceFile(id, slug+"/st.csv", "sha", "2026-01-01T00:00:00Z", 0)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+33 -22
View File
@@ -205,31 +205,22 @@ async function refreshAll() {
// --- import and retag ----------------------------------------------------- // --- import and retag -----------------------------------------------------
function runImport(force) { async function runImport(force) {
return importing(() => api('POST', '/api/import', { force: !!force })); if (state.busy) return;
}
// importing runs a request that ends in an import — the Import button, or an
// upload — and reports it the same way. It answers whether the request
// succeeded; per-file failures inside the import still count as success.
async function importing(request) {
if (state.busy) return false;
state.busy = true; state.busy = true;
const btn = document.getElementById('import'); const btn = document.getElementById('import');
btn.disabled = true; btn.disabled = true;
btn.textContent = 'Importing…'; btn.textContent = 'Importing…';
setStatus('importing… parsing PDF statements can take a while'); setStatus('importing… parsing PDF statements can take a while');
try { try {
const res = await request(); const res = await api('POST', '/api/import', { force: !!force });
state.status = res.status; state.status = res.status;
state.error = res.failed ? `${res.failed} file(s) failed to import` : ''; state.error = res.failed ? `${res.failed} file(s) failed to import` : '';
renderBanners(); renderBanners();
showImportDetails(res); showImportDetails(res);
await refreshAll(); await refreshAll();
return true;
} catch (e) { } catch (e) {
setError(e); setError(e);
return false;
} finally { } finally {
state.busy = false; state.busy = false;
btn.disabled = false; btn.disabled = false;
@@ -317,8 +308,8 @@ function fileList(list) {
panel.append(h('table', {}, panel.append(h('table', {},
h('thead', {}, h('tr', {}, h('th', {}, 'Account'), h('th', {}, 'File'), h('th', { class: 'num' }, 'Size'), h('thead', {}, h('tr', {}, h('th', {}, 'Account'), h('th', {}, 'File'), h('th', { class: 'num' }, 'Size'),
h('th', {}, 'Modified'), h('th', {}, ''), h('th', {}, 'Status'), h('th', {}, 'Modified'), h('th', {}, ''), h('th', {}, 'Status'),
h('th', { class: 'num', title: 'Transactions this file brought into the index; rows already brought by an overlapping statement count there' }, 'Added'), h('th', { class: 'num', title: 'Transactions the file holds, and how many of them it was the first to bring in — an overlapping statement imported earlier already holds the rest' }, 'Transactions'),
h('th', {}, 'Imported'))), h('th', {}, 'Imported'), h('th', {}, ''))),
h('tbody', {}, list.map((f) => { h('tbody', {}, list.map((f) => {
const [mark, cls, text] = FILE_STATUS[f.status] || ['', '', f.status]; const [mark, cls, text] = FILE_STATUS[f.status] || ['', '', f.status];
return h('tr', {}, return h('tr', {},
@@ -331,8 +322,11 @@ function fileList(list) {
h('td', { class: 'muted' }, f.modified), h('td', { class: 'muted' }, f.modified),
h('td', { class: 'mark ' + cls }, mark), h('td', { class: 'mark ' + cls }, mark),
h('td', { class: cls || 'muted' }, text), h('td', { class: cls || 'muted' }, text),
h('td', { class: 'num' }, f.status === 'new' ? '' : f.added), h('td', { class: 'num' }, f.status === 'new' ? '' : `${f.rows} ${f.rows === 1 ? 'row' : 'rows'} · ${f.added} new`),
h('td', { class: 'muted' }, localTime(f.importedAt))); h('td', { class: 'muted' }, localTime(f.importedAt)),
h('td', { class: 'actions' }, h('button', {
type: 'button', class: 'danger', onclick: () => removeFile(f),
}, f.status === 'missing' ? 'Forget' : 'Delete')));
})))); }))));
return h('div', {}, h('h3', { class: 'section' }, 'Statements ', return h('div', {}, h('h3', { class: 'section' }, 'Statements ',
h('span', { class: 'sub' }, `· ${list.length} file(s)` + (pending ? ` · ${pending} need attention` : ''))), panel); h('span', { class: 'sub' }, `· ${list.length} file(s)` + (pending ? ` · ${pending} need attention` : ''))), panel);
@@ -348,6 +342,22 @@ function localTime(iso) {
return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())} ${p(d.getHours())}:${p(d.getMinutes())}`; return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())} ${p(d.getHours())}:${p(d.getMinutes())}`;
} }
// removeFile deletes a statement for good. It does not import: rows another
// statement also holds come back at the next Import, so the question asked
// here is only about what this file alone brought in.
function removeFile(f) {
const where = `${f.account}/${f.name}`;
const rows = f.status === 'new' ? 'It has not been imported, so no transactions change.'
: `The ${f.added} transaction(s) it brought in leave the index; any another statement also holds come back at the next Import.`;
const question = f.status === 'missing'
? `Forget ${where}? The file is already gone from disk. ${rows}`
: `Delete ${where} permanently? The file is removed from disk and cannot be restored from here. ${rows}`;
if (!confirm(question)) return;
api('POST', '/api/files/delete', { account: f.account, name: f.name })
.then((res) => { setStatus(res.status); return refreshAll(); })
.catch(setError);
}
function formatSize(bytes) { function formatSize(bytes) {
if (bytes < 1024) return `${bytes} B`; if (bytes < 1024) return `${bytes} B`;
if (bytes < 1024 * 1024) return `${Math.round(bytes / 1024)} KB`; if (bytes < 1024 * 1024) return `${Math.round(bytes / 1024)} KB`;
@@ -377,7 +387,7 @@ function uploader(o) {
onchange: (e) => choose(e.target.files), onchange: (e) => choose(e.target.files),
}); });
const chosen = h('div', { class: 'chosen muted' }, 'No files chosen'); const chosen = h('div', { class: 'chosen muted' }, 'No files chosen');
const send = h('button', { type: 'button', class: 'primary', disabled: true, onclick: submit }, 'Upload and import'); const send = h('button', { type: 'button', class: 'primary', disabled: true, onclick: submit }, 'Upload');
const zone = h('div', { const zone = h('div', {
class: 'dropzone', class: 'dropzone',
tabindex: 0, tabindex: 0,
@@ -404,10 +414,11 @@ function uploader(o) {
account: account.value, account: account.value,
files: await Promise.all(files.map(async (f) => ({ name: f.name, data: await base64(f) }))), files: await Promise.all(files.map(async (f) => ({ name: f.name, data: await base64(f) }))),
}; };
if (await importing(() => api('POST', '/api/upload', payload))) { const res = await api('POST', '/api/upload', payload);
picker.value = ''; picker.value = '';
choose([]); choose([]);
} setStatus(res.status);
await refreshAll();
} catch (e) { } catch (e) {
setError(e); setError(e);
} finally { } finally {
@@ -419,7 +430,7 @@ function uploader(o) {
h('div', { class: 'toolbar' }, h('label', {}, 'Account ', account), send), h('div', { class: 'toolbar' }, h('label', {}, 'Account ', account), send),
zone, picker, chosen, zone, picker, chosen,
h('div', { class: 'hint muted' }, h('div', { class: 'hint muted' },
'Files are saved into the account folder, next to the statements already there, and imported. ' + 'Files are saved into the account folder, next to the statements already there; press Import to read them. ' +
'A file with the same name and different contents is refused rather than replaced.')); 'A file with the same name and different contents is refused rather than replaced.'));
return panel; return panel;
} }
+204 -15
View File
@@ -1,7 +1,9 @@
package web package web
import ( import (
"errors"
"fmt" "fmt"
"io/fs"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
@@ -23,10 +25,30 @@ func init() {
parser.Register("webtest", func(acc *config.Account) (parser.Parser, error) { parser.Register("webtest", func(acc *config.Account) (parser.Parser, error) {
return csvParser{digits: acc.Digits()}, nil return csvParser{digits: acc.Digits()}, nil
}) })
parser.Register("webnamed", func(acc *config.Account) (parser.Parser, error) {
return namingParser{csvParser{digits: acc.Digits()}}, nil
})
} }
type csvParser struct{ digits int } type csvParser struct{ digits int }
// namingParser is csvParser for a bank whose downloads are named unhelpfully:
// it names a statement after its first date, as nlb does after the statement
// date, and says nothing for a statement with no rows.
type namingParser struct{ csvParser }
func (namingParser) StatementName(path string) (string, error) {
body, err := os.ReadFile(path)
if err != nil {
return "", err
}
lines := strings.Split(strings.TrimSpace(string(body)), "\n")
if len(lines) < 2 {
return "", nil
}
return "stmt_" + strings.Split(lines[1], ",")[0], nil
}
func (p csvParser) Parse(path string, _ *config.Account) ([]parser.RawTxn, error) { func (p csvParser) Parse(path string, _ *config.Account) ([]parser.RawTxn, error) {
body, err := os.ReadFile(path) body, err := os.ReadFile(path)
if err != nil { if err != nil {
@@ -81,32 +103,38 @@ const checkingTOML = "currency = \"EUR\"\nparser = \"webtest\"\n"
const statement = "date,description,amount\n2026-02-01,LIDL SOFIA,-12.50\n2026-02-03,SALARY,1000.00\n" const statement = "date,description,amount\n2026-02-01,LIDL SOFIA,-12.50\n2026-02-03,SALARY,1000.00\n"
func TestUploadSavesAndImports(t *testing.T) { // An upload only saves: the file waits on the statements list as new until
// the user presses Import.
func TestUploadSavesWithoutImporting(t *testing.T) {
dir, h := newUploadServer(t, checkingTOML) dir, h := newUploadServer(t, checkingTOML)
req := uploadReq{Account: "checking", Files: []uploadFile{{Name: "2026-02.csv", Data: []byte(statement)}}} req := uploadReq{Account: "checking", Files: []uploadFile{{Name: "2026-02.csv", Data: []byte(statement)}}}
var res importJSON var res status
call(t, h, "POST", "/api/upload", req, http.StatusOK, &res) call(t, h, "POST", "/api/upload", req, http.StatusOK, &res)
if !strings.Contains(res.Status, "uploaded 1 file(s) to checking") || !strings.Contains(res.Status, "2 new") { if res.Status != "uploaded 1 file(s) to checking · press Import to read them" {
t.Errorf("status = %q", res.Status) t.Errorf("status = %q", res.Status)
} }
if got, err := os.ReadFile(filepath.Join(dir, "2026-02.csv")); err != nil || string(got) != statement { if got, err := os.ReadFile(filepath.Join(dir, "2026-02.csv")); err != nil || string(got) != statement {
t.Errorf("file on disk = %q, %v", got, err) t.Errorf("file on disk = %q, %v", got, err)
} }
var txns struct{ Rows []txnRow } if got := descriptions(t, h); got != "" {
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns) t.Errorf("upload imported %s; it should only save", got)
if len(txns.Rows) != 2 { }
t.Errorf("index holds %d rows, want the 2 uploaded", len(txns.Rows)) var files struct{ Files []fileRow }
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
if len(files.Files) != 1 || files.Files[0].Status != "new" {
t.Errorf("files = %+v, want the upload waiting as new", files.Files)
} }
// The same file again is not an error, and adds nothing. call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
call(t, h, "POST", "/api/upload", req, http.StatusOK, &res) if got := descriptions(t, h); got != "SALARY,LIDL SOFIA" {
if !strings.Contains(res.Status, "uploaded 0 file(s)") || !strings.Contains(res.Status, "1 already there") { t.Errorf("after Import the index holds %s", got)
t.Errorf("re-upload status = %q", res.Status)
} }
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
if len(txns.Rows) != 2 { // The same file again is not an error, and saves nothing.
t.Errorf("re-upload left %d rows, want 2", len(txns.Rows)) call(t, h, "POST", "/api/upload", req, http.StatusOK, &res)
if res.Status != "uploaded 0 file(s) to checking (1 already there)" {
t.Errorf("re-upload status = %q", res.Status)
} }
} }
@@ -203,7 +231,8 @@ func TestFileListStatuses(t *testing.T) {
t.Errorf("%s status = %q, want %q", name, got[name].Status, want) t.Errorf("%s status = %q, want %q", name, got[name].Status, want)
} }
} }
if got["a.csv"].Added != 2 || got["c.csv"].Added != 1 || got["a.csv"].ImportedAt == "" { if got["a.csv"].Rows != 2 || got["a.csv"].Added != 2 || got["c.csv"].Rows != 1 || got["c.csv"].Added != 1 ||
got["a.csv"].ImportedAt == "" {
t.Errorf("a.csv = %+v, c.csv = %+v", got["a.csv"], got["c.csv"]) t.Errorf("a.csv = %+v, c.csv = %+v", got["a.csv"], got["c.csv"])
} }
if got["a.csv"].Size != int64(len(statement)) { if got["a.csv"].Size != int64(len(statement)) {
@@ -251,3 +280,163 @@ func TestServeFileServesOnlyStatements(t *testing.T) {
} }
} }
} }
// A parser that names its statements decides the stored name, so the bank's
// "download (3).pdf" lands as something that sorts by date. A reissue of the
// same date is numbered rather than refused, and a statement that does not
// say keeps the name it came with.
func TestUploadNamesStatementsTheParserCanName(t *testing.T) {
dir, h := newUploadServer(t, "currency = \"EUR\"\nparser = \"webnamed\"\n")
upload := func(files ...uploadFile) string {
t.Helper()
var res status
call(t, h, "POST", "/api/upload", uploadReq{Account: "checking", Files: files}, http.StatusOK, &res)
return res.Status
}
reissue := "date,description,amount\n2026-02-01,LIDL SOFIA,-12.50\n2026-02-04,ZARA,-30.00\n"
other := "date,description,amount\n2026-03-01,KAUFLAND,-9.00\n"
status := upload(uploadFile{Name: "download (3).csv", Data: []byte(statement)})
if !strings.Contains(status, "download (3).csv → stmt_2026-02-01.csv") {
t.Errorf("status = %q, want the rename named", status)
}
// The same statement downloaded again is recognised under its new name.
if status := upload(uploadFile{Name: "download (4).csv", Data: []byte(statement)}); !strings.Contains(status, "1 already there") {
t.Errorf("second download: status = %q", status)
}
// A different statement of the same date, and two of one date in a batch.
upload(uploadFile{Name: "x.csv", Data: []byte(reissue)},
uploadFile{Name: "y.csv", Data: []byte(reissue + "2026-02-05,BOLT,-4.00\n")},
uploadFile{Name: "Z.CSV", Data: []byte(other)}, // a chosen name is lowercase
uploadFile{Name: "empty.csv", Data: []byte("date,description,amount\n")})
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
var names []string
for _, e := range entries {
names = append(names, e.Name())
}
want := config.AccountFile + " empty.csv stmt_2026-02-01.csv stmt_2026-02-01_2.csv stmt_2026-02-01_3.csv stmt_2026-03-01.csv"
if strings.Join(names, " ") != want {
t.Errorf("folder = %v, want %s", names, want)
}
}
func writeFile(t *testing.T, path, body string) {
t.Helper()
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
func descriptions(t *testing.T, h http.Handler) string {
t.Helper()
var txns struct{ Rows []txnRow }
call(t, h, "GET", "/api/transactions", nil, http.StatusOK, &txns)
var out []string
for _, r := range txns.Rows {
out = append(out, r.Description)
}
return strings.Join(out, ",")
}
// Removing a statement deletes the file and takes out what it brought in. A
// row an overlapping statement also holds was stored under the file imported
// first; the next Import brings it back from the other one.
func TestRemoveFileKeepsWhatAnotherStatementHolds(t *testing.T) {
dir, h := newUploadServer(t, checkingTOML)
writeFile(t, filepath.Join(dir, "a.csv"), "date,description,amount\n2026-01-30,ONLY IN A,-1.00\n2026-02-01,SHARED,-2.00\n")
writeFile(t, filepath.Join(dir, "b.csv"), "date,description,amount\n2026-02-01,SHARED,-2.00\n2026-02-03,ONLY IN B,-3.00\n")
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
// Each holds two rows; b.csv was imported second, so its shared row
// stayed with a.csv and it brought in only one.
var before struct{ Files []fileRow }
call(t, h, "GET", "/api/files", nil, http.StatusOK, &before)
if f := before.Files; len(f) != 2 || f[0].Rows != 2 || f[0].Added != 2 || f[1].Rows != 2 || f[1].Added != 1 {
t.Errorf("files = %+v, want a.csv 2 rows · 2 new and b.csv 2 rows · 1 new", f)
}
var res status
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusOK, &res)
if !strings.HasPrefix(res.Status, "deleted checking/a.csv, 2 transaction(s) dropped") {
t.Errorf("status = %q", res.Status)
}
// Deleting does not import, so the shared row is gone for now, and b.csv
// says it is due to be read again.
if got := descriptions(t, h); got != "ONLY IN B" {
t.Errorf("index holds %s right after the delete, want only b's own row", got)
}
var files struct{ Files []fileRow }
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
if len(files.Files) != 1 || files.Files[0].Status != "changed" {
t.Errorf("files = %+v, want b.csv marked to be re-read", files.Files)
}
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
if got := descriptions(t, h); got != "ONLY IN B,SHARED" {
t.Errorf("after Import the index holds %s, want the shared row restored from b", got)
}
if _, err := os.Stat(filepath.Join(dir, "a.csv")); !errors.Is(err, fs.ErrNotExist) {
t.Errorf("a.csv is still on disk: %v", err)
}
if entries, _ := os.ReadDir(dir); len(entries) != 2 {
t.Errorf("folder holds %d entries, want account.toml and b.csv only", len(entries))
}
call(t, h, "GET", "/api/files", nil, http.StatusOK, &files)
if len(files.Files) != 1 || files.Files[0].Name != "b.csv" || files.Files[0].Status != "imported" {
t.Errorf("files = %+v, want only b.csv, imported", files.Files)
}
}
// A removed leg takes its transfer pairing with it, and the other leg is left
// unpaired — which is the truth once one side is gone.
func TestRemoveFileUnpairsItsTransfers(t *testing.T) {
dir, h := newUploadServer(t, checkingTOML)
writeFile(t, filepath.Join(filepath.Dir(dir), config.RulesFile), `
[[transfer]]
from_account = "checking"
from_desc = "*OUT*"
to_account = "checking"
to_desc = "*IN*"
`)
writeFile(t, filepath.Join(dir, "out.csv"), "date,description,amount\n2026-02-01,MOVE OUT,-5.00\n")
writeFile(t, filepath.Join(dir, "in.csv"), "date,description,amount\n2026-02-02,MOVE IN,5.00\n")
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
var res status
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "in.csv"}, http.StatusOK, &res)
if !strings.Contains(res.Status, "1 transfer leg(s) unpaired") {
t.Errorf("status = %q, want the remaining leg reported unpaired", res.Status)
}
}
// A file already gone from disk is forgotten by the index; a name the index
// never had is not found.
func TestRemoveFileForgetsAMissingOne(t *testing.T) {
dir, h := newUploadServer(t, checkingTOML)
writeFile(t, filepath.Join(dir, "a.csv"), statement)
call(t, h, "POST", "/api/import", importReq{}, http.StatusOK, nil)
if err := os.Remove(filepath.Join(dir, "a.csv")); err != nil {
t.Fatal(err)
}
var res status
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusOK, &res)
if !strings.HasPrefix(res.Status, "forgot checking/a.csv, 2 transaction(s) dropped") {
t.Errorf("status = %q", res.Status)
}
if got := descriptions(t, h); got != "" {
t.Errorf("index still holds %s", got)
}
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "a.csv"}, http.StatusNotFound, nil)
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: "../rules.toml"}, http.StatusBadRequest, nil)
if _, err := os.Stat(filepath.Join(dir, config.AccountFile)); err != nil {
t.Fatal(err)
}
call(t, h, "POST", "/api/files/delete", removeFileReq{Account: "checking", Name: config.AccountFile}, http.StatusNotFound, nil)
}