Drop counterparty, the generic parsers and .money/
counterparty was a structured field only nlb could fill honestly. revolut and traderepublic invented one by running an IBAN-shaped regex over the description they had just built, and the two spellings disagreed -- SI56 1234 5678 9012 345 against SI56123456789012345 -- so a literal rule pattern that worked on one account silently matched nothing on another. It is gone from the model, the index, the rule keys, ls --wide and the rules screen. nlb now appends its IBAN column to the end of the description, where the other two already keep theirs, so match = "*SI56*" works everywhere. That changes those descriptions and with them their fingerprints, so a statement overlapping an already-imported period will re-add rather than dedupe those rows until the index is rebuilt. An index built by an older binary drops the column when it is opened. The index itself moves from .money/index.db up to index.db beside rules.toml. Nothing looks in the old location, so an existing one has to be moved by hand -- otherwise the tool quietly starts a fresh index and the manual tags in the old file, the only thing statements cannot reproduce, stay behind in it. The csv and cmd parsers are gone along with the [csv] and [cmd] config they carried. cmd shelled out to the Python extractors, which were ported to Go and deleted, so it bridged to nothing; csv was a generic column-mapped fallback that no account used, and between them they were the largest configuration surface in the tool. A bank is now described in Go, where it can be tested. The importer tests register their own three-column parser rather than borrow a bank's, so they stay about the directory walk, dedupe and per-file error reporting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -2,5 +2,4 @@
|
|||||||
/cmd/money/money
|
/cmd/money/money
|
||||||
|
|
||||||
# Never commit a data root that happens to live inside the repo.
|
# Never commit a data root that happens to live inside the repo.
|
||||||
.money/
|
|
||||||
*.db
|
*.db
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ cmd/money/main.go subcommands; the TUI is the default
|
|||||||
internal/config rules.toml, account.toml, XDG config, data-root resolution
|
internal/config rules.toml, account.toml, XDG config, data-root resolution
|
||||||
internal/model Account, Transaction, amount formatting, description normalisation
|
internal/model Account, Transaction, amount formatting, description normalisation
|
||||||
internal/glob the `*` / `?` matcher used by rules (linear time, no backtracking)
|
internal/glob the `*` / `?` matcher used by rules (linear time, no backtracking)
|
||||||
internal/parser Parser interface + registry; csv, cmd, nlb, revolut, traderepublic
|
internal/parser Parser interface + registry; nlb, revolut, traderepublic
|
||||||
internal/store SQLite index (modernc.org/sqlite, no cgo)
|
internal/store SQLite index (modernc.org/sqlite, no cgo)
|
||||||
internal/importer directory walk, dedupe, balance checks
|
internal/importer directory walk, dedupe, balance checks
|
||||||
internal/rules applies ordered rules to rule_* columns only
|
internal/rules applies ordered rules to rule_* columns only
|
||||||
@@ -32,8 +32,8 @@ any time, and it is the first thing to preserve when touching the schema or the
|
|||||||
rules engine. Covered by `TestManualTagSurvivesRetag`.
|
rules engine. Covered by `TestManualTagSurvivesRetag`.
|
||||||
|
|
||||||
**Statements are the source of truth; the index is disposable.** Deleting
|
**Statements are the source of truth; the index is disposable.** Deleting
|
||||||
`.money/index.db` and re-importing must reproduce everything except manual tags
|
`index.db` at the root of the data directory and re-importing must reproduce
|
||||||
and manual transfer marks.
|
everything except manual tags and manual transfer marks.
|
||||||
|
|
||||||
**Dedupe is by fingerprint**: `sha256(date | amount | normalised description |
|
**Dedupe is by fingerprint**: `sha256(date | amount | normalised description |
|
||||||
ordinal)`, where the ordinal distinguishes identical lines *within one
|
ordinal)`, where the ordinal distinguishes identical lines *within one
|
||||||
@@ -46,7 +46,7 @@ milliseconds. That is the checksum skip working, not a failure.
|
|||||||
conversion, and totals are never summed across currencies.
|
conversion, and totals are never summed across currencies.
|
||||||
|
|
||||||
**First matching rule wins**, so transfer rules belong above general tag rules.
|
**First matching rule wins**, so transfer rules belong above general tag rules.
|
||||||
A rule setting several of `match` / `counterparty` / `type` requires all of them.
|
A rule setting both `match` and `type` requires both of them.
|
||||||
`config.AppendRule` therefore appends — never prepends — so saving from the
|
`config.AppendRule` therefore appends — never prepends — so saving from the
|
||||||
rule builder cannot shadow a rule the user wrote by hand.
|
rule builder cannot shadow a rule the user wrote by hand.
|
||||||
|
|
||||||
@@ -100,6 +100,15 @@ compresses runs of spaces, so columns shift with font and page size — derive
|
|||||||
positions from the header line (`traderepublic`) or from the line being parsed
|
positions from the header line (`traderepublic`) or from the line being parsed
|
||||||
(`nlb`).
|
(`nlb`).
|
||||||
|
|
||||||
|
**The other side's account number belongs in the description, not in a field
|
||||||
|
of its own.** There used to be a `Counterparty` field; only `nlb` could fill it
|
||||||
|
honestly, `revolut` and `traderepublic` invented it with an IBAN-shaped regex
|
||||||
|
over the description, and the two disagreed on spacing, so one rule pattern
|
||||||
|
could not serve both. Now `nlb` appends its IBAN column to the end of the
|
||||||
|
description — at the end, and not in the position it held on the page, so an
|
||||||
|
IBAN wrapped across continuation lines stays contiguous for a glob to match.
|
||||||
|
A new parser must do the same rather than reintroduce a structured field.
|
||||||
|
|
||||||
## Verifying
|
## Verifying
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ transfers so they never count as spending.
|
|||||||
```
|
```
|
||||||
~/money/ # the data root (see "Where the data root lives")
|
~/money/ # the data root (see "Where the data root lives")
|
||||||
rules.toml # tag + transfer rules, in order
|
rules.toml # tag + transfer rules, in order
|
||||||
.money/index.db # SQLite index (rebuildable; safe to delete*)
|
index.db # SQLite index (rebuildable; safe to delete*)
|
||||||
checking/
|
checking/
|
||||||
account.toml # currency + how to parse this bank's exports
|
account.toml # currency + how to parse this bank's exports
|
||||||
2026-01.csv
|
2026-01.csv
|
||||||
@@ -63,7 +63,7 @@ money import # extract new transactions from every statement
|
|||||||
money import --force # re-parse statements even if unchanged
|
money import --force # re-parse statements even if unchanged
|
||||||
money retag # re-apply rules.toml to everything already imported
|
money retag # re-apply rules.toml to everything already imported
|
||||||
money ls --untagged # what still needs a tag
|
money ls --untagged # what still needs a tag
|
||||||
money ls --wide # also show type, counterparty and reported balance
|
money ls --wide # also show type and reported balance
|
||||||
money ls --account checking --month 2026-01
|
money ls --account checking --month 2026-01
|
||||||
money report --month 2026-01 # spending by tag
|
money report --month 2026-01 # spending by tag
|
||||||
money accounts # balances
|
money accounts # balances
|
||||||
@@ -175,9 +175,8 @@ Rules are evaluated in file order and the **first match wins**, so put transfer
|
|||||||
rules above general tag rules. Patterns are globs (`*` and `?`) matched
|
rules above general tag rules. Patterns are globs (`*` and `?`) matched
|
||||||
case-insensitively, with whitespace collapsed.
|
case-insensitively, with whitespace collapsed.
|
||||||
|
|
||||||
A rule matches on `match` (the description), `counterparty` (the other side's
|
A rule matches on `match` (the description) and `type` (the bank's own
|
||||||
account number) and `type` (the bank's own classification). Setting several is
|
classification). Setting both is an "and": both must match.
|
||||||
an "and": all must match.
|
|
||||||
|
|
||||||
`note` is free text for you, never for the matcher: why the rule is there, or
|
`note` is free text for you, never for the matcher: why the rule is there, or
|
||||||
what the unrecognisable payee behind the glob actually is. It shows in the last
|
what the unrecognisable payee behind the glob actually is. It shows in the last
|
||||||
@@ -206,12 +205,13 @@ match = "*FROM CHECKING*"
|
|||||||
transfer = true
|
transfer = true
|
||||||
tag = "transfer"
|
tag = "transfer"
|
||||||
|
|
||||||
# Transfers are often only identifiable by the counterparty IBAN, whatever
|
# Transfers are often only identifiable by the other side's account number,
|
||||||
# the description happens to say.
|
# whatever the rest of the description happens to say. Every bank parser keeps
|
||||||
|
# that number in the description, so an ordinary glob finds it.
|
||||||
[[rule]]
|
[[rule]]
|
||||||
counterparty = "SI56*"
|
match = "*SI56*"
|
||||||
transfer = true
|
transfer = true
|
||||||
tag = "transfer"
|
tag = "transfer"
|
||||||
|
|
||||||
# A rule can be limited to one account, and can require several patterns.
|
# A rule can be limited to one account, and can require several patterns.
|
||||||
[[rule]]
|
[[rule]]
|
||||||
@@ -236,43 +236,20 @@ Every account folder needs one. `currency` and `parser` are required.
|
|||||||
```toml
|
```toml
|
||||||
name = "Main Checking"
|
name = "Main Checking"
|
||||||
currency = "EUR"
|
currency = "EUR"
|
||||||
parser = "csv"
|
parser = "nlb"
|
||||||
# minor_digits = 2 # decimal places for this currency
|
# minor_digits = 2 # decimal places for this currency
|
||||||
# include = ["*.csv"] # only treat matching files as statements
|
# include = ["*.csv"] # only treat matching files as statements
|
||||||
```
|
```
|
||||||
|
|
||||||
### `parser = "csv"`
|
### Parsers
|
||||||
|
|
||||||
Column positions are 0-based.
|
Three are built in, ported from the original Python extractors. A statement
|
||||||
|
layout is described in Go rather than in a table of column indexes, so there is
|
||||||
```toml
|
nothing else to configure — `parser` names one of these and that is all.
|
||||||
[csv]
|
|
||||||
delimiter = "," # default ","
|
|
||||||
skip_rows = 1 # header rows to drop
|
|
||||||
date = { col = 0, layout = "02.01.2006" } # Go reference layout
|
|
||||||
description = { col = 3 }
|
|
||||||
amount = { col = 4, decimal = ",", thousands = "." }
|
|
||||||
# invert = true # if outflows are written as positive
|
|
||||||
```
|
|
||||||
|
|
||||||
For statements with separate debit and credit columns, replace `amount`:
|
|
||||||
|
|
||||||
```toml
|
|
||||||
debit = { col = 4 } # both written as positive numbers
|
|
||||||
credit = { col = 5 }
|
|
||||||
```
|
|
||||||
|
|
||||||
Amount parsing is deliberately tolerant: `1.234,56`, `-45.20`, `45,20-`,
|
|
||||||
`(45.20)` and `45.20 EUR` all work.
|
|
||||||
|
|
||||||
### Bank-specific parsers
|
|
||||||
|
|
||||||
Three are built in, ported from the original Python extractors. They need no
|
|
||||||
`[csv]` block — the layout is baked in.
|
|
||||||
|
|
||||||
| `parser` | Statement | Notes |
|
| `parser` | Statement | Notes |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `nlb` | NLB izpisek PDF | Wrapped descriptions and the counterparty IBAN are folded in from continuation lines. |
|
| `nlb` | NLB izpisek PDF | Wrapped descriptions are folded in from continuation lines, and the IBAN column is appended to the description. |
|
||||||
| `traderepublic` | Trade Republic PDF | Handles both the single-line and the stacked layout by measuring column positions. |
|
| `traderepublic` | Trade Republic PDF | Handles both the single-line and the stacked layout by measuring column positions. |
|
||||||
| `revolut` | `account-statement*.csv` | Skips non-COMPLETED rows, folds the fee into the amount. |
|
| `revolut` | `account-statement*.csv` | Skips non-COMPLETED rows, folds the fee into the amount. |
|
||||||
|
|
||||||
@@ -286,33 +263,19 @@ The two PDF parsers shell out to `pdftotext -layout` (poppler-utils), exactly
|
|||||||
as the Python versions did; its layout reconstruction is what makes the
|
as the Python versions did; its layout reconstruction is what makes the
|
||||||
column-based parsing work.
|
column-based parsing work.
|
||||||
|
|
||||||
|
Amount parsing is shared and deliberately tolerant: `1.234,56`, `-45.20`,
|
||||||
|
`45,20-`, `(45.20)` and `45.20 EUR` all work, whichever parser reads them.
|
||||||
|
|
||||||
**Revolut and multiple currencies.** One export can hold several currencies,
|
**Revolut and multiple currencies.** One export can hold several currencies,
|
||||||
but an account here has exactly one. Rows in other currencies are skipped and
|
but an account here has exactly one. Rows in other currencies are skipped and
|
||||||
reported at import. To keep them, give that currency its own account folder
|
reported at import. To keep them, give that currency its own account folder
|
||||||
with its own `currency` and `minor_digits` (JPY wants `minor_digits = 0`) and
|
with its own `currency` and `minor_digits` (JPY wants `minor_digits = 0`) and
|
||||||
put a copy or symlink of the export in it.
|
put a copy or symlink of the export in it.
|
||||||
|
|
||||||
### `parser = "cmd"`
|
## Adding a parser
|
||||||
|
|
||||||
Runs an external extractor and reads normalised CSV (`date,description,amount`)
|
A new bank means a new parser. Drop it in `internal/parser` and register it —
|
||||||
from its stdout. This is how PDF statements and any existing Python extractor
|
no other package changes:
|
||||||
are handled without porting them first.
|
|
||||||
|
|
||||||
```toml
|
|
||||||
parser = "cmd"
|
|
||||||
[cmd]
|
|
||||||
argv = ["python3", "../extract_bankx.py", "{{file}}"]
|
|
||||||
skip_rows = 1 # if the script prints a header
|
|
||||||
# layout = "2006-01-02" # date format the script emits (default)
|
|
||||||
```
|
|
||||||
|
|
||||||
`{{file}}` is replaced with the statement's path, and the command runs with the
|
|
||||||
account folder as its working directory, so relative script paths work.
|
|
||||||
|
|
||||||
## Adding a native parser
|
|
||||||
|
|
||||||
When a Python extractor is ported to Go, drop it in `internal/parser` and
|
|
||||||
register it — no other package changes:
|
|
||||||
|
|
||||||
```go
|
```go
|
||||||
func init() {
|
func init() {
|
||||||
|
|||||||
+4
-4
@@ -232,7 +232,7 @@ func cmdLs(root string, args []string) error {
|
|||||||
search := fs.String("search", "", "only descriptions containing this text")
|
search := fs.String("search", "", "only descriptions containing this text")
|
||||||
untagged := fs.Bool("untagged", false, "only transactions with no effective tag")
|
untagged := fs.Bool("untagged", false, "only transactions with no effective tag")
|
||||||
limit := fs.Int("limit", 0, "maximum rows (0 = no limit)")
|
limit := fs.Int("limit", 0, "maximum rows (0 = no limit)")
|
||||||
wide := fs.Bool("wide", false, "also show type, counterparty and reported balance")
|
wide := fs.Bool("wide", false, "also show type and reported balance")
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := fs.Parse(args); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -256,7 +256,7 @@ func cmdLs(root string, args []string) error {
|
|||||||
|
|
||||||
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
|
w := tabwriter.NewWriter(os.Stdout, 0, 0, 2, ' ', 0)
|
||||||
if *wide {
|
if *wide {
|
||||||
fmt.Fprintln(w, "DATE\tACCOUNT\tAMOUNT\tCUR\tBALANCE\tTAG\tT\tTYPE\tCOUNTERPARTY\tDESCRIPTION")
|
fmt.Fprintln(w, "DATE\tACCOUNT\tAMOUNT\tCUR\tBALANCE\tTAG\tT\tTYPE\tDESCRIPTION")
|
||||||
} else {
|
} else {
|
||||||
fmt.Fprintln(w, "DATE\tACCOUNT\tAMOUNT\tCUR\tTAG\tT\tDESCRIPTION")
|
fmt.Fprintln(w, "DATE\tACCOUNT\tAMOUNT\tCUR\tTAG\tT\tDESCRIPTION")
|
||||||
}
|
}
|
||||||
@@ -270,9 +270,9 @@ func cmdLs(root string, args []string) error {
|
|||||||
if t.BalanceMinor != nil {
|
if t.BalanceMinor != nil {
|
||||||
balance = model.FormatMinor(*t.BalanceMinor, t.MinorDigits)
|
balance = model.FormatMinor(*t.BalanceMinor, t.MinorDigits)
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||||
t.Date, t.AccountSlug, t.FormatAmount(), t.Currency, balance,
|
t.Date, t.AccountSlug, t.FormatAmount(), t.Currency, balance,
|
||||||
t.Tag(), transfer, t.Type, t.Counterparty, t.Description)
|
t.Tag(), transfer, t.Type, t.Description)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
|
||||||
|
|||||||
@@ -18,9 +18,8 @@ const (
|
|||||||
RulesFile = "rules.toml"
|
RulesFile = "rules.toml"
|
||||||
// AccountFile is the per-account config inside each account folder.
|
// AccountFile is the per-account config inside each account folder.
|
||||||
AccountFile = "account.toml"
|
AccountFile = "account.toml"
|
||||||
// StateDir holds the rebuildable SQLite index.
|
// IndexFile is the rebuildable SQLite index, at the root of the data
|
||||||
StateDir = ".money"
|
// directory alongside rules.toml.
|
||||||
// IndexFile is the SQLite index inside StateDir.
|
|
||||||
IndexFile = "index.db"
|
IndexFile = "index.db"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -31,10 +30,6 @@ type Rule struct {
|
|||||||
Tag string `toml:"tag"`
|
Tag string `toml:"tag"`
|
||||||
Transfer bool `toml:"transfer"`
|
Transfer bool `toml:"transfer"`
|
||||||
Account string `toml:"account"` // optional: restrict to one account slug
|
Account string `toml:"account"` // optional: restrict to one account slug
|
||||||
// Counterparty matches the other side's account number, which for
|
|
||||||
// movements between the user's own accounts is often the only reliable
|
|
||||||
// signal. Optional; when set, it must match as well as Match.
|
|
||||||
Counterparty string `toml:"counterparty"`
|
|
||||||
// Type matches the bank's own classification, e.g. Revolut's CARD_PAYMENT.
|
// Type matches the bank's own classification, e.g. Revolut's CARD_PAYMENT.
|
||||||
// Optional; when set, it must match as well as Match.
|
// Optional; when set, it must match as well as Match.
|
||||||
Type string `toml:"type"`
|
Type string `toml:"type"`
|
||||||
@@ -62,8 +57,8 @@ func LoadRules(root string) (*Rules, error) {
|
|||||||
return nil, fmt.Errorf("%s: %w", path, err)
|
return nil, fmt.Errorf("%s: %w", path, err)
|
||||||
}
|
}
|
||||||
for i, rule := range r.Rule {
|
for i, rule := range r.Rule {
|
||||||
if rule.Match == "" && rule.Counterparty == "" && rule.Type == "" {
|
if rule.Match == "" && rule.Type == "" {
|
||||||
return nil, fmt.Errorf("%s: rule %d has no match, counterparty or type pattern", path, i+1)
|
return nil, fmt.Errorf("%s: rule %d has no match or type pattern", path, i+1)
|
||||||
}
|
}
|
||||||
if rule.Tag == "" && !rule.Transfer {
|
if rule.Tag == "" && !rule.Transfer {
|
||||||
return nil, fmt.Errorf("%s: rule %d (%q) sets neither tag nor transfer", path, i+1, rule.Match)
|
return nil, fmt.Errorf("%s: rule %d (%q) sets neither tag nor transfer", path, i+1, rule.Match)
|
||||||
@@ -79,8 +74,8 @@ func LoadRules(root string) (*Rules, error) {
|
|||||||
// The file is rewritten through a temporary file so a failure part-way cannot
|
// The file is rewritten through a temporary file so a failure part-way cannot
|
||||||
// leave the user with a truncated config.
|
// leave the user with a truncated config.
|
||||||
func AppendRule(root string, r Rule) error {
|
func AppendRule(root string, r Rule) error {
|
||||||
if r.Match == "" && r.Counterparty == "" && r.Type == "" {
|
if r.Match == "" && r.Type == "" {
|
||||||
return fmt.Errorf("a rule needs a match, counterparty or type pattern")
|
return fmt.Errorf("a rule needs a match or type pattern")
|
||||||
}
|
}
|
||||||
if r.Tag == "" && !r.Transfer {
|
if r.Tag == "" && !r.Transfer {
|
||||||
return fmt.Errorf("a rule needs a tag or transfer = true")
|
return fmt.Errorf("a rule needs a tag or transfer = true")
|
||||||
@@ -254,7 +249,6 @@ func formatRule(r Rule) string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
write("match", r.Match)
|
write("match", r.Match)
|
||||||
write("counterparty", r.Counterparty)
|
|
||||||
write("type", r.Type)
|
write("type", r.Type)
|
||||||
write("account", r.Account)
|
write("account", r.Account)
|
||||||
write("tag", r.Tag)
|
write("tag", r.Tag)
|
||||||
@@ -267,41 +261,6 @@ func formatRule(r Rule) string {
|
|||||||
return b.String()
|
return b.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Column locates one field in a CSV row.
|
|
||||||
type Column struct {
|
|
||||||
Col int `toml:"col"`
|
|
||||||
Layout string `toml:"layout"` // date only, Go reference layout
|
|
||||||
Decimal string `toml:"decimal"` // amount only, default "."
|
|
||||||
Thousands string `toml:"thousands"` // amount only, default ""
|
|
||||||
}
|
|
||||||
|
|
||||||
// CSVConfig describes how to read a delimited statement.
|
|
||||||
type CSVConfig struct {
|
|
||||||
Delimiter string `toml:"delimiter"`
|
|
||||||
SkipRows int `toml:"skip_rows"`
|
|
||||||
Encoding string `toml:"encoding"` // "" or "utf-8"; other encodings unsupported for now
|
|
||||||
Date Column `toml:"date"`
|
|
||||||
Description Column `toml:"description"`
|
|
||||||
Amount *Column `toml:"amount"` // single signed column...
|
|
||||||
Debit *Column `toml:"debit"` // ...or a debit/credit pair
|
|
||||||
Credit *Column `toml:"credit"`
|
|
||||||
// Invert flips the sign of the parsed amount, for statements that report
|
|
||||||
// outflows as positive numbers.
|
|
||||||
Invert bool `toml:"invert"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// CmdConfig runs an external extractor (e.g. one of the existing Python
|
|
||||||
// scripts) and reads normalised CSV from its stdout.
|
|
||||||
type CmdConfig struct {
|
|
||||||
// Argv is the command to run. The literal token "{{file}}" is replaced
|
|
||||||
// with the absolute path of the statement being imported.
|
|
||||||
Argv []string `toml:"argv"`
|
|
||||||
// Layout is the date layout the script emits; defaults to 2006-01-02.
|
|
||||||
Layout string `toml:"layout"`
|
|
||||||
// SkipRows skips leading rows of the script's output (e.g. a header).
|
|
||||||
SkipRows int `toml:"skip_rows"`
|
|
||||||
}
|
|
||||||
|
|
||||||
// Account is a parsed account.toml.
|
// Account is a parsed account.toml.
|
||||||
type Account struct {
|
type Account struct {
|
||||||
Slug string // folder name, filled in by LoadAccounts
|
Slug string // folder name, filled in by LoadAccounts
|
||||||
@@ -312,9 +271,7 @@ type Account struct {
|
|||||||
Parser string `toml:"parser"`
|
Parser string `toml:"parser"`
|
||||||
// Include restricts which files in the folder are treated as statements.
|
// Include restricts which files in the folder are treated as statements.
|
||||||
// Defaults to every regular file except account.toml and dotfiles.
|
// Defaults to every regular file except account.toml and dotfiles.
|
||||||
Include []string `toml:"include"`
|
Include []string `toml:"include"`
|
||||||
CSV *CSVConfig `toml:"csv"`
|
|
||||||
Cmd *CmdConfig `toml:"cmd"`
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Digits returns the configured minor-unit scale, defaulting to 2.
|
// Digits returns the configured minor-unit scale, defaulting to 2.
|
||||||
@@ -376,7 +333,7 @@ func loadAccount(dir, slug, cfgPath string) (*Account, error) {
|
|||||||
|
|
||||||
// IndexPath returns the location of the SQLite index for a data root.
|
// IndexPath returns the location of the SQLite index for a data root.
|
||||||
func IndexPath(root string) string {
|
func IndexPath(root string) string {
|
||||||
return filepath.Join(root, StateDir, IndexFile)
|
return filepath.Join(root, IndexFile)
|
||||||
}
|
}
|
||||||
|
|
||||||
// UserConfig is the small file in the user's config directory that says where
|
// UserConfig is the small file in the user's config directory that says where
|
||||||
|
|||||||
@@ -169,7 +169,6 @@ func importFile(root string, db *store.DB, acc *config.Account, accountID int64,
|
|||||||
Date: t.Date,
|
Date: t.Date,
|
||||||
Description: t.Description,
|
Description: t.Description,
|
||||||
AmountMinor: t.AmountMinor,
|
AmountMinor: t.AmountMinor,
|
||||||
Counterparty: t.Counterparty,
|
|
||||||
Type: t.Type,
|
Type: t.Type,
|
||||||
BalanceMinor: t.BalanceMinor,
|
BalanceMinor: t.BalanceMinor,
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
package importer
|
package importer
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"git.petrovv.com/nikola/money/internal/config"
|
"git.petrovv.com/nikola/money/internal/config"
|
||||||
"git.petrovv.com/nikola/money/internal/parser"
|
"git.petrovv.com/nikola/money/internal/parser"
|
||||||
@@ -15,14 +17,49 @@ import (
|
|||||||
const accountTOML = `
|
const accountTOML = `
|
||||||
name = "Checking"
|
name = "Checking"
|
||||||
currency = "EUR"
|
currency = "EUR"
|
||||||
parser = "csv"
|
parser = "test"
|
||||||
[csv]
|
|
||||||
skip_rows = 1
|
|
||||||
date = { col = 0, layout = "2006-01-02" }
|
|
||||||
description = { col = 1 }
|
|
||||||
amount = { col = 2 }
|
|
||||||
`
|
`
|
||||||
|
|
||||||
|
// These tests are about the directory walk, the checksum skip, dedupe and
|
||||||
|
// per-file error reporting -- not about any bank's layout. Driving them with a
|
||||||
|
// real bank parser would drag that bank's quirks (Revolut's fee folding and
|
||||||
|
// COMPLETED filter, the PDF parsers' dependency on pdftotext) into every
|
||||||
|
// fixture, so they register the smallest parser that will do instead.
|
||||||
|
func init() {
|
||||||
|
parser.Register("test", func(acc *config.Account) (parser.Parser, error) {
|
||||||
|
return testParser{digits: acc.Digits()}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// testParser reads "date,description,amount" with one header row.
|
||||||
|
type testParser struct{ digits int }
|
||||||
|
|
||||||
|
func (p testParser) Parse(path string, acc *config.Account) ([]parser.RawTxn, error) {
|
||||||
|
body, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var txns []parser.RawTxn
|
||||||
|
for i, line := range strings.Split(strings.TrimSpace(string(body)), "\n") {
|
||||||
|
if i == 0 || strings.TrimSpace(line) == "" {
|
||||||
|
continue // header
|
||||||
|
}
|
||||||
|
fields := strings.Split(line, ",")
|
||||||
|
if len(fields) != 3 {
|
||||||
|
return nil, fmt.Errorf("row %d: got %d fields, want 3", i+1, len(fields))
|
||||||
|
}
|
||||||
|
if _, err := time.Parse("2006-01-02", fields[0]); err != nil {
|
||||||
|
return nil, fmt.Errorf("row %d: %w", i+1, err)
|
||||||
|
}
|
||||||
|
amount, err := parser.ParseAmount(fields[2], ".", "", p.digits)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("row %d: %w", i+1, err)
|
||||||
|
}
|
||||||
|
txns = append(txns, parser.RawTxn{Date: fields[0], Description: fields[1], AmountMinor: amount})
|
||||||
|
}
|
||||||
|
return txns, nil
|
||||||
|
}
|
||||||
|
|
||||||
// newRoot builds a data root with one account and the given statement files.
|
// newRoot builds a data root with one account and the given statement files.
|
||||||
func newRoot(t *testing.T, statements map[string]string) (string, *store.DB, []*config.Account, *rules.Engine) {
|
func newRoot(t *testing.T, statements map[string]string) (string, *store.DB, []*config.Account, *rules.Engine) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|||||||
@@ -33,10 +33,6 @@ type Transaction struct {
|
|||||||
SourceFileID int64
|
SourceFileID int64
|
||||||
SourcePath string
|
SourcePath string
|
||||||
|
|
||||||
// Counterparty is the other side's account number (an IBAN, where the
|
|
||||||
// statement gives one). Transfers between the user's own accounts are
|
|
||||||
// often only distinguishable by it.
|
|
||||||
Counterparty string
|
|
||||||
// Type is the bank's own classification, e.g. Revolut's CARD_PAYMENT.
|
// Type is the bank's own classification, e.g. Revolut's CARD_PAYMENT.
|
||||||
Type string
|
Type string
|
||||||
// BalanceMinor is the running balance the statement reported after this
|
// BalanceMinor is the running balance the statement reported after this
|
||||||
|
|||||||
@@ -1,124 +0,0 @@
|
|||||||
package parser
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"context"
|
|
||||||
"encoding/csv"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"os/exec"
|
|
||||||
"strings"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.petrovv.com/nikola/money/internal/config"
|
|
||||||
)
|
|
||||||
|
|
||||||
func init() {
|
|
||||||
Register("cmd", newCmdParser)
|
|
||||||
}
|
|
||||||
|
|
||||||
// FileToken is replaced with the statement's absolute path in a cmd parser's argv.
|
|
||||||
const FileToken = "{{file}}"
|
|
||||||
|
|
||||||
// cmdRunTimeout bounds an extractor run so a hung script cannot wedge an import.
|
|
||||||
const cmdRunTimeout = 2 * time.Minute
|
|
||||||
|
|
||||||
// cmdParser runs an external extractor and reads normalised CSV from its
|
|
||||||
// stdout: date,description,amount with any further columns ignored. This is
|
|
||||||
// the bridge that lets the existing Python extractors be used unchanged.
|
|
||||||
type cmdParser struct {
|
|
||||||
cfg config.CmdConfig
|
|
||||||
digits int
|
|
||||||
}
|
|
||||||
|
|
||||||
func newCmdParser(acc *config.Account) (Parser, error) {
|
|
||||||
if acc.Cmd == nil || len(acc.Cmd.Argv) == 0 {
|
|
||||||
return nil, fmt.Errorf("account %s: parser \"cmd\" requires [cmd] with a non-empty argv", acc.Slug)
|
|
||||||
}
|
|
||||||
cfg := *acc.Cmd
|
|
||||||
if cfg.Layout == "" {
|
|
||||||
cfg.Layout = "2006-01-02"
|
|
||||||
}
|
|
||||||
if !hasFileToken(cfg.Argv) {
|
|
||||||
return nil, fmt.Errorf("account %s: [cmd] argv must contain %s so the script knows which file to read",
|
|
||||||
acc.Slug, FileToken)
|
|
||||||
}
|
|
||||||
return &cmdParser{cfg: cfg, digits: acc.Digits()}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func hasFileToken(argv []string) bool {
|
|
||||||
for _, a := range argv {
|
|
||||||
if strings.Contains(a, FileToken) {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *cmdParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
|
||||||
argv := make([]string, len(p.cfg.Argv))
|
|
||||||
for i, a := range p.cfg.Argv {
|
|
||||||
argv[i] = strings.ReplaceAll(a, FileToken, path)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), cmdRunTimeout)
|
|
||||||
defer cancel()
|
|
||||||
|
|
||||||
cmd := exec.CommandContext(ctx, argv[0], argv[1:]...)
|
|
||||||
cmd.Dir = acc.Dir
|
|
||||||
var stdout, stderr bytes.Buffer
|
|
||||||
cmd.Stdout = &stdout
|
|
||||||
cmd.Stderr = &stderr
|
|
||||||
|
|
||||||
if err := cmd.Run(); err != nil {
|
|
||||||
msg := strings.TrimSpace(stderr.String())
|
|
||||||
if ctx.Err() == context.DeadlineExceeded {
|
|
||||||
return nil, fmt.Errorf("extractor %v timed out after %s", argv, cmdRunTimeout)
|
|
||||||
}
|
|
||||||
if msg != "" {
|
|
||||||
return nil, fmt.Errorf("extractor %v failed: %w: %s", argv, err, msg)
|
|
||||||
}
|
|
||||||
return nil, fmt.Errorf("extractor %v failed: %w", argv, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return p.parseOutput(&stdout, argv)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *cmdParser) parseOutput(out io.Reader, argv []string) ([]RawTxn, error) {
|
|
||||||
r := csv.NewReader(out)
|
|
||||||
r.FieldsPerRecord = -1
|
|
||||||
r.LazyQuotes = true
|
|
||||||
|
|
||||||
var txns []RawTxn
|
|
||||||
for row := 0; ; row++ {
|
|
||||||
rec, err := r.Read()
|
|
||||||
if err == io.EOF {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("extractor %v: output row %d: %w", argv, row+1, err)
|
|
||||||
}
|
|
||||||
if row < p.cfg.SkipRows || isBlank(rec) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if len(rec) < 3 {
|
|
||||||
return nil, fmt.Errorf("extractor %v: output row %d has %d columns, want date,description,amount",
|
|
||||||
argv, row+1, len(rec))
|
|
||||||
}
|
|
||||||
d, err := time.Parse(p.cfg.Layout, strings.TrimSpace(rec[0]))
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("extractor %v: output row %d: date %q does not match layout %q",
|
|
||||||
argv, row+1, rec[0], p.cfg.Layout)
|
|
||||||
}
|
|
||||||
amount, err := ParseAmount(rec[2], ".", "", p.digits)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("extractor %v: output row %d: %w", argv, row+1, err)
|
|
||||||
}
|
|
||||||
txns = append(txns, RawTxn{
|
|
||||||
Date: d.Format("2006-01-02"),
|
|
||||||
Description: strings.TrimSpace(rec[1]),
|
|
||||||
AmountMinor: amount,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
return txns, nil
|
|
||||||
}
|
|
||||||
@@ -1,175 +0,0 @@
|
|||||||
package parser
|
|
||||||
|
|
||||||
import (
|
|
||||||
"encoding/csv"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"os"
|
|
||||||
"strings"
|
|
||||||
"time"
|
|
||||||
|
|
||||||
"git.petrovv.com/nikola/money/internal/config"
|
|
||||||
)
|
|
||||||
|
|
||||||
func init() {
|
|
||||||
Register("csv", newCSVParser)
|
|
||||||
}
|
|
||||||
|
|
||||||
// csvParser reads a delimited statement using column positions from
|
|
||||||
// account.toml. Amounts come either from one signed column, or from a
|
|
||||||
// debit/credit pair.
|
|
||||||
type csvParser struct {
|
|
||||||
cfg config.CSVConfig
|
|
||||||
digits int
|
|
||||||
}
|
|
||||||
|
|
||||||
func newCSVParser(acc *config.Account) (Parser, error) {
|
|
||||||
if acc.CSV == nil {
|
|
||||||
return nil, fmt.Errorf("account %s: parser \"csv\" requires a [csv] section", acc.Slug)
|
|
||||||
}
|
|
||||||
cfg := *acc.CSV
|
|
||||||
if cfg.Amount == nil && cfg.Debit == nil && cfg.Credit == nil {
|
|
||||||
return nil, fmt.Errorf("account %s: [csv] needs either amount or debit/credit columns", acc.Slug)
|
|
||||||
}
|
|
||||||
if cfg.Amount != nil && (cfg.Debit != nil || cfg.Credit != nil) {
|
|
||||||
return nil, fmt.Errorf("account %s: [csv] sets both amount and debit/credit; pick one", acc.Slug)
|
|
||||||
}
|
|
||||||
if cfg.Date.Layout == "" {
|
|
||||||
return nil, fmt.Errorf("account %s: [csv] date needs a layout, e.g. layout = \"02.01.2006\"", acc.Slug)
|
|
||||||
}
|
|
||||||
return &csvParser{cfg: cfg, digits: acc.Digits()}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *csvParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
|
||||||
f, err := os.Open(path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
defer f.Close()
|
|
||||||
|
|
||||||
r := csv.NewReader(f)
|
|
||||||
r.FieldsPerRecord = -1 // statements are ragged more often than not
|
|
||||||
r.LazyQuotes = true
|
|
||||||
if d := p.cfg.Delimiter; d != "" {
|
|
||||||
runes := []rune(d)
|
|
||||||
if len(runes) != 1 {
|
|
||||||
return nil, fmt.Errorf("delimiter %q must be a single character", d)
|
|
||||||
}
|
|
||||||
r.Comma = runes[0]
|
|
||||||
}
|
|
||||||
|
|
||||||
var out []RawTxn
|
|
||||||
for row := 0; ; row++ {
|
|
||||||
rec, err := r.Read()
|
|
||||||
if err == io.EOF {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("%s: row %d: %w", path, row+1, err)
|
|
||||||
}
|
|
||||||
if row < p.cfg.SkipRows {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if isBlank(rec) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
txn, err := p.row(rec)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("%s: row %d: %w", path, row+1, err)
|
|
||||||
}
|
|
||||||
out = append(out, txn)
|
|
||||||
}
|
|
||||||
return out, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *csvParser) row(rec []string) (RawTxn, error) {
|
|
||||||
var t RawTxn
|
|
||||||
|
|
||||||
raw, err := field(rec, p.cfg.Date.Col)
|
|
||||||
if err != nil {
|
|
||||||
return t, fmt.Errorf("date column: %w", err)
|
|
||||||
}
|
|
||||||
d, err := time.Parse(p.cfg.Date.Layout, strings.TrimSpace(raw))
|
|
||||||
if err != nil {
|
|
||||||
return t, fmt.Errorf("date %q does not match layout %q", raw, p.cfg.Date.Layout)
|
|
||||||
}
|
|
||||||
t.Date = d.Format("2006-01-02")
|
|
||||||
|
|
||||||
desc, err := field(rec, p.cfg.Description.Col)
|
|
||||||
if err != nil {
|
|
||||||
return t, fmt.Errorf("description column: %w", err)
|
|
||||||
}
|
|
||||||
t.Description = strings.TrimSpace(desc)
|
|
||||||
|
|
||||||
switch {
|
|
||||||
case p.cfg.Amount != nil:
|
|
||||||
raw, err := field(rec, p.cfg.Amount.Col)
|
|
||||||
if err != nil {
|
|
||||||
return t, fmt.Errorf("amount column: %w", err)
|
|
||||||
}
|
|
||||||
v, err := ParseAmount(raw, p.cfg.Amount.Decimal, p.cfg.Amount.Thousands, p.digits)
|
|
||||||
if err != nil {
|
|
||||||
return t, err
|
|
||||||
}
|
|
||||||
t.AmountMinor = v
|
|
||||||
default:
|
|
||||||
// Debit/credit pair: exactly one of the two carries a value, and both
|
|
||||||
// are written as positive numbers.
|
|
||||||
debit, err := p.optional(rec, p.cfg.Debit)
|
|
||||||
if err != nil {
|
|
||||||
return t, fmt.Errorf("debit column: %w", err)
|
|
||||||
}
|
|
||||||
credit, err := p.optional(rec, p.cfg.Credit)
|
|
||||||
if err != nil {
|
|
||||||
return t, fmt.Errorf("credit column: %w", err)
|
|
||||||
}
|
|
||||||
if debit != 0 && credit != 0 {
|
|
||||||
return t, fmt.Errorf("both debit (%d) and credit (%d) are set", debit, credit)
|
|
||||||
}
|
|
||||||
t.AmountMinor = credit - abs(debit)
|
|
||||||
}
|
|
||||||
|
|
||||||
if p.cfg.Invert {
|
|
||||||
t.AmountMinor = -t.AmountMinor
|
|
||||||
}
|
|
||||||
return t, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// optional parses a column that may legitimately be blank, as debit/credit
|
|
||||||
// columns always are for half the rows.
|
|
||||||
func (p *csvParser) optional(rec []string, c *config.Column) (int64, error) {
|
|
||||||
if c == nil {
|
|
||||||
return 0, nil
|
|
||||||
}
|
|
||||||
raw, err := field(rec, c.Col)
|
|
||||||
if err != nil {
|
|
||||||
return 0, err
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(raw) == "" {
|
|
||||||
return 0, nil
|
|
||||||
}
|
|
||||||
return ParseAmount(raw, c.Decimal, c.Thousands, p.digits)
|
|
||||||
}
|
|
||||||
|
|
||||||
func field(rec []string, i int) (string, error) {
|
|
||||||
if i < 0 || i >= len(rec) {
|
|
||||||
return "", fmt.Errorf("index %d out of range, row has %d columns", i, len(rec))
|
|
||||||
}
|
|
||||||
return rec[i], nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func isBlank(rec []string) bool {
|
|
||||||
for _, f := range rec {
|
|
||||||
if strings.TrimSpace(f) != "" {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
func abs(v int64) int64 {
|
|
||||||
if v < 0 {
|
|
||||||
return -v
|
|
||||||
}
|
|
||||||
return v
|
|
||||||
}
|
|
||||||
@@ -1,112 +0,0 @@
|
|||||||
package parser
|
|
||||||
|
|
||||||
import (
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"git.petrovv.com/nikola/money/internal/config"
|
|
||||||
)
|
|
||||||
|
|
||||||
func writeFile(t *testing.T, name, content string) string {
|
|
||||||
t.Helper()
|
|
||||||
path := filepath.Join(t.TempDir(), name)
|
|
||||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
return path
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCSVSignedAmountColumn(t *testing.T) {
|
|
||||||
path := writeFile(t, "st.csv", `Date,Ref,Description,Amount
|
|
||||||
02.01.2026,X1,LIDL SOFIA 1234,"-45,20"
|
|
||||||
03.01.2026,X2,ACME PAYROLL,"1.500,00"
|
|
||||||
|
|
||||||
04.01.2026,X3,COFFEE,"-3,50"
|
|
||||||
`)
|
|
||||||
acc := &config.Account{Slug: "checking", Currency: "EUR", Parser: "csv", CSV: &config.CSVConfig{
|
|
||||||
SkipRows: 1,
|
|
||||||
Date: config.Column{Col: 0, Layout: "02.01.2006"},
|
|
||||||
Description: config.Column{Col: 2},
|
|
||||||
Amount: &config.Column{Col: 3, Decimal: ",", Thousands: "."},
|
|
||||||
}}
|
|
||||||
p, err := For(acc)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
got, err := p.Parse(path, acc)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
want := []RawTxn{
|
|
||||||
{Date: "2026-01-02", Description: "LIDL SOFIA 1234", AmountMinor: -4520},
|
|
||||||
{Date: "2026-01-03", Description: "ACME PAYROLL", AmountMinor: 150000},
|
|
||||||
{Date: "2026-01-04", Description: "COFFEE", AmountMinor: -350},
|
|
||||||
}
|
|
||||||
if len(got) != len(want) {
|
|
||||||
t.Fatalf("got %d txns, want %d: %+v", len(got), len(want), got)
|
|
||||||
}
|
|
||||||
for i := range want {
|
|
||||||
if got[i] != want[i] {
|
|
||||||
t.Errorf("txn %d = %+v, want %+v", i, got[i], want[i])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCSVDebitCreditPair(t *testing.T) {
|
|
||||||
path := writeFile(t, "st.csv", `2026-02-01;RENT;800.00;
|
|
||||||
2026-02-05;SALARY;;2500.00
|
|
||||||
`)
|
|
||||||
acc := &config.Account{Slug: "checking", Currency: "EUR", Parser: "csv", CSV: &config.CSVConfig{
|
|
||||||
Delimiter: ";",
|
|
||||||
Date: config.Column{Col: 0, Layout: "2006-01-02"},
|
|
||||||
Description: config.Column{Col: 1},
|
|
||||||
Debit: &config.Column{Col: 2},
|
|
||||||
Credit: &config.Column{Col: 3},
|
|
||||||
}}
|
|
||||||
p, err := For(acc)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
got, err := p.Parse(path, acc)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
if len(got) != 2 {
|
|
||||||
t.Fatalf("got %d txns, want 2: %+v", len(got), got)
|
|
||||||
}
|
|
||||||
if got[0].AmountMinor != -80000 {
|
|
||||||
t.Errorf("debit row = %d, want -80000", got[0].AmountMinor)
|
|
||||||
}
|
|
||||||
if got[1].AmountMinor != 250000 {
|
|
||||||
t.Errorf("credit row = %d, want 250000", got[1].AmountMinor)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCSVConfigErrors(t *testing.T) {
|
|
||||||
cases := map[string]*config.Account{
|
|
||||||
"no csv section": {Slug: "a", Parser: "csv"},
|
|
||||||
"no amount columns": {Slug: "a", Parser: "csv", CSV: &config.CSVConfig{
|
|
||||||
Date: config.Column{Layout: "2006-01-02"},
|
|
||||||
}},
|
|
||||||
"amount and debit together": {Slug: "a", Parser: "csv", CSV: &config.CSVConfig{
|
|
||||||
Date: config.Column{Layout: "2006-01-02"},
|
|
||||||
Amount: &config.Column{Col: 1},
|
|
||||||
Debit: &config.Column{Col: 2},
|
|
||||||
}},
|
|
||||||
"no date layout": {Slug: "a", Parser: "csv", CSV: &config.CSVConfig{
|
|
||||||
Amount: &config.Column{Col: 1},
|
|
||||||
}},
|
|
||||||
}
|
|
||||||
for name, acc := range cases {
|
|
||||||
if _, err := For(acc); err == nil {
|
|
||||||
t.Errorf("%s: expected an error", name)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestUnknownParser(t *testing.T) {
|
|
||||||
if _, err := For(&config.Account{Slug: "a", Parser: "nope"}); err == nil {
|
|
||||||
t.Error("expected an error for an unregistered parser")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+19
-12
@@ -18,8 +18,8 @@ func init() {
|
|||||||
// nlbParser reads an NLB izpisek PDF.
|
// nlbParser reads an NLB izpisek PDF.
|
||||||
//
|
//
|
||||||
// A transaction starts on a line beginning with a dd.mm.yy date and ends with
|
// A transaction starts on a line beginning with a dd.mm.yy date and ends with
|
||||||
// the signed amount and the running balance. Long descriptions and the
|
// the signed amount and the running balance. Long descriptions and the other
|
||||||
// counterparty account wrap onto indented continuation lines below.
|
// side's account number wrap onto indented continuation lines below.
|
||||||
type nlbParser struct {
|
type nlbParser struct {
|
||||||
digits int
|
digits int
|
||||||
}
|
}
|
||||||
@@ -37,7 +37,9 @@ var (
|
|||||||
`(?P<balance>[-\x{2212}]?[\d.,]+[-\x{2212}]?)\s*$`)
|
`(?P<balance>[-\x{2212}]?[\d.,]+[-\x{2212}]?)\s*$`)
|
||||||
// Columns inside a line are separated by four or more spaces.
|
// Columns inside a line are separated by four or more spaces.
|
||||||
nlbFieldSplit = regexp.MustCompile(`\s{4,}`)
|
nlbFieldSplit = regexp.MustCompile(`\s{4,}`)
|
||||||
// A Slovenian IBAN, which is the counterparty account when present.
|
// A Slovenian IBAN, which NLB prints in a column of its own. It is split
|
||||||
|
// out only so that its wrapped fragments can be rejoined; the result goes
|
||||||
|
// onto the end of the description, where every other parser keeps it.
|
||||||
nlbAccount = regexp.MustCompile(`^SI\d{2}(?:\s?\d{4}){3}\s?\d{3}$`)
|
nlbAccount = regexp.MustCompile(`^SI\d{2}(?:\s?\d{4}){3}\s?\d{3}$`)
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -63,9 +65,12 @@ func (p *nlbParser) Parse(path string, acc *config.Account) ([]RawTxn, error) {
|
|||||||
// be tested against captured pdftotext output.
|
// be tested against captured pdftotext output.
|
||||||
func parseNLBText(text string, digits int) ([]RawTxn, error) {
|
func parseNLBText(text string, digits int) ([]RawTxn, error) {
|
||||||
var (
|
var (
|
||||||
txns []RawTxn
|
txns []RawTxn
|
||||||
accts []string // counterparty per transaction, built up alongside
|
// The account column per transaction, built up alongside because it
|
||||||
descAt int // description column of the last transaction line
|
// wraps in fragments of its own and has to be rejoined before it can
|
||||||
|
// be appended to the description.
|
||||||
|
trailing []string
|
||||||
|
descAt int // description column of the last transaction line
|
||||||
)
|
)
|
||||||
|
|
||||||
for _, page := range pages(text) {
|
for _, page := range pages(text) {
|
||||||
@@ -84,7 +89,7 @@ func parseNLBText(text string, digits int) ([]RawTxn, error) {
|
|||||||
return nil, fmt.Errorf("line %d: %w", i+1, err)
|
return nil, fmt.Errorf("line %d: %w", i+1, err)
|
||||||
}
|
}
|
||||||
txns = append(txns, txn)
|
txns = append(txns, txn)
|
||||||
accts = append(accts, account)
|
trailing = append(trailing, account)
|
||||||
descAt = col
|
descAt = col
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -104,20 +109,22 @@ func parseNLBText(text string, digits int) ([]RawTxn, error) {
|
|||||||
last := len(txns) - 1
|
last := len(txns) - 1
|
||||||
txns[last].Description += " " + parts[0]
|
txns[last].Description += " " + parts[0]
|
||||||
if len(parts) > 1 {
|
if len(parts) > 1 {
|
||||||
accts[last] = strings.TrimSpace(accts[last] + " " + parts[1])
|
trailing[last] = strings.TrimSpace(trailing[last] + " " + parts[1])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The account column goes last rather than in the position it occupied on
|
||||||
|
// the page, so that an IBAN wrapped over several lines stays contiguous
|
||||||
|
// and a glob can match it.
|
||||||
for i := range txns {
|
for i := range txns {
|
||||||
txns[i].Counterparty = accts[i]
|
txns[i].Description = strings.Join(strings.Fields(txns[i].Description+" "+trailing[i]), " ")
|
||||||
txns[i].Description = strings.Join(strings.Fields(txns[i].Description), " ")
|
|
||||||
}
|
}
|
||||||
return txns, nil
|
return txns, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseNLBLine parses one transaction line, returning the transaction, the
|
// parseNLBLine parses one transaction line, returning the transaction, the
|
||||||
// counterparty account found among its columns, and the column at which the
|
// account number found among its columns, and the column at which the
|
||||||
// description starts, which is where its wrapped lines will sit.
|
// description starts, which is where its wrapped lines will sit.
|
||||||
func parseNLBLine(line string, digits int) (RawTxn, string, int, error) {
|
func parseNLBLine(line string, digits int) (RawTxn, string, int, error) {
|
||||||
trimmed := strings.TrimRight(line, " \t\r")
|
trimmed := strings.TrimRight(line, " \t\r")
|
||||||
@@ -154,7 +161,7 @@ func parseNLBLine(line string, digits int) (RawTxn, string, int, error) {
|
|||||||
return RawTxn{}, "", 0, fmt.Errorf("balance %q: %w", balance, err)
|
return RawTxn{}, "", 0, fmt.Errorf("balance %q: %w", balance, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The middle holds the description and, sometimes, the counterparty IBAN.
|
// The middle holds the description and, sometimes, the IBAN.
|
||||||
var (
|
var (
|
||||||
account string
|
account string
|
||||||
desc []string
|
desc []string
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import "testing"
|
|||||||
|
|
||||||
// A page as pdftotext -layout renders it: a header, transaction lines ending
|
// A page as pdftotext -layout renders it: a header, transaction lines ending
|
||||||
// in amount and balance, and indented continuation lines carrying wrapped
|
// in amount and balance, and indented continuation lines carrying wrapped
|
||||||
// descriptions and the counterparty IBAN.
|
// descriptions and the other side's IBAN.
|
||||||
// The description column sits at 15, which is what the script's
|
// The description column sits at 15, which is what the script's
|
||||||
// CONTINUATION_INDENT tells us about the real layout.
|
// CONTINUATION_INDENT tells us about the real layout.
|
||||||
const nlbPage = ` NLB d.d.
|
const nlbPage = ` NLB d.d.
|
||||||
@@ -35,14 +35,12 @@ func TestParseNLBText(t *testing.T) {
|
|||||||
if first.Date != "2026-01-02" {
|
if first.Date != "2026-01-02" {
|
||||||
t.Errorf("date = %q, want 2026-01-02", first.Date)
|
t.Errorf("date = %q, want 2026-01-02", first.Date)
|
||||||
}
|
}
|
||||||
// The continuation line is appended to the description.
|
// The continuation line is appended to the description, and the IBAN in
|
||||||
if first.Description != "PLACILO S KARTICO LIDL SOFIA 4412" {
|
// its second column goes last so that it stays contiguous however many
|
||||||
|
// lines it wrapped over.
|
||||||
|
if first.Description != "PLACILO S KARTICO LIDL SOFIA 4412 SI56 1234 5678 9012 345" {
|
||||||
t.Errorf("description = %q", first.Description)
|
t.Errorf("description = %q", first.Description)
|
||||||
}
|
}
|
||||||
// ...and the IBAN in its second column becomes the counterparty.
|
|
||||||
if first.Counterparty != "SI56 1234 5678 9012 345" {
|
|
||||||
t.Errorf("counterparty = %q", first.Counterparty)
|
|
||||||
}
|
|
||||||
if first.AmountMinor != -4520 {
|
if first.AmountMinor != -4520 {
|
||||||
t.Errorf("amount = %d, want -4520", first.AmountMinor)
|
t.Errorf("amount = %d, want -4520", first.AmountMinor)
|
||||||
}
|
}
|
||||||
@@ -57,9 +55,9 @@ func TestParseNLBText(t *testing.T) {
|
|||||||
t.Errorf("description = %q", txns[1].Description)
|
t.Errorf("description = %q", txns[1].Description)
|
||||||
}
|
}
|
||||||
|
|
||||||
// A transaction with no continuation line keeps an empty counterparty.
|
// A transaction with no continuation line gets nothing appended.
|
||||||
if txns[2].Counterparty != "" {
|
if txns[2].Description != "PRENOS NA VARCEVALNI" {
|
||||||
t.Errorf("counterparty = %q, want empty", txns[2].Counterparty)
|
t.Errorf("description = %q, want no trailing account number", txns[2].Description)
|
||||||
}
|
}
|
||||||
|
|
||||||
// A trailing minus marks a negative balance.
|
// A trailing minus marks a negative balance.
|
||||||
|
|||||||
@@ -1,11 +1,10 @@
|
|||||||
// Package parser turns a statement file into raw transactions.
|
// Package parser turns a statement file into raw transactions.
|
||||||
//
|
//
|
||||||
// Every bank needs its own extraction logic, so parsers are looked up by name
|
// Every bank needs its own extraction logic, so parsers are looked up by name
|
||||||
// from a registry. Two are built in: "csv" for delimited exports with
|
// from a registry that each one joins from an init. A parser becomes usable by
|
||||||
// configurable columns, and "cmd" for shelling out to an external extractor
|
// putting its name in an account.toml; there is no generic column-mapped
|
||||||
// (which is how the existing Python scripts are used until they are ported).
|
// parser, because a statement layout is better described in Go, where it can
|
||||||
// Ported extractors register themselves here and become usable by putting
|
// be tested, than in a table of column indexes.
|
||||||
// their name in an account.toml.
|
|
||||||
package parser
|
package parser
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -23,9 +22,6 @@ type RawTxn struct {
|
|||||||
Description string
|
Description string
|
||||||
AmountMinor int64 // signed; negative is an outflow
|
AmountMinor int64 // signed; negative is an outflow
|
||||||
|
|
||||||
// Counterparty is the other side's account number, where the statement
|
|
||||||
// gives one. Optional.
|
|
||||||
Counterparty string
|
|
||||||
// Type is the bank's own classification of the transaction. Optional.
|
// Type is the bank's own classification of the transaction. Optional.
|
||||||
Type string
|
Type string
|
||||||
// BalanceMinor is the running balance after this transaction, when the
|
// BalanceMinor is the running balance after this transaction, when the
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"regexp"
|
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@@ -36,9 +35,6 @@ type revolutParser struct {
|
|||||||
warnings []string
|
warnings []string
|
||||||
}
|
}
|
||||||
|
|
||||||
// revolutIBAN finds a counterparty account inside a description.
|
|
||||||
var revolutIBAN = regexp.MustCompile(`\b[A-Z]{2}\d{2}[A-Z0-9]{11,30}\b`)
|
|
||||||
|
|
||||||
// Columns the parser needs; a missing one is a hard error rather than a
|
// Columns the parser needs; a missing one is a hard error rather than a
|
||||||
// silently empty field.
|
// silently empty field.
|
||||||
var revolutColumns = []string{
|
var revolutColumns = []string{
|
||||||
@@ -164,18 +160,26 @@ func (p *revolutParser) row(get func(string) string, digits int) (RawTxn, error)
|
|||||||
desc += fmt.Sprintf(" (fee %s)", model.FormatMinor(fee, digits))
|
desc += fmt.Sprintf(" (fee %s)", model.FormatMinor(fee, digits))
|
||||||
}
|
}
|
||||||
|
|
||||||
counterparty := revolutIBAN.FindString(desc)
|
|
||||||
|
|
||||||
return RawTxn{
|
return RawTxn{
|
||||||
Date: d.Format("2006-01-02"),
|
Date: d.Format("2006-01-02"),
|
||||||
Description: desc,
|
Description: desc,
|
||||||
AmountMinor: amount - fee,
|
AmountMinor: amount - fee,
|
||||||
Counterparty: counterparty,
|
|
||||||
Type: strings.TrimSpace(get("Type")),
|
Type: strings.TrimSpace(get("Type")),
|
||||||
BalanceMinor: &balance,
|
BalanceMinor: &balance,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// isBlank reports whether a CSV record holds nothing but whitespace, which is
|
||||||
|
// what a trailing newline in the export reads as.
|
||||||
|
func isBlank(rec []string) bool {
|
||||||
|
for _, f := range rec {
|
||||||
|
if strings.TrimSpace(f) != "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
// headerIndex maps a Revolut column name to its position.
|
// headerIndex maps a Revolut column name to its position.
|
||||||
type headerIndex map[string]int
|
type headerIndex map[string]int
|
||||||
|
|
||||||
|
|||||||
@@ -1,12 +1,24 @@
|
|||||||
package parser
|
package parser
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"git.petrovv.com/nikola/money/internal/config"
|
"git.petrovv.com/nikola/money/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// writeFile drops a statement in a temporary directory and returns its path.
|
||||||
|
func writeFile(t *testing.T, name, content string) string {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(t.TempDir(), name)
|
||||||
|
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
// A Revolut export: out of date order, mixed currencies, a pending row, a row
|
// A Revolut export: out of date order, mixed currencies, a pending row, a row
|
||||||
// with a fee, and a transfer carrying an IBAN.
|
// with a fee, and a transfer carrying an IBAN.
|
||||||
const revolutCSV = `Type,Product,Started Date,Completed Date,Description,Amount,Fee,Currency,State,Balance
|
const revolutCSV = `Type,Product,Started Date,Completed Date,Description,Amount,Fee,Currency,State,Balance
|
||||||
@@ -60,9 +72,6 @@ func TestRevolutParse(t *testing.T) {
|
|||||||
if !strings.Contains(transfer.Description, "(fee 0.35)") {
|
if !strings.Contains(transfer.Description, "(fee 0.35)") {
|
||||||
t.Errorf("description = %q, want a fee note", transfer.Description)
|
t.Errorf("description = %q, want a fee note", transfer.Description)
|
||||||
}
|
}
|
||||||
if transfer.Counterparty != "SI56123456789012345" {
|
|
||||||
t.Errorf("counterparty = %q, want the IBAN from the description", transfer.Counterparty)
|
|
||||||
}
|
|
||||||
if transfer.BalanceMinor == nil || *transfer.BalanceMinor != 73441 {
|
if transfer.BalanceMinor == nil || *transfer.BalanceMinor != 73441 {
|
||||||
t.Errorf("balance = %v, want 73441", transfer.BalanceMinor)
|
t.Errorf("balance = %v, want 73441", transfer.BalanceMinor)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -32,7 +32,6 @@ var (
|
|||||||
trAmount = regexp.MustCompile(`[-\x{2212}]?€\s?[-\x{2212}]?[\d,]+\.\d{2}`)
|
trAmount = regexp.MustCompile(`[-\x{2212}]?€\s?[-\x{2212}]?[\d,]+\.\d{2}`)
|
||||||
trToken = regexp.MustCompile(`\S+`)
|
trToken = regexp.MustCompile(`\S+`)
|
||||||
trFullDate = regexp.MustCompile(`^\d{2} [A-Z][a-z]{2} \d{4}$`)
|
trFullDate = regexp.MustCompile(`^\d{2} [A-Z][a-z]{2} \d{4}$`)
|
||||||
trIBAN = regexp.MustCompile(`\b[A-Z]{2}\d{2}[A-Z0-9]{11,30}\b`)
|
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@@ -197,7 +196,6 @@ func parseTradeRepublicBlock(lines []string, cols map[string]int, digits int) (R
|
|||||||
Date: d.Format("2006-01-02"),
|
Date: d.Format("2006-01-02"),
|
||||||
Description: desc,
|
Description: desc,
|
||||||
AmountMinor: amount,
|
AmountMinor: amount,
|
||||||
Counterparty: trIBAN.FindString(desc),
|
|
||||||
Type: strings.Join(words["TYPE"], " "),
|
Type: strings.Join(words["TYPE"], " "),
|
||||||
BalanceMinor: amounts["BALANCE"],
|
BalanceMinor: amounts["BALANCE"],
|
||||||
}, true, nil
|
}, true, nil
|
||||||
|
|||||||
@@ -120,9 +120,6 @@ func TestParseTradeRepublicText(t *testing.T) {
|
|||||||
if transfer.Description != "Standing order to sav-ings SI56123456789012345" {
|
if transfer.Description != "Standing order to sav-ings SI56123456789012345" {
|
||||||
t.Errorf("description = %q, want the wrapped fragment joined without a space", transfer.Description)
|
t.Errorf("description = %q, want the wrapped fragment joined without a space", transfer.Description)
|
||||||
}
|
}
|
||||||
if transfer.Counterparty != "SI56123456789012345" {
|
|
||||||
t.Errorf("counterparty = %q", transfer.Counterparty)
|
|
||||||
}
|
|
||||||
if transfer.AmountMinor != -50000 {
|
if transfer.AmountMinor != -50000 {
|
||||||
t.Errorf("amount = %d, want -50000", transfer.AmountMinor)
|
t.Errorf("amount = %d, want -50000", transfer.AmountMinor)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -37,16 +37,15 @@ func (e *Engine) Match(accountSlug string, t model.Transaction) *config.Rule {
|
|||||||
|
|
||||||
// MatchIndex returns the position of the first rule matching a transaction, or
|
// MatchIndex returns the position of the first rule matching a transaction, or
|
||||||
// -1 if none does. Every pattern a rule sets must match: a rule with both
|
// -1 if none does. Every pattern a rule sets must match: a rule with both
|
||||||
// match and counterparty is an "and", not an "or".
|
// match and type is an "and", not an "or".
|
||||||
//
|
//
|
||||||
// The position matters as well as the rule: because the first match wins, a
|
// The position matters as well as the rule: because the first match wins, a
|
||||||
// rule that is fully shadowed by an earlier one never applies to anything, and
|
// rule that is fully shadowed by an earlier one never applies to anything, and
|
||||||
// only the index reveals that.
|
// only the index reveals that.
|
||||||
func (e *Engine) MatchIndex(accountSlug string, t model.Transaction) int {
|
func (e *Engine) MatchIndex(accountSlug string, t model.Transaction) int {
|
||||||
var (
|
var (
|
||||||
description = model.NormalizeDescription(t.Description)
|
description = model.NormalizeDescription(t.Description)
|
||||||
counterparty = model.NormalizeDescription(t.Counterparty)
|
kind = model.NormalizeDescription(t.Type)
|
||||||
kind = model.NormalizeDescription(t.Type)
|
|
||||||
)
|
)
|
||||||
for i := range e.rules {
|
for i := range e.rules {
|
||||||
r := &e.rules[i]
|
r := &e.rules[i]
|
||||||
@@ -56,9 +55,6 @@ func (e *Engine) MatchIndex(accountSlug string, t model.Transaction) int {
|
|||||||
if r.Match != "" && !glob.Match(r.Match, description) {
|
if r.Match != "" && !glob.Match(r.Match, description) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if r.Counterparty != "" && !glob.Match(r.Counterparty, counterparty) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if r.Type != "" && !glob.Match(r.Type, kind) {
|
if r.Type != "" && !glob.Match(r.Type, kind) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -201,25 +201,16 @@ func TestAccountScopedRule(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Movements between the user's own accounts are often only identifiable by
|
// Besides the description, a rule can match the bank's own classification of
|
||||||
// the counterparty IBAN, so rules can match on it.
|
// the transaction, on its own or alongside a description glob.
|
||||||
func TestCounterpartyAndTypeRules(t *testing.T) {
|
func TestTypeRules(t *testing.T) {
|
||||||
engine := New(&config.Rules{Rule: []config.Rule{
|
engine := New(&config.Rules{Rule: []config.Rule{
|
||||||
{Counterparty: "SI56123456789012345", Tag: "transfer", Transfer: true},
|
|
||||||
{Type: "CARD_PAYMENT", Match: "*LIDL*", Tag: "groceries"},
|
{Type: "CARD_PAYMENT", Match: "*LIDL*", Tag: "groceries"},
|
||||||
{Type: "ATM", Tag: "cash"},
|
{Type: "ATM", Tag: "cash"},
|
||||||
}})
|
}})
|
||||||
|
|
||||||
// Counterparty alone is enough, whatever the description says.
|
|
||||||
tag, transfer := engine.ApplyTxn("checking", model.Transaction{
|
|
||||||
Description: "Standing order", Counterparty: "SI56123456789012345",
|
|
||||||
})
|
|
||||||
if tag != "transfer" || !transfer {
|
|
||||||
t.Errorf("tag=%q transfer=%v, want transfer/true", tag, transfer)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A rule setting several patterns requires all of them to match.
|
// A rule setting several patterns requires all of them to match.
|
||||||
tag, _ = engine.ApplyTxn("checking", model.Transaction{
|
tag, _ := engine.ApplyTxn("checking", model.Transaction{
|
||||||
Description: "LIDL SOFIA", Type: "CARD_PAYMENT",
|
Description: "LIDL SOFIA", Type: "CARD_PAYMENT",
|
||||||
})
|
})
|
||||||
if tag != "groceries" {
|
if tag != "groceries" {
|
||||||
|
|||||||
+24
-11
@@ -1,6 +1,6 @@
|
|||||||
// Package store is the SQLite index over the statements. It is entirely
|
// Package store is the SQLite index over the statements. It is entirely
|
||||||
// rebuildable: delete .money/index.db and re-import to get it back, except for
|
// rebuildable: delete index.db and re-import to get it back, except for manual
|
||||||
// manual tags and manual transfer overrides, which live only here.
|
// tags and manual transfer overrides, which live only here.
|
||||||
package store
|
package store
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -48,7 +48,6 @@ CREATE TABLE IF NOT EXISTS transactions (
|
|||||||
date TEXT NOT NULL,
|
date TEXT NOT NULL,
|
||||||
description TEXT NOT NULL,
|
description TEXT NOT NULL,
|
||||||
amount_minor INTEGER NOT NULL,
|
amount_minor INTEGER NOT NULL,
|
||||||
counterparty TEXT NOT NULL DEFAULT '',
|
|
||||||
type TEXT NOT NULL DEFAULT '',
|
type TEXT NOT NULL DEFAULT '',
|
||||||
balance_minor INTEGER,
|
balance_minor INTEGER,
|
||||||
rule_tag TEXT,
|
rule_tag TEXT,
|
||||||
@@ -65,12 +64,17 @@ CREATE INDEX IF NOT EXISTS idx_txn_account ON transactions(account_id);
|
|||||||
// migrations bring an index created by an older build up to date. SQLite
|
// migrations bring an index created by an older build up to date. SQLite
|
||||||
// errors on a duplicate column, which is how we detect "already applied".
|
// errors on a duplicate column, which is how we detect "already applied".
|
||||||
var migrations = []string{
|
var migrations = []string{
|
||||||
`ALTER TABLE transactions ADD COLUMN counterparty TEXT NOT NULL DEFAULT ''`,
|
|
||||||
`ALTER TABLE transactions ADD COLUMN type TEXT NOT NULL DEFAULT ''`,
|
`ALTER TABLE transactions ADD COLUMN type TEXT NOT NULL DEFAULT ''`,
|
||||||
`ALTER TABLE transactions ADD COLUMN balance_minor INTEGER`,
|
`ALTER TABLE transactions ADD COLUMN balance_minor INTEGER`,
|
||||||
}
|
}
|
||||||
|
|
||||||
// migrate applies any column that this index is missing.
|
// dropped are columns an older build created that this one no longer reads.
|
||||||
|
// Nothing indexes or constrains them, so they can simply go; leaving them
|
||||||
|
// would keep a NOT NULL column alive that no INSERT here ever names.
|
||||||
|
var dropped = []string{"counterparty"}
|
||||||
|
|
||||||
|
// migrate brings an index created by an older build up to date, adding the
|
||||||
|
// columns it lacks and removing the ones it should no longer have.
|
||||||
func migrate(db *sql.DB) error {
|
func migrate(db *sql.DB) error {
|
||||||
have, err := columns(db, "transactions")
|
have, err := columns(db, "transactions")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -85,6 +89,15 @@ func migrate(db *sql.DB) error {
|
|||||||
return fmt.Errorf("migrate (%s): %w", stmt, err)
|
return fmt.Errorf("migrate (%s): %w", stmt, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for _, name := range dropped {
|
||||||
|
if !have[name] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
stmt := fmt.Sprintf(`ALTER TABLE transactions DROP COLUMN %s`, name)
|
||||||
|
if _, err := db.Exec(stmt); err != nil {
|
||||||
|
return fmt.Errorf("migrate (%s): %w", stmt, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -118,7 +131,7 @@ func addedColumn(stmt string) string {
|
|||||||
// Open opens (creating if needed) the index at path.
|
// Open opens (creating if needed) the index at path.
|
||||||
func Open(path string) (*DB, error) {
|
func Open(path string) (*DB, error) {
|
||||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||||
return nil, fmt.Errorf("create state dir: %w", err)
|
return nil, fmt.Errorf("create index dir: %w", err)
|
||||||
}
|
}
|
||||||
sqlDB, err := sql.Open("sqlite", path)
|
sqlDB, err := sql.Open("sqlite", path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -224,11 +237,11 @@ func (d *DB) InsertTransaction(t model.Transaction) (bool, error) {
|
|||||||
res, err := d.sql.Exec(`
|
res, err := d.sql.Exec(`
|
||||||
INSERT INTO transactions
|
INSERT INTO transactions
|
||||||
(account_id, source_file_id, fingerprint, date, description, amount_minor,
|
(account_id, source_file_id, fingerprint, date, description, amount_minor,
|
||||||
counterparty, type, balance_minor, rule_tag, rule_transfer)
|
type, balance_minor, rule_tag, rule_transfer)
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, NULLIF(?, ''), ?)
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, NULLIF(?, ''), ?)
|
||||||
ON CONFLICT(account_id, fingerprint) DO NOTHING`,
|
ON CONFLICT(account_id, fingerprint) DO NOTHING`,
|
||||||
t.AccountID, t.SourceFileID, t.Fingerprint, t.Date, t.Description,
|
t.AccountID, t.SourceFileID, t.Fingerprint, t.Date, t.Description,
|
||||||
t.AmountMinor, t.Counterparty, t.Type, balance, t.RuleTag, boolToInt(t.RuleTransfer))
|
t.AmountMinor, t.Type, balance, t.RuleTag, boolToInt(t.RuleTransfer))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, fmt.Errorf("insert transaction: %w", err)
|
return false, fmt.Errorf("insert transaction: %w", err)
|
||||||
}
|
}
|
||||||
@@ -253,7 +266,7 @@ func (d *DB) Transactions(f Filter) ([]model.Transaction, error) {
|
|||||||
q := `
|
q := `
|
||||||
SELECT t.id, t.account_id, a.slug, a.currency, a.minor_digits,
|
SELECT t.id, t.account_id, a.slug, a.currency, a.minor_digits,
|
||||||
t.fingerprint, t.date, t.description, t.amount_minor,
|
t.fingerprint, t.date, t.description, t.amount_minor,
|
||||||
COALESCE(s.path, ''), t.counterparty, t.type, t.balance_minor,
|
COALESCE(s.path, ''), t.type, t.balance_minor,
|
||||||
COALESCE(t.rule_tag, ''), COALESCE(t.manual_tag, ''),
|
COALESCE(t.rule_tag, ''), COALESCE(t.manual_tag, ''),
|
||||||
t.rule_transfer, t.manual_transfer
|
t.rule_transfer, t.manual_transfer
|
||||||
FROM transactions t
|
FROM transactions t
|
||||||
@@ -293,7 +306,7 @@ func (d *DB) Transactions(f Filter) ([]model.Transaction, error) {
|
|||||||
)
|
)
|
||||||
if err := rows.Scan(&t.ID, &t.AccountID, &t.AccountSlug, &t.Currency, &t.MinorDigits,
|
if err := rows.Scan(&t.ID, &t.AccountID, &t.AccountSlug, &t.Currency, &t.MinorDigits,
|
||||||
&t.Fingerprint, &t.Date, &t.Description, &t.AmountMinor, &t.SourcePath,
|
&t.Fingerprint, &t.Date, &t.Description, &t.AmountMinor, &t.SourcePath,
|
||||||
&t.Counterparty, &t.Type, &balance,
|
&t.Type, &balance,
|
||||||
&t.RuleTag, &t.ManualTag, &ruleTransfer, &manualTransfer); err != nil {
|
&t.RuleTag, &t.ManualTag, &ruleTransfer, &manualTransfer); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-5
@@ -599,16 +599,13 @@ func (m *Model) reloadRuleList() error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// rulePattern renders whichever patterns a rule sets, labelled so a
|
// rulePattern renders whichever patterns a rule sets, labelled so a type rule
|
||||||
// counterparty or type rule is not mistaken for a description one.
|
// is not mistaken for a description one.
|
||||||
func rulePattern(r config.Rule) string {
|
func rulePattern(r config.Rule) string {
|
||||||
var parts []string
|
var parts []string
|
||||||
if r.Match != "" {
|
if r.Match != "" {
|
||||||
parts = append(parts, r.Match)
|
parts = append(parts, r.Match)
|
||||||
}
|
}
|
||||||
if r.Counterparty != "" {
|
|
||||||
parts = append(parts, "counterparty:"+r.Counterparty)
|
|
||||||
}
|
|
||||||
if r.Type != "" {
|
if r.Type != "" {
|
||||||
parts = append(parts, "type:"+r.Type)
|
parts = append(parts, "type:"+r.Type)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -388,7 +388,7 @@ func newRuleModel(t *testing.T) (*Model, *store.DB, string) {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
|
|
||||||
db, err := store.Open(filepath.Join(root, ".money", "index.db"))
|
db, err := store.Open(config.IndexPath(root))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -437,8 +437,8 @@ func newRuleModel(t *testing.T) (*Model, *store.DB, string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
m := New(root, db, []*config.Account{
|
m := New(root, db, []*config.Account{
|
||||||
{Slug: "checking", Currency: "EUR", Parser: "csv"},
|
{Slug: "checking", Currency: "EUR", Parser: "revolut"},
|
||||||
{Slug: "savings", Currency: "EUR", Parser: "csv"},
|
{Slug: "savings", Currency: "EUR", Parser: "revolut"},
|
||||||
}, rules.New(&config.Rules{}))
|
}, rules.New(&config.Rules{}))
|
||||||
if err := m.reload(); err != nil {
|
if err := m.reload(); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -1125,8 +1125,8 @@ func newEmptyModel(t *testing.T, accounts []*config.Account) *Model {
|
|||||||
// disk, but nothing is registered in the index until an import runs.
|
// disk, but nothing is registered in the index until an import runs.
|
||||||
func TestEmptyAccountsWithConfiguredFolders(t *testing.T) {
|
func TestEmptyAccountsWithConfiguredFolders(t *testing.T) {
|
||||||
m := newEmptyModel(t, []*config.Account{
|
m := newEmptyModel(t, []*config.Account{
|
||||||
{Slug: "checking", Currency: "EUR", Parser: "csv"},
|
{Slug: "checking", Currency: "EUR", Parser: "revolut"},
|
||||||
{Slug: "savings", Currency: "EUR", Parser: "csv"},
|
{Slug: "savings", Currency: "EUR", Parser: "revolut"},
|
||||||
})
|
})
|
||||||
|
|
||||||
view := m.View()
|
view := m.View()
|
||||||
|
|||||||
Reference in New Issue
Block a user