Add job board scraper: modular scrapers, SQLite storage, web UI
Scrapes slo-tech.com (Ljubljana) every 24h, stores jobs in SQLite and tracks which are new vs. seen. New sites can be added as modules in src/scrapers/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,41 @@
|
||||
# Job Board
|
||||
|
||||
Scrapes job sites once a day, stores results in SQLite (`data/jobs.sqlite`), and shows them in a web UI where you can mark jobs as seen.
|
||||
|
||||
No runtime dependencies. It uses Bun's built-in SQLite, HTTP server and HTMLRewriter.
|
||||
|
||||
## Run
|
||||
|
||||
```sh
|
||||
bun install
|
||||
bun start # http://localhost:3000
|
||||
```
|
||||
|
||||
The server scrapes every 24h. On startup it waits out the rest of the 24h since the last scrape (or scrapes right away if none happened in the last 24h).
|
||||
You can also click **Scrape now** in the UI, or run `bun run scrape` once (e.g. from cron).
|
||||
|
||||
Env vars: `PORT` (3000), `SCRAPE_INTERVAL_HOURS` (24), `DB_PATH` (`data/jobs.sqlite`).
|
||||
|
||||
## UI
|
||||
|
||||
- **New / Seen / All** tabs. Opening a job, or clicking *Mark seen*, moves it to Seen.
|
||||
- **no longer listed** means the job was missing from that source's latest successful scrape.
|
||||
- Search filters on title, company and tags. Recent scrape runs (and errors) are listed at the bottom.
|
||||
|
||||
## Adding a site
|
||||
|
||||
1. Create `src/scrapers/<site>.ts` exporting an object that implements `Scraper` (`src/scrapers/types.ts`):
|
||||
```ts
|
||||
export const mySite: Scraper = {
|
||||
id: "my-site", // stable, stored in DB
|
||||
name: "My Site",
|
||||
async scrape() {
|
||||
const html = await fetchHtml("https://..."); // helper in ./http.ts
|
||||
return [/* { externalId, title, url, company?, location?, tags?, postedAt? } */];
|
||||
},
|
||||
};
|
||||
```
|
||||
2. Add it to the list in `src/scrapers/index.ts`.
|
||||
|
||||
`externalId` must stay stable for a job, because it is what decides whether a job is new.
|
||||
For another slo-tech city, just add `sloTech("Maribor")` to the list.
|
||||
Reference in New Issue
Block a user