feat: dedup and format the output
This commit is contained in:
8 files changed
+153
-54
No files matched your search
@@ -9,10 +9,12 @@ A Go service that fetches market data (crypto/equities/futures/forex) and news,
|
||||
## Commands
|
||||
|
||||
```sh
|
||||
go build -o stonks ./cmd/stonks # build
|
||||
go vet ./... # static checks
|
||||
gofmt -l . # list any unformatted files (should be empty); gofmt -w to fix
|
||||
go test ./... # no test files exist yet — this is a no-op until some are added
|
||||
make build # go build -o stonks ./cmd/stonks
|
||||
make install # build, then install to ~/.local/bin/stonks (matches systemd/stonks.service's ExecStart)
|
||||
make vet # go vet ./...
|
||||
make fmt # gofmt -l . — list any unformatted files (should be empty); gofmt -w to fix
|
||||
make test # go test ./... — no test files exist yet, this is a no-op until some are added
|
||||
make clean # remove the built binary
|
||||
```
|
||||
|
||||
Run locally (needs secrets loaded into the environment first — see `config/secrets.env.example`):
|
||||
@@ -22,7 +24,7 @@ set -a; source config/secrets.env; set +a
|
||||
go run ./cmd/stonks --config config/config.yaml --timeout 120s
|
||||
```
|
||||
|
||||
There is no Makefile or CI config — the commands above are the whole toolchain.
|
||||
There is no CI config — the Makefile plus the commands above are the whole toolchain.
|
||||
|
||||
## Architecture
|
||||
|
||||
@@ -59,7 +61,9 @@ Two interchangeable modes behind one `Analyzer` interface, selected by `analyzer
|
||||
- `api.go`: calls the Messages API directly via `anthropic-sdk-go`, billed per token.
|
||||
- `subscription.go`: shells out to the `claude` CLI (`claude -p`, reading the prompt from stdin), billed against the host's logged-in Claude Pro/Max subscription instead of metered API usage. Requires `claude login` to have been run once on the deploying host/user.
|
||||
|
||||
`prompt.go` builds the user-turn text from `fetch.Result` and holds `SystemPrompt`, which is the single source of truth for the report's structure (Crypto/Equities/Futures/Forex/Watch sections) **and** instructs Claude to append a `===RECOMMENDATIONS_JSON===` marker followed by a JSON buy/sell list as the last thing in its response. `split.go`'s `SplitReport` parses that trailing block into `report.Report{Text, Recommendations}` via `extractJSONObject` (grabs from the first `{` to the last `}` after the marker — tolerant of stray code fences/prose Claude sometimes adds despite being asked not to, observed in practice with multi-agent mode on). If the marker's missing or the JSON still fails to parse, it degrades to a text-only report (logs a warning in `main.go`) rather than failing the run. Keep the marker instruction and the parser in sync if you touch either.
|
||||
`prompt.go` builds the user-turn text from `fetch.Result` and holds `SystemPrompt`, which is the single source of truth for the report's structure (fixed Summary/Crypto/Equities/Futures/Forex/Watch section headers, in that order, every run) **and** instructs Claude to append a `===RECOMMENDATIONS_JSON===` marker followed by a JSON buy/sell list as the last thing in its response. `split.go`'s `SplitReport` parses that trailing block into `report.Report{Text, Recommendations}` via `extractJSONObject` (grabs from the first `{` to the last `}` after the marker — tolerant of stray code fences/prose Claude sometimes adds despite being asked not to, observed in practice with multi-agent mode on). If the marker's missing or the JSON still fails to parse, it degrades to a text-only report (logs a warning in `main.go`) rather than failing the run. Keep the marker instruction and the parser in sync if you touch either.
|
||||
|
||||
Because the timer fires every 5 hours, `main.go` calls `archive.LoadLatest` before building the prompt to read back the previous run's `report.md` and passes it into `BuildUserPrompt` as `previousReport`. `SystemPrompt` instructs Claude to treat that as a "Previous Briefing" section: dedup commentary against it (fresh prices are never skipped, but unchanged analysis should collapse to a one-liner), and only carry forward previously-mentioned info if it's still genuinely important (new regulation, unresolved macro event). This is why archiving is no longer optional — see below.
|
||||
|
||||
**Multi-agent mode** (`advisor.go`, optional via `analyzer.advisors` in config): the same data prompt is also piped to other one-shot CLIs (`gemini`, `codex exec -`, ...) concurrently. This is deliberately **not** a fourth `Analyzer` — advisors don't produce the final report shape, they're extra input to it. `main.go` runs them after building the base prompt, logs failures as warnings (same non-fatal pattern as fetchers — a failing/unauthenticated advisor is just excluded), then `AppendAdvisorSection` appends the successful ones to the prompt before the real `analyzer.Analyze` call. `SystemPrompt` tells Claude to treat that section as opinions to weigh, not ground truth to defer to.
|
||||
|
||||
@@ -69,9 +73,9 @@ Two interchangeable modes behind one `Analyzer` interface, selected by `analyzer
|
||||
|
||||
`discord.go` sends the narrative text as one or more chunked messages (Discord's 2000-char `content` limit), then **always** sends a second message with two separate embeds — green "🟢 Buy" and red "🔴 Sell" — even when a list is empty (states "No buy/sell recommendations today." rather than omitting the embed). If you change the recommendation rendering, keep the two embeds separate; that was an explicit choice, not a simplification opportunity.
|
||||
|
||||
### Archiving (`internal/archive`, optional)
|
||||
### Archiving (`internal/archive`, always on)
|
||||
|
||||
`archive.Save`, called from `main.go` when `config.Archive.Enabled`, writes one timestamped subdirectory per run under `archive.dir` (default `~/.local/state/stonks/runs`) containing `fetched.json`, `prompt.txt` (the *final* prompt, post-advisor-section), `analysis_raw.md` (pre-`SplitReport`), `report.md`, `recommendations.json`. This is a debugging aid, not part of the pipeline — a save failure is logged and non-fatal, same pattern as everything else here.
|
||||
`archive.Save`, called unconditionally from `main.go`, writes one timestamped subdirectory per run under `archive.dir` (default `~/.local/state/stonks/runs`) containing `fetched.json`, `prompt.txt` (the *final* prompt, post-advisor-section), `analysis_raw.md` (pre-`SplitReport`), `report.md`, `recommendations.json`. It's still a debugging aid, but it's also load-bearing now: `archive.LoadLatest` reads the most recent run's `report.md` back at the start of the next run so the prompt can dedup against it (see `prompt.go` above). A save failure is logged and non-fatal — it just means the *next* run won't have anything to dedup against, same non-fatal pattern as everything else here.
|
||||
|
||||
## Config and secrets
|
||||
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
BINARY := stonks
|
||||
PREFIX := $(HOME)/.local/bin
|
||||
|
||||
.PHONY: build install clean vet fmt test
|
||||
|
||||
build:
|
||||
go build -o $(BINARY) ./cmd/stonks
|
||||
|
||||
install: build
|
||||
mkdir -p $(PREFIX)
|
||||
install -m 0755 $(BINARY) $(PREFIX)/$(BINARY)
|
||||
|
||||
clean:
|
||||
rm -f $(BINARY)
|
||||
|
||||
vet:
|
||||
go vet ./...
|
||||
|
||||
fmt:
|
||||
gofmt -l .
|
||||
|
||||
test:
|
||||
go test ./...
|
||||
@@ -7,7 +7,7 @@ See [PLAN.md](PLAN.md) for the full design, cost analysis, and rationale.
|
||||
## Build
|
||||
|
||||
```sh
|
||||
go build -o stonks ./cmd/stonks
|
||||
make build
|
||||
```
|
||||
|
||||
## Configure
|
||||
@@ -37,15 +37,15 @@ codex login
|
||||
|
||||
A failing or unauthenticated advisor is logged as a warning and simply excluded — it never fails the run. Remove an entry (or the whole `advisors:` list) to disable multi-agent mode.
|
||||
|
||||
### Archiving runs (optional)
|
||||
### Archiving runs
|
||||
|
||||
Set `archive.enabled: true` in `config.yaml` to save each run's fetched data, full prompt (including any advisor section), raw and parsed analysis to `archive.dir` (default `~/.local/state/stonks/runs`), one timestamped subdirectory per run: `fetched.json`, `prompt.txt`, `analysis_raw.md`, `report.md`, `recommendations.json`. Useful for debugging prompt or parsing issues without reproducing a live run. Off by default — these accumulate over time with no automatic cleanup.
|
||||
Every run saves its fetched data, full prompt (including any advisor section), raw and parsed analysis to `archive.dir` (default `~/.local/state/stonks/runs`), one timestamped subdirectory per run: `fetched.json`, `prompt.txt`, `analysis_raw.md`, `report.md`, `recommendations.json`. This is always on — besides being useful for debugging prompt or parsing issues, the next run reads back `report.md` from the most recent subdirectory so it can avoid repeating unchanged commentary from ~5 hours ago. These accumulate over time with no automatic cleanup.
|
||||
|
||||
## Install as a user-scope systemd service
|
||||
|
||||
```sh
|
||||
mkdir -p ~/.local/bin ~/.config/systemd/user
|
||||
go build -o ~/.local/bin/stonks ./cmd/stonks
|
||||
mkdir -p ~/.config/systemd/user
|
||||
make install # builds and installs to ~/.local/bin/stonks
|
||||
cp systemd/stonks.service systemd/stonks.timer ~/.config/systemd/user/
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now stonks.timer
|
||||
|
||||
+20
-13
@@ -52,7 +52,14 @@ func run(configPath string, timeout time.Duration) error {
|
||||
return fmt.Errorf("build analyzer: %w", err)
|
||||
}
|
||||
|
||||
prompt := analyze.BuildUserPrompt(cfg.Targets, result)
|
||||
previousReport, err := archive.LoadLatest(cfg.Archive)
|
||||
if err != nil {
|
||||
// Missing/unreadable previous report just means this briefing can't
|
||||
// dedup against it — not fatal.
|
||||
log.Printf("warning: could not load previous archived report for dedup: %v", err)
|
||||
}
|
||||
|
||||
prompt := analyze.BuildUserPrompt(cfg.Targets, result, previousReport)
|
||||
|
||||
if len(cfg.Analyzer.Advisors) > 0 {
|
||||
advisorResults := analyze.RunAdvisors(ctx, cfg.Analyzer.Advisors, prompt)
|
||||
@@ -76,18 +83,18 @@ func run(configPath string, timeout time.Duration) error {
|
||||
log.Printf("warning: could not parse recommendations block: %v", err)
|
||||
}
|
||||
|
||||
if cfg.Archive.Enabled {
|
||||
dir, archErr := archive.Save(cfg.Archive, archive.Run{
|
||||
Result: result,
|
||||
Prompt: prompt,
|
||||
RawResponse: raw,
|
||||
Report: rep,
|
||||
}, time.Now())
|
||||
if archErr != nil {
|
||||
log.Printf("warning: could not save run archive: %v", archErr)
|
||||
} else {
|
||||
log.Printf("stonks: run archived to %s", dir)
|
||||
}
|
||||
dir, archErr := archive.Save(cfg.Archive, archive.Run{
|
||||
Result: result,
|
||||
Prompt: prompt,
|
||||
RawResponse: raw,
|
||||
Report: rep,
|
||||
}, time.Now())
|
||||
if archErr != nil {
|
||||
// Archiving failure is non-fatal to this run, but it does mean the
|
||||
// next run won't be able to dedup against this one.
|
||||
log.Printf("warning: could not save run archive: %v", archErr)
|
||||
} else {
|
||||
log.Printf("stonks: run archived to %s", dir)
|
||||
}
|
||||
|
||||
if err := runOutputs(ctx, cfg, rep); err != nil {
|
||||
|
||||
+4
-4
@@ -52,9 +52,9 @@ outputs:
|
||||
params:
|
||||
webhook_env_var: DISCORD_WEBHOOK_URL
|
||||
|
||||
# Optional: save each run's fetched data + full prompt + raw/parsed analysis
|
||||
# to disk (one timestamped subdirectory per run) — useful for debugging
|
||||
# prompt or parsing issues without reproducing a live run. Off by default.
|
||||
# Always on: saves each run's fetched data + full prompt + raw/parsed
|
||||
# analysis to disk (one timestamped subdirectory per run). This is both a
|
||||
# debugging aid and the source of the previous run's report.md, which the
|
||||
# prompt reads back to dedup each briefing against the last one.
|
||||
archive:
|
||||
enabled: false
|
||||
dir: ~/.local/state/stonks/runs
|
||||
+26
-11
@@ -12,22 +12,29 @@ import (
|
||||
|
||||
// SystemPrompt instructs Claude how to analyze the fetched data and format
|
||||
// its response for posting to Discord.
|
||||
const SystemPrompt = `You are a financial analyst producing a concise daily briefing for a personal Discord channel.
|
||||
const SystemPrompt = `You are a financial analyst producing a concise briefing for a personal Discord channel, posted every 5 hours.
|
||||
|
||||
You will be given today's prices (with percent change) and recent news headlines for a fixed watchlist of cryptocurrencies, NASDAQ equities, futures (metals, crude oil), and forex pairs.
|
||||
|
||||
Structure your response in Markdown, formatted for Discord (short lines, ` + "`**bold**`" + ` section headers, no HTML, no huge tables):
|
||||
Structure your response in Markdown, formatted for Discord (short lines, ` + "`**bold**`" + ` section headers, no HTML, no huge tables). Use exactly these section headers, in exactly this order, every single time — do not rename, reorder, merge, or skip a section (write "nothing notable" under a section rather than omitting it):
|
||||
|
||||
1. A one-paragraph market-wide summary.
|
||||
2. A "Crypto" section: brief commentary per asset grounded in the given price/news data.
|
||||
3. A "NASDAQ Equities" section: same, grouped sensibly (e.g. by sector) if useful.
|
||||
4. A "Futures" section (gold, silver, crude oil).
|
||||
5. A "Forex" section covering the given currency pairs.
|
||||
6. A short "Watch" section flagging anything time-sensitive (earnings, macro events implied by the news) for the next day or two.
|
||||
1. **Summary** — one paragraph, market-wide.
|
||||
2. **Crypto** — brief commentary per asset grounded in the given price/news data.
|
||||
3. **NASDAQ Equities** — same, grouped sensibly (e.g. by sector) if useful.
|
||||
4. **Futures** — gold, silver, crude oil.
|
||||
5. **Forex** — the given currency pairs.
|
||||
6. **Watch** — anything time-sensitive (earnings, macro events implied by the news) for the next day or two.
|
||||
|
||||
Keep the whole message under roughly 1500 words. Ground every claim in the provided data — do not fabricate news or prices. If data for an asset is missing, say so briefly rather than inventing a view. End with a one-line disclaimer that this is not financial advice.
|
||||
|
||||
If the user turn includes a section titled "Other AI Analysts' Independent Takes", those are additional perspectives from other AI models given the same data — not ground truth and not your own reasoning. Weigh them alongside the price/news data, note explicitly where you agree or disagree and why, but reach your own independent conclusion. Do not simply restate their conclusions.
|
||||
## Deduplicating against the previous briefing
|
||||
|
||||
If the user turn includes a section titled "Previous Briefing", that was posted about 5 hours ago to the same channel — treat it as what the reader already knows, not as data to re-derive conclusions from. Since reports repeat several times a day:
|
||||
- Do not re-explain news items, price moves, or commentary that already appeared there and have not materially changed since. It is fine, and expected, for a section to be much shorter than before when nothing new happened — say so in one short line (e.g. "Crypto: no material change since last briefing.") rather than restating the same analysis in new words.
|
||||
- Always still report today's current price and percent change for every asset — that number itself is never a repeat, only the commentary around it can be.
|
||||
- Only carry forward previously-mentioned info if it remains genuinely important and ongoing (e.g. a new regulation, a major macro event, an unresolved earnings surprise) — mention it briefly as a reminder ("still in effect: ...") rather than re-explaining it in full.
|
||||
- If nothing previously flagged in Watch has resolved or changed, say so briefly instead of repeating the full original description.
|
||||
- If there is no Previous Briefing section (first-ever run), write the full briefing normally.
|
||||
|
||||
After the full briefing above, on its own line, output exactly:
|
||||
===RECOMMENDATIONS_JSON===
|
||||
@@ -36,12 +43,20 @@ followed immediately by a single JSON object (no markdown code fence, no comment
|
||||
"action" must be exactly "buy" or "sell". Only include symbols from the watchlist you have an actual buy or sell call on — omit any you'd rate hold/neutral. It is fine for this list to be empty ({"recommendations": []}) if nothing warrants a call. This JSON must be the very last thing in your response.`
|
||||
|
||||
// BuildUserPrompt renders the fetched result into the user-turn text sent
|
||||
// alongside SystemPrompt.
|
||||
func BuildUserPrompt(targets config.Targets, result fetch.Result) string {
|
||||
// alongside SystemPrompt. previousReport is the narrative text of the last
|
||||
// archived briefing (see archive.LoadLatest), used so Claude can dedup this
|
||||
// briefing against it; pass "" if none exists yet (e.g. first-ever run).
|
||||
func BuildUserPrompt(targets config.Targets, result fetch.Result, previousReport string) string {
|
||||
var b strings.Builder
|
||||
|
||||
fmt.Fprintf(&b, "Date: %s\n\n", time.Now().Format("2006-01-02"))
|
||||
|
||||
if previousReport != "" {
|
||||
b.WriteString("## Previous Briefing (posted ~5 hours ago — see system prompt for how to use this)\n\n")
|
||||
b.WriteString(previousReport)
|
||||
b.WriteString("\n\n")
|
||||
}
|
||||
|
||||
b.WriteString("## Prices\n\n")
|
||||
writeQuoteSection(&b, "Crypto", targets.Crypto, result.Quotes)
|
||||
writeQuoteSection(&b, "NASDAQ Equities", targets.Equities, result.Quotes)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Package archive optionally saves a local copy of each run's fetched data
|
||||
// and analysis, for debugging prompt/parsing issues without reproducing a
|
||||
// live run.
|
||||
// Package archive saves a local copy of each run's fetched data and
|
||||
// analysis. It doubles as the source of the previous run's report, which
|
||||
// the prompt uses to dedup successive briefings against each other.
|
||||
package archive
|
||||
|
||||
import (
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -31,13 +32,9 @@ type Run struct {
|
||||
// path. Callers should treat a non-nil error as non-fatal — archiving is a
|
||||
// debugging aid, not part of the report pipeline.
|
||||
func Save(cfg config.ArchiveConfig, run Run, now time.Time) (string, error) {
|
||||
base := cfg.Dir
|
||||
if base == "" {
|
||||
base = defaultDir
|
||||
}
|
||||
base, err := expandHome(base)
|
||||
base, err := resolveDir(cfg)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("archive: %w", err)
|
||||
return "", err
|
||||
}
|
||||
|
||||
dir := filepath.Join(base, now.UTC().Format("20060102T150405Z"))
|
||||
@@ -70,6 +67,59 @@ func Save(cfg config.ArchiveConfig, run Run, now time.Time) (string, error) {
|
||||
return dir, nil
|
||||
}
|
||||
|
||||
// LoadLatest reads back the most recent previously-archived run's
|
||||
// report.md, for the prompt to dedup the next briefing against. Returns
|
||||
// ("", nil) if no prior run exists yet (e.g. first-ever run) — callers
|
||||
// should treat that as "no previous report" rather than an error.
|
||||
func LoadLatest(cfg config.ArchiveConfig) (string, error) {
|
||||
base, err := resolveDir(cfg)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
entries, err := os.ReadDir(base)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return "", nil
|
||||
}
|
||||
return "", fmt.Errorf("archive: list %q: %w", base, err)
|
||||
}
|
||||
|
||||
var runDirs []string
|
||||
for _, e := range entries {
|
||||
if e.IsDir() {
|
||||
runDirs = append(runDirs, e.Name())
|
||||
}
|
||||
}
|
||||
if len(runDirs) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
sort.Strings(runDirs) // timestamp-named dirs sort chronologically
|
||||
latest := runDirs[len(runDirs)-1]
|
||||
|
||||
data, err := os.ReadFile(filepath.Join(base, latest, "report.md"))
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return "", nil
|
||||
}
|
||||
return "", fmt.Errorf("archive: read previous report: %w", err)
|
||||
}
|
||||
return string(data), nil
|
||||
}
|
||||
|
||||
// resolveDir returns cfg.Dir (or defaultDir if empty) with "~" expanded.
|
||||
func resolveDir(cfg config.ArchiveConfig) (string, error) {
|
||||
base := cfg.Dir
|
||||
if base == "" {
|
||||
base = defaultDir
|
||||
}
|
||||
base, err := expandHome(base)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("archive: %w", err)
|
||||
}
|
||||
return base, nil
|
||||
}
|
||||
|
||||
// expandHome replaces a leading "~" with the current user's home directory.
|
||||
func expandHome(path string) (string, error) {
|
||||
if path != "~" && !strings.HasPrefix(path, "~/") {
|
||||
|
||||
@@ -76,13 +76,13 @@ type AdvisorConfig struct {
|
||||
}
|
||||
|
||||
// ArchiveConfig controls saving a local copy of each run's fetched data and
|
||||
// analysis — useful for debugging prompt/parsing issues without needing to
|
||||
// reproduce a live run.
|
||||
// analysis. Archiving is always on: besides being a debugging aid, the
|
||||
// previous run's report.md is read back on every run to let the prompt
|
||||
// dedup against it (see analyze.BuildUserPrompt).
|
||||
type ArchiveConfig struct {
|
||||
Enabled bool `yaml:"enabled"`
|
||||
// Dir is the directory each run's files are written under, one
|
||||
// timestamped subdirectory per run. Supports a leading "~". Defaults to
|
||||
// "~/.local/state/stonks/runs" if enabled with Dir left empty.
|
||||
// "~/.local/state/stonks/runs" if left empty.
|
||||
Dir string `yaml:"dir"`
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user