Rendered at 21:34:43 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
kennywinker 18 minutes ago [-]
Compaction has been a pretty painful part of local llm usage. Scrapping the current context and parsing almosy 128k of context then generating something like 5-10k tokens - that can take quite a while when you’re working with 10t/s-45t/s (depending on the model).
I pretty much just start a new session whenever i fill the context.
Gecko4072 34 minutes ago [-]
Can someone recommend a Hermes alternative that is less token hungry? Pi did not work well for my use case.
pkulak 7 minutes ago [-]
I keep my max context really small for personal assistant agents; they don't need it. Especially since compaction keeps anything important around anyway. I use 60k with Pi.
skinfaxi 26 minutes ago [-]
I found hermes to be really lightweight, though I am on a relatively older version and built a custom plugin to lazily load mcps (that's probably in hermes proper by now). Compared to kilo it seems to consume far fewer tokens.
3 hours ago [-]
randomblock1 30 minutes ago [-]
TLDR: It keeps ~20k tokens of recent conversations, then hands the rest of the conversation to another model with a special system & user prompt. This then fills out a template with relevant information.
I pretty much just start a new session whenever i fill the context.
See: https://github.com/earendil-works/pi/blob/main/packages/codi...