Memory & Context Optimization

Why Your Agent Forgets Everything

Out of the box, agent memory doesn't work the way most people expect. You tell your agent something on Monday. By Wednesday, it's gone. You explain a preference three times and it still doesn't stick. This isn't a bug — it's a misunderstanding of how the system works.

Three things are working against you:

• Context compaction. When the conversation gets too long, the runtime compresses older messages to make room for new ones. Important details get summarized away or dropped entirely.
• Memory defaults vary and change. Every agent runtime handles memory differently, and defaults shift between versions. Don't assume auto-capture is on — check your current config rather than relying on a default.
• No structure to what gets saved. Even with auto-capture on, everything goes into a flat daily log. Critical business decisions sit next to casual chit-chat. When the agent searches memory, it retrieves noise alongside signal.

The fix isn't a third-party plugin (though those help). The fix starts with a deliberate memory architecture that you own, in plain files, independent of any one runtime.

The Core Memory Layers

The memory systems that work are layered and built on files, with embedding-backed semantic search on top. Mine uses embedding-backed search plus compaction-triggered memory flushes — not just passive file recall. There are four core layers:

The agent needs two ways into memory: semantic search across all indexed files, and targeted reads of a specific date or document.

Those four layers are the foundation. The operating model I built on top uses a 3-tier architecture: session memory (governance files + MEMORY.md), a knowledge graph (structured notes in a numbered vault), and an ingestion pipeline that promotes research and transcripts into canonical notes. This guide explains the thinking behind it.

The Hybrid Memory Model

The single most effective memory optimization is separating durable knowledge from ephemeral logs — and in production, separating both from live operational state. My live install uses three layers: MEMORY.md for durable facts, daily logs for search history, and runtime-derived state for operational control. MEMORY.md is for what should always be true. Current blockers, freshness, approvals, and delivery failures are better treated as live operating state, not repeated prose.

MEMORY.md — Durable Facts and Standing Context

This is the file your agent reads when it needs to recall important facts. It should be structured, concise, and manually maintained. Think of it as the agent's institutional knowledge. MEMORY.md is for what has been true for weeks, not what is happening right now.

# MEMORY.md

## Owner
- Name: Joshua Thacker
- Location: Sacramento, CA
- Primary business: SearcherOS (SaaS for SMB acquisition screening)
- Goal: $100K ARR by December 2026

## Preferences
- Communication style: direct, concise, no fluff
- Timezone: America/Los_Angeles
- Morning person — available before 2 PM, family time after
- Never send emails without explicit approval

## Current Priorities (updated weekly)
- SearcherOS: shipping notification system this sprint
- Content: publishing the agent guide series
- Marketing: LinkedIn content cadence — 3x/week

## Key Decisions
- 2026-02-15: Chose Anthropic over OpenAI for prompt-injection resistance
- 2026-02-20: Moved from EC2 to Mac Mini M4 Pro — saved $150/month
- 2026-03-01: Added Bridget (CCO) to agent hierarchy

Keep MEMORY.md under 1,000 words. It loads into the context window, so every word costs tokens. Be ruthless about what earns a place here. If it's not referenced at least weekly, it doesn't belong.

Daily Logs — The Ephemeral Record

Daily logs go to memory/60-daily/YYYY-MM-DD.md. With auto-capture enabled, the runtime writes a summary of each session to the day's log file. These are searchable but aren't loaded into the context window by default.

The key insight: daily logs are for search, not for context. The agent doesn't read them unless it specifically searches for something. This means they can be verbose without wasting tokens.

# memory/60-daily/2026-03-08.md

## Session 1 — SearcherOS bug fix
- Fixed notification delivery failure on webhook timeout
- Root cause: retry logic wasn't exponential
- Deployed v2.4.1 at 10:30 AM PT

## Session 2 — Content planning
- Drafted outline for Memory Optimization guide
- Decided to cover pre-compaction flush technique
- Riker to promote on LinkedIn Thursday

The Memory Pipeline: Daily → Weekly → Long-Term

The 2-file system is the foundation. But to really make memory work at scale, you need a pipeline that progressively distills information from raw session logs into durable knowledge. This is the system I use:

Layer 1: Daily Logs (Raw)

Every session, every agent writes to memory/60-daily/YYYY-MM-DD.md. These logs capture everything: decisions made, tasks completed, bugs found, ideas discussed. They're verbose and unfiltered. That's the point — you want the raw record.

Layer 2: Weekly Summaries (Distilled)

Every Friday (or as part of a cron job), the agent summarizes the week's daily logs into a single memory/70-weekly/YYYY-WW.md file. The weekly summary strips out the noise — the false starts, the debugging dead ends, the casual conversation — and keeps only what mattered:

# memory/70-weekly/2026-W10.md

## Key Decisions
- Switched notification system from polling to webhooks
- Approved Riker's LinkedIn content calendar through March
- Declined partnership offer from XYZ — unit economics didn't work

## Metrics
- SearcherOS: 47 new signups, 12 conversions, $2,400 MRR
- Content: 3 LinkedIn posts published, 8,200 impressions
- Agent costs: $34.50 in API tokens this week

## Blockers Resolved
- Webhook timeout bug (Bill, deployed v2.4.1)
- DNS propagation delay on joshuathacker.com (resolved)

## Carried Forward
- SearcherOS onboarding flow redesign (Bill, in progress)
- Agent guide series (Bridget, 2 of 4 complete)

Weekly summaries are compact enough to be useful for search but structured enough to tell the story of what happened that week.

Layer 3: Long-Term Memory (Promoted)

When patterns emerge across weekly summaries — recurring decisions, stable preferences, proven strategies — promote them to MEMORY.md. This is the institutional knowledge layer. It's what the agent loads every session.

The promotion criteria is simple: if something has been true for three consecutive weeks, it belongs in MEMORY.md. If it was a one-time decision, it stays in the weekly summary. The daily log gets archived after 90 days.

memory/60-daily/ (raw, verbose)
  ↓ summarized every Friday
memory/70-weekly/ (distilled, structured)
  ↓ patterns promoted after 3+ weeks
MEMORY.md (permanent, curated, <1000 words)

This pipeline means the agent always has the right level of detail available: MEMORY.md for instant context, weekly summaries for recent history, and daily logs for forensic recall when needed.

The Numbered Vault

Daily logs and weekly summaries are just two directories in a larger system. My production install uses a numbered Obsidian-style vault with 10 directories, each with a specific purpose:

memory/00-home/       — Navigation entrypoints
memory/10-maps/       — Maps of systems, domains, and note locations
memory/15-specs/      — Formal local specs
memory/20-domains/    — Stable domain truths
memory/30-decisions/  — Durable decisions and policies
memory/40-research/   — Canonical research takeaways
memory/50-operations/ — Operating procedures, scoreboards, and workflows
memory/60-daily/      — Chronological daily notes
memory/70-weekly/     — Weekly synthesis notes
memory/99-archive/    — Compressed historical detail

The vault separates concerns. Decisions live in 30-decisions/, not scattered across daily logs. Research findings land in 40-research/, not dumped into MEMORY.md. Each directory has a canonical purpose, and the agent navigates the graph starting from 00-home/index.md.

You don't need to build this by hand. Describe the structure to your agent, including what belongs in each directory, and let it build and maintain the vault.

Verify Your Memory Configuration

Before anything else, confirm that memory persistence is working. The simplest test: tell your agent a specific fact, end the session, start a new one, and ask for it back. If your agent starts every conversation fresh, persistence is not configured — check your runtime's docs for the version you have installed.

Pre-Compaction Memory Flush

This is the technique that changed how I think about agent memory. Here's the problem: when a conversation gets long, the runtime compacts the context window. Older messages get summarized or dropped. If you had important context in those messages, it's gone.

The fix: before the conversation hits the compaction threshold, the agent saves critical context to its daily log. You can trigger this manually (“Save everything important from this conversation to your memory files”) but you should not rely on remembering to do it. Build it into every agent's SOUL.md as a standing instruction:

# In SOUL.md — add this to EVERY agent:

## Memory Protocol (CRITICAL — do not skip)
Before EVERY long session ends, and whenever the conversation
approaches 50% of the context window:
1. Summarize all key decisions, facts, and action items
2. Write them to today's daily log (memory/60-daily/YYYY-MM-DD.md)
3. Promote anything that should be permanent to MEMORY.md
4. Confirm with me before compaction occurs
5. At end of week, summarize daily logs into weekly summary

NEVER let a session end without writing to the daily log.
Lost context cannot be recovered.

The difference is night and day. Before I added this protocol, I was constantly re-explaining context to agents. After, the daily logs capture everything, the weekly summaries distill it, and MEMORY.md holds the permanent knowledge. Nothing falls through the cracks.

Hybrid Search Configuration

When the agent searches memory, it uses a combination of vector (semantic) search and text (keyword) search. The default settings are fine for casual use, but for production setups with multiple agents and weeks of accumulated memory, you want to tune the search.

The Settings I Use

The exact config keys depend on your search layer. The values carry over.

What each setting does:

• Vector weight 0.7, text weight 0.3: Semantic search finds conceptually related content even when the exact words don't match. Text search catches exact terms, names, and identifiers. The 70/30 split favors meaning over keywords.
• MMR (Maximal Marginal Relevance) with lambda 0.7: Reduces redundant results. Without MMR, a search for “SearcherOS pricing” might return five nearly identical daily log entries. MMR diversifies the results so you get the most informative hits.
• Temporal decay with 30-day half-life: Newer memories rank higher than older ones. A decision from last week is more relevant than one from two months ago. The 30-day half-life means a memory's weight drops by 50% every 30 days.

Shared Memory Across Agents

In a multi-agent setup, agents need to share knowledge without contaminating each other's individual context. The team structure is on the agent hierarchy; the memory side is here.

Shared Knowledge Directory

Create a _shared/knowledge/ directory as the canonical surface for cross-agent truth. Symlink it into each agent's workspace. Only facts that are durable (1-2+ weeks), relevant to multiple agents, and should remain consistent across the organization belong here.

mkdir -p ~/agents/_shared/knowledge
ln -s ~/agents/_shared ~/agents/bob/_shared
ln -s ~/agents/_shared ~/agents/bill/_shared

Workspace-CEO Shared State (My Live Setup)

In my setup, the coordination surface is a shared folder in the CEO agent's workspace. Operational state (blockers, approvals, delivery failures, freshness) lives as runtime-derived state here, not as repeated prose in MEMORY.md. Canonical cross-agent truth routes through _shared/knowledge/. STATUS.md exists as a projection for human readability — it is not the canonical operational source.

Index Shared Files for Search

Symlinked files are often readable but not automatically indexed for semantic search. Check that your search layer indexes the shared path; most let you add extra directories to the index.

Keep shared knowledge in .md files. If you need to share structured data (JSON, CSV), convert the relevant parts to Markdown first.

Third-Party Memory Options

File-based memory is solid once configured. But if you need capabilities beyond what file-based memory provides, three options are gaining traction in the community:

Mem0

Adds persistent cross-session memory with a 30-second setup. Mem0 sits alongside your runtime's native memory and provides a separate knowledge store that persists across sessions without relying on MEMORY.md. It's the most recommended memory plugin in the community right now.

Best for: getting cross-session memory working quickly with minimal configuration.

MemU (MEMU)

Designed for always-on agents running 24/7 (like on a Mac Mini). MemU adds long-term memory with prediction capabilities — the agent can anticipate what you'll need based on patterns in past behavior. Free and open-source.

Best for: always-on setups where you want the agent to proactively surface relevant information without being asked.

Obsidian Vault

Pairing your agents with Obsidian gives a “second brain” approach. The agent reads and writes to your Obsidian vault, giving it access to your entire personal knowledge base. Notes, bookmarks, research, and structured references all become searchable context.

Best for: users who already maintain an Obsidian vault and want to bridge their personal knowledge management with their agent's memory.

Memory Anti-Patterns

Things I've seen go wrong — and things I got wrong myself:

• Dumping everything into MEMORY.md. If your MEMORY.md is 3,000 words, it's a daily log pretending to be curated knowledge. MEMORY.md is for the 20% of information that drives 80% of decisions. Everything else goes in daily logs.
• Never pruning daily logs. After 90 days, your memory directory has hundreds of files. Temporal decay helps with search ranking, but the index still grows. Archive or delete logs older than 90 days unless they contain critical decisions.
• Treating automatic capture as the strategy. Whatever your version's session capture does, it doesn't know what's important. You still need to manually promote key decisions and facts to MEMORY.md. Automatic capture is the safety net, not the strategy.
• Skipping the pre-compaction flush. Long sessions without a flush are the #1 cause of “I told the agent this yesterday and it forgot.” Build the flush protocol into your SOUL.md and make it automatic.

Putting It All Together

A properly configured memory system has these components:

• Memory persistence verified across sessions (behavior varies by runtime)
• A structured MEMORY.md under 1,000 words with owner info, preferences, and key decisions — durable facts only, not live state
• A numbered vault (memory/00-home/ through memory/99-archive/) with daily logs in memory/60-daily/ and weekly summaries in memory/70-weekly/
• A pre-compaction flush protocol in SOUL.md
• Hybrid search tuned to 0.7 vector / 0.3 text with MMR and temporal decay
• A shared _shared/knowledge/ surface across agents, indexed for search
• Operational state (blockers, approvals, freshness) tracked as live runtime state, not repeated in MEMORY.md prose

Get this right and your agents remember what matters, forget what doesn't, and share knowledge cleanly across the team. Get it wrong and you'll spend more time re-explaining context than doing actual work.