Building Oracle Agent Memory: Why your agents need real memory

When we talk about agent memory, we look to compose memory formation, retrieval, and persistent storage.

You’ve probably seen an agent like Claude Code or Codex re-read the same file it just read.

Most agents do not have durable memory by default. They have context for the current run, but not persistent, managed experience across runs.

Context isn’t agent memory

When an agent performs useful work, like reading documents, running queries, and synthesizing a useful procedure – what happens at the end of the stateless session? The useful artifact is discarded.

When a new agent comes along and begins related work, that agent must inefficiently reassemble context. Not only does this repeat work and superfluously consume tokens, it amplifies the opportunity for common errors like hallucinations.

Diagram showing two agent sessions: without agent memory, we repeat work.

Agent memory is not just long-term storage. It is a managed state system that converts agent experience into durable, scoped, retrievable, and revisable knowledge. Context functions as working memory during a run; persistent memory requires formation, retrieval, consolidation, governance, and evaluation.

Markdown, files, wikis, and prompts

Many users and builders I talk to are trying to implement memory with scratchpads: files, prompts, and the like.

However, I think scratchpads are workarounds:

  • Files don’t scale well in multi-tenant systems, with distributed reads and writes. They also lack ACID capabilities and transactions.
  • It’s difficult to apply fine-grained security to files. For example, memory access to specific agents, workflows, teams, or organizations.
  • File search is typically done with directory traversal and grep-like tools. Separate stores are needed for vector search, graph traversal, or other advanced search methods.

Taking this line of thought to what I think is its logical conclusion, you’ll end up building your own database. This probably isn’t what you set out to do.

Where is agent memory today?

Talking to customers and users, we’ve seen common, useful patterns emerge around memory.

  • Memory Formation: When we store memories, we store distilled experiences and structured facts, not raw transcripts. Memories may link to related memories or source material.
  • Hybrid Retrieval: Vector search alone is usually not sufficient. Hybrid retrieval fuses vector, lexical, relational entity, graph, and other feedback signals for comprehensive results using a combined, weighted score.
  • Consolidation: Unmanaged memories get messy. A memory system must be capable of merging duplicates, resolving conflicts, and evolving over time.
  • Temporality and Forgetting: Memories should have a creation time, access time, and validity windows. Old or irrelevant memories should be forgotten.
  • Evaluation and Feedback: The system must measure use and return it as feedback. If a memory was helpful or harmful, this should adjust its score.
  • Governance: Secrets/PII must be redacted, access should be scoped to the agent/user, and memories reviewed to raise quality.

This is why Memory is your moat. You can easily swap a model or harness out, but memory is derived from your business or application behavior and experiences. This is difficult for competitors to copy.

The agent memory loop

In the agent memory loop, useful facts or experiences should propagate from local session data to the memory layer and persistence storage.

  1. During an agent session, the agent does work and creates a useful fact.
  2. That fact should be redacted, enriched, and persisted to the database.
  3. On a new agent session containing similar context, this fact should be recalled.
  4. The fact evolves with related data from successive sessions and temporality. Whether the fact was used, rejected, helpful/not helpful must be included in the memory feedback.

In the agent loop, memory APIs (read/write/admin) should be invoked by the agent runtime tool capabilities. This can be through hooks, MCP, skills, or your custom harness integration.

A basic example works with hooks: at the beginning of a session, the agent queries the memory layer for a context card. At the end of the session (or in the middle) if a useful fact is created, the agent submits transcripts to the memory layer for review.

Memory formation

When we persist memories to storage, we don’t simply dump transcripts in a database. Good memories are formed, which means write-path distillation.

  • First, secrets, PII, and irrelevant information must be redacted. Redaction must occur before metadata enrichment, embedding, or other work.
  • The memory should be enriched: raw transcripts should be distilled to a compact memory with metadata (timestamps, related memories).
  • The agent, workload, organization, etc. should be added as scope. Scope is essential for structured access.
  • The memory content is converted to a vector embedding for similarity search.
  • Distinct slices of the source transcript (the data that created the memory) can be saved as related episodes, and linked relationally or in a graph.
  • Any organization-specific policies may also be applied before write.

The memory object persisted in the database then contains primary keys, blob content, JSON metadata (tags, feedback, scope), vector embeddings, temporal facts, and relational/graph edges.

It’s a great practice to also add auditing events during write: the memory was used at this time, by this agent, to do this task.

Recall === Retrieval

When we recall a memory, we’re talking about hybrid retrieval. Hybrid queries combine distinct search branches in a single round-trip transaction to efficiently find related context.

For example, a hybrid query may combine vector search for semantic matches, lexical search for keyword matches, graph edges for related memories, and predicates like recency and feedback scores.

If you’d like to see how hybrid search can work using vector embeddings and lexical indexing, check out my Hybrid Search example on GitHub.

Memories Evolve

Memory isn’t a simple repository with read/write access. For memory to be effective, it must evolve over time through consolidation, error handling, and forgetting.

Without proper controls, memory systems can encounter the following issues:

  • Stale memory: When something changes in your project, old lessons still rank highly. For example, if you changed the build system in your project, the old tools and processes may be recalled to confuse your agents. Agents should be able to evaluate memories against the current state of the system, providing feedback to accept or reject the memory.
  • Bad scope: memories should be scoped to the agent, workflow, team, organization, etc., on both read and write. Without scoping, private state leaks into another user’s recall – similar to logging into ChatGPT and seeing someone else’s message history.
  • Poisoned memory: This is a worst case scenario where an untrusted source plants a compromised instruction in your memory layer. Without proper guardrails, redaction, and review, agents may read data from malicious websites which could end up as a durable fact.
  • Conflicts: Conflicts arise when two similar memories disagree on the current state. Deduplication, consolidation, and review feedback are key to resolving conflicts.
  • Over-recall and Irrelevance: Hybrid search may return too many weak matches, or matches that are semantically similar but irrelevant. These signals are usually related to a weak hybrid search algorithm and require tuning to fix.
  • Data leakage and Silent Drift: Over time, your working memory will grow. What worked great with thousands of memories may fall over with millions. Careful review, consolidation, and forgetting are required to maintain the efficacy of the memory layer.

Feedback and evaluation close the memory loop: ultimately, memory quality must be measurable. Measure signals around retrieval correctness, whether a memory was helpful, and if it was harmful. The overall downstream usefulness of a memory should feed ranking and lifecycle decisions.

Build your own agent memory

The next step is to experiment with agent memory using a storage platform capable of hybrid search, secure governance, mixed storage. Generally, this would be a multi-model database capable of co-locating vectors, lexical search, graph traversal, spatial data, and more.

Because I work in DevRel at Oracle, I’ll give you the opinionated database solution: The Oracle Agent Memory Python SDK is a great place to start. It includes notebooks that implement agent memory concepts on top of Oracle AI Database.

All of these memory features (and more) are available in Oracle AI Database Free.

Response

Leave a Reply