andersswanson.dev

Agent memory is becoming one of the most important ways to differentiate an AI application. It’s getting much cheaper to turn a good idea into a working app.

A small, skilled team with capable models and modern coding tools can get surprisingly far in a couple days. Increasingly, features that took months and a specialized team are faster to prototype, and faster for competitors to copy.

I think the code still matters: the system must still work under load, handle failure gracefully, and earn user trust. A poorly planned, vibe-coded app won’t scale. But the advantage of “having the best product code” is much weaker when everyone has access to similar models and tools.

What’s more difficult to copy than the feature? The knowledge behind the feature. In this article, we’ll look at memory that goes beyond a simple RAG pipeline.

Agent memory must leverage your knowledge base

Your data differentiates the work that your agents can do.

Think about the data you’ve built up over time: for an ecommerce business, there are records of customer interactions, transactions, product usage, support cases, policy decisions, documents, and exceptions no one anticipated on the initial release. This is the data that can give an agent memory about the product you’ve built, and it’s also data a competitor cannot easily replicate.

Agent memory is really a data problem: lifecycle, retrieval, and evolution. Agents need more than chat history, they must have a reliable way to find the right context and use it (security/governance are also concerns!).

A pile of connectors isn’t agent memory

Fragmented systems can’t automatically produce queryable memory. Data is stored in separate systems, with different access patterns and levels of visibility or search. For example, Confluence for documentation, a support ticket system, Jira for product work, Slack for decisions made in the moment, and email.

You can build an agent on top of this: pull content from each system, and send it to an LLM in a loop. But, there are limits. Relationships are difficult to model, and questions about structured facts across systems become hard to answer:

• Is a customer from a support thread entitled to a refund according to specific policy documents or their contract?
• Is a result current, authorized for this user, and connected to the same customer or product?
• Is the conversation relevant for the current geographical region, known history, and prior incidents?

I think a multi-model database (like Oracle AI Database) fits as the foundation for memory. A multi-model database gives you a place to work with relational data, JSON documents, vectors, graphs, and more. You can ingest and normalize source data into a memory layer (preserving source IDs and links), and query it with the retrieval mode the question needs; e.g., graph + AI vector search.

Hybrid search for context-aware results

RAG complements the memory story: A document retrieved from a vector index may help an agent explain a policy, but memory also needs to know which policy version is current, which customer it applies to, whether that customer is entitled to an exception, and whether the requesting user can see any of it.

Hybrid search is the combination of different search methods, like vector and lexical search, to find context-aware results, including governed security and logging. It’s how an agent combines semantic evidence with exact, current, and authorized facts.

For example, an agent could use one query to:

• Find semantically similar support and Slack content with vector search
• Filter based on exact product name or error code with full-text search
• Join results to current customer, entitlement, and ticket status with relational data
• Apply a location/service-area constraint when geography matters
• Follow graph links to related people, organizations, cases, and issues
• Secure results with row and column level security so the agent doesn’t access data it shouldn’t (Oracle Deep Data Security)
• Lastly, record what the agent accessed, and why.

The utility of a multi-model database like Oracle AI Database is that JSON, graph, text, spatial, and vector can be combined into one query pipeline, in one round-trip, to efficiently evaluate hybrid queries on memory data.

Once source data is ingested and linked in the memory layer, a single database query can combine semantic retrieval with structured filters, relationships, and access controls. This reduces handoffs between retrieval systems and keeps the memory layer easier to govern and operate.

A simple end-to-end example

Let’s imagine a simple example: a customer contacts support for a refund on a product that didn’t work as expected. The memory layer can then facilitate resolving the case like so:

1. Customer and order: Retrieve the order, purchase date, product, tier, and refund status with a relational query.
2. Related evidence: Use vector + lexical search to find similar support cases and discussions in the memory layer.
3. Current policy: Use exact relational search to find the refund rule.
4. Graph Relationships: Follow links to the customer’s organization, contract, incidents, and approved exceptions.
5. Authorized context: Return only the records the support agent is permitted to access.
6. Update memory: Update access times, provide feedback, and promote redacted transcript for potential memory formation.

Memory must keep learning

Memory is more than a document dump: useful memory must change with real-time feedback, or else risk becoming outdated. The memory layer must capture feedback, surface stale knowledge, and log why an agent used a particular piece of data.

Every memory record should carry enough context to be evaluated later:

• Source identity: Where did it come from? A customer interaction, a team, a workflow, or an agent?
• Freshness: When was it created, last verified, and last used?
• Access: Who is allowed to retrieve it?
• Usefulness: Did it help answer a similar question, or was it ignored, corrected, or down-ranked?
• Supersession: How are memories forgotten when a related policy, product, or decision changes?

Not every action should be a persisted memory. Write memory selectively: redact the content, retain links to the source material, attach who or what created it, and allow the record to be corrected, expired, or removed. Proper implementation of these techniques reduce the risk of over-recall, conflicts, and irrelevant retrieval.

Memory layers are still an integration layer

A well-implemented memory layer over a multi-model database will makes it possible to retrieve, govern, and secure your data for effective use by agents. You’ll still need connectors, cleanup, or stewardship of your data.

Tools like vector search, graph queries, and LLMs can be used by anyone. What can set you apart is building a memory layer around the knowledge you/your business has accumulated, and continuously improving it through real-time feedback.

Oracle AI Database can provide the foundation for agent memory because it brings the data shapes agents need: relational records, documents, vectors, graph relationships, spatial context, and security controls into one database platform.

References

• Oracle Agent Memory Python package
• Building Agent Memory: Why Your Agents Need a Real Memory Layer
• Oracle Deep Data Security
• The Value Proposition of a Multi-Model Database