Making personal experience compound across AI tools

A knowledge base that compounds experience

The purpose of my personal KB is to make experience reusable in an agentic AI system. A useful lesson from one task should help with the next. An assistant should be able to retrieve the context that matters, apply it to the current problem, and capture new learning for future use.

This is also about making AI more useful to me personally. My preferences, previous decisions and accumulated experience provide context that a general model does not have. Keeping that context in one knowledge base gives different assistants a shared foundation, while letting me change models and tools.

The intended loop is straightforward: retrieve relevant experience → work on a task → capture a learning → index it → retrieve it when it becomes useful again. New AI-written notes are raw captures; saving and indexing them does not make them approved knowledge.

From a local Obsidian MCP to a cloud service

The first generation used an Obsidian Markdown vault, a local retrieval index and a local MCP server. Local agents could search and fetch notes through defined tools. It established that my existing notes could become usable context for AI.

It also exposed two limits. The computer hosting the MCP server had to be running, and cloud-based AI such as ChatGPT could not directly reach that local service. The usefulness of the KB depended on where an assistant ran.

The second generation moves the service boundary to AWS. Private Markdown and retrieval snapshots live in S3, and a public HTTPS MCP endpoint handles authenticated requests. Compatible desktop and cloud clients can connect through OAuth, with permissions enforced by the backend. Each client still needs to complete onboarding and receive its own access grant.

Why this matters beyond the technology

• Carry experience between tasks: lessons can become retrievable context instead of disappearing into individual conversations.
• Spend less effort rebuilding context: assistants can find relevant notes without repeated copy-and-paste or per-chat uploads.
• Keep continuity across AI tools: knowledge stays with me as I switch clients or models.
• Remove the laptop dependency: the cloud service can serve authorised clients while the original computer is offline.
• Expose useful capabilities: search, fetch and bounded note capture give AI access to the KB through a defined interface.

These are the outcomes the system is designed to support. The measured evidence so far is retrieval quality on a synthetic benchmark; I have not measured a productivity improvement from everyday use.

Architecture: context in, learning back

AWS API Gateway fronts a Lambda MCP service. Google sign-in establishes identity; the backend checks the owner, registered client, scope and allowed path for each tool call. Google authentication and KB authorisation are separate decisions.

Markdown is the canonical source in a private, versioned S3 bucket, encrypted at rest. A separate private bucket holds disposable SQLite retrieval snapshots with lexical and vector indexes. Cohere Embed Multilingual v3 on Amazon Bedrock in Sydney generates document and query embeddings.

The capture_learning tool persists a raw note under the permitted Ideas/ boundary before queueing ingestion. An SQS worker chunks the note, generates embeddings and publishes an updated snapshot. This makes new learning retrievable without treating an embedding database as the source of truth.

Engineering an AI system with reviewable evidence

I used an AI-native development workflow across requirements, code, evaluation and deployment. SpecKit records the requirements, design and implementation tasks. AI-assisted changes are reviewed against those artefacts, explicit tool contracts and executable checks.

There are several different questions to answer: does the code work, does access control hold, can retrieval find the right note, and can the system be deployed repeatably? Service tests exercise HTTP and Lambda behaviour, storage and authorisation. Retrieval evaluation measures ranking quality. CloudFormation and a pinned release pipeline make infrastructure and application changes reviewable.

That combination matters because a successful tool call alone says little about whether the assistant found the right context or respected the intended access boundary.

Evaluate retrieval before choosing a default

I compared lexical, pure vector and hybrid retrieval rather than assuming that hybrid must be best. An early 14-query fixture favoured pure vector on top-one accuracy. I expanded the evaluation before choosing the default: 20 synthetic notes and 33 judged queries, including paraphrases, similar notes with different answers, exact identifiers and English/Chinese cross-language retrieval.

Hybrid also reached 7/7 exact-identifier top one and 6/6 cross-language top three. Those results support hybrid as the current default on this fixture. They do not establish that it is best for every corpus or query.

The evaluation uses synthetic content. It checks the embedding model together with chunking, indexing and ranking; it does not isolate embedding quality, measure generated answer quality, or prove relevance on my private notes. An owner-judged question set from regular use is the next step.

CI/CD: quality gates at the right stage

GitHub Actions runs credential-free infrastructure validation and synthetic service checks on pull requests. The current AWS release path is an on-demand CodeBuild job started at an explicitly selected source commit. CodeBuild uses its scoped AWS service role to package and deploy through CloudFormation.

The live Bedrock retrieval benchmark is mandatory when the pinned embedding model or retrieval configuration changes. Unrelated changes still run the normal validation and service tests, while avoiding a paid model benchmark on every deployment. A failing required quality gate blocks the release.

After deployment, health and unauthenticated rejection checks confirm basic service behaviour. An authenticated client still needs an OAuth and permission acceptance check. Those checks answer different questions from the retrieval benchmark.

Current status and next evidence

The cloud service, private S3 note copy and full hybrid index are deployed at the raw API Gateway endpoint. The desktop OAuth callback rejection has been fixed and deployed; registration now reaches authorisation. Completing client sign-in and verifying its KB permissions remain separate onboarding steps.

The certificate for mcp.midu.com.au is issued. API Gateway custom-domain creation remains blocked by an AWS organisation permission, so the custom hostname is not live yet.

The next evidence should come from everyday use: owner-judged retrieval questions, whether captured learnings help with later tasks, and how reliably the service handles freshness, latency and client access. Those checks will show whether the knowledge loop is actually compounding useful experience.