Enterprise Knowledge Search
New Computer needed to improve Dot’s long-term memory retrieval so the assistant could answer personalized queries accurately across diverse intents without exposing private user data during experimentation.
Reported impact
New Computer improved retrieval quality for Dot, sped prompt iteration on highly variable conversational behavior, and converted a new wave of users more effectively by making the personal-AI experience feel more accurate and bespoke over time.
New Computer designed Dot around an agentic memory system where memories are created or pre-calculated first and only retrieved later, which makes retrieval quality a core product problem rather than a back-end detail. To tune that system safely, the team created cohorts of synthetic users with LLM-generated backstories, seeded their memory databases from initial conversations, and stored queries plus the full set of candidate memories in LangSmith datasets. Retrieval experiments compared simple semantic baselines against keyword, BM25, and meta-field-filtered methods using labeled examples and metrics for precision, recall, and F1. The same workflow was then extended to Dot’s dynamic conversational prompt, where LangSmith comparison views made prompt regressions visible and the built-in playground let the team adjust prompts without leaving the tool. This turned privacy-preserving retrieval tuning into a repeatable experimental loop.
Synthetic-user conversations seed memory databases, user-style queries and the available memories are stored in LangSmith datasets, multiple retrieval methods are evaluated against labeled relevance sets, and the best memories are fed back into Dot’s dynamic conversational prompt for personalized responses.
40% higher precision
50% higher recall
creating a cohort of synthetic users with LLM-generated backstories
more than 45% of which converted to the app’s paid tier
the full set of available memories in a LangSmith dataset
Review how this reference was scored and trace each normalized claim back to its documented source.
Scoring methodology
Customer Reference Value measures usefulness as a customer reference. Evidence Confidence measures the quality and traceability of available evidence, not independent verification.
Traceability
Field coverage shows which facts each source supports. Dates are shown only when documented by the source.
The official privacy policy names "Foobarbaz Inc. dba New Computer" and lists the address "251 Little Falls Drive, Wilmington, DE 19808."
LinkedIn's Organization JSON-LD lists numberOfEmployees value 3 and sameAs "https://new.computer".
Related
LangChain Sample Dec 16, 2024 AppFolio AppFolio needed a natural-language interface that could help property managers query portfolio state, execute bulk actions, and troubleshoot complex workflows while preserving low latency, observability, and prompt… performance significantly increased from ~40% to ~80% LangChain · LangGraph · LangSmith LangChain Sample Jul 9, 2024 Wordsmith Wordsmith needed to observe, evaluate, and continuously optimize increasingly complex legal-assistant workflows that spanned multiple data sources, model providers, and experiment branches without slowing development or… compare its performance to GPT-4o within an hour LangSmith · Slack · Statsig 
MongoDB Sample Sep 9, 2026 Squary AI Squary AI needed to eliminate slow, outage-prone legal search infrastructure that combined MongoDB with OpenSearch and hurt document review speed, cost, and trust. At Quatrini Law Group, Squary AI analyzed more than 460,000 pages and saved over 400 attorney and paralegal hours. Amazon Bedrock · Claude · MongoDB Atlas 