Sync a data source

When your users connect another app to yours, such as a CRM, a helpdesk, a drive or a mailbox, what it holds for each of them can live in their memory and stay current. Four rules do it:

1. Each item under its own id. Add it with an external_id. Sent again unchanged, it costs nothing; changed, only what changed is learned, and what was removed is taken back. See Your own ids.
2. One label for the app. Add every item with the same label, such as {"channel": "crm"}, so recall can keep to it and one call reaches everything from it. See Labels.
3. What is gone, goes. A run that read everything ends with delete_labelled(labels, keep=...), which deletes what the app no longer has.
4. Disconnecting forgets it all. delete_labelled(labels) deletes everything from the app, and nothing the user added another way.

The whole sync

from geniffy import Geniffy

geniffy = Geniffy()                               # reads GENIFFY_API_KEY
LABELS = {"channel": "crm"}


def sync(user_id: str, records: list[dict]) -> None:
    """What the app holds for one user now: each record with its id, a title and its text."""
    mem = geniffy.space(f"user_{user_id}")
    for record in records:
        mem.memories.add(record["text"], title=record["title"], external_id=f"crm:{record['id']}", labels=LABELS)
    mem.sources.delete_labelled(LABELS, keep={f"crm:{record['id']}" for record in records})


def disconnect(user_id: str) -> int:
    """The user disconnected the app: everything from it goes. Returns how many records."""
    return geniffy.space(f"user_{user_id}").sources.delete_labelled(LABELS)

import { Geniffy } from "geniffy";

const geniffy = new Geniffy(); // reads GENIFFY_API_KEY
const LABELS = { channel: "crm" };

/** What the app holds for one user now: each record with its id, a title and its text. */
export async function sync(userId: string, records: { id: string; title: string; text: string }[]): Promise<void> {
  const mem = geniffy.space(`user_${userId}`);
  for (const record of records) {
    await mem.memories.add({ text: record.text, title: record.title, externalId: `crm:${record.id}`, labels: LABELS });
  }
  await mem.sources.deleteLabelled(LABELS, { keep: records.map((record) => `crm:${record.id}`) });
}

/** The user disconnected the app: everything from it goes. Resolves to how many records. */
export async function disconnect(userId: string): Promise<number> {
  return geniffy.space(`user_${userId}`).sources.deleteLabelled(LABELS);
}

Run it on a schedule, with what the app holds for each user read through that user's own sign-in to it, so a token only ever reaches what its user can see.

End with keep only after a read that succeeded. An empty list from a read that failed quietly would delete everything from the app, so let a failed read raise and stop the run.

Each user's, or everyone's

What a user brings from their own account, such as their mail, their drive or their CRM notes, goes in their own space, and nothing else can read it. What every user should be able to draw on, such as your help centre, your policies or a repository, goes in a space of its own, and your assistant recalls from it beside the user's:

shared = geniffy.space("site_help.acme.com").context(question)     # what everyone may know
known = geniffy.space(f"user_{user_id}").context(question)         # what is known about this user
system = f"<knowledge>\n{shared}\n</knowledge>\n\n<memory>\n{known}\n</memory>"

Put only what everyone may see in a shared space; then nothing one user said can reach another through it.

When the app says what changed

Reading everything each run is the simplest sync, and fine for a few hundred items: an unchanged one costs nothing. For more, most APIs list what changed since a token you keep, such as Drive's changes list, Gmail's history and Microsoft Graph's delta query. Read everything on the first run and keep the token. After that, add what changed, and delete what was removed by its id with mem.sources.delete(external_id=...). When the token runs out, read everything again and end with keep, and memory is put right.

Ready to copy

Gmail, Google Drive, Notion and OneDrive each have a complete sync, tested against a stand-in of that app's API.