Website
Keep a website in a memory of its own: your help centre, your prices, your opening hours. This sync reads
the site's sitemap, fetches each page and adds it under its address, in a space for the site. Run it on a
schedule. A page whose lastmod hasn't changed isn't fetched again, a page that changed teaches only what
changed, and a page the sitemap no longer lists is deleted with what it taught. Your assistant then recalls
from the site beside each user's own memory.
Install
pip install httpx geniffyuv add httpx geniffySet GENIFFY_API_KEY from API keys in the Geniffy app.
The sync
import xml.etree.ElementTree as ET
from urllib.parse import urlparse
import httpx
from geniffy import BadRequestError, Geniffy
geniffy = Geniffy() # reads GENIFFY_API_KEY
LABELS = {"channel": "website"}
SITEMAP = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
def web() -> httpx.Client:
return httpx.Client(timeout=30, follow_redirects=True, headers={"User-Agent": "site-sync/1.0"})
def listed(client: httpx.Client, sitemap_url: str) -> dict[str, str]:
"""Every page the sitemap lists, with when it last changed ("" when it doesn't say). A sitemap index is
followed to each sitemap it names."""
pages, todo = {}, [sitemap_url]
while todo:
root = ET.fromstring(client.get(todo.pop()).raise_for_status().content)
for entry in root:
loc = (entry.findtext(SITEMAP + "loc") or "").strip()
if entry.tag == SITEMAP + "sitemap" and loc:
todo.append(loc)
elif entry.tag == SITEMAP + "url" and loc:
pages[loc] = (entry.findtext(SITEMAP + "lastmod") or "").strip()
return pages
def space_for(sitemap_url: str) -> str:
return "site_" + urlparse(sitemap_url).hostname
def sync(sitemap_url: str, last: dict[str, str] | None = None) -> dict[str, str]:
"""Bring a website into its own space. Returns what the sitemap listed: pass it in next time, and a page
whose lastmod hasn't changed isn't fetched again."""
mem, last = geniffy.space(space_for(sitemap_url)), last or {}
with web() as client:
pages = listed(client, sitemap_url)
for url, lastmod in pages.items():
if lastmod and last.get(url) == lastmod:
continue # unchanged since the last run
got = client.get(url)
if got.status_code != 200 or "html" not in got.headers.get("content-type", ""):
continue # down for now, or not a page: memory keeps what it had
try:
mem.memories.add_file(got.content, filename="page.html", title=url, external_id=f"web:{url}",
labels=LABELS)
except BadRequestError: # a page with no text in it
pass
mem.sources.delete_labelled(LABELS, keep={f"web:{url}" for url in pages})
return pagesRun it on a schedule, and keep what each run returns for the next: pages = sync(SITEMAP_URL, pages). Your
server fetches the pages itself, so a site behind a sign-in works too: give web() the cookie or header it
needs. Pages without a lastmod are fetched every run, and one that hasn't changed teaches nothing new.
Recall from it
The site has a space of its own, so every user's conversation can draw on it, beside what is known about that user:
site = geniffy.space(space_for(SITEMAP_URL)).context(question)
user = geniffy.space(f"user_{user_id}").context(question)
system = f"<site>\n{site}\n</site>\n\n<memory>\n{user}\n</memory>"How it behaves
- Each page is one source, under its address and titled by it, so what recall gives back cites the page. Changed, only the paragraphs that changed are learned. See Your own ids.
- A page the sitemap no longer lists goes, with what it taught. A page that is down for a run keeps what it had until it is back.
- Only pages are read: images, PDFs and anything else the sitemap lists that isn't HTML are left out. For PDFs, add them as files.
- One space for each site, named by its host, so two sites never mix. To forget a site, erase its
space:
geniffy.forget_space(space_for(SITEMAP_URL)).