Hi Jacob, thanks for sharing! Witchcraft looks super interesting—starring the repo now.
The late interaction (ColBERT / MaxSim token-level) approach is fascinating for local search. Retaining per-token interaction matrices gives incredible semantic retrieval precision without losing fine-grained token context.
From an architectural standpoint, the main trade-off I explored with Hillock was moving away from dense token-embedding matrices entirely toward an explicit neuro-symbolic split:
1. Symbolic Grounding: Hard Subject-Predicate-Object (SPO) triples in relational SQLite tables to eliminate vector drift for exact factual memory.
2. Subsymbolic VSA Gating: A 10,000-dimensional Vector Symbolic Architecture (HDC) hypervector space on CPU using subword n-grams and GloVe SimHash projections for <1ms gating and pronoun resolution.
3. Edge VRAM Footprint: Keeping the whole engine under 1.2 GB VRAM on a GTX 1070 by using a CUDA bi-encoder pipeline (Fastcoref + MiniLM + GLiREL) for non-generative document parsing.
I'm curious—how do you manage the index footprint and per-token memory overhead in Witchcraft when scaling to larger local document collections?
There is no memory overhead, in that everything resides on disk/in the sqlite database, and is read from there on demand. That said, sqlite for blob storage is a bit inefficient, so the next version will use tightly packed structs on disk. I also managed to finetune an embedding down to 96d, so in the coming version the storage for the index is getting close to parity with the size of the text being indexed.
The late interaction (ColBERT / MaxSim token-level) approach is fascinating for local search. Retaining per-token interaction matrices gives incredible semantic retrieval precision without losing fine-grained token context.
From an architectural standpoint, the main trade-off I explored with Hillock was moving away from dense token-embedding matrices entirely toward an explicit neuro-symbolic split:
1. Symbolic Grounding: Hard Subject-Predicate-Object (SPO) triples in relational SQLite tables to eliminate vector drift for exact factual memory. 2. Subsymbolic VSA Gating: A 10,000-dimensional Vector Symbolic Architecture (HDC) hypervector space on CPU using subword n-grams and GloVe SimHash projections for <1ms gating and pronoun resolution. 3. Edge VRAM Footprint: Keeping the whole engine under 1.2 GB VRAM on a GTX 1070 by using a CUDA bi-encoder pipeline (Fastcoref + MiniLM + GLiREL) for non-generative document parsing.
I'm curious—how do you manage the index footprint and per-token memory overhead in Witchcraft when scaling to larger local document collections?
Excited to dig deeper into your codebase!