“Nenex: A Neural Personal Wiki Idea”, 2023-09-13 (; backlinks):
Proposal for a personal wiki built on neural nets: all edits are logged & used to finetune a NN assistant in realtime.
Existing personal wikis or personal knowledge management tools (eg. Roam, Obsidian, org-mode) make highly limited use of the wave of powerful language & code-generating neural networks (LLMs like GPT-4), limited to minor improvements such as suggesting relevant links or offering copyediting suggestions.
This is due less to a lack of neural network capabilities than the difficulty of integrating them into document systems all designed in paradigms long predating LLMs. If Vannevar Bush or Douglas Engelbart were designing a ‘neural wiki’ from the ground up to be a ‘tool for thought’, taking GPT-4-level LLMs for granted, what would that look like?
It would probably not look like existing tools, which take a hypertext approach of a collection of independent nodes referencing each other and version-controlled as text files. Simple text-file-based approaches like copying in entire documents quickly run into performance limits or small but non-trivial error rates.
A more natural approach would be to draw inspiration from DL scaling paradigms in treating ‘everything as a sequence prediction task’: in this LLM-centric wiki paradigm (Nenex), the wiki would not be file/node-centric but edit-centric.
Instead of being hobbled by cloud providers optimizing for simplicity & jealous of their genericized chatbot models, you log all actions to train a local LLM to imitate you, and use the system alternating between taking actions and approving/disapproving execution of predicted actions. As data accumulates, the LLM learns not simply tool usage or generic text prediction, but prediction of your text, with your unique references, preferences, and even personality/values.
The wiki is represented not as a set of static files with implicit history, but in more of a revision-control system or functional programming style as a history of edits in a master log; the LLM simply learns to predict the next action in the log (using ‘dynamic evaluation’ finetuning for scalability).
All user edits, reference additions, spellchecks or new vocabulary addition, summarization, updates of now-outdated pages etc, are just more actions for the LLM to learn to predict on the fly. It can flexibly use embeddings & retrieval, simple external tools (such as downloading research papers), & operate over an API. A Nenex’s LLM can be easily upgraded by training new models on the Nenex log, additionally trained on all relevant information (private or public), and incorporate arbitrary feedback from the user.
A Nenex would interactively tailor itself to a user’s writing style, knowledge, existing corpus, and enable semantic features unavailable in other systems, such as searching a personal wiki for pages that need updating given updates to other pages.