# obsidian-node-canvas During the hackathon I **conceived**, **prototyped** and **put to use** a new *tool*: **_obsidian-node-canvas_** **pitch video**: [obsidian-node-canvas video](https://www.youtube.com/watch?v=0ZQYGhy3d9E) **pitch presentation**: maybe unusual but hopefully highly effective - see [here](https://raw.githubusercontent.com/Davidiusdadi/obsidian-node-canvas/master/examples/tutorial/tutorial.png) and "follow the arrows". **mock up screens**: - there is a working prototype but (described below) - [mockups of planned further development](https://github.com/Davidiusdadi/obsidian-node-canvas/tree/master/examples/hackathon/mockup) (see the 3 `png`-files there) - planned developments also have a dedicated section below ## What is it **[Obsidan-Node-Canvas](https://github.com/Davidiusdadi/obsidian-node-canvas)** is a `javascript`-based workflow/scripting engine, build on top of [obsidian](https://obsidian.md/) (the popular and powerful journaling, knowledge bases, and project management tool), specifically on it's [canvas feature](https://obsidian.md/canvas). The name "obsidian-node-canvas" was chosen as this tool combines obsidian canvas with the power of [nodejs](https://nodejs.org). **Essentially we are turning a canvas into essentially a executable workflow.** Workflows are made out of markdown-"nodes"/snippets - whose code is used as programmatic instructions. Markdown-code-blocks will be executed and the visual canvas-arrows are used to control the flow of the program. The real power of such a "canvas" lies not only in the **ability to execute arbitrary Javascript** but also in the ability to intuitively **tie in together custom nodes** that e.g.: - interface with LLMs - interface with the one's obsidian vaults - tie in e.g. REST-APIs Here a visual excerpt of 2 styles of LLM-invocation that are supported already. The screenshoted canvas here is not *conceptual* but **runnable**:  Running it (via a command line call) will produce the following output:  To see the whole canvas or an image of it see [repo/examples/tutorial](https://github.com/Davidiusdadi/obsidian-node-canvas/tree/master/examples/tutorial) You could reproduce this yourself following the `README.md`-instructions. ## Putting obsidian-node-canvas to use During prototyping obsidian-node-canvas has already been put to use: ### Use case 1: ingest Moos Knowledge Base into community map In collaboration with other hackathon-participants - - Anton with the Community Map (https://moos-map.de/) - Luiz with the Moos Knowledge Base I wrote a `.canvas` that: - goes through communities/places/projects in Luiz's vault - around ~70 in number - resolves their addresses (if present) into geo-cooridantes - and places them on the https://moos-map.de/ via API *If developed further* this could be turned into a synchronization tool between both "spaces". Source: [repo/examples/hackathon/cofi-berlin-to-moos-map](https://github.com/Davidiusdadi/obsidian-node-canvas/tree/master/examples/hackathon/cofi-berlin-to-moos-map) ### Use case 2: visualize project description embeddings I figured that https://moos-map.de/ could benefit from similarity correlation via [word embeddings](https://platform.openai.com/docs/guides/embeddings). This would allow to suggest similar places independent of geographic location. Also it's not technically part of this submission I build such an [embedding API]( https://github.com/utopia-os/utopia-mesh-api) and [deployed it](https://lionfish-app-a8os7.ondigitalocean.app/places?vector=1). Via a `.canvas` file I then consumed that API and prototyped a visualization as `svg`-file-render:  This render was created early in the hackathon when there had been only a few places ingested manually. The image shows a [t-distributed stochastic neighbor embedding](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding) of the projects based on their description-texts (pulled from embedding API mentioned above). *If developed further* it could grow into a way to discover related places visually. Similar places would bunch together by way of the algorithm Source: [repo/examples/hackathon/map-word-vector-t-sne](https://github.com/Davidiusdadi/obsidian-node-canvas/tree/master/examples/hackathon/map-word-vector-t-sne) ### Use case 3: web crawling + LLM to find addresses On the way of wanting to feed data into https://moos-map.de/ I wanted to utilize LLMs in combination with web crawling. - I manually pulled [gaianet.earth/ecosystem](https://airtable.com/appVC7loBsmy3fcJw/shrb2A5dyfF0XofUr/tblQVaLTcxW9ARaoV?backgroundColor=purple&viewControls=on) into a [gist](https://gist.github.com/Davidiusdadi/ba43c44fd2966bf2ce2366ddafc6cd4f) - wrote the `.canvas` that will: - use the gist address list as start point - then use [Puppeteer](https://pptr.dev/) to fetch each page - find and extract addressed via LLM - and finally resolve the address via [a geo API](https://photon.komoot.io/) into geo-coordinates The resulting dataset was not ingested as it is pretty good but not flawless as - some sites e.g. mention addresses that do not belong to them-self but e.g. to related events. - some detected addresses are not specific enough - e.g. only refer to a city *If developed further* this could be developed further: - to produce and ingest a higher quality data set - to produce a bigger dataset by building a more fully featured [web crawler](https://en.wikipedia.org/wiki/Web_crawler) - that would e.g. follow links, explore domains thoroughly, utilize search engines - by utilizing learnings form the [Berlin Mapathon](https://github.com/utopia-os/Berlin-Mapathon) (another hackathon-submission) Source: [repo/examples/hackathon/gaia-net-ecosystem-crawler](https://github.com/Davidiusdadi/obsidian-node-canvas/tree/master/examples/hackathon/gaia-net-ecosystem-crawler) ### technical note This being a very technical submission I want to **highlight a few technical challenges** that were overcome and which will impact further development: - [CommonJS](https://en.wikipedia.org/wiki/CommonJS) / [ESM](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) compatibility issues where solved via [esbuild]( https://esbuild.github.io/) - initially [bun.sh](https://bun.sh/ )was to be used instead of [nodejs](https://nodejs.org) but this was dropped as bun does not support [jsdom](https://www.npmjs.com/package/jsdom) which is important for for some web-crawling features like e.g. [readability](https://github.com/mozilla/readability) DOM-transforms. - to make the code in the canvas runnable - `ts` nodes are transpiled to JavaScript - `import` statements are converted into `await import` ([more info](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import)) - `yaml` is not just being parsed but basic [nunjucks](https://mozilla.github.io/nunjucks/) templating is supported **Implementation detail**: The execution engine that was build to always execute one node at a time. This makes the engine fully predictable. No race conditions. :) Building the engine itself, rather than implementing the use cases, has been the hardest part of the project so far. E.g. users need to be able to not only *direct* and *split* flow via arrows but also reunite flows. This resulted in 2 engine features: - **joining** flows together e.g. based on a join criteria (canvas: [cofi-berlin-to-moos-map](https://github.com/Davidiusdadi/obsidian-node-canvas/tree/master/examples/hackathon/cofi-berlin-to-moos-map)) - this required keeping track of how a node has already been called - if a join is not possible yet at runtime, internally an `exception` is used to abort the premature node invocation - side note: this is like an SQL inner join - **aggregating** flows - aka "wait for everything before me to complete" (canvas: [tutorial](https://github.com/Davidiusdadi/obsidian-node-canvas/tree/master/examples/tutorial)) - in addition to tracking node invocations this requires tracking of "ancestor"-nodes - only when no ancestor is scheduled any more the aggregation is complete and the node will be invoked # Further implementation plans The use-cases show that obsidian-node-canvas can solve technical problems for individuals and communities and can be extended to allows even richer and intuitive workflows. The general agenda is to make it better. - here also again the [mockups of planned further development](https://github.com/Davidiusdadi/obsidian-node-canvas/tree/master/examples/hackathon/mockup) - A full feature wish list is here: [README.md#plans](https://github.com/Davidiusdadi/obsidian-node-canvas/blob/master/README.md#plans) The most immediate further plans are: - to stabilize API - add tutorial and documentation - allow intuitive use of: - LLMs - via [ollama](https://ollama.com/) - obsidian vault data - via [obsidian-dataview](https://blacksmithgu.github.io/obsidian-dataview/) - [Puppeteer](https://pptr.dev/) browser automation - allow reusable nodes and canvas (go fully: [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)) - canvas need to be able to call each other - generally replace the need for manual-js-hacking with specialized well documented node types - extend flow logic - to allow loops - to allow more granular aggregation - allow to use chart to act as a web-server - allow visual debugging via a web-server - allow consumption via `npx` , as library, via bun Thanks for making it all the way through. 🙏