Understanding node system
The Scope node system enables third-party extensions to provide custom pipelines. This document describes the architectural design and data flows that enable node discovery, installation, and lifecycle management.
Architecture Layers
Desktop App (Electron) → Frontend (React) → Backend (FastAPI) → Nodes
Key Technologies
- pluggy: Python hook system for pipeline registration and discovery
- uv: Fast Python package manager for dependency resolution and installation
- Electron IPC: Communication bridge between desktop app and frontend
System Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Desktop App (Electron) │
│ ┌─────────────┐ ┌──────────────┐ ┌───────────────────┐ │
│ │ Deep Links │───▶│ IPC Bridge │◀──▶│ Python Process │ │
│ │ File Browse │ │ │ │ Manager │ │
│ └─────────────┘ └──────────────┘ └───────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Frontend (React) │
│ ┌─────────────┐ ┌──────────────┐ ┌───────────────────┐ │
│ │ Settings UI │───▶│ API Client │───▶│ State Management │ │
│ └─────────────┘ └──────────────┘ └───────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│ HTTP
▼
┌─────────────────────────────────────────────────────────────────┐
│ Backend (FastAPI) │
│ ┌─────────────┐ ┌──────────────┐ ┌───────────────────┐ │
│ │ REST API │───▶│ Node Manager │───▶│ Pipeline Registry │ │
│ └─────────────┘ └──────────────┘ └───────────────────┘ │
│ │ │
│ ┌────────┴────────┐ │
│ ▼ ▼ │
│ ┌────────────┐ ┌────────────┐ │
│ │ Dependency │ │ Venv │ │
│ │ Validator │ │ Snapshot │ │
│ └────────────┘ └────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Node Discovery
Nodes integrate with Scope through Python’s entry point mechanism:
- Nodes declare entry points in their
pyproject.toml under [project.entry-points."scope"]
- The backend uses pluggy hooks to discover installed nodes at startup
- Each node implements a
register_pipelines hook to register its pipeline implementations
- The Pipeline Registry maintains a mapping of pipeline IDs to their implementations
Node Sources
| Source | Description |
|---|
| PyPI | Standard Python packages |
| Git | Direct repository installation |
| Local | File system paths (editable mode for development) |
Installation Flow
User Frontend Backend Desktop App
│ │ │ │
│ Click Install │ │ │
│───────────────────▶│ │ │
│ │ POST /nodes │ │
│ │────────────────────▶│ │
│ │ │ Validate deps │
│ │ │────────┐ │
│ │ │◀───────┘ │
│ │ │ Capture venv │
│ │ │────────┐ │
│ │ │◀───────┘ │
│ │ │ Install via uv │
│ │ │────────┐ │
│ │ │◀───────┘ │
│ │ 200 OK │ │
│ │◀────────────────────│ │
│ │ Request restart │ │
│ │─────────────────────────────────────────────▶
│ │ │ │ Respawn
│ │ │ │────────┐
│ │ │ │◀───────┘
│ │ Poll health │ │
│ │────────────────────▶│ │
│ │ Refresh pipelines │ │
│ │────────────────────▶│ │
- User initiates install (UI or deep link)
- Frontend sends install request to backend API
- Backend validates dependencies won’t conflict with existing environment
- Backend captures current venv state (for rollback)
- Backend resolves and installs dependencies via uv
- Backend updates node registry
- Frontend triggers server restart
- Server restarts with new node loaded
- Frontend polls until server is healthy
- Frontend refreshes pipeline list
Rollback: If installation fails at any step, the venv is restored to its captured state.
Node Update Flow
The update flow has two phases: detection (happens automatically when nodes are listed) and execution (triggered by the user). The execution phase reuses the installation flow with an upgrade: true flag.
Update Detection
Frontend Backend
│ │
│ GET /nodes │
│───────────────────────────────▶│
│ │ For each node:
│ │ _check_node_update()
│ │────────┐
│ │ │ Compare installed
│ │ │ vs latest version
│ │◀───────┘
│ Node list │
│ (with update_available flags) │
│◀───────────────────────────────│
- Frontend fetches the node list via
GET /nodes
- Backend iterates over installed nodes and calls
_check_node_update() for each
- For PyPI nodes, the installed version is compared against the latest version on PyPI
- For Git nodes, the installed commit hash is compared against the latest commit on the remote
- Local nodes are skipped (use Reload instead)
- Each node in the response includes an
update_available flag
- Frontend displays an update badge on nodes where the flag is
true
Update Execution
User Frontend Backend Desktop App
│ │ │ │
│ Click │ │ │
│ Update │ │ │
│──────────▶│ │ │
│ │ POST /nodes │ │
│ │ {upgrade: true} │ │
│ │────────────────────▶│ │
│ │ │ Capture venv │
│ │ │────────┐ │
│ │ │◀───────┘ │
│ │ │ Compile with │
│ │ │ --upgrade-package │
│ │ │────────┐ │
│ │ │◀───────┘ │
│ │ │ Sync deps │
│ │ │────────┐ │
│ │ │◀───────┘ │
│ │ 200 OK │ │
│ │◀────────────────────│ │
│ │ Request restart │ │
│ │─────────────────────────────────────────────▶
│ │ │ │ Respawn
│ │ │ │────────┐
│ │ │ │◀───────┘
│ │ Poll health │ │
│ │────────────────────▶│ │
│ │ Refresh pipelines │ │
│ │────────────────────▶│ │
- User clicks the Update button on a node
- Frontend sends
POST /nodes with the node spec and upgrade: true
- Backend captures the current venv state (for rollback)
- Backend runs
uv pip compile with --upgrade-package targeting only the node package
- Backend syncs the environment with the newly resolved dependencies
- Backend updates the node registry
- Frontend triggers server restart
- Server restarts with the updated node loaded
- Frontend polls until server is healthy
- Frontend refreshes pipeline list
Rollback: If the update fails at any step, the venv is restored to its pre-update state, just like a failed installation.
Source-Specific Update Behavior
| Source | Detection Method | Notes |
|---|
| PyPI | Compares installed version against latest version on PyPI | Standard version comparison |
| Git | Compares installed commit hash against latest remote commit | Detects new commits on the default branch |
| Local | Skipped | Local nodes use Reload instead |
Uninstallation Flow
- User initiates uninstall
- Backend unloads any active pipelines from the node
- Backend removes node from registry
- Backend uninstalls package via uv
- Frontend triggers server restart
- Frontend refreshes pipeline list
Manual Reload Flow
For local/editable nodes, developers can trigger a reload to pick up code changes:
- Developer modifies code
- Developer clicks Reload button
- Frontend requests server restart
- Server restarts with fresh Python module imports
- Code changes take effect
In standalone mode (without the desktop app), the reload flow is the same except the server performs a self-restart instead of being respawned by the desktop app (see Standalone Mode below).Developer Frontend Backend Desktop App
│ │ │ │
│ Modify code │ │ │
│────────┐ │ │ │
│◀───────┘ │ │ │
│ Click Reload │ │ │
│───────────────▶│ │ │
│ │ Request restart │ │
│ │───────────────────────────────────────────▶│
│ │ │ │ Respawn server
│ │ │ │────────┐
│ │ │ │◀───────┘
│ │ Poll health │ │
│ │────────────────────▶│ │
│ │ Refresh pipelines │ │
│ │────────────────────▶│ │
Deep Link Installation
External sources can facilitate node installation via protocol URLs:
daydream-scope://install-node?package=<spec>
| Source | Raw Spec | Encoded URL |
|---|
| PyPI | my-node | daydream-scope://install-node?package=my-node |
| Git | git+https://github.com/user/repo.git | daydream-scope://install-node?package=git%2Bhttps%3A%2F%2Fgithub.com%2Fuser%2Frepo.git |
- External source opens the deep link URL
- Desktop app receives URL via OS protocol handler
- If app is starting: stores pending deep link for later processing
- Once frontend is loaded: sends action via IPC to renderer
- Frontend opens settings with Nodes tab and pre-filled package spec
- User confirms installation
Component Responsibilities
Backend Components
| Component | Responsibility |
|---|
| Node Manager | Singleton that manages node lifecycle (install, uninstall, update, reload) |
| Dependency Validator | Pre-validates that new packages won’t break existing environment |
| Venv Snapshot | Captures and restores environment state for safe rollback |
| Pipeline Registry | Maps pipeline IDs to their implementations and source nodes |
| REST API | Exposes node operations to frontend |
Frontend Components
| Component | Responsibility |
|---|
| Settings Dialog | User interface for node management |
| API Client | HTTP calls to backend node endpoints |
| Restart Coordinator | Handles server restart and health polling |
| Pipeline Context | Refreshes available pipelines after changes |
Desktop App Components
| Component | Responsibility |
|---|
| Python Process Manager | Spawns/respawns backend server, handles restart signals |
| Deep Link Handler | Receives and parses protocol URLs |
| IPC Bridge | Communicates between main process and renderer |
| File Browser | Native dialog for selecting local node directories |
The File Browser is a desktop convenience feature. In standalone mode, users can type local paths directly into the node installation input field.
Server Restart Protocol
Managed Mode (Desktop App)
When running in the desktop app, server restarts are handled automatically:
- Backend exits with code 42 (signals intentional restart)
- Desktop app waits for port release
- Desktop app respawns server process
- Frontend polls health endpoint until ready
Frontend Backend Desktop App
│ │ │
│ POST /restart │ │
│────────────────────▶│ │
│ │ Exit code 42 │
│ │─────────────────────▶│
│ │ │ Wait for port
│ │ │────────┐
│ │ │◀───────┘
│ │ │ Respawn server
│ │◀─────────────────────│
│ Poll /health │ │
│────────────────────▶│ │
│ 200 OK │ │
│◀────────────────────│ │
│ Resume ops │ │
│────────┐ │ │
│◀───────┘ │ │
- Exit code 42 signals intentional restart (not a crash)
- Brief wait ensures the port is released before respawn
- Frontend polls health endpoint until server is ready
- This ensures proper Python module reloading since imports are cached
Standalone Mode
When running the server directly (uv run daydream-scope):
- Unix/macOS: Uses
os.execv() to replace the current process in-place
- Windows: Spawns a new subprocess and exits the old one
Data Storage
Node state is persisted in the user’s data directory:
| Data | Location | Purpose |
|---|
| Node list | ~/.daydream-scope/nodes/nodes.txt | Installed package specs |
| Resolved deps | ~/.daydream-scope/nodes/resolved.txt | Lock file for reproducibility; baseline for update detection |
| Venv backup | ~/.daydream-scope/nodes/freeze.txt | Rollback state |
Error Handling
The node system uses defensive error handling at each stage:
| Error Type | Detection | Recovery |
|---|
| Dependency conflicts | Dry-run compilation before install | Installation blocked with clear error message |
| Installation failures | Exception during uv install | Venv rolled back to pre-install state |
| Runtime errors | Pipeline execution failure | Pipeline unloaded without affecting others |
| Network errors | Health check timeouts | Frontend retries with exponential backoff |
See Also
