Matthew Grotke 0cec7d69c9 Development

2026-06-06 14:55:29 -04:00

19 KiB

Raw Blame History

Developer Guide

This guide covers how to add features, write action handlers, and follow the conventions used throughout the codebase.

Architecture Overview

This app is the web UI for Routlin, a Linux-based router software package. It runs as a Docker container and manages the same configuration files that Routlin's core scripts read and apply to the system. The app does not manage the system directly; it edits config.json and queues commands that Routlin's core.py executes.

The app is a Flask application in a Docker image.

app/                  Python source (baked into image)
  factory.py          Converts content.json item trees into HTML strings
  view_page.py        Routes, data loaders, token assembly, layout rendering
  navbar.json         Navigation structure
  pages/              One subdirectory per page
    <pagename>/
      content.json    Declarative page layout
      action.py       Flask Blueprint for POST actions on this page
      page.js?        Optional page-specific client-side JS (auto-included if present)
  sanitize.py         Input sanitization (strips dangerous characters)
  config_utils.py     Config I/O, change history (SQLite), command queue
  authorized_accounts.json  Web UI user accounts (separate from Routlin users)

www/                  Live-mounted at runtime (./www:/www)
  styles.css          Application stylesheet (inlined in dev, served as /www/styles.css in production)
  common.js           Shared client-side JS (inlined in dev, served as /www/common.js in production)
  icons/              SVG icon files (always inlined by factory.py)

                      # host directory mounted into container: $HOME/routlin -> /routlin_location
routlin_location/    Routlin install dir
  config.json         Main Routlin configuration (read on every request, written on save)
  validation.py       Shared validation module (imported via PYTHONPATH)
  core.py             Apply script invoked to push config changes to the system
  ddns.log            DDNS update log
  blocklists/         Downloaded blocklist files (*.con)
  .dashboard-snapshots  SQLite database storing change history and revert records
  .health             Health status JSON written by Routlin
  .dashboard-queue    Pending commands waiting to be applied
  .dashboard-done     Record of completed commands
  .dashboard-pending  Commands queued but not yet run
  .dashboard-last-run Timestamp of the last apply run
  .dashboard-lock     Lock file indicating an apply is in progress
  .ddns-last-ip-*     Cached public IP files, one per DDNS provider
  .ddns-last-service  Timestamp of the last DDNS service check
  .<iface>.pub        WireGuard public key files (e.g. .wg0.pub)

Shared Resources with Routlin

The Routlin install directory ($HOME/routlin on the host) is mounted into the container at /routlin_location and added to PYTHONPATH. This is the primary integration point between the dashboard and the router software.

Files and directories under /routlin_location that the app reads or writes:

Path	Access	Description
`config.json`	read/write	Main Routlin configuration. The app reads this on every request and writes it on every save action.
`validation.py`	import	Shared validation module imported as `import validation as validate`. Contains field validators and `validate_config()` used by all action handlers.
`core.py`	exec	Routlin's apply script. The app invokes it as `python3 core.py --apply` or `--update-blocklists` to push config changes to the system.
`ddns.log`	read	DDNS update log shown in the DDNS page.
`blocklists/`	read	Directory of downloaded blocklist files (`*.con`). The app reads them to count entries and report last-updated timestamps.
`.dashboard-snapshots`	read/write	SQLite database storing change history groups and field-level diffs, used for the change history and revert feature.
`.health`	read	JSON file written by Routlin describing service and configuration health status. The dashboard reads it to display the health panel and auto-queue fixes.
`.dashboard-queue`	read/write	Pending commands waiting to be applied.
`.dashboard-done`	read/write	Record of completed commands.
`.dashboard-pending`	read/write	Commands that have been queued but not yet run.
`.dashboard-last-run`	read	Timestamp of the last dashboard apply run.
`.dashboard-lock`	read	Lock file checked to determine if an apply is currently in progress.
`.ddns-last-ip-*`	read	One file per DDNS provider, written by Routlin with the last-known public IP. The app reads all matching files to display the current IP.
`.ddns-last-service`	read	Timestamp written by Routlin each time the DDNS service runs. Displayed as "Last checked" on the DDNS page.
`.<iface>.pub`	read	WireGuard public key files (e.g. `.wg0.pub`). Read by the VPN page to display the server public key.

The container also mounts /sys/class/net and /sys/devices read-only to read live network interface state (link status, speed, MAC address, MTU), and /etc/localtime read-only for correct timezone display.

Code Style

ASCII only in all files. No em dashes, en dashes, curly quotes, or any non-ASCII character. If a sentence needs a pause that would normally use a dash, restructure it.
No comments unless the WHY is non-obvious. Well-named functions and variables explain the what. Comments belong on hidden constraints, subtle invariants, or workarounds for specific external bugs.
No docstrings on obvious functions. Reserve them for functions with non-obvious behavior or important invariants to document.
import validation as validate - the module is aliased because validate is a common local variable name and shadowing it causes confusion.
Imports at top, no inline imports except in rare cases where a circular import cannot otherwise be avoided (e.g. the from flask import abort inside serve_view).

Naming Conventions

Underscore prefix

Use _ prefix only for tiny helpers used in exactly one place. Do not use it for:

Module-level constants (NAVBAR_FILE, not _NAVBAR_FILE)
Functions reused by more than one caller
Imports (import factory, not import factory as _factory)

Good uses of _:

A two-line inner function defined inside a larger function and called once
A short closure that is never referenced outside its enclosing scope

File path constants

Define all file paths as named constants at the top of the module, using os.path.join:

PAGES_DIR    = os.path.join(APP_DIR,     'pages')
NAVBAR_FILE  = os.path.join(APP_DIR,     'navbar.json')
CSS_FILE     = os.path.join(DATA_DIR,    'styles.css')

Never hardcode paths inline.

Section headers in Python files

Use the # Label ===... style for major section breaks:

# File loaders ======================================================

# Config data loaders ===============================================

# Routes ============================================================

HTML Building

All HTML in factory.py and view_page.py is built by string concatenation using f-strings, not a template engine. Every value that comes from user data or config must be passed through e() before interpolation:

from factory import e

html = f'<td class="table-cell">{e(row["description"])}</td>'

e() wraps html.escape(). Never skip it for user-controlled or config-sourced values.

When building multi-line HTML inside view_page.py for a token, follow the same e() discipline and assign the result to tokens['MY_TOKEN']. Mark values as Markup() only when they are already-safe assembled HTML that must not be double-escaped at the outer render stage.

Key principle: factory.py is a pure HTML builder. It has no routing, no data loading, and no Flask context other than reading the session for access level. All data loading and token assembly happens in view_page.py. The two modules are kept separate to avoid circular imports; view_page.py injects factory.load_datasource = load_datasource at startup.

Adding a New Page

1. Create the page directory

app/pages/<pagename>/
  __init__.py         Empty file (makes it a Python package)
  content.json        Page layout
  action.py           POST action handlers

<pagename> becomes the URL: /pagename.

2. Write content.json

Every content.json starts with a client_requirement and an items array:

{
  "client_requirement": "client_is_viewer+",
  "items": [
    {
      "type": "header_page_title",
      "items": [
        { "type": "h1", "text": "My Page" },
        { "type": "p",  "text": "Description here." }
      ]
    }
  ]
}

See TYPES.md for the full set of item types and their fields. See TYPES.md#access-control for the full set of client_requirement values.

3. Write action.py

from pathlib import Path
from flask import Blueprint, request, redirect, flash
from auth import require_level
from config_utils import load_config, record_group, diff_fields, verify_config_hash
import sanitize
import validation as validate

_PAGE = Path(__file__).parent.name
bp = Blueprint(_PAGE, __name__)

_PAGE is the canonical page name and redirect target. Using Path(__file__).parent.name means it stays correct even if the directory is renamed.

4. Register the blueprint

In app/main.py (or wherever blueprints are registered), add the new action blueprint alongside the existing ones.

5. Add page.js (optional)

If the page needs client-side behavior not shared with other pages, create app/pages/<pagename>/page.js. It is automatically included - no registration needed. See the "Page-Specific JavaScript" section for guidance on what belongs here vs common.js.

Token System

Tokens are %LIKE_THIS% placeholders in content.json string fields. They are substituted before the page is rendered. Tokens carry dynamic data (config values, computed HTML, JSON arrays) from Python into the declarative layout without embedding Python logic in the JSON.

Tokens are assembled in view_page.py's collect_tokens() function:

tokens['VLAN_SUBNET'] = vlan.get('subnet', '')
tokens['EXISTING_VLAN_IDS_JSON'] = json.dumps([v['vlan_id'] for v in vlans])

String tokens that resolve to a JSON array or object are automatically parsed back into Python structures by factory.expand_fields(), so they can be serialized correctly into data-fields attributes.

Token values are not HTML-escaped by default. When a token is used in an HTML attribute or text node, factory.py calls e() on it. When a token resolves to raw HTML meant for injection (e.g. PENDING_ACTIONS_HTML), it is used as-is and must be safe before it enters the token map.

Action Handler Pattern

Every POST action follows the same sequence: parse, validate input, check config hash, mutate config, validate config, save.

@bp.route('/action/bannedips/addip_add', methods=['POST'])
@require_level('administrator')
def addip_add():
    # 1. Parse and validate individual fields first, before touching config.
    raw = request.form.get('ip', '').strip()
    ip = validate.banned_ip(raw)
    if not ip:
        flash('Invalid IP address.', 'error')
        return redirect(f'/{_PAGE}')

    description = sanitize.text(request.form.get('description', ''))

    # 2. Check config hash (stale-config detection / CSRF guard).
    if not verify_config_hash(request.form.get('config_hash', '')):
        flash('Configuration was modified by another session. Please refresh and try again.', 'error')
        return redirect(f'/{_PAGE}')

    # 3. Load, mutate, validate the full config.
    cfg = load_config()
    entry = {'ip': ip, 'description': description, 'enabled': True}
    cfg.setdefault('banned_ips', []).append(entry)

    errors = validate.validate_config(cfg)
    if errors:
        for msg in errors:
            flash(msg, 'error')
        return redirect(f'/{_PAGE}')

    # 4. Record, save, and queue.
    changes = diff_fields({}, entry)
    flash(record_group(cfg, 'banned_ips', 'ip', ip, changes, 'core apply'), 'success')
    return redirect(f'/{_PAGE}')

Rules:

Always redirect back to /{_PAGE} after a POST, never render HTML directly.
flash() is the only feedback mechanism. Use 'error' or 'success' as the category.
Parse all user input before checking the config hash. If input is invalid you want to bail out cheaply before acquiring anything.
Always call validate.validate_config(cfg) on the mutated config before saving, even for simple operations.
record_group(cfg, parent_path, item_key, item_value, changes, cmd) saves the config, records the change in the SQLite history, queues the command, and returns a human-readable flash message string. Use diff_fields(before, after) to produce the changes argument.

Row index actions

For table row operations, extract the row index and bounds-check it:

def _row_index():
    try:
        return int(request.form.get('row_index', ''))
    except (ValueError, TypeError):
        return None

idx = _row_index()
if idx is None:
    flash('Invalid request.', 'error')
    return redirect(f'/{_PAGE}')

items = cfg.get('my_list', [])
if idx < 0 or idx >= len(items):
    flash('Entry not found.', 'error')
    return redirect(f'/{_PAGE}')

Input Handling

Sanitize vs. validate

sanitize.* functions strip or normalize raw input so it is safe to store. They do not return errors; they return a cleaned value.
validate.* functions check whether a value is semantically correct for a specific field. They return a cleaned/normalized value on success or a falsy value (empty string, None) on failure.

description = sanitize.text(request.form.get('description', ''))  # always safe
ip = validate.banned_ip(raw)                                       # returns None on failure

Checkbox fields

Checkboxes are absent from the form body when unchecked. Both forms are equivalent and acceptable:

enabled  = request.form.get('enabled') == 'on'   # explicit value comparison
logging  = 'logging' in request.form              # presence check

JSON fields from record_editor

record_editor submits its data as a JSON string in a hidden input. Parse it defensively:

import json
raw = request.form.get('server_identities', '[]')
try:
    identities = json.loads(raw)
    if not isinstance(identities, list):
        raise ValueError
except (ValueError, TypeError):
    flash('Invalid identity data.', 'error')
    return redirect(f'/{_PAGE}')

Access Control

In content.json

Any item can carry "client_requirement": "client_is_administrator+". Items that fail the check are omitted from the rendered output entirely. See TYPES.md#access-control for the full table.

The page-level client_requirement is inherited by all child items that do not declare their own. This means you can set client_is_viewer+ at the page level and only gate specific cards or buttons at a higher level.

In action.py

@require_level('administrator')
def my_action():
    ...

Valid levels: nothing, viewer, administrator, manager. The decorator enforces the minimum rank and handles the redirect automatically.

Config Persistence

load_config() reads config.json fresh on every call. Do not cache it across a request.

record_group(cfg, parent_path, item_key, item_value, changes, cmd) saves the config, records a change group in the SQLite history database, and queues the apply command. diff_fields(before, after) computes the field-level diff to pass as changes. Always take a copy.deepcopy of the relevant sub-object before mutating so before is accurate.

verify_config_hash(hash) confirms the config has not changed since the form was rendered. Always check it before mutating the config.

Client-Side Validation

Attach a validate field to any field item in content.json to enable live validation. The value is one or more VALIDATION_* flag names joined by |:

{ "type": "field", "name": "nat_ip", "validate": "VALIDATION_IPV4_FORMAT|VALIDATION_UNRESTRICTED" }

Factory converts the flag names to a bitmask integer stored in data-validate. The client-side bigValidate() function reads this mask and runs only the enabled checks on every keystroke, marking the field valid (green), incomplete (yellow), or invalid (red). The form submit button stays disabled until all validated fields pass.

See TYPES.md#validation for the full flag table.

VALIDATION_UNRESTRICTED is special: when present, factory automatically reads the current restricted VLAN subnets from config and injects them into data-existing-ids on the input. No extra token is needed.

For record_editor fields, use validate (or the legacy alias valtype) on each field definition. The same VALIDATION_* flags apply.

For data that must be passed to a JS validator (such as existing VLAN IDs for uniqueness checking), use a data-* HTML attribute on the input rather than a global JS variable. The existing_ids field on a field item emits data-existing-ids on the rendered input.

Page-Specific JavaScript

If a page needs client-side behavior that is not shared with any other page, put it in app/pages/<pagename>/page.js. view_page.py automatically appends this file to the inline script bundle when it exists.

app/pages/ddns/page.js               -- credential provider toggle
app/pages/physicalinterfaces/page.js -- interface config card wiring
app/pages/actions/page.js            -- history revert button state
app/pages/networklayout/page.js      -- is_vpn -> mdns_reflection sync

page.js runs after common.js, so all shared utilities (htmlEsc, showCard, tablePickerCloseAll, etc.) are available.

In production mode (PRODUCTION_MODE=1 env var), common.js and styles.css are served as separate files from /www/ with Cache-Control: public, max-age=86400 and an mtime-based ?v= query string for cache busting. page.js and the generated bigValidate() function are always inlined regardless of mode.

Rules for what belongs in page.js vs common.js:

page.js: behavior that only activates on one page (specific field names, CSS classes, or form actions that appear nowhere else).
common.js: shared infrastructure used by multiple pages, or global event handlers (click-to-close, UUID hover highlight, stat card edit mode).
Inline <script> from factory: per-element wiring emitted by factory at render time (form validation, table-picker row wiring). Factory uses document.currentScript.previousElementSibling to scope the script to the element it follows.

JS Table Workers

Non-standard input_type values in inline_edit row actions require a table worker. factory.py detects any input_type not in STANDARD_INPUT_TYPES = {'text', 'password', 'number', 'checkbox', 'select', 'textarea'} and emits a <script> block via build_table_worker_script() that registers the worker using registerTableWorker(id, impl).

Workers implement:

renderCell(fDef, td, val, row) - returns true if handled, false to fall through to default text input
afterRowOpen(tr, row) - optional; called after all cells are rendered; use it to wire cross-field listeners

The worker ID is derived from the table's datasource field by stripping the config: or live: prefix. The only current non-standard type is credentials on the DDNS accounts table.

19 KiB Raw Blame History