Introduction

trueblocks-dalleserver is a thin, resilient HTTP façade around the separate trueblocks-dalle library. It turns a request of the form:

/dalle/<series>/<ethereum-address>?generate=1

into an annotated PNG image plus a bundle of prompt artifacts on disk, while surfacing live progress, structured errors, health, and lightweight metrics.

What this book covers (server only):

• Process & architecture (routing, middleware, concurrency model)
• Request parsing, validation and error response contract
• Progress polling contract returned by /dalle/...
• Circuit breaker + retry layers protecting OpenAI enhancement calls
• Metrics & health endpoints (JSON + Prometheus exposition)
• File system layout and robustness wrappers
• Configuration (flags, environment, .env loader)
• Preview gallery implementation
• Testing approach & extensibility hooks

What this book deliberately does not re‑document:

• Prompt assembly, attribute derivation, enhancement semantics
• The DalleDress or progress phase internal field-by-field description
• Series definition format or authoring guidance

For those topics, consult the upstream trueblocks-dalle book:

https://github.com/TrueBlocks/trueblocks-dalle/tree/develop/book

If reading inside a cloned mono workspace, the library book lives under the dalle/book directory.

High‑Level Flow

1. Client hits /dalle/<series>/<addr>?generate=1.
2. Handler validates series (against dalle.ListSeries()) and address format.
3. If an annotated image already exists and generation not forced, it is served directly.
4. Otherwise a background goroutine attempts dalle.GenerateAnnotatedImage(...) guarded by an in‑library per key lock (TTL from config) so duplicate concurrent requests coalesce.
5. While generation proceeds, repeated polls return a JSON progress document (sourced from the library progress package) until done=true.
6. On completion the PNG becomes available at the same URL (without ?generate) and under /files/<series>/annotated/<addr>.png and appears in the /preview gallery.

All prompt and image heavy lifting is delegated to the library; this server focuses on orchestration, resilience, and presentation.

Design Goals

Proceed to Getting Started to run the server locally.

Getting Started

This section launches the server locally using only the components in this repository. Prompt generation, attribute derivation and series logic reside in the separate trueblocks-dalle library—refer to its book for deeper internals.

Prerequisites

• Go 1.21+ (check with go version)
• (Optional) An OpenAI API key for real enhancement + image generation. Without it the server automatically switches to mock/skip mode (still exercising progress + caching paths).

Clone & Enter

git clone https://github.com/TrueBlocks/trueblocks-dalleserver.git
cd trueblocks-dalleserver

Environment Setup

Copy and edit an env file:

cp .env.example .env

Populate at minimum (fish shell example):

set -x OPENAI_API_KEY "sk-..."  # optional; omit to run in skip image mode

You can also export at runtime or rely on your shell profile. The minimal config for a mock run is literally nothing—absence of OPENAI_API_KEY implies skip image.

Run

make run

Output (abridged):

[status] reporter started (interval=2s)
Starting server on :8080

Visit:

• List series: http://localhost:8080/series
• Trigger generation: http://localhost:8080/dalle/simple/0xf503017d7baf7fbc0fff7492b751025c6a78179b?generate=1
• Poll progress (repeat same URL; done=true when finished)
• Gallery: http://localhost:8080/preview

Forcing Regeneration

Remove a cached annotated image by adding ?remove=1 then re‑issuing ?generate=1.

Graceful Shutdown

Ctrl+C sends SIGINT. The server initiates a 10s timeout graceful shutdown (http.Server.Shutdown) after which it force closes.

Next

Proceed to Running the Server for flags, port overrides and data directory details.

Running the Server

This section expands on invocation, flags, and shutdown behavior.

Invocation Forms

Simplest (build + run):

make run

Manual (explicit build then run):

go build -o dalleserver .
./dalleserver --port=9090

Flags (parsed once)

Note: repeated flag parsing during tests is ignored without failing.

Environment Variables (server owned)

Environment variables consumed only by the library (e.g. enhancement timeouts, quality) are intentionally not duplicated here—see the library book.

Skip / Mock Behavior

If no API key is detected the server still starts (emitting a warning). Progress phases execute up to the point of image acquisition which is simulated, producing annotated outputs fast for development.

Graceful Shutdown

SIGINT / SIGTERM triggers a 10s graceful shutdown window via http.Server.Shutdown. In-flight requests get that window to complete; after timeout the server force closes.

Timeouts & Protection

OpenAI enhancement gets its own context deadline (60s) inside openai_client.go with an additional client-level timeout buffer.

Status Printer

A background goroutine prints a concise table of active generations every 2s (to stderr). It is purely diagnostic and has no API surface.

Next

See Usage & Endpoints for request patterns, progress polling and the preview gallery.

Usage & Endpoints

All JSON responses include success and a correlation request_id (8-char UUID prefix).

Root (/)

Plain text enumeration of primary endpoints. Not intended for automation.

Series Listing (/series)

GET /series

Example:

{
	"success": true,
	"data": {"series":["simple"], "count":1},
	## Image Generation & Progress (`/dalle/<series>/<address>`)

	Forms:

	```
	GET /dalle/<series>/<address>
	GET /dalle/<series>/<address>?generate=1
	GET /dalle/<series>/<address>?remove=1
	```

	| Condition | Plain GET | `?generate=1` | `?remove=1` |
	|-----------|-----------|---------------|-------------|
	| Annotated PNG exists | Streams PNG | Returns progress (cacheHit true) | Deletes PNG (text confirmation) |
	| PNG missing; idle | `{}` (no spawn) | Spawns generation goroutine; returns early progress or `{}` | (No-op) |
	| PNG missing; active generation | Progress JSON | Same progress (lock prevents duplicate) | (No-op) |

	Validation errors → 400 with codes: `INVALID_SERIES`, `INVALID_ADDRESS`, `MISSING_PARAMETER`.

	The progress JSON (poll until `done=true`) is produced by the library; server only adds `request_id`.

	### Locking & Concurrency
	Per-key (series,address) lock with TTL (`--lock-ttl`) coalesces concurrent generation requests. Duplicate triggers only observe progress.

	### Removal
	Only the annotated PNG is removed; prompts persist.

	## Preview Gallery (`/preview`)
	HTML template enumerating `<data>/output/<series>/annotated/*.png` grouped by series, newest first, client-side filter input.

	## Static Files (`/files/`)
	Serve any file under the output directory (read-only). Example: `/files/simple/annotated/0xabc...png`.

	## Health (`/health`)

	Modes via `check` query parameter:

	| Mode | URL | Meaning | Codes |
	|------|-----|---------|-------|
	| Full | `/health` | Composite status + components (filesystem, openai, memory, disk_space). | 200 / 503 |
	| Liveness | `/health?check=liveness` | Process responsive. | 200 |
	| Readiness | `/health?check=readiness` | Ready unless overall unhealthy. | 200 / 503 |

	OpenAI status reflects circuit breaker state (CLOSED→healthy, HALF_OPEN→degraded, OPEN→unhealthy).

	## Metrics (`/metrics`)

	| Request | Format | Purpose |
	|---------|--------|---------|
	| `/metrics` | Prometheus text | Counters, circuit breaker state gauges, response time quantiles, error breakdowns. |
	| `/metrics?format=json` | JSON | Structured snapshot (same underlying data). |

	Sample (abridged):

	```
	# Request ID: deadbeef
GET /metrics

Returns a placeholder Prometheus-style line:

	```

	## Error Shape

	```json
	{
		"success": false,
		"error": {"code":"INVALID_SERIES","message":"Invalid series name","details":"Series 'foo' not found","timestamp":1730000000,"request_id":"deadbeef"},
		"request_id": "deadbeef"
	}
	```

	Generation failures surface inside progress JSON (`error` field) with HTTP 200 to maintain polling flow.

	## Auth & CORS
	Not implemented. Apply upstream (reverse proxy / gateway) if needed.

	## Versioning
	No version prefix; additive changes preferred. Breaking changes should use new endpoints.
dalleserver_up 1

Progress & Polling Contract

The server exposes progress transparently by forwarding the library’s snapshot for an in‑flight generation. This chapter focuses on how clients should consume it rather than restating the library schema.

Lifecycle Phases (High Level)

Typical sequence (library defined):

setup → base_prompts → enhance_prompt → image_prep → image_wait → image_download → annotate → completed

Phases may be marked skipped (e.g. enhancement when running in skip mode).

Polling Pattern

1. Kick off (or re-use) generation:

   GET /dalle/<series>/<address>?generate=1
   

2. Poll without changing parameters (optionally retain ?generate=1; lock prevents duplicate work) once per second until done=true.
3. When done=true and no error, re-request without generate to stream the final PNG directly.

Fish loop example:

while true
    curl -s "http://localhost:8080/dalle/simple/<addr>?generate=1" | jq '.percent, .current, .done'
    sleep 1
end

Percent & ETA

percent and etaSeconds reflect exponential moving averages (EMA) tracked across prior successful runs (cache hits excluded). Expect more accuracy as the system observes more generations.

Cache Hits

If the annotated PNG already exists before generation starts: snapshot returns quickly with cacheHit=true, done=true, minimal or empty phase timings.

Error Semantics

Progress snapshot may include a non-empty error while done=true; client should surface the message and avoid retry loops unless user refires manually (e.g. after clearing causes like rate limiting). HTTP status still 200 in this case—polling contract relies on payload state not transport errors.

Stability & Forward Compatibility

The progress object is additive: new fields may appear; existing names are stable. Clients should ignore unknown fields.

Where to Find Field Details

For a full enumeration of snapshot and DalleDress fields, consult the trueblocks-dalle book (progress section). This server layer intentionally avoids duplicating that documentation to prevent drift.

Observability Coupling

Response time metrics collected in middleware are independent of progress timing; phase averages feed only the progress percent/ETA calculation inside the library.

Future Extensions (Server-Level)

• Optional WebSocket push (reduces polling overhead)
• Soft cancellation endpoint (e.g. DELETE /dalle/<series>/<address>)
• Streaming Server-Sent Events wrapper around snapshot diffs

Configuration

Server configuration merges (in precedence order): command‑line flags → environment variables (including those injected from .env) → internal defaults.

This chapter documents only variables consumed directly by server code. Library‑specific knobs (prompt enhancement timeouts, image quality, etc.) are described in the trueblocks-dalle book.

.env Loader

loadDotEnv() reads a local .env file early in startup. Format: KEY=VALUE, comments start with #. Existing environment keys are never overridden. Quotes around values are stripped.

Flags

Flags are parsed once (subsequent parsing attempts in tests are ignored silently).

Environment Variables (Server)

Derived / Implicit Behavior

Sample .env

# Minimal development (mock) run – leave key blank for fast iteration
# OPENAI_API_KEY=sk-...
TB_DALLE_SKIP_IMAGE=1
TB_DALLE_PORT=8080

Observability Defaults

No explicit logging mode flags exist; logs are plain text via the shared logger package and mirrored to stderr. The metrics collector keeps only the last 1000 response time samples for percentile calculation.

Extending Configuration

If adding new server-level toggles prefer:

1. Environment variable (document here)
2. Flag (if runtime override is operationally important)
3. Conservative default preserving existing behavior

Then expose through Config struct if the value needs to be widely accessed.

Preview Gallery

/preview renders an HTML page listing annotated images grouped by series.

Implementation outline (handlePreview):

1. Walk output/ recursively.
2. Select PNG files whose path includes /annotated/.
3. Extract series (first path segment) and address (filename sans extension).
4. Collect modification time and relative path.
5. Group by series, sort each group by descending modification time.
6. Render a Go html/template with a client-side JavaScript filter.

Each figure shows:

• Image (square container with object-fit: contain)
• Address (EIP-55 case)
• Timestamp (mod time formatted YYYY-MM-DD HH:MM:SS)

Testing & Benchmarks

The test suite exercises request parsing, error shaping, locking behavior, progress handling, and failure resilience without requiring real OpenAI calls.

Modes

Image generation is skipped automatically when OPENAI_API_KEY is absent (or TB_DALLE_SKIP_IMAGE=1), enabling fast deterministic tests. The library still produces progress objects with simulated phases.

Key Tests (Representative)

Running

make test

Race detector:

make race

Benchmarks

Benchmark targets measure prompt + orchestration overhead (image path still skipped unless a key is supplied). Use:

make bench

Focused benchmark (see makefile):

make benchmark

Baselines capture JSON artifacts for regression tracking:

make bench-baseline

Artifacts stored under benchmarks/ with timestamped filenames; latest symlink / file pointer updated for easy diffing.

Adding New Tests

• Favor table-driven tests for handlers (inputs: path + query, expected HTTP status + JSON code/message fields).
• When stubbing generation, overwrite generateAnnotatedImage with a closure returning a temp file path or error.
• For metrics assertions, snapshot counters before and after the action; assert monotonic increases.

Flakiness Guidance

Avoid time-based sleeps for progress; directly call progress retrieval functions from the library after triggering a generation in tests. If unavoidable, keep sleeps short (<50ms) and document the rationale.

References

Primary server source files:

Library documentation (prompts, phases, DalleDress schema):

https://github.com/TrueBlocks/trueblocks-dalle/tree/develop/book

Artifacts directory layout (created under the library’s data dir, typically $HOME/.local/share/trueblocks/dalle/output):

output/<series>/annotated/<address>.png
output/<series>/prompt/... (and related prompt text subfolders)

Future enhancements & design rationale notes live inline as comments within the corresponding Go files (search for TODO: or Future markers when exploring the codebase).