Quickstart

Overview

This guide will take you through the process of sending video input to our StreamDiffusion pipeline. You will learn how to adjust parameters to create a variety of visual effects, utilize live streaming and audio interactivity features, generate real-time visuals, and view the resulting output video.

API Auth

The use of the API key is currently subsidized for a limited time, and we will provide an update on pricing in the future.

The API uses Bearer auth. Include an Authorization header on every request:

Authorization: Bearer <YOUR_API_KEY>

Keep your API keys secure. Do not commit them to source control or share them publicly.

API Documentation

The full API reference is available in the sidebar and contains more in-depth descriptions of all parameters.

Creating Your First App

Building on top of our StreamDiffusion pipeline consists of three parts:

Creating a Stream object
Sending in video and playing the output
Setting StreamDiffusion parameters

1. Create a Stream

First, we need to create a ‘Stream’ object. This will provide us with an input URL (to send in video) and an output URL (to play back the modified video).

All examples use Bash so you can copy/paste into a terminal.

Create Stream Request

When specifying the pipeline_id, it also needs to match the right model_id.Ex. pip_SD-turbo corresponds to stabilityai/sd-turbo

# Choose your AI video processing pipeline:
# - pip_SD-turbo: Fast generation with SD-turbo (default)
# - pip_SDXL-turbo: High-quality with SDXL-turbo  
# - pip_SD15: Stable Diffusion 1.5 for more control
# - pip_SDXL-turbo-faceid: SDXL with face identity with ipAdapters
PIPELINE_ID="pip_SD-turbo"

# Change this to the API key you generated
DAYDREAM_API_KEY="<YOUR_API_KEY>"

curl -X POST \
  "https://api.daydream.live/v1/streams" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${DAYDREAM_API_KEY}" \
  -d "{\"pipeline_id\":\"${PIPELINE_ID}\"}"

Create Stream Response

{
  "id": "str_gERGnGZE4331XBxW",
  "output_playback_id": "0d1crgzijlcsxpw4",
  "whip_url": "https://ai.livepeer.com/live/video-to-video/stk_abc123/whip"
}

2. Send video and play the output

Now we’ll start sending in video and view the processed output.

Install OBS.
Copy the whip_url from the Create Stream response.
In OBS → Settings → Stream: choose WHIP as the Service and paste the whip_url as the Server. Leave the Bearer Token blank and save the settings.
Under the Sourcessection, add a video source for the stream.(Ex: Video Capture Device )
Under the Controls section, select Start Streaming to start the stream.
Copy the output_playback_id from the Create Stream response and open: https://lvpr.tv/?v=<your output_playback_id>
You should now see your output video playing.

Prefer to use your own player?

Fetch playback URLs from Livepeer Studio’s Playback endpoint: curl "https://livepeer.studio/api/playback/<your playback id>"
Choose either the HLS or WebRTC endpoint.
Configure your chosen player with that URL.

3. Set StreamDiffusion parameters

Send a PATCH to https://api.daydream.live/v1/streams/<YOUR_STREAM_ID> to control the effect. You only need to include the parameters you want to change.

{
  "params": {
    // Base model options:
    // - "stabilityai/sd-turbo" for SD2.1 models
    // - "prompthero/openjourney-v4" for SD1.5 models  
    // - "stabilityai/sdxl-turbo" for SDXL models
    "model_id": "stabilityai/sd-turbo",
    "prompt": "superman",
    "prompt_interpolation_method": "slerp",
    "normalize_prompt_weights": true,
    "normalize_seed_weights": true,
    "negative_prompt": "blurry, low quality, flat, 2d",
    "num_inference_steps": 50,
    "seed": 42,
    "t_index_list": [0, 8, 17],
    // Different ControlNets are required for different models
    // See the ControlNets section below for model-specific options
    "controlnets": [
      {
        "conditioning_scale": 0,
        "control_guidance_end": 1,
        "control_guidance_start": 0,
        "enabled": true,
        "model_id": "thibaud/controlnet-sd21-openpose-diffusers",
        "preprocessor": "pose_tensorrt",
        "preprocessor_params": {}
      },
      {
        "conditioning_scale": 0,
        "control_guidance_end": 1,
        "control_guidance_start": 0,
        "enabled": true,
        "model_id": "thibaud/controlnet-sd21-hed-diffusers",
        "preprocessor": "soft_edge",
        "preprocessor_params": {}
      },
      {
        "conditioning_scale": 0,
        "control_guidance_end": 1,
        "control_guidance_start": 0,
        "enabled": true,
        "model_id": "thibaud/controlnet-sd21-canny-diffusers",
        "preprocessor": "canny",
        "preprocessor_params": {"high_threshold": 200, "low_threshold": 100}
      },
      {
        "conditioning_scale": 0,
        "control_guidance_end": 1,
        "control_guidance_start": 0,
        "enabled": true,
        "model_id": "thibaud/controlnet-sd21-depth-diffusers",
        "preprocessor": "depth_tensorrt",
        "preprocessor_params": {}
      },
      {
        "conditioning_scale": 0,
        "control_guidance_end": 1,
        "control_guidance_start": 0,
        "enabled": true,
        "model_id": "thibaud/controlnet-sd21-color-diffusers",
        "preprocessor": "passthrough",
        "preprocessor_params": {}
      }
    ]
  }
}

Example PATCH Request

# Update stream parameters
STREAM_ID="<YOUR_STREAM_ID>"
DAYDREAM_API_KEY="<YOUR_API_KEY>"

curl -X PATCH \
  "https://api.daydream.live/v1/streams/${STREAM_ID}" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${DAYDREAM_API_KEY}" \
  -d '{
    "params": {
      "prompt": "superman",
      "num_inference_steps": 50,
      "seed": 42
    }
  }'

IP Adapter Availability: IP adapters for style conditioning are only available for SD1.5 (prompthero/openjourney-v4), SDXL (stabilityai/sdxl-turbo), and SDXL-faceid models. They are not supported by SD2.1 models like stabilityai/sd-turbo.Base64 for IP Adapter image input is also supported.

Parameter change examples

Experiment with any of the parameters. A few useful ones:

Parameter	Description
`prompt`	Guides the model toward a desired visual style or subject.
`negative_prompt`	Tells the model what not to produce (e.g., discourages low quality, flat, blurry results).
`num_inference_steps`	Higher values improve quality at the cost of speed/FPS.
`seed`	Ensures reproducibility across runs. Change it to introduce variation.

ControlNets

ControlNets guide image generation by providing extra structural inputs (poses, edges, depth maps, colors) that help the model interpret the input video. They impact performance differently, so experiment with which ones to enable for your use case.

Don’t toggle the enabled field to turn ControlNets on/off, as it currently triggers a pipeline reload. Set conditioning_scale to 0 to effectively disable a ControlNet, or raise it above 0 to enable its influence.

To enable a ControlNet, increase its conditioning_scale:

{
  "conditioning_scale": 0.22,
  "control_guidance_end": 1,
  "control_guidance_start": 0,
  "enabled": true,
  "model_id": "thibaud/controlnet-sd21-openpose-diffusers",
  "preprocessor": "pose_tensorrt",
  "preprocessor_params": {}
}

Available ControlNets by model type: SD2.1 Models (stabilityai/sd-turbo):

thibaud/controlnet-sd21-openpose-diffusers: Body and hand pose tracking to maintain human poses in the output
thibaud/controlnet-sd21-hed-diffusers: Soft edge detection preserving smooth edges and contours
thibaud/controlnet-sd21-canny-diffusers: Sharp edge preservation with crisp outlines and details
thibaud/controlnet-sd21-depth-diffusers: Preserves spatial depth and 3D structure of objects and faces
thibaud/controlnet-sd21-color-diffusers: Color composition passthrough to maintain palette and composition

SD1.5 Models (prompthero/openjourney-v4):

lllyasviel/control_v11f1p_sd15_depth: Depth-based guidance for spatial structure preservation
lllyasviel/control_v11f1e_sd15_tile: Tile-based pattern control for texture preservation
lllyasviel/control_v11p_sd15_canny: Canny edge detection for detailed outline preservation

SDXL Models (stabilityai/sdxl-turbo):

xinsir/controlnet-depth-sdxl-1.0: High-resolution depth guidance for SDXL models
xinsir/controlnet-canny-sdxl-1.0: SDXL-optimized canny edge detection
xinsir/controlnet-tile-sdxl-1.0: Tile-based control for SDXL texture generation

Documentation

API Reference

Parameters

References

Overview

API Auth

API Documentation

Creating Your First App

1. Create a Stream

Create Stream Request

Create Stream Response

2. Send video and play the output

3. Set StreamDiffusion parameters

Example PATCH Request

Parameter change examples

ControlNets

Documentation

API Reference

Parameters

References

​Overview

​API Auth

​API Documentation

​Creating Your First App

​1. Create a Stream

​Create Stream Request

​Create Stream Response

​2. Send video and play the output

​3. Set StreamDiffusion parameters

​Example PATCH Request

​Parameter change examples

​ControlNets

Overview

API Auth

API Documentation

Creating Your First App

1. Create a Stream

Create Stream Request

Create Stream Response

2. Send video and play the output

3. Set StreamDiffusion parameters

Example PATCH Request

Parameter change examples

ControlNets