updated on Study

Unleashing creative power of Stable Diffusion – Showcase

TL;DR

Made possible using automatic1111 + API enabled + some extra code in Jupiter notebook which you can see below.

I have been experimenting with Stable Diffusion for quite some time and I wanted to find a way to take back some of the control over creative process. Automatic1111 UI gets us 95% there (and hopefully 100% in the near future) but img2img UI was still missing something. It turned out, the solution was just a few lines of code and I can’t express how immensely satisfying it is to create art in this way!

While I used Photoshop, this would work with any editor.

To make this happen, I hacked together a bit of a python script to to assist with live paining (right hand side) and convert it in near-real-time to a stylised image using Stable Diffusion (left hand side).

Instructions

  1. Graphics card with at least 4GB VRAM (I ran this on RTX A4000 16GB – it’s very loud!)
  2. Set up Stable Diffusion (SD) and ensure it works in UI mode
  3. Enable API for AUTOMATIC1111 and start the UI
  4. Create promt.txt, base.psd and a Jupiter notebook with code below and run it 🙂
  5. I used Photoshop to modify base.psd and saved manually at regular intervals. Code checks if the image (based.psd) is modified and if it is, will create and display a new img2img rendition via SD API. Same applies for promt.txt, so you can change the prompt on the fly, though I tend to keep it same for most of the process.

Python code to make this work in Photoshop

import webuiapi # if missing, run: pip install webuiapi
from IPython.display import clear_output
import os
import time
from pathlib import Path
from psd_tools import PSDImage
from PIL import Image

# create API client
api = webuiapi.WebUIApi(sampler='Euler a', steps=20)

file_base = "base.psd"
file_prompt = "prompt.txt"
f_base_new = 0
f_prompt_new = 0

while True:
    f_base = os.path.getmtime(file_base)
    f_prompt = os.path.getmtime(file_prompt)
    if f_base == f_base_new and f_prompt==f_prompt_new:
        time.sleep(0.5)
    else:
        f_base_new = f_base
        f_prompt_new = f_prompt
        prompt_txt = Path(file_prompt).read_text()
        
        psd = PSDImage.open(file_base)
        im = psd.composite()
        result2 = api.img2img(
            images=[im], 
            prompt=prompt_txt, 
            negative_prompt = "poorly drawn, photorealistic, watermark, logo, text, bad anatomy, missing fingers,missing body part,mangled hands",
            steps=40,
            styles=[],
            seed=515553, 
            cfg_scale=7, 
            width=512,
            height=724,
            denoising_strength=0.55)
        clear_output(wait=True)
        display(result2.image)

Python code to make this work with any PNG file using any editor

import webuiapi  # if missing, run: pip install webuiapi
from IPython.display import clear_output
import os
import time
from pathlib import Path
from PIL import Image

# create API client
api = webuiapi.WebUIApi(sampler='Euler a', steps=20)

file_base = "base.png"
file_prompt = "prompt.txt"
f_base_new = 0
f_prompt_new = 0

while True:
    f_base = os.path.getmtime(file_base)
    f_prompt = os.path.getmtime(file_prompt)
    if f_base == f_base_new and f_prompt==f_prompt_new:
        time.sleep(0.5)
    else:
        f_base_new = f_base
        f_prompt_new = f_prompt
        prompt_txt = Path(file_prompt).read_text()
        with Image.open(file_base) as im:
            result2 = api.img2img(
                images=[im], 
                prompt=prompt_txt, 
                negative_prompt = "poorly drawn, photorealistic, watermark, logo, text, bad anatomy, missing fingers,missing body part,mangled hands",
                steps=40,
                styles=[],
                seed=515553, 
                cfg_scale=7, 
                width=512,
                height=724,
                denoising_strength=0.6)
            clear_output(wait=True)
            display(result2.image)

Other details

Model: deliberate v2

Prompt: (((amazing Impasto style oil artwork of))) (a simple glass of negroni with ice) on wooden table, next to a vase of flowers, books on bookshelf

Negative Prompt: poorly drawn, photorealistic, watermark, logo, text, bad anatomy, missing fingers,missing body part,mangled hands

Recommended Articles