Starting from Stable Diffusion, many scholars and independent developers have created derivative generative models that allow truly remarkable effects to be achieved by exploiting the benefits of artificial intelligence (AI). Lvmin Zhang, known for being the creator of ControlNetrecently introduced the concept of Layer Diffusion publishing and sharing a document that describes in detail the technique used.
Drawing inspiration from Stable DiffusionZhang has not only theorized but also created a new system that allows you to create new images with AI by applying a transparent background. The effect obtained is comparable to the intended operation crop an image and remove the background from a photo.
What is Layer Diffusion and how does it work in brief
The author of the project developed a solution called “latent transparency” which enriches the latent diffusion models such as Stable Diffusion and allows you to generate images that present transparencies. Zhang proposed the idea to him because the market lacks effective tools for remove the background from the images produced by generative models, and this despite the growing demand for such functions from the market.
The diffusion is a process by which a complex probability distribution is gradually “diffused” or “simplified” during the sampling process. In simpler terms, the model starts with a “noisy” or random image and then updates it bit by bit, gradually reducing uncertainty and generating amore defined image. This diffusion process is driven by a stochastic approach that determines how the image is developed step by step.
The concept of “latent transparency” refers to Zhang’s proposed scheme for making part of an image transparent. The term “latent” indicates that this transparency feature is incorporated and manipulated into the latent representation of the image. The latter is an abstract space that captures the significant characteristics of the image in a compressed and interpretable format.
To ensure the consistency and the harmonious fusion between levels of the same image, the LoRA models (Low-Rank Adaptation) specially trained allow you to adapt the operation of the system to the different conditions of the levels of each image.
How to crop images and remove the background using Stable Diffusion WebUI Forge
In another article we saw how to speed up the results obtainable with Stable Diffusion and significantly reduce processing times image generation.
Anyone who has already installed and configured locally Stable Diffusion WebUI Forgea solution developed by Zhang himself, can now add support for the additional module that allows you to crop the generated images quickly and easily.
To proceed, simply access the local web interface of Stable Diffusion WebUI Forge (address localhost in the browser bar), click on the tab Extensions so up Install from URL. The following address must be entered in the appropriate field: https://github.com/layerdiffusion/sd-forge-layerdiffuse
With one click on the button Installmodel Layer Diffusion, briefly described above, will in turn be loaded on the system in use. After the confirmation message appears, you must click on the button Apply and restart UI.
The presence of the drop-down menu LayerDiffusion, confirms that the installation was successful. You just need to check the box Enabled is active and that the preferred method is Only generate transparent image (attention injection).
Use “ad hoc” prompts to remove the background from the generated images
At this point you can enter a prompt for generate an image, as already seen in the article focused on Stable Diffusion WebUI Forge. In his examples, Zhang suggests resorting, for example, to the Juggernaut XL model.
In the Layer Diffusion GitHub repository you can find several example prompt, to produce a vast array of images. There is also a video that shows the different effects that can be obtained by adjusting the various adjustments.
Indeed, as Zhang demonstrates in this example, it is possible to select From Background to Blending from the drop-down menu Method and pass a pre-existing image to the generative model.
The researcher “handed” the image of a bench surrounded by greenery to the model and then requested, via prompt, the addition of an elderly person sitting on the same bench. The result is the one visible in the figure: the photo of the bench in the box LayerDiffusion and now “populated” with the image of a person of a certain age.
The great thing is that, by dragging the further image produced by the generative model, it is possible to cut out the “protagonist” and remove the background.
Opening image credit: iStock.com – Vertigo3d