Abstract
A method, apparatus, non-transitory computer readable medium, and system for conditional text-to-image synthesis include obtaining a prompt input and a color input. The prompt input describes an image element, and the color input indicates a color palette. Embodiments then perform an optimization of an intermediate noise map by computing a color loss based on the color input to obtain a conditioned noise map. Subsequently, embodiments generate, using an image generation model, a synthetic image based on the prompt input and the conditioned noise map, wherein the synthetic image depicts the image element and includes colors from the color palette. Some embodiments are further configured to optimize an intermediate noise map based on a shape input, where the shape input indicates a shape for the image element.