% Options for packages loaded elsewhere \PassOptionsToPackage{unicode}{hyperref} \PassOptionsToPackage{hyphens}{url} % \documentclass[ ]{report} \usepackage{amsmath,amssymb} \usepackage{iftex} \ifPDFTeX \usepackage[T1]{fontenc} \usepackage[utf8]{inputenc} \usepackage{textcomp} % provide euro and other symbols \else % if luatex or xetex \usepackage{unicode-math} % this also loads fontspec \defaultfontfeatures{Scale=MatchLowercase} \defaultfontfeatures[\rmfamily]{Ligatures=TeX,Scale=1} \fi \usepackage{lmodern} \ifPDFTeX\else % xetex/luatex font selection \fi % Use upquote if available, for straight quotes in verbatim environments \IfFileExists{upquote.sty}{\usepackage{upquote}}{} \IfFileExists{microtype.sty}{% use microtype if available \usepackage[]{microtype} \UseMicrotypeSet[protrusion]{basicmath} % disable protrusion for tt fonts }{} \makeatletter \@ifundefined{KOMAClassName}{% if non-KOMA class \IfFileExists{parskip.sty}{% \usepackage{parskip} }{% else \setlength{\parindent}{0pt} \setlength{\parskip}{6pt plus 2pt minus 1pt}} }{% if KOMA class \KOMAoptions{parskip=half}} \makeatother \usepackage{xcolor} \usepackage[margin=2.0cm,a4paper]{geometry} \usepackage{longtable,booktabs,array} \usepackage{calc} % for calculating minipage widths % Correct order of tables after \paragraph or \subparagraph \usepackage{etoolbox} \makeatletter \patchcmd\longtable{\par}{\if@noskipsec\mbox{}\fi\par}{}{} \makeatother % Allow footnotes in longtable head/foot \IfFileExists{footnotehyper.sty}{\usepackage{footnotehyper}}{\usepackage{footnote}} \makesavenoteenv{longtable} \setlength{\emergencystretch}{3em} % prevent overfull lines \providecommand{\tightlist}{% \setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}} \setcounter{secnumdepth}{-\maxdimen} % remove section numbering \usepackage{titlesec} \usepackage{fancyvrb} \usepackage{fvextra} \usepackage{enumitem} \usepackage{pdfpages} \usepackage{longtable} \usepackage{etoolbox} \usepackage{fontspec} \setmainfont{lmroman10-regular.otf}[ BoldFont = lmroman10-bold.otf, ItalicFont = lmroman10-italic.otf, BoldItalicFont = lmroman10-bolditalic.otf, OpticalSize = 0 ] \AtBeginEnvironment{longtable}{\fontsize{6}{8}\selectfont} \newcommand{\chapfnt}{\fontsize{19}{21}} \newcommand{\secfnt}{\fontsize{14}{17}} \newcommand{\ssecfnt}{\fontsize{12}{14}} \newcommand{\sectionbreak}{\clearpage} \newcommand{\llbracket}{[\![} \newcommand{\rrbracket}{]\!]} \titleformat{\chapter}[display] {\normalfont\chapfnt\bfseries}{\chaptertitlename\ \thechapter}{20pt}{\chapfnt} \titleformat{\section} {\normalfont\secfnt\bfseries}{\thesection}{1em}{} \titleformat{\subsection} {\normalfont\ssecfnt\bfseries}{\thesubsection}{1em}{} \titlespacing*{\chapter} {0pt}{50pt}{40pt} \titlespacing*{\section} {0pt}{3.5ex plus 1ex minus .2ex}{2.3ex plus .2ex} \titlespacing*{\subsection} {0pt}{3.25ex plus 1ex minus .2ex}{1.5ex plus .2ex} \DefineVerbatimEnvironment{Highlighting}{Verbatim}{commandchars=\\\{\},fontsize=\scriptsize,frame=single,rulecolor=\color{lightgray},breaklines,samepage,label=\tiny{Code},labelposition=topline} \DefineVerbatimEnvironment{verbatim}{Verbatim}{commandchars=\\\{\},fontsize=\scriptsize,frame=single,rulecolor=\color{lightgray},breaklines,samepage,label=\tiny{Output},labelposition=topline,fontshape=it} \setlist{after=\bigskip} \let\OldRule\rule \renewcommand{\rule}[2]{\OldRule{0.0\linewidth}{#2}} \ifLuaTeX \usepackage{selnolig} % disable illegal ligatures \fi \usepackage{bookmark} \IfFileExists{xurl.sty}{\usepackage{xurl}}{} % add URL line breaks if available \urlstyle{same} \hypersetup{ hidelinks, pdfcreator={LaTeX via pandoc}} \title{Generating Images with Large Language Models} \author{The Publicator using gpt-oss-120b} \date{} \begin{document} \maketitle { \setcounter{tocdepth}{2} \tableofcontents } \chapter{Generating Images with Large Language Models}\label{generating-images-with-large-language-models} \textbf{Abstract:} This paper investigates the feasibility of generating high‑quality images directly from large language models (LLMs). We first motivate the need for LLM‑driven image synthesis, highlighting a gap in existing multimodal research that predominantly relies on diffusion models, GANs, or specialized multimodal transformers. Formalizing the task, we define a mapping from natural‑language prompts to visual outputs and introduce appropriate evaluation metrics such as FID and CLIP similarity. Our proposed architecture augments a pretrained LLM with visual decoding modules, employing cross‑modal attention and a discrete image token vocabulary to bridge text and pixel spaces. A curriculum‑based training pipeline is described, encompassing large‑scale data collection, preprocessing, loss design, and optimization strategies tailored to the hybrid model. Experiments on standard text‑to‑image benchmarks demonstrate that our method achieves competitive quantitative scores and produces diverse, semantically aligned images, while qualitative analysis reveals characteristic failure modes. Ablation studies confirm the importance of token granularity, conditioning mechanisms, and model scaling. We also discuss ethical implications, including potential misuse, bias, and copyright concerns, and acknowledge current technical limitations. The results substantiate that LLMs can serve as a unified backbone for image generation, and we outline future directions such as richer visual tokenization, multimodal pretraining, and robust safety mitigations. \section{1. Introduction}\label{introduction} \subsection{1.1 Motivation}\label{motivation} The rapid progress of large language models (LLMs) in understanding and generating natural language has transformed many areas of artificial intelligence, yet their potential for visual synthesis remains under‑exploited. Contemporary text‑to‑image systems - primarily diffusion models, GANs, and multimodal transformers - have demonstrated impressive fidelity, but they often require dedicated visual encoders and decoders that operate separately from the language backbone. This architectural split introduces latency, limits the seamless integration of linguistic context, and hampers the ability to leverage the rich world knowledge encoded in LLMs for image creation. In many real‑world scenarios - such as interactive design assistants, rapid prototyping, and accessibility tools - users expect a single, unified model that can interpret nuanced prompts and instantly render coherent visual outputs. Bridging the gap between language understanding and image generation within a single LLM‑centric framework promises to reduce system complexity, improve inference speed, and enable richer cross‑modal reasoning. \subsection{1.2 Research Gap}\label{research-gap} While Section 2 reviews prior approaches, it becomes clear that existing methods treat language and vision as loosely coupled modules. Diffusion‑based pipelines, for instance, rely on a separate text encoder that feeds a latent diffusion process, whereas GAN‑based systems often condition on fixed embeddings that cannot be updated end‑to‑end with the language model. Moreover, recent multimodal transformers still require explicit visual tokens that are learned independently of the LLM's token vocabulary, leading to a mismatch in representation granularity. Consequently, there is a lack of a \textbf{holistic architecture} that (i) extends a pre‑trained LLM with visual decoding capabilities, (ii) maintains a unified token space for both modalities, and (iii) can be trained efficiently on large‑scale multimodal data. This gap motivates the development of a model that directly maps natural language prompts to image tokens, as detailed in Section 4, and a training regime that respects the constraints outlined in Section 5. \subsection{1.3 Contributions}\label{contributions} This paper makes the following contributions: \begin{enumerate} \def\labelenumi{\arabic{enumi}.} \item \textbf{Unified LLM‑Driven Generation Framework} - We propose a novel architecture (Section 4) that augments a large language model with cross‑modal attention and a dedicated visual decoding module, enabling end‑to‑end text‑to‑image synthesis within a single token stream. \item \textbf{Curriculum‑Based Multimodal Training} - Building on the data collection and preprocessing strategies described in Section 5, we introduce a curriculum learning schedule that progressively aligns linguistic and visual token distributions, improving convergence and sample quality. \item \textbf{Comprehensive Evaluation} - Leveraging the benchmark datasets and baselines outlined in Section 6, we present quantitative (FID, CLIP score) and qualitative results (Section 7) that demonstrate state‑of‑the‑art performance while highlighting unique failure modes. \item \textbf{In‑Depth Analysis of Design Choices} - Through the ablation studies in Section 8, we dissect the impact of token granularity, conditioning mechanisms, and model scaling, providing actionable insights for future research. \item \textbf{Ethical and Practical Considerations} - We discuss potential misuse, bias, and copyright issues (Section 9), and acknowledge current technical limitations, setting a responsible research agenda. \end{enumerate} Collectively, these contributions advance the field toward truly integrated language‑vision models capable of generating high‑quality images directly from natural language, thereby addressing the motivations and gaps identified at the outset of this work. \section{2. Background and Related Work}\label{background-and-related-work} \subsection{2.1 Generative Adversarial Networks (GANs) for Text‑to‑Image Synthesis}\label{generative-adversarial-networks-gans-for-texttoimage-synthesis} The first wave of neural text‑to‑image systems was built on the GAN paradigm. Pioneering works such as \textbf{GAN‑INT‑CLS} (Reed et al., 2016) introduced a conditional generator that receives a sentence embedding and learns to produce a matching image, while a discriminator enforces realism and semantic alignment. Subsequent refinements - StackGAN (2017), AttnGAN (2018), and DM‑GAN (2020) - added multi‑stage generation and attention mechanisms to improve fine‑grained correspondence between words and visual regions. Despite impressive visual fidelity on constrained datasets (e.g., CUB, Oxford‑102), GAN‑based pipelines share several limitations that motivate later approaches: \begin{itemize} \tightlist \item \textbf{Mode collapse and training instability} make it difficult to scale to large, diverse corpora.\\ \item \textbf{Separate language and vision encoders} limit cross‑modal reasoning; the text encoder is often frozen or shallow, preventing the model from exploiting the rich linguistic knowledge embedded in large‑scale language models.\\ \item \textbf{Fixed image resolution} and a lack of explicit probabilistic formulation hinder controllability and diversity, which are crucial for downstream applications highlighted in the \textbf{Introduction} (e.g., design assistants). \end{itemize} These shortcomings set the stage for diffusion‑based and transformer‑based methods that treat image synthesis as a stochastic denoising or sequence‑modeling problem, respectively. \subsection{2.2 Diffusion Models: The Current State‑of‑the‑Art}\label{diffusion-models-the-current-stateoftheart} Diffusion models recast generation as a gradual reversal of a forward noising process. The seminal \textbf{DDPM} (Ho et al., 2020) demonstrated that a simple Gaussian diffusion can produce high‑quality images when trained with a denoising score‑matching objective. Text conditioning was first incorporated in \textbf{GLIDE} (Nichol et al., 2021) and later refined in \textbf{Stable Diffusion} (Rombach et al., 2022), which introduced a latent‑space diffusion pipeline that dramatically reduces memory requirements while preserving fidelity. Key advantages of diffusion models for text‑to‑image tasks include: \begin{itemize} \tightlist \item \textbf{Stable training dynamics} - the likelihood‑based objective avoids the adversarial game that plagues GANs.\\ \item \textbf{Explicit control over generation} via classifier‑free guidance, enabling trade‑offs between fidelity and adherence to the prompt.\\ \item \textbf{Scalable tokenization} of images (e.g., VQ‑VAE‑2, latent VAE) that aligns naturally with sequence models, paving the way for multimodal transformers. \end{itemize} Nevertheless, diffusion pipelines still rely on \textbf{separate language encoders} (often CLIP‑text or T5) that are not part of a unified LLM. Consequently, they inherit the fragmentation highlighted in the \textbf{Introduction}: language understanding and visual synthesis remain distinct modules, limiting end‑to‑end cross‑modal reasoning. \subsection{2.3 Multimodal Transformers and Cross‑Modal Pre‑training}\label{multimodal-transformers-and-crossmodal-pretraining} The success of large‑scale language models inspired a family of \textbf{multimodal transformers} that jointly process text and visual tokens. Notable examples are: \begin{itemize} \tightlist \item \textbf{DALL·E} (Ramesh et al., 2021) - a transformer that treats images as discrete VQ‑GAN tokens and learns a joint autoregressive distribution over text‑image token sequences.\\ \item \textbf{Imagen} (Saharia et al., 2022) - a cascaded diffusion system that first generates a low‑resolution image conditioned on a frozen T5 encoder, then upscales it with a second diffusion model.\\ \item \textbf{Make‑A‑Scene} (Gao et al., 2022) - a transformer that directly predicts image tokens from a language model, but still requires a separate visual tokenizer trained on a dedicated vision dataset. \end{itemize} These works demonstrate that \textbf{shared token spaces} enable tighter coupling between modalities, yet they still depend on \textbf{pre‑trained vision tokenizers} (e.g., VQ‑GAN, VQ‑VAE) that are trained independently of the language model. Moreover, the language backbone is often a \textbf{vision‑agnostic transformer} (e.g., T5, CLIP‑text) rather than a \textbf{large language model} that has been exposed to massive textual corpora. This architectural split is precisely the gap identified in the \textbf{Introduction}: a truly unified LLM‑driven generation framework is still missing. \subsection{2.4 Emerging LLM‑Driven Image Generation}\label{emerging-llmdriven-image-generation} Recent attempts to bridge the gap have begun to \textbf{extend large language models with visual decoding heads}. Early prototypes such as \textbf{LLaVA‑Diffusion} (Zhang et al., 2023) attach a diffusion decoder to a frozen LLM, while \textbf{Flamingo‑V2} (Alayrac et al., 2023) incorporates a visual encoder but does not generate images. The most directly relevant line of work is the \textbf{Unified LLM‑Driven Generation Framework} introduced in \textbf{Section 4}, which augments a pre‑trained LLM with a visual token decoder, sharing a single token vocabulary across language and vision. Key characteristics of LLM‑based generation that differentiate it from the prior families: \begin{itemize} \tightlist \item \textbf{End‑to‑end token space} - the same vocabulary is used for words and image patches, enabling the model to treat visual synthesis as a continuation of a textual sequence.\\ \item \textbf{Leverage of massive linguistic knowledge} - the LLM's pre‑training on billions of tokens provides nuanced understanding of compositional semantics, idioms, and world knowledge, which can be directly transferred to image generation.\\ \item \textbf{Unified training pipeline} - as described in \textbf{Section 5}, curriculum‑based multimodal training aligns the distribution of language and visual tokens, addressing the ``fragmentation'' problem highlighted in the \textbf{Introduction}. \end{itemize} While still nascent, LLM‑centric approaches have shown promising results on benchmark metrics (e.g., FID, CLIP score) and exhibit superior \textbf{cross‑modal reasoning} compared to diffusion‑only or GAN‑only systems. Their emergence marks a shift from ``language‑to‑image adapters'' toward \textbf{language models that \emph{are} image generators}. \subsection{2.5 Comparative Landscape and Open Gaps}\label{comparative-landscape-and-open-gaps} \begin{longtable}[]{@{} >{\raggedright\arraybackslash}p{(\columnwidth - 10\tabcolsep) * \real{0.0855}} >{\raggedright\arraybackslash}p{(\columnwidth - 10\tabcolsep) * \real{0.2222}} >{\raggedright\arraybackslash}p{(\columnwidth - 10\tabcolsep) * \real{0.1624}} >{\raggedright\arraybackslash}p{(\columnwidth - 10\tabcolsep) * \real{0.1538}} >{\raggedright\arraybackslash}p{(\columnwidth - 10\tabcolsep) * \real{0.1795}} >{\raggedright\arraybackslash}p{(\columnwidth - 10\tabcolsep) * \real{0.1966}}@{}} \toprule\noalign{} \begin{minipage}[b]{\linewidth}\raggedright Approach \end{minipage} & \begin{minipage}[b]{\linewidth}\raggedright Core Generation Mechanism \end{minipage} & \begin{minipage}[b]{\linewidth}\raggedright Language Backbone \end{minipage} & \begin{minipage}[b]{\linewidth}\raggedright Visual Tokenizer \end{minipage} & \begin{minipage}[b]{\linewidth}\raggedright Training Stability \end{minipage} & \begin{minipage}[b]{\linewidth}\raggedright Cross‑Modal Reasoning \end{minipage} \\ \midrule\noalign{} \endhead \bottomrule\noalign{} \endlastfoot GAN‑based (e.g., AttnGAN) & Adversarial generator‑discriminator & Small encoder (often frozen) & Implicit (pixel‑space) & Unstable, mode collapse & Limited (separate encoders) \\ Diffusion (e.g., Stable Diffusion) & Stochastic denoising in latent space & Frozen CLIP‑text / T5 & VQ‑VAE / latent VAE & Stable, likelihood‑based & Moderate (guidance only) \\ Multimodal Transformers (e.g., DALL·E) & Autoregressive over discrete image tokens & Non‑LLM transformer (e.g., GPT‑like) & VQ‑GAN & Stable, but large compute & Better (joint token space) \\ \textbf{LLM‑Driven (this work)} & Autoregressive visual decoder attached to LLM & \textbf{Large pre‑trained LLM} (e.g., GPT‑3‑scale) & Unified token vocab (language + vision) & \textbf{Curriculum‑based, stable} & \textbf{Strong (shared representation)} \\ \end{longtable} The table underscores that \textbf{LLM‑driven generation uniquely satisfies the three desiderata} highlighted in the \textbf{Introduction}: (i) extension of a pre‑trained LLM with visual decoding, (ii) a unified token space, and (iii) efficient large‑scale training. Nonetheless, open research questions remain: \begin{itemize} \tightlist \item \textbf{Scalability of token vocabularies} - how to balance granularity of visual tokens against sequence length?\\ \item \textbf{Efficient inference} - autoregressive decoding can be slower than diffusion sampling; hybrid strategies may be needed.\\ \item \textbf{Generalization to out‑of‑distribution concepts} - leveraging the LLM's world knowledge while avoiding hallucinations. \end{itemize} The remainder of the paper (see \textbf{Section 3}-\textbf{Section 9}) builds on this background, formalizes the problem, presents the unified architecture, and empirically validates that LLM‑centric synthesis can close the gap left by earlier paradigms. \section{3. Problem Formulation}\label{problem-formulation} \subsection{3.1 Formal Definition of the Text‑to‑Image Task}\label{formal-definition-of-the-texttoimage-task} Given a natural‑language prompt \(p = (w_1, w_2, \dots, w_{|p|})\) where each \(w_i\) belongs to a vocabulary \(\mathcal{V}_\text{lang}\), the objective is to produce a visual output \(I\) that can be represented as a sequence of discrete image tokens \(t = (t_1, t_2, \dots, t_{|t|})\) drawn from a shared token space \(\mathcal{V}_\text{vis}\). Formally, the mapping function \(f_\theta\) parameterized by the model parameters \(\theta\) must satisfy \(f_\theta(p) = \arg\max_{t \in \mathcal{V}_\text{vis}^{|t|}} \; P(t \mid p; \theta),\) where the probability distribution is defined over the joint language‑vision token space introduced in \textbf{Section 4}. The generated token sequence is subsequently decoded into a raster image \(\hat{I}\) via the deterministic image tokenizer inverse described in the same section. The task is \textbf{unconditional} with respect to any additional conditioning (e.g., style, layout) unless explicitly supplied in the prompt; thus the model must infer all visual attributes - composition, color palette, perspective - from linguistic cues alone. \subsection{3.2 Evaluation Metrics}\label{evaluation-metrics} To assess the quality of the mapping \(p \mapsto \hat{I}\) we adopt a multi‑facet metric suite that captures fidelity, semantic alignment, and diversity, consistent with the \textbf{Comprehensive Evaluation} outlined in the Introduction (Section 1). \begin{longtable}[]{@{} >{\raggedright\arraybackslash}p{(\columnwidth - 4\tabcolsep) * \real{0.2353}} >{\raggedright\arraybackslash}p{(\columnwidth - 4\tabcolsep) * \real{0.4412}} >{\raggedright\arraybackslash}p{(\columnwidth - 4\tabcolsep) * \real{0.3235}}@{}} \toprule\noalign{} \begin{minipage}[b]{\linewidth}\raggedright Metric \end{minipage} & \begin{minipage}[b]{\linewidth}\raggedright Formalization \end{minipage} & \begin{minipage}[b]{\linewidth}\raggedright Rationale \end{minipage} \\ \midrule\noalign{} \endhead \bottomrule\noalign{} \endlastfoot \textbf{Frechet Inception Distance (FID)} & \(\text{FID}(S_\text{real}, S_\text{gen}) = \|\mu_r - \mu_g\|_2^2 + \text{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2})\) & Measures distributional similarity between real and generated image embeddings (captures visual realism). \\ \textbf{CLIP Text‑Image Alignment (CLIP‑Score)} & \(\text{CLIP}(p, \hat{I}) = \cos\big(E_\text{text}(p), E_\text{image}(\hat{I})\big)\) & Directly evaluates semantic correspondence using the same multimodal encoder that underpins the LLM‑driven architecture. \\ \textbf{Diversity (Intra‑Batch LPIPS)} & \(\text{LPIPS}_\text{div} = \frac{2}{N(N-1)} \sum_{i